NetScaler attack mitigation

How can I stop the bad guys trying to break in?

Hello!

One of the features of running an Internet facing service, is the chance that your service will get noticed by someone looking to access your system who is not authorised. There some common tools freely available to scan vasts ranges of Internet facing addresses for services to attack. This typically will be carried out by a robot or hacker of some sort. The question is:

How do I stop this kind of attack? What steps are needed to keep the administration overhead low?

This piece will detail some of the steps that you should take to minimise your exposure to these kinds of attack with a combination of settings and processes.

Who is this for?

Any NetScaler administrator with a internet facing service, in fact, any service that has an authorisation option enabled. As attacks can come from 'internal’ sources too. The role of an administrator is to ensure that a service is secure and that attacks do not lead to data loss, exfiltration or general compromise.

Who provided help to create this?

Many thanks to: Steven Wright, Simha Kaipa, Hemang Raval, Karsten Feldhusen, Martin Nygaard Jensen and Rick Davis for all the suggestions.

Background

When you have a NetScaler providing an internet service, you want to only allow authorised users access. The challenge, is how to ensure that person accesses is who they say they are? Typically, users have a password, in combination with a username this gives the administrator some level of security. The problem is that, this ‘single factor authentication’ is a poor starting point. Almost no internet facing system can just use this option, hence the prevalence of One Time Password(OTP) options. OTP gives a better level of security, but it is good practise to combine it with other measures to have good levels of security.

Other reading?

An internet search of this topic, yielded a few hits.

• From 2013! A former colleague created this. It is very old now, and things have moved on (classic polices).

• Released 2017 and updated in 2021. There is this support doc here, which just talks about some password options. These are covered in the text below, but need to be combined with other techniques.

What is the configuration baseline?

A good starting point for a NetScaler configuration, and something that could make a big difference to the fundamental security that is in place is to follow one of these reference designs. It can be common for NetScaler to be deployed in different ways by different engineers, these designs have been created to create a level of consistency.

Who created these? Steven Wright works in the NetScaler business unit, these designs are based on extensive real world experience and feedback from other product managers and engineers that he works with.

Simple steps to prevent brute force attacks

The first item in the following section, is a preferred way to handle brute force attacks, there are other configuration options discussed, some quite new. However, if you are under attack today, this one step will halt the attack in its tracks.

1. Authentication flow to apply and test the OTP factor first.

When the user is presented with the login boxes for username, password and OTP factor, it can be common to setup the appliance to have a user password to be tested first, before the OTP factor. One of the elements of the reference design above is that they always apply the OTP factor first. The benefit of this approach is if the OTP is not known, a password (regardless if it's wrong) will never be sent from the NetScaler to the LDAPS server unless OTP is successful, completely eliminating the problem of brute force attempts hammering AD until successful. 

Having a token based factor as part of the authentication, does make the job of breaking in harder. Access passwords do change, but the frequency at which they are changed is likely every few months. The token typically changes every 30 seconds. A password plus token approach is very common. This process flow provides a level of protection to the ‘weaker or more static’ password that the user has.

Pros: It is simple and effective, and can be transparent to the user. The settings are defined by the admin.

Cons: Not much. The issue is typically, this nuance(LDAP first, rather than OTP first) is missed during the NetScaler setup, weakening the overall setup.

2. IP Reputation with a responder policy

This reputation feature detailed here, can also be used to potentially reject IP addresses that have been black listed for one reason or another(malware etc). IP addresses will be marked as bad by a web service that the NetScaler can access, a simple responder policy, can be used to inspect the source IP of the client request and decide if the connection is to be allowed.

Pros: A simple responder policy, not much over head.

Cons: Needs the premium feature set.

3. Geo blocking IP ranges

If the service is typically only to offer access to users in a specific country or locality, why have access to the broader world? A Geo database is included with NetScaler, that could be considered as ‘good enough’ to add this layer of security.

Pros: Another, simple responder policy, not much over head.

Cons: Available on all NetScaler types, needs to be included with other techniques.

4. Allow and Deny lists of IPs

Using a call out in combination with a ‘list’, this could be Deny list or Allow list. The steps are listed here. The Deny list approach means that the admin will monitor an block denied access requests from the same IP. The Allow list is the opposite, only allowing addresses that have a successful authenticated.

Pros: It can filter out bad IP addresses.

Cons: Can be an admin heavy way of doing things, every IP address (following the Deny list approach) needs to be blocked, bad actors can change addresses quickly.

5. Using Action analytics to dynamically create an ACL in memory for the threat actors’ source IP

When this was first suggested, it is a very cool way to address an attack. The NetScaler has its configuration modified to address the behaviour of the attacker. Action Analytics can be used to monitor the logs and add/perform an ACL addition to block the requesting Source IP using a callout.

It is a somewhat over engineered approach. The bad guys get you to change the NetScaler setup. There are possibly other options that get the job done with less complexity.

Pros: Its cool! It is amazing what a NetScaler can do.

Cons: This does require a level of expertise to setup the NetScaler that is ‘uncommon’. Does it really need to be done? Again, option 1 is a more simple solution.

Nice to have/other options?

Rate limit the VPN Vserver

Set rate limit for NS GW VPN vserver and have appropriate responder policy. This should help with brute force attack when automated bots are sending several combination of user credentials at high rate.

Pros: Simple

Cons: Needs to be added to other options. Could be part of a layered defence. Likely not needed if step 1 is done properly.

Max Login and timeout

This CTX230464 support document has some simple policy steps to defend the user password. Adding a ‘Max login attempts’ and a timeout for those attempts does give the admin setting that will trigger if user fails to login within 5 attempts. The account would be locked for 15 minutes. This would push up the time to break in, assuming this is the only option set.

Pros: It is a simple setting on the gateway vserver. Should be enabled by default.

Cons: If the attack is consistent, the admin will have multiple requests for ‘unlock’ requests from the legitimate user. This is best when combined with other techniques. Another point about relying on just this option, Max login is unlikely to help very much against a broad brute force attack against many users. If, a dictionary of hundreds of likely usernames and a list of random passwords for each, it would be trivial to ensure it didn't perform multiple attempts for the same account within a period. Rather than trying three logins per minute for one user, it would be possble to try one login per minute for three users, elongating the per user time but neatly avoiding tripping the max login flag.

WAF on Gateway/AAA Vserver

The details are listed here. Starting from NetScaler release 14.1 build 21.57, you can protect the NetScaler Gateway virtual servers, traffic management virtual servers, and authentication virtual servers against malicious attacks by applying Web Application Firewall protection. This functionality provides an additional layer of security to your deployments as it filters incoming requests by validating them using the API schema and then authenticates users to access applications.

Pros: Simple to add, not dependant on having the WAF module licensed.

Cons: It is quite a new capability.

Implement Google Captcha (requires nFactor).

The option described here. This is used to make the user/requester do something only a human can do. This assumes that other techniques have not been successful.

Pros: Stops a BOT.

Cons: Can be annoying for the real users, or is that just me?

Leverage BOT Protection

NetScaler bot management provides the following benefits:

• Defend against bots, scripts, and toolkits. Provides real-time threat mitigation using static signature based defence and device fingerprinting.

• Neutralise automated basic and advanced attacks. Prevents attacks, such as App layer DDoS, password spraying, password stuffing, price scrapers, and content scrapers.

Inform the ISP

A python script could be a good option. The logic would likely be to parse data on the 3P syslog server, query splunk, etc. Match login attempts against IPs within a defined period, set a threshold value for max attempts, find the abuse contact with an automated 'whois' lookup, drop them an email with stock text. "Hey my friend, your IP address A.B.C.D has tried to login to my server 500 times or more in the last hour, you might want to look into that, etc". 

People say an attacker can create a new VM to attack from in a couple of minutes, but the reality is that it's usually a little more time consuming as you need to fill in at least some forms for a new account and pay some tiny number of dollars. If I were attacking someone and needed to google for a new VM provider, make an account, create a VM, re-upload my naughty scripts, and pay another $5 every day, it would get tiresome quite quickly even if that process only took me five minutes; I'd move on and bother someone else, especially if I were seeing no success. 

Pros: Could be used with other techniques. Automation is key!

Cons: Create a script in python to this this might require some skills!

Summary

Two factor authentication for NetScaler can be augmented with some of the above steps to keep you gateways/AAA vservers better protected against brute force attack. The document, also talks about other techniques, however using a validated baseline and setting the OTP option to validate first is the best defence.