Changelog
- 11.01.2021: Added information about the new Citrix ADC Gateway (formerly NetScaler) firmware releases, which solve the memory leak issue with
-helloVerifiyRequest - 24.12.2020: Added information about the official Citrix Knowledge Center article CTX289674
Added a final summary, that repeats all possible solutions
Maked it a lot clearer, that-helloVerifiyRequestdoesn’t seem to work well - 22.12.2020: Added a warning note, that
-helloVerifiyRequestdoesn’t work on all Citrix ADC (NetScaler) firmware versions - 21.12.2020: Added a third possible solution regarding
-helloVerifiyRequest - 21.12.2020: Initial version
The situation
During the night from Saturday (19.12.2020) to Sunday (20.12.2020) our Zabbix Monitoring informed us, that several Citrix Gateway VPX (50) appliances were at its license cap. We investigated the situation and soon found out, that we had 0 ICA sessions on most of them, hence no explanation for the traffic.
The search
Before we started the investigation, we took each and every Citrix Gateway VPX offline, to gather time for the research.
We considered several possible root causes for the issue, for example:
- Malware on the VPX appliance
(although all are up-to-date, but you may never know) - VPX being part of a botnet
- Malware inside the customers network, leveraging the VPX in some form or another
- Bruteforce attacks, although all appliances have 2FA or MFA configured
- DDoS
After quite some tests we were quite sure, that only the last option would be viable:
To proof our theories, we brought test appliances back online and only blocked the potential source IPs of the DDoS at the corporate firewall level. The bandwidth consumption on the Citrix VPX immediately went down to zero.
One possible solution
We then started blocking the source IPs one by one on the corporate firewall, but soon realized, that this would be an impossible task, although we saw immediate relief. So we decided to completely block UDP:443 targeting the customers Gateway VIP on the corporate firewall level, as we are luckily not dependent on Citrix EDT.
The next day
During Monday morning (21.12.2020) we found additional evidence in the EUC community:
And on Twitter, that we were not alone, and that our conclusions were somewhat correct:
Anyone seen UDP reflect DDoS attacks on #citrix #netscaler lately??
— Marius Sandbu (@msandbu) December 20, 2020
It seems a worldwide UDP:443 (EDT) DDOS attack against #NetScaler #gateway is active since last night. I found these source IP addresses of the attackers in my nstraces:
45.200.42.0/24
220.167.109.0/24
45.248.9.195
206.71.159.131
46.229.195.108
117.27.239.154
13.69.68.47
(1/3) pic.twitter.com/AuAg72BsEY— Daniel Weppeler (@_DanielWep) December 21, 2020
We then double checked all our managed corporate Firewalls, if we correctly blocked UDP:443 DTLS for our Citrix Gateway customers.
Another possible solution
Additionally we had one case, where we weren’t able to block UDP:443 on the corporate firewall, so we disabled DTLS on the Citrix Gateway virtual server instead.
Third possible solution
This solution is only recommended, if you are on firmware version:
- 13.0-71.44 and later
- 12.1-60.19 and later
- 11.1-65.16 and later
Otherwise, you will have memory leaks which will crash your Citrix ADC, see paragraph below!
By now several Citrix ADC experts did came up with a third solution, which does not require to prevent any EDT/UDP DTLS connection on port 443. With the following command, you can enable that those requests are properly validated by Citrix ADC. They also mention, that this feature comes with a small performance degradation on the initial connection. On the other hand you might not need to alter your corporate firewall rules, and are still able to benefit from EDT protocol:
set ssl dtlsProfile nsdtls_default_profile -helloVerifyRequest ENABLED
If you are making use of Citrix ADC and have enabled DTLS/EDT (UDP via port 443) you might need to run this command: "set ssl dtlsProfile nsdtls_default_profile -helloVerifyRequest ENABLED". This will prevent you from future UDP amplification attacks. #NetScaler #CitrixADC
— Anton van Pelt (@AntonvanPelt) December 21, 2020
if you want to stick with @Citrix @CitrixAppDesktp @CitrixNetwork ADC NetScaler with #EDT turned-on, you might mitigate the attack surface by this:
set dtlsProfile nsdtls_default_profile -helloVerifyRequest ENABLED
there will be a small perf degradation on initial handshake https://t.co/Esl96QU1HU
— Thorsten Rood (@ThorstenRood) December 21, 2020
Update regarding -helloVerifiyRequest
Update: This issue has been solved, if you are on firmware version:
- 13.0-71.44 and later
- 12.1-60.19 and later
- 11.1-65.16 and later
There are confirmed cases on Twitter and in my blog comments (see below), that the -helloVerifiyRequest option doesn’t work reliable on all different Citrix ADC (NetScaler) firmware versions out there. Please test this setting properly in your environment. I don’t have an exact list, but there are several firmware versions out there, where -helloVerifiyRequest will lead to a memory leak, which will potentially crash you Citrix ADC after a few hours.
Yes, we‘re facing a very version- and config-dependent result. It‘s not crashing per se, and we‘re changing a factory default with that setting. In case of trouble, simply block UDP at the firewall edge and failback to native TCP-based HDX until a fixed firmware is available.
— Thorsten Rood (@ThorstenRood) December 22, 2020
Zabbix Monitoring
After the dust was settled, I also added an additional Trigger to our MSP Zabbix monitoring, were we compare now, if the traffic on the VPX is unusual high, in comparison to the number of active ICA sessions.
It’s very simple, but gets the job done:
{Template SNMP Citrix Gateway:Throughput.avg(15m)} > {Template SNMP Citrix Gateway:aaaCurICAOnlyConn.avg(15m)}
Citrix Knowledge Center Article
In the meantime (24.12.2020 here in Europe) Citrix has published an official statement regarding this subject in their Knowledge Center:
https://support.citrix.com/article/CTX289674
The official statement from Citrix is, to disable the DTLS feature on the Citrix Gateway virtual server.
Additionally Citrix announces an upcoming enhancement for the Citrix ADC (NetScaler) Firmware, where this special attack will be eliminated.
This is to be expected 12.01.2021.
Update 11.02.2021: Citrix has released new firmware versions. These firmware fixes the problems with -helloVerifiyRequest and makes it a working solution. With the current firmware and -helloVerifiyRequest enabled, you are safe from the DDoS amplify attack, while still being able to offer the Citrix EDT protocol.
Summary
There are three possible solutions, I list those in my personal preferred order:
- Set
-helloVerifyRequest ENABLEDin the DTLS profile.
It has been reported by many comments here on my blog and on Twitter, that there are Citrix ADC firmware versions, which will have a memory leak and crash a few hours later. This was fixed by Citrix, please check: CTX289674. Fixed in:- 13.0-71.44 and later
- 12.1-60.19 and later
- 11.1-65.16 and later
- Block UDP:443 traffic targeting the Citrix Gateway VIP at the corporate firewall level, to prevent your states table being overwhelmed.
- Disable the DTLS feature on the Citrix Gateway virtual server, as recommended by Citrix, if you are not ready for the new firmware version, yet.
Final words
To find out, if you are under attack, take a look at Citrix ADC Dashboard and look at the current Throughput and ask yourself if this makes sense:
Additionally you might want to tcpdump and/or Wireshark you corporate Firewall WAN interface and search for UDP:443 traffic pointing at your Citrix Gateway VIP, if it looks somewhat similar to my Wireshark screenshot above.
Also subscribe to the on-going discussion in the World of EUC Slack, which is a tremendous source for everything going on in the EUC space.
Thank you for this! I have a couple of clients being hammered with this.
Good job fellas
We have been seeing this for the last 48 hours against multiple netscalers also.
Thanks. It is a very useful article.
The 3rd solution doesn’t work. Memory is keep increasing and finally the whole netscaler die. Finally i use the first solution and now is monitoring.
I read several possible comments on Twitter, about the third solution being a good solution, but blocking the traffic at the corporate firewall level is still a valid and reliable option! Good you got it solved for now!
Hi Marco,
Thanks for sharing these details. Enabling -helloVerifyRequest causes NetScaler NS13.0: Build 61.48.nc to become unresponsive. Not recommended. Best to disable DTLS entirely until further notice.
Best regards,
Great thanks!
It really helped us.
Hi,
Good job.
What will be the impact to SSL VPN user service by disabling UDP/443 at the edge?
finding conflicting info from citrix about this.
thanks
To be honest, I don’t know, as we don’t make use of the SSL VPN feature. We have all our Citrix Gateways in ICA proxy mode.
I would either disable DTLS or block with firewall. The “set ssl dtlsProfile nsdtls_default_profile -helloVerifyRequest ENABLED” causes a memory leak (validated in v11.1 and v12.1) and you will lose access to your ADCs.
Yes, I know. I already update the blog post with that new information, did you see that? Do you think I should make this more clear?
At our company we didn’t use that method, but I added it, as many people wanted to know it.
We simply blocked UDP 443 at the corporate firewall since Sunday and are happy at the moment.
We have identified the same issue in my compagny. After a exchange with Citrix Support a public post must arrived. The Citrix response could be to use the option “Hello verify” or disable DTLS if you not use EDT but they don’t say if there is an issue with a memory leak it’s depend of the Netscaler build…
We had same issue.
on our ELK I registered more than 3400 unique external IP adresses that are participating in the attack.
So blocking the IPs one by one not the berst way to prevent the DDOS.
Yes, I agree. We simply blocked UDP:443 targeting the VIP at the corporate firewall level.
Citrix has released a CTX Article acknowledging this event..
Threat Advisory – DTLS Amplification Distributed Denial of Service Attack on Citrix ADC
https://support.citrix.com/article/CTX289674
Thank you very much. I will update the blog post accordingly.
I manage to mitigate this by learning the attack layout, you will see the payload of the packets and its all the same they are just using blocks of ip addresses to send small amout of traffic, they to capture the traffic and read the packet data. You will see the similarities of every single packets.
Acknowledged by Citrix now:
https://support.citrix.com/article/CTX289674
But they downplay the severity in a rather dismissive manner “the scope of attack is limited to a small number of customers around the world”. Every ADC I looked at was being attacked.
Thank you very much. I will update the blog post accordingly.
For us, about 90% of our public Citrix Gateway devices were under attack.
I also received some comments about Citrix ADC (NetScaler) Admins who weren’t affected and asked me for traces.
Also, maybe not everyone was DTLS enabled or makes use of EDT.
Hello Marco,
Does this threat affects the NS VPN which are exposed to internet?
What is we have VPN gateway with Site-Site corporate access?
I agree, Stephen. I believe all Citrix Gateways with EDT enabled are affected. Since there are lists of Gateways available online, the statement by Citrix is misleading at best. Sure, it is true that not all Gateways have EDT enabled and udp port 443 exposed. Additionally I’ve found the attacks are intermittent and not all environments are sufficiently impacted by the traffic generated for the attack to be detected and attributed to this vulnerability. In any case it’s good news that an update is coming!
I do not believe by default DTLS is enabled (unless some version/builds set that by default now). So unless you are trying to utilize EDT for ICA and DTLS for full VPN, the 443/UDP would not be listening. There is a third option which would be someone turning the DTLS knob to enabled without understanding why they did that.. Maybe the enhancement the CTX article is talking about is fixing the memory leak in “-helloVerifyRequest ENABLED”.
Has anyone been able to utilize the “hello verify” without losing NSIP access to their MPX/VPX instances after a few hours? The memory leak time to loss of NSIP access may vary between version/build and the amount of malicious traffic.
My NSVPN is doing tons of pings to the following IPs. DTLS was disabled and the device rebooted but to no avail. I wonder if my NS has been compromised, anyone else seeing this?
193.231.169.133
193.231.169.155
193.231.169.199
193.231.169.144
Please read the following articles, to check if your appliance is compromised. If the tests are positive, immediately replace the device.
https://www.citrix.com/blogs/2020/01/22/citrix-and-fireeye-mandiant-share-forensic-tool-for-cve-2019-19781/
https://support.citrix.com/article/CTX269180
https://github.com/citrix/ioc-scanner-CVE-2019-19781
https://www.citrix.com/blogs/2020/01/24/citrix-releases-final-fixes-for-cve-2019-19781/
What OID(s) are you using in your Zabbix template to get the ‘Template SNMP Citrix Gateway:Throughput’ value? I’ve got the aaaCurICAOnlyConn OID from the Citrix docs, but can’t work out how you’ve pulled the single Throughput value.
Citrix has updated its CTX article to include newly released firmware to mitigate this issue. https://support.citrix.com/article/CTX289674
It appears they may have fixed the memory leak with “-HelloVerifyRequest ENABLED” in the new firmware versions.
Coo, let’s find out! I’ll be pushing the update for a few customers this evening.
All my customers with DTLS enabled (whether manually enabled or not) were affected: udp/443 was blocked at the firewall to mitigate.
No issues to report with the updated 12.1 and 13.0 releases after 24 hrs. Enabling helloVerifyRequest would’ve hung up the NetScaler way earlier on previous releases.
DTLS/EDT humming along nicely once again. Will be scheduling further upgrades.
Hi Marco,
We are also using Zabbix to monitor our NS.
Would you be able to share your template in regards to this “{Template SNMP Citrix Gateway:Throughput.avg(15m)} > {Template SNMP Citrix Gateway:aaaCurICAOnlyConn.avg(15m)}”.
If the template is from an online source could you share?
If none of those are possible could you share the names and OID’s of the values used in that trigger.
Thank you sir.