While GRIMM engineers were not able to obtain a device or the firmware for a full analysis, the device in question looks like a Linux-based rack-mounted server that sits inside the firewall and mediates all kinds of access for clients accessing it via a web interface. Based on the available information, it appears weaknesses in the web application within the device have been the root of multiple problems, including the most recent vulnerability.
Takeaway
This was not a supply chain attack, this was a sophisticated actor becoming intimately familiar with a target and exploiting it persistently, continuing to find new vulnerabilities and tailoring their malware to blend in with the software. The signs are subtle but detectable: changes in hashes on the device, strange authentication behavior appearing in logs, small changes in network traffic such as new HTTP verbs and extra parameters being used. The attackers display impressive knowledge of the system in order to sustain persistent access, but they aren't invisible and omniscient. They would have been caught by something as routine as comparing binaries on the device to known configurations, though that's something the device appears to not allow users to do easily. The attackers employed log-cleaning tools after they gained access, but logging to a centralized server may have allowed defenders to notice the initial or successive access attempts.
Recommendation
Internet-facing network devices must be understood and their behavior logged and tracked centrally. Any device or service with high levels of administrative privilege or purview over an enterprise network should be heavily scrutinized. Security assessments will help increase confidence a device is not trivially exploitable, but sophisticated actors may still be able to find new vulnerabilities. A layered defense-in-depth strategy is the best protection, and devices should not be treated as black boxes whose behaviors and characteristics are only known and understood by the developers of the device.
Bug Details
From SA44784: This vulnerability includes an authentication by-pass vulnerability that can allow an unauthenticated user to perform remote arbitrary file execution on the Pulse Connect Secure gateway.
PCS 9.0R3 and up (Fixed in 9.1R.11.4)
- Using the blacklisting feature to disable the "URL-Based Attack"
- CVE-2021-22893 can be mitigated by importing the Workaround-2104.xml file.
- Disables "Windows File Share Browser" and "Pulse Secure Collaboration"
- Workaround-2014.xml does not work 9.0R1 - 9.0R4.1 or 9.1R1-9.1R2. If your PCS is running one of these versions, upgrade before doing the import
- The workaround is not recommended for a license server. We recommend minimizing who can connect to a license server. For example, place a license server on a management VLAN, or have a firewall enforce source-IP restrictions.
Attacks
Attackers seen exploiting four vulnerabilities, only the last one is new:
- CVE-2019-11510 / SA44101 - Unauthenicated remote file access https://kb.pulsesecure.net/articles/Pulse_Security_Advisories/SA44101
- CVE-2020-8243 / SA44588 - Unauth RCE in admin web interface https://kb.pulsesecure.net/articles/Pulse_Security_Advisories/SA44588
- CVE-2020-8260 / SA44601 - Authenticated RCE via uncontrolled gzip extraction https://kb.pulsesecure.net/articles/Pulse_Security_Advisories/SA44601
- CVE-2021-22893 / SA44784 https://kb.pulsesecure.net/articles/Pulse_Security_Advisories/SA44784
Note that the Pulse Secure security advisories contain multiple vulnerabilities nested under each and include a litany of web bugs (XXE, XSS, cookie injection, zip slip, URL sanitization problems, SSRF). Primary commonality is the web admin interface.
Recommendations from Pulse Secure on what administrators can do:
- Check the logs for unusual authentication requests
- Log unauthenticated requests via a "Unauthenticated Request option" that is off by default
- Disable admin access from the external port, which is the default setting