On April 20, 2021, Mandiant published detailed results of our
investigations into
Pulse Secure devices by suspected Chinese espionage operators.
This blog post is intended to provide an update on our findings, give
additional recommendations to network defenders, and discuss potential
implications for U.S.-China strategic relations.
Figure 1: Organizations with compromised
Pulse Secure devices by vertical and geographic location
Pulse Secure continues to work closely with Mandiant, affected
customers, government partners, and other forensic experts to address
these issues. Pulse Secure’s parent company, Ivanti, has released
patches to proactively address software vulnerabilities and issued
updated
Advisories and
to assist customers. (Please see the Forensics, Remediation, and
Hardening Guidelines section for additional details.)
Mandiant is tracking 16 malware families exclusively designed to
infect Pulse Secure VPN appliances and used by several cyber espionage
groups which we believe are affiliated with the Chinese government.
Between April 17th and 20th, 2021, Mandiant
incident responders observed UNC2630 access dozens of compromised
devices and remove webshells like ATRIUM and SLIGHTPULSE.
Both UNC2630 and UNC2717 display advanced tradecraft and go to
impressive lengths to avoid detection. The actors modify file
timestamps and regularly edit or delete forensic evidence such as
logs, web server core dumps, and files staged for exfiltration. They
also demonstrate a deep understanding of network appliances and
advanced knowledge of a targeted network. This tradecraft can make it
difficult for network defenders to establish a complete list of tools
used, credentials stolen, the initial intrusion vector, or the
intrusion start date.
We continue to suspect that multiple groups including UNC2630 and
UNC2717 are responsible for this activity, despite the use of similar
exploits and tools. There is a high degree of variation in attacker
actions within victim environments, with actors inconsistently using a
combination of tools and command and control IP addresses.
Reverse engineers on the FLARE team have identified four additional
malware families specifically designed to manipulate Pulse Secure
devices (Table 1). These utilities have similar functions to the 12
previously documented malware families: harvesting credentials and
sensitive system data, allowing arbitrary file execution, and removing
forensic evidence. Please see the Technical Annex for detailed
analysis of these code families.
|
|
|
BLOODMINE
| valign="top"> | |
BLOODBANK
| valign="top"> | |
CLEANPULSE
| UNC2630 | |
RAPIDPULSE
| valign="top"> |
Table 1: New malware families identified
Initial Compromise
The actors leveraged several vulnerabilities in Pulse Secure VPN
appliances. Mandiant observed the use of the recently patched
vulnerability CVE-2021-22893 to compromise fully patched Pulse Secure
appliances as well as previously disclosed vulnerabilities from 2019
and 2020. In many cases, determining the initial exploitation vector
and timeframe was not possible to determine because the actors altered
or deleted forensic evidence, or the appliance had undergone
subsequent code upgrades thereby destroying evidence related to the
initial exploitation.
Establish Foothold
In some cases, Mandiant observed the actors create their own Local
Administrator account outside of established credential management
controls on Windows servers of strategic value. This allowed the actor
to maintain access to systems with short-cycle credential rotation
policies and provided a sufficient level of access to operate freely
within their target environment. The actors also maintained their
foothold into the targeted environments exclusively through Pulse
Secure webshells and malware without relying on backdoors deployed on
internal Windows or Linux endpoints.
Escalate Privileges
Mandiant observed the actors use three credential harvesting
techniques on Windows systems:
In addition to these privilege escalation techniques, the actors
specifically targeted separate privileged accounts belonging to
individuals whose unprivileged accounts were previously compromised
(likely through the Pulse Secure credential harvesting malware
families). It is unclear how the account associations were made by the actor.
Internal Reconnaissance
Mandiant found evidence that the actors renamed their own
workstations that they connected to the VPN of victim networks to
mimic the naming convention of their target environment. This practice
aligns with the actor’s objective for long-term persistence and
evading detection and demonstrates a familiarity with the internal
hostnames in the victim environment.
The actors operated solely by utilizing Windows-based utilities to
carry out tasks. Some of the utilities observed were net.exe,
quser.exe, powershell.exe, powershell_ise.exe, findstr.exe,
netstat.exe, cmd.exe, reg.exe and tasklist.exe.
Move Laterally
Most lateral movement originated from compromised Pulse Secure VPN
appliances to internal systems within the environment. While
connected to the Pulse VPN appliance, the actor’s system was assigned
an IP address from the Pulse VPN DHCP pool and they moved laterally
throughout the environments by leveraging the Remote Desktop Protocol
(RDP), the Secure Shell Protocol (SSH), and browser-based
communication to HTTPS hosted resources. The actors also accessed
other resources such as Microsoft M365 cloud environments using stolen
credentials they had previously acquired.
Mandiant also observed the actors targeting ESXi host servers. The
actor enabled SSH on ESXi hosts that were previously disabled via the
web interface. When their operations on the system were finished, the
actors disabled SSH on the ESXi host again and cleared or preemptively
disabled all relevant logging associated with the performed
activities. This includes authentication, command history, and message
logging on the system.
Maintain Presence
Mandiant observed the threat actor maintain persistence by
compromising the upgrade process on the Pulse Secure Appliance.
Persistence was primarily achieved by modifying the legitimate
DSUpgrade.pm file to install the ATRIUM webshell across each upgrade
performed by an administrator. The actor likely chose DSUpgade.pm to
host their patch logic as it is a core file in the system upgrade
procedure, ensuring the patch is applied during updates. The patcher
modifies content in /tmp/data as this directory holds the extracted
upgrade image the newly upgraded system will boot into. This results
in a persistence mechanism which allows the actor to maintain access
to the system across updates.
The actors also achieved persistence in other cases by prepending a
bash script to the file /bin/umount normally used to unmount a Linux
filesystem. This binary was targeted by the actor because it is
executed by the Pulse Secure appliance during a system upgrade. The
actor’s script verifies that the umount binary executes with a
specific set of arguments, which are identical to the arguments used
by the Pulse Secure appliance to executes the binary. The inserted
malicious bash script remounts the filesystem as read-write and
iterates through a series of bash routines to inject the ATRIUM
webshell, hide SLOWPULSE from a legacy file integrity bash script,
remove or add itself from the umount file, and validate the web
process was running after a reboot to return the filesystem back to read-only.
Complete Mission
The threat actor’s objectives appear to be stealing credentials,
maintaining long-term persistent access to victim networks, and
accessing or exfiltrating sensitive data. Mandiant has observed the attackers:
Analysis of new malware families is included in the Technical Annex
to enable defenders to quickly assess if their respective appliances
have been affected. Relevant MITRE ATT&CK techniques, Yara rules
and hashes are published on
GitHub page.
To begin an investigation, Pulse Secure users should contact their
Customer Support Representative for assistance completing the
following steps:
To remediate a compromised Pulse Secure appliance:
To secure the appliance and assist with future investigations,
consider implementing the following:
In collaboration with intelligence analysts at BAE Systems Applied
Intelligence, Mandiant has identified dozens of organizations across
the defense, government, telecommunications, high tech, education,
transportation, and financial sectors in the U.S. and Europe that have
been compromised via vulnerabilities in Pulse Secure VPNs. Historic
Mandiant and BAE investigations identified a significant number of
these organizations as previous APT5 targets.
Notably, compromised organizations operate in verticals and
industries aligned with Beijing’s strategic objectives as outlined in
China’s 14th Five Year Plan. Many manufacturers also compete with
Chinese businesses in the high tech, green energy, and
telecommunications sectors. Despite this, we have not directly
observed the staging or exfiltration of any data by Chinese espionage
actors that could be considered a violation of the Obama-Xi agreement.
Targets of Chinese cyber espionage operations are often selected for
their alignment with national strategic goals, and there is a strong
correlation between pillar industries listed in policy white papers
and targets of Chinese cyber espionage activity.
China has outlined eight key areas of vital economic interest for
development and production which it views as essential to maintaining
global competitiveness, under the following categories: energy,
healthcare, railway transportation, telecommunications, national
defense and stability, advanced manufacturing, network power, and
sports and culture.
Historical Context
In the
Line Drawn report, Mandiant documented a
significant decline in the volume of Chinese cyberespionage activity
in 2014 and assessed that the restructuring of China's military and
civilian intelligence agencies significantly impacted Chinese cyber
operations. Then, in September 2015, President Xi of China concluded a
bilateral agreement with U.S. President Obama to prohibit
state-sponsored theft of intellectual property for the purpose of
providing commercial advantage. Commercial IP theft has historically
been a prominent characteristic of Chinese cyber espionage activity.
In 2018 we conducted an extensive
of Chinese cyber espionage operations, both before and after the
official announcement of the PLA reforms and bilateral agreement to
determine if there were any corresponding changes in the tactics,
techniques, and procedures (TTPs) used during Chinese cyberespionage
operations. We observed two important changes in the type of
information stolen and the geographic distribution of the targets.
Changes in Chinese Espionage Activity between 2019 and 2021
Based on developments observed between 2019-2021, Mandiant Threat
Intelligence assesses that most Chinese APT actors now concentrate on
lower-volume but more-sophisticated, stealthier operations collecting
strategic intelligence to support Chinese strategic political,
military, and economic goals. While some of the technical changes may
be the result of the restructuring of China's military and civilian
organizations, some changes possibly reflect larger technical trends
in cyber operations overall.
Redline Withdrawn?
The Obama-Xi agreement prohibits the theft of intellectual property
with purely commercial applications for the purpose of gaining a
competitive advantage. It does not cover government or diplomatic
information, sensitive business communications, IT data, PII, or
intellectual property with military or dual use applications.
Given the narrow definition of commercial intellectual property
theft and the limited availability of forensic evidence, it is
possible that our assessment will change with the discovery of new information.
Evidence collected by Mandiant over the past decade suggests that
norms and diplomatic agreements do not significantly limit China's use
of its cyber threat capabilities, particularly when serving
high-priority missions.
The greater ambition and risk tolerance demonstrated by Chinese
policymakers since 2019 indicates that the tempo of Chinese
state-sponsored activity may increase in the near future and that the
Chinese cyber threat apparatus presents a renewed and serious threat
to US and European commercial entities.
Mandiant would like to thank analysts at BAE Systems Applied
Intelligence, Stroz Friedberg, and Pulse Secure for their hard work,
collaboration and partnership. The team would also like to thank Scott
Henderson, Kelli Vanderlee, Jacqueline O'Leary, Michelle Cantos, and
all the analysts who worked on Mandiant’s Red Line Redrawn
project. The team would also like to thank Mike Dockry, Josh
Villanueva, Keith Knapp, and all the incident responders who worked on
these engagements.
The following table contains specific FireEye product detection
names for the malware families associated with this updated information.
|
|
Network Security Email Detection On Demand Malware File Malware File Storage Scanning | |
Endpoint Security | |
Helix |
|
BLOODMINE
BLOODMINE is a utility for parsing Pulse Secure Connect log files.
It extracts information related to logins, Message IDs and Web
Requests and copies the relevant data to another file.
The sample takes three command line arguments
It parses the input file for login status codes:
AUT31504 |
AUT24414 |
AUT22673 |
AUT22886 |
AUT23574 |
It parses the input file for web results code WEB20174. If it finds
a web result code, it looks for file extensions:
.css |
.jpg |
.png |
.gif |
.ico |
.js |
.jsp |
These strings indicate the type of data that is collected from web requests:
Web login, IP: %s, User: %s, |
Agent login, IP: %s, User: %s, |
Logout, IP: %s, User: %s, Realm: |
Session end, IP: %s, User: %s, |
New session, IP: %s, User: %s, |
Host check, Policy: %s |
WebRequest completed, IP: %s, |
BLOODBANK
BLOODBANK is a credential theft utility that parses two LMDB (an in
memory database) files and expects an output file to be given at the
command prompt. BLOODBANK takes advantage of a legitimate process that
supports Single Sign On functionality and looks for plaintext
passwords when they are briefly loaded in memory.
The utility parses the following two files containing password
hashes or plaintext passwords:
BLOODBANK expects an output file as a command line parameter,
otherwise it prints file open error. It contains the following strings
which it likely tries to extract and target.
PRIMARY |
SECONDARY |
remoteaddr |
user@ |
logicUR |
logicTim |
passw@ |
userAge |
realm |
Sourc |
CLEANPULSE
CLEANPULSE is a memory patching utility that may be used to prevent
certain log events from occurring. The utility inserts two strings
from the command line into the target process and patches code to
conditionally circumvent a function call in the original executable.
File Name | File | Size | |
dsrlog | ELF.X86 | 13332 |
|
The utility expects to be run from the command line as follows:
drslog <pid> <code2_string>
<code3_string> <command>
Where <pid> is the pid process ID to patch in memory,
<code2_string> and <code3_string> are two strings to write
into the target process, and <command> is either 'e' or 'E' for
installation or 'u' or 'U' for uninstallation.
During installation (using the 'e' or 'E' <command>), the
<code2_string> <code3_string> command line strings are
written to the target process at hard-coded memory addresses, a small
amount of code is written, and a jump instruction to the code snippet
is patched in memory of the target process. The added code checks
whether an argument is equal to either <code2_string>
<code3_string> strings, and if, so skips a function call in the
target process.
During uninstall (using the 'u' or 'U' <command>) the patch
jump location is overwritten with what appears to be the original 8
bytes of instructions, and the two additional memory buffers and the
code snippet appear to be overwritten with zeros.
The CLEANPULSE utility is highly specific to a victim environment.
It does not contain any validation code when patching (i.e. verifying
that code is expected prior to modifying it), and it contains
hard-coded addresses to patch.
The target code to patch appears to be the byte sequence: 89 4C 24
08 FF 52 04. This appears as the last bytes in the patched code, and
is the 8-bytes written when the uninstall 'u' command is given.
These bytes correspond to the following two instructions:
.data:0804B138 89 4C 24 |
.data:0804B13C FF 52 |
This byte sequence occurs at the hard-coded patch address the
utility expects, dslogserver. Based on status and error messages in
nearby functions the executable dslogserver appears to be related to
log event handling, and the purpose of the CLEANPULSE utility may be
to prevent certain events from being logged.
There are several un-referenced functions that appear to have been
taken from the open source project PUPYRAT. It is likely that the
actor re-purposed this open source code, using PUPYRAT as a simple
template project.
RAPIDPULSE
RAPIDPULSE is a webshell capable of arbitrary file read. As is
common with other webshells, RAPIDPULSE exists as a modification to a
legitimate Pulse Secure file.
The webshell modifies the legitimate file's main routine which
compares the HTTP query parameter with key name: deviceid to a
specific key with value. If the parameter matches, then the sample
uses an RC4 key to decrypt HTTP query parameter with key name:
hmacTime. This decrypted value is a filename which the sample then
opens, reads, RC4 encrypts with the same key, base64 encodes, then
writes to stdout. The appliance redirects stdout as the response to
HTTP requests. This serves as an encrypted file download for the attacker.
In our public report, we noted two code families that
manipulate check_integrity.sh, a legitimate script used during a
normal system upgrade. This validation script was modified by the
actor to exit early so that it would not perform the intended checks.
Per Ivanti, the validation provided by check_integrity.sh is a
separate validation feature and not the same as the
Checker Tool (ICT) available on their website. They recommend
that organizations use the online ICT to confirm that hashes of files
on their Pulse Secure devices match Ivanti’s list of known good
hashes. Please note that the ICT does not scan the rollback partition.