Hacking Operational Technology for Defense: Lessons Learned From OT Red
Teaming Smart Meter Control InfrastructureHigh-profile security incidents in the past decade have brought
increased scrutiny to cyber security for operational technology (OT).
However, there is a continued perception across critical
infrastructure organizations that OT networks are isolated from public
networks—such as the Internet. In Mandiant’s experience, the concept
of an ‘air gap’ separating OT assets from external networks rarely
holds true in practice.
In 2018, we released a blog post presenting the tools and techniques
that href="/content/fireeye-www/en_US/blog/threat-research/2019/04/triton-actor-ttp-profile-custom-attack-tools-detections.html">TEMP.Veles
used during the TRITON incident to traverse from an external
compromise of the information technology (IT) network of a critical
infrastructure organization to the safety systems located deep in the
OT network. We regularly reproduce this approach in our OT-focused red
team engagements to expose similar attack paths across client
infrastructure and to identify environment specific opportunities to
prevent and detect network propagation techniques across intermediary systems.
In this blog post, we share another case study from one of our href="/content/fireeye-www/en_US/blog/threat-research/2020/08/hands-on-introduction-to-mandiant-approach-to-ot-red-teaming.html">OT
Red Team engagements to illustrate the tactics, techniques, and
procedures (TTPs) that can be leveraged by sophisticated threat actors
to breach the protected perimeter between an IT network and an OT
network. We also examine some of the different types of critical
information often found in IT networks that an attacker can leverage
during later stages of the Targeted Attack Lifecycle. The goal of this
engagement was to access an endpoint meter control infrastructure for
a state-wide smart grid environment from the Internet and turn it off.
To hear our experts relay more on this and other OT Red Team lessons
learned, join our href="https://www.brighttalk.com/webcast/12695/480199?utm_source=FireEye&utm_medium=brighttalk&utm_campaign=480199">FireEye
Mandiant Virtual Summit session.
Visit our href="/content/fireeye-www/en_US/products/industrial-systems-and-critical-infrastructure-security.html">website
to learn more about Mandiant’s OT security practice or contact us
directly to request href="/content/fireeye-www/en_US/mandiant.html">Mandiant
services or href="/content/fireeye-www/en_US/mandiant/threat-intelligence.html">threat intelligence.
Building the Foundation: Information Gathering for IT-OT Network Propagation
Targeted OT attacks attempting to cause physical impacts require
planning. A sophisticated actor who is motivated to disrupt or modify
an industrial process from a public network will necessarily need to
maintain access to the victim environment and remain undetected for
enough time to accomplish their objective. Throughout this time, the
actor will strive to learn about the control process to formulate the
attack, figure out how to pivot to the OT systems and bypass security
controls, and sometimes even develop or deploy custom OT malware.
Similar to the techniques used by TEMP.Veles to reach the OT network
during the href="/content/fireeye-www/en_US/blog/threat-research/2019/04/triton-actor-ttp-profile-custom-attack-tools-detections.html">TRITON
incident, Mandiant’s experience during red team engagements
highlights that collecting information from IT network assets plays a
crucial role in targeted OT attacks. As a result, the internal
reconnaissance phase for OT targeted attacks begins in the enterprise
network, where the actor obtains knowledge and resources to propagate
from an initial compromise in the IT network to remote access in the
OT network. Detailed information collected about the target, their
security operations, and their environment can also support an actor's
attempts at remaining undetected while expanding operations.
src="https://www.fireeye.com/content/dam/fireeye-www/blog/images/ot-lessons-learned/fig1.jpg" alt="" />
Figure 1: Targeted OT attack from a
public network
Thinking Like an Adversary: How to Turn Off Smart Energy Meters
The ideal scenario for an attacker targeting OT systems is to
achieve their objective while remaining undetected. Mandiant’s Red
Team works with clients across critical infrastructure industries to
simulate attack scenarios in which actors can accomplish this goal by
gaining access to OT systems via compromise of external facing IT
networks. During these engagements, we emulate real actor behaviors to
learn about our target and to determine the best paths for IT/OT
network propagation.
For this engagement, we simulated an end-to-end OT-specific attack
scenario in which we tested the security controls and response
capabilities of an organization to protect smart grid meter control
infrastructure from an external attacker. Mandiant leveraged
weaknesses in people, process, and technology to gain remote access
from the public Internet and to achieve a set of pre-approved
objectives in the OT environment.
Establishing a Foothold in the IT Network
Mandiant conducted a spear phishing exercise to gain initial access
into the client’s enterprise network from the Internet. We defined a
combination of two different phishing scenarios that we deployed to
test email filtering and egress monitoring controls:
- Embedded link for a malicious file hosted on a Mandiant owned
domain on the Internet - Email attachment for a Microsoft
Office document with auto - executable macro code
This exercise allowed our team to achieve code execution on a user
workstation in the enterprise environment and to establish an
unattributable egress communication path to a Mandiant hosted Cobalt
Strike Command and Control (C&C) server on the Internet. After
establishing a stable communication path with workstations in the
enterprise environment, we utilized the following publicly available
offensive security tools (OST) to escalate privileges and to obtain
domain administrator level access:
ldapsearch to
enumerate information in the enterprise domain
href="https://github.com/PowerShellMafia/PowerSploit">PowerSploit
to exploit common security misconfigurations in IT
href="https://github.com/FortyNorthSecurity/WMImplant">WMImplant
to move laterally from one system to another in the internal
network
Mimikatz
to extract credentials for local user and domain administrator
accounts
As domain administrators, we gained unrestricted access to a variety
of resources connected to the enterprise domain (e.g. server
resources, file shares, IT applications, and administrator consoles
for IT systems). During the initial stages of our engagement, our
actions were in no way different to other less sophisticated
intrusions on industrial organizations, such as href="https://www.fireeye.com/blog/threat-research/2020/07/financially-motivated-actors-are-expanding-access-into-ot.html">financially-motivated compromises.
Defining Our Path to the OT Network
Similar to real world threat actors carrying out targeted OT
attacks, Mandiant’s OT Red Team dedicates significant effort for
internal reconnaissance in the IT network to develop a logical mapping
of the extended network architecture and discover targets of interest
(people, processes, or technology). The information we acquire helps
us to (a) define paths to propagate from the IT to the OT network and
(b) achieve our final objective in the OT network without raising
alarms. During OT Red Team engagements across different industries, we
follow a real attacker’s cost-benefit analysis to determine which
sources or methods are most likely to help us obtain that information.
src="https://www.fireeye.com/content/dam/fireeye-www/blog/images/ot-lessons-learned/fig2.jpg" alt="" />
Figure 2: Information sources and target
information from enterprise networks
For this engagement, we leveraged the domain administrator
credentials obtained in the previous phase to gain access to Microsoft
System Center Configuration Manager (SCCM) in the IT network. Logged
into the SCCM console, we leveraged software deployment features for
collection to establish C&C over user workstations belonging to
pre-selected departments in the client organization.
Mandiant chose the specific groups based on the names of their
departments and the description attributes, which suggested a high
likelihood of member users with high privilege access for network
infrastructure or application management. This included members of the
following groups: network management, firewall administration, control
engineering, and smart meter operations.
Access to user workstations of target employees in these departments
enabled us to:
- Capture keystrokes to obtain remote desktop protocol (RDP)
credentials for the OT network by using a Cobalt Strike modified
script - Login to department file shares and extract OT system
design documents - Extract network design documents and
backup files for OT firewall configurations found in the firewall
management console - Find plaintext credentials for OT
management systems from operation manuals
Internal reconnaissance in the IT network not only allowed us to
obtain remote access credentials for the OT network, but to also gain
a deeper understanding of the business processes and technical control
system operations in the OT environment by reviewing internal
OT-specific operational procedures and security documentation such as
asset inventories and configurations.
Propagating to the OT Network
During the process of propagation from IT to OT networks, an actor
will leverage previously compromised systems, credentials, or
applications to access systems in higher security zones—such as OT
demilitarized zones (DMZ). Based on our observations during multiple
red teaming engagements and research, the most likely attack vectors
for propagation are:
src="https://www.fireeye.com/content/dam/fireeye-www/blog/images/ot-lessons-learned/table1.jpg" alt="" />
Table 1: Most likely attack vectors for
IT/OT propagation
For this engagement, we initially analyzed the system architecture
to define the best path to follow. Engineers from the target
organization were required to use multi-factor-authentication (MFA) to
gain remote access to jumpbox servers in the OT network. While not
impossible, bypassing this setup would require more time and
resources. We instead decided to search for other plausible attack
propagation paths.
src="https://www.fireeye.com/content/dam/fireeye-www/blog/images/ot-lessons-learned/fig3.jpg" alt="" />
Figure 3: Formal communication path from
enterprise to OT network
Reviewing the firewall configuration files, we identified a
dedicated communication path for management access to a Microsoft
Windows patch management server in a DMZ between the IT network and
the OT network. This patch management server was running on a virtual
machine in the DMZ network, while the administration console for the
underlying hypervisor software itself was hosted in the IT network.
Mandiant logged into the administration console for the hypervisor
software using IT network domain administrator credentials. We then
leveraged guest machine administration features via direct console
access to execute commands on the patch management server in the DMZ
network. The compromise of the patch management server in the DMZ
allowed us to pivot via SMB connections to Microsoft Windows-based
intermediary systems in the OT network.
src="https://www.fireeye.com/content/dam/fireeye-www/blog/images/ot-lessons-learned/fig4.jpg" alt="" />
Figure 4: Remote attack propagation path
from IT network to OT network
Lastly, we compromised Microsoft Windows server systems in the OT
network to complete the objectives of the exercise. Using OT
credentials retrieved in the previous phases, we authenticated to the
SMB service (using single factor authentication) by pivoting through
the patch management server in the DMZ network. This enabled us to
execute remote console commands on management servers (such as the
domain controller) in the OT network.
With access to the domain controller in the core OT network, we
extracted credentials for high privilege domain administrator accounts
in the OT network using DCSYNC and Mimikatz. Using these accounts, we
gained control of management servers, application servers, and
operator workstations in the OT network. Mandiant was also able to use
compromised credentials to login to the human machine interface (HMI)
portal for the meter control infrastructure and issue a disconnect
command for a target endpoint meter in the smart grid environment.
Strategic Collection and Detection Opportunities During
Reconnaissance and Network Propagation
Although specific capabilities such as malware and tooling vary
amongst incidents, internal reconnaissance and network propagation are
consistently needed for sophisticated adversaries to expand remote
operations from external networks to OT systems. Focusing collection,
detection, and hunting efforts on assets or information that are
likely to be compromised during these phases presents defenders with
strategic opportunities to hunt for and detect targeted adversary
activity before it poses a risk to control systems.
- In a previous blog post stating our approach to OT security,
we highlighted that IT networks close to production networks and OT
intermediary systems remain the best zones to detect OT targeted
attacks, a.k.a. “ href="/content/fireeye-www/en_US/blog/threat-research/2019/12/fireeye-approach-to-operational-technology-security.html">The
Funnel of Opportunity”. As actors pivot across systems and
networks to gather information and elevate privileges, they leave
footprints that can be tracked before they propagate to critical
systems. - An actor who covertly performs internal
reconnaissance and propagates to the OT network is already
positioned to cause damage on mission critical assets and is
unlikely to be discovered. Early detection of adversary activity
before reaching critical OT systems will decrease the dwell time and
the risk of an incident. - OT defenders can prioritize
collection and detection, alert triage, and incident response
efforts by becoming familiar with the types of information and
services that OT focused threat actors commonly search for during
internal reconnaissance in IT networks and network propagation
across OT intermediary systems. - Understanding where this
information resides presents defenders with a catalog of systems and
networks to focus collection and detection efforts on. Defenders can
create tailored detections to hunt for adversary activity pursuing
this information, prioritize alert response efforts, and identify
additional security controls to implement. Mandiant red teaming in
OT can help organizations identify which data is valuable for
attackers to support their network propagation efforts and which
systems are most likely to be compromised by attackers targeting OT
networks.
Visit our website for href="/content/fireeye-www/en_US/products/industrial-systems-and-critical-infrastructure-security.html">more
information or to request href="/content/fireeye-www/en_US/mandiant.html">Mandiant
services or href="/content/fireeye-www/en_US/mandiant/threat-intelligence.html">threat intelligence.
Source:
Hacking Operational Technology for Defense: Lessons Learned From OT Red
Teaming Smart Meter Control Infrastructure
