Mandiant's Nick Bennett and Jake Valletta discussed data stacking
at MIRcon™ last month. If you were unable to attend the talk, we
will discuss this data analysis technique here on the M-Unition
blog.
What is Data Stacking?
Data stacking is the
application of frequency analysis to large volumes of similar data
in an effort to isolate and identify anomalies. In short, data
stacking is an investigative technique that can be used to find a
needle in a digital haystack. It involves an iterative process of
reducing large amounts of data into manageable chunks that can be
consumed and investigated.
At Mandiant we frequently
investigate large organizations with hundreds of thousands of hosts.
Looking at this data with no pre-processing and reduction, such as
data stacking, is infeasible. We commonly use data stacking to
discover unknown malware and new Indicators of Compromise (IOCs)
during our investigations.
When investigating targeted
attacks, malicious data should be much less common throughout the
enterprise than benign data, allowing investigators to concentrate
their efforts on audit rows that appear different. (We will discuss
the "different" in different rows later in the post.)
Given a large set of data, the investigator determines what
attributes make the unusual data rows stand out and may indicate
they are malicious. These attributes then become the grouping
criteria used to create the frequency analysis calculations. The
frequency, or count, is then used to determine investigative
starting points for possible anomalies within the enterprise.
Data Stacking in Four Steps
There are four
basic steps involved in the data stacking process. Most of the time
we perform steps three and four several times - grouping, filtering
and reducing the data on different sets of attribute as discussed
below. The four basic steps are as follows:
href="https://www.fireeye.com/content/dam/legacy/ammo/Fig-1_Data-Stacking-in-Four-Steps1.png">
class=" wp-image-3735 " title="Fig 1_Data Stacking in Four Steps"
src="https://www.fireeye.com/content/dam/legacy/ammo/Fig-1_Data-Stacking-in-Four-Steps1.png"
alt="" width="397" height="75" />Figure 1 - Data Stacking in
Four Steps
First, we acquire data from across the environment to
find anomalies across all hosts. It is imperative to get data from
as many hosts in the environment as possible. At Mandiant we
typically use Mandiant Intelligent Response® (MIR), but
the data can come from any source. As long as you have a
sufficiently large amount of similar data from multiple systems, the
data can be aggregated and analyzed. In the examples given below we
discuss stacking of data collected with MIR -- specifically the
services audits.
We usually skip step 2 listed above.
However, we mention it to show the iterative approach to data
reduction. This step is stacking the data without reducing the data
set to interesting attributes, which is explained in step 3
below.
At step 3, we are trying to determine a good subset of
attributes for stacking. The list of attributes in a standard MIR
Services Audit is shown below:
href="https://www.fireeye.com/content/dam/legacy/ammo/Figure-2_Standard-MIR-Services-Audit.png">
class=" wp-image-3739 "
title="Figure 2_Standard MIR Services Audit"
src="https://www.fireeye.com/content/dam/legacy/ammo/Figure-2_Standard-MIR-Services-Audit.png"
alt="" width="212" height="320" />Figure 2 - Standard MIR
Services Audit
The investigator's goal is to determine a set of
attributes that can be used to stack the data. If you pick something
that is unique to every system you will end up with a grouping that
means nothing. For example, using the process ID of a running
service is meaningless for stacking. If you group and count services
based on that attribute, you will end up with an even distribution,
with approximately the same counts for every row. Also, the fact
that something is running with a particular process ID, with no
other context, doesn't help a great deal in determining the nature
of the process.
Likewise, if you pick an attribute set that
is not sufficiently unique, you'll run into the opposite problem:
not being able to identify the "evilness" because you
don't have enough data. For instance, if you only stack (or group)
on the service name, you may be able to identify malware that is
installed with random names, but chances are you will not find
malware that uses more sophisticated techniques to disguise itself.
So, only looking at the names, will not find a malicious service
that is called "Disk Driver" (a common service name), but
is installed in C: EMP. If you were to group the data on service
name and service path you would be able to easily identify such
instances.
Determining a useful set of attributes to group the
data by, thus reducing the data set, is a balancing act. There are a
number of standard attribute sets that investigators at Mandiant
use, but even then, a lot depends on the environment, the type of
malware, the type of data available to the investigator and so
on.
That is why you would typically repeat the grouping/data
reduction/analysis process more than once, employing different
attribute sets and different ways to slice the data.
A set of
attributes commonly used for Service Audits by Mandiant
investigators is:
href="https://www.fireeye.com/content/dam/legacy/ammo/Figure-3_Set-of-Attributes.png">
class=" wp-image-3740 " title="Figure 3_Set of Attributes"
src="https://www.fireeye.com/content/dam/legacy/ammo/Figure-3_Set-of-Attributes.png"
alt="" width="124" height="95" />Figure 3 - Set of
Attributes
Using this smaller grouping of attributes, we are able
to obtain better and more manageable frequency distribution counts.
This distribution makes flagging rows for more detailed analysis
much easier. This makes analysis easier because the stacked data now
has rows at each end of the spectrum. Rows that have been seen on a
large number of systems (typically ignored) and rows that have only
been seen on a few systems. As we learned earlier, these rows that
we have only seen on a few systems are probably where targeted
attackers are hiding. With data stacking, false positives are not
uncommon, and reducing the number of rows that require investigation
makes discovery of unknown threats more feasible.
Other
investigators may choose different attribute groupings for their
stacking purposes. Most, however, have the same goal in mind: data
reduction in an effort to make manual inspection of results possible
and manageable in a relatively short time frame.
Last is step
4, looking for anomalies, which can be very tedious. Some rows
(frequently mass malware) will immediately jump out of the frequency
distribution, making identification easy. Sophisticated attackers
are typically more subtle and therefore, harder to discover. We hope
to give you a few pointers in discovering these camouflaged
outliers.
As an example, let's consider a small set of services
data that we obtained and stacked on service name, service path and
service DLL.
href="https://www.fireeye.com/content/dam/legacy/ammo/Figure-4_Service-Data.png">
class=" wp-image-3741 " title="Figure 4_Service Data"
src="https://www.fireeye.com/content/dam/legacy/ammo/Figure-4_Service-Data.png"
alt="" width="402" height="80" />Figure 4 - Service Data
In
the example above, we have two rows representing the
"Seclogon" service. One has nearly 5,600 systems with the
same attributes. We know the count roughly represents systems
because the service names must be unique, so a single system cannot
have two services with the same name allowing that system to
increment the count more than once. The other row has a much lower
count of two. From the table, we can see the difference lies in the
naming of the Service DLL, seclogon.dll vs. selogon.dll (note the
missing 'c'). In this situation the investigator would acquire the
"selogon.dll" file to determine if it's malicious.
The iprip service exhibits the same distribution: one row has a
high incidence and two rows that are uncommon. The uncommon rows are
worth investigating further for different reasons. The first has an
odd spelling of the Service DLL name and the other has the DLL in a
temp directory -- a common location where attackers place malicious
service DLLs. It is easier for malware authors to drop files into
these directories than the more protected directories, such as
system32.
It is important to note here that just because
something is not like the others doesn't automatically mean that it
is evil. For example, it is common to find a few services that are
located in C:winnt. Windows NT 4 systems are not commonly found
these days but are still legitimate installs. Therefore, services on
NT systems commonly show up in the unique rows because they are rare
in 2012, but that doesn't make them malicious. Uniqueness provides a
starting point for more thorough investigation and that is one of
the main purposes of data stacking.
Consider the following
data stack:
href="https://www.fireeye.com/content/dam/legacy/ammo/Figure-5_Data-Stack.png">
class=" wp-image-3742 " title="Figure 5_Data Stack"
src="https://www.fireeye.com/content/dam/legacy/ammo/Figure-5_Data-Stack.png"
alt="" width="423" height="68" />Figure 5 - Data Stack
The
psexec service listed above, in and of itself, is not malicious, but
is frequently used by attackers to execute tools on systems
throughout an enterprise. However, depending on the network
administration practices, the presence of this service may or may
not be a red flag. It might be that the red flag in this example is
the presence of three different versions. Or perhaps that is
expected due to other administrators using different versions.
Determining the concern level at this point requires further
investigation.
Another services audit where we are
concentrating on the Service DLLs:
href="https://www.fireeye.com/content/dam/legacy/ammo/Figure-6_Services-Audit.png">
class=" wp-image-3743 " title="Figure 6_Services Audit"
src="https://www.fireeye.com/content/dam/legacy/ammo/Figure-6_Services-Audit.png"
alt="" width="410" height="89" />Figure 6 - Services
Audit
Here we see a listing of services named "bits".
Looking at the table above we see that each row has a unique MD5.
Using a feature in MIR, the agent tries to verify the signature of
the file associated with the service. Signatures are used to ensure
that a file is the same as what the original author created. If a
single bit in the file changes so will the signature, and therefore
it would not match a previously known good signature. Shown above,
we see one service where the agent could not verify the signature.
This could mean several things: that the file was modified, that the
file does not have a signature, or that the MIR agent was not able
to properly verify it. Either way, this is worth further
investigation.
In the situation where the MIR agent is unable
to properly verify a signature (for whatever reason: an error, lack
of internet connectivity and so on) you may find other rows that
have the same MD5 and a verified signature. That means you can
ignore the fact that a signature didn't verify correctly for one row
because another row has the exact same file that did verify.
We have shown you just a few examples of the usefulness and data
reduction capabilities of performing data stacking. At Mandiant we
will stack and slice and analyze the services data set several times
using different sets of attributes: analyzing services as a whole;
and then, service paths separately; then, service DLLs and
associated MD5s separately. The process is more art than science and
there is no universal procedure. Generally, it depends a lot on the
environment and the attacker, but practice makes perfect.
Conclusion
At Mandiant, we do not limit our
data stacking analysis to service audits. We stack many different
types of audits in an effort to discover all aspects of a
compromise. For example, we have stacked particular registry keys
(think runkeys and shim cache entries), file data from file listing
audits (careful! too much data alert!), user accounts and even
Master Boot Records (MBR) from physical disks to identify
rootkits.
In all instances, the goal is the same: group and
reduce the data set to identify the outliers and make manual
analysis of an entire environment feasible.
