Shining a Light on SolarCity: Practical Exploitation of the X2e IoT
Device (Part Two)In this post, we continue our analysis of the href="http://cms.digi.com/resources/documentation/digidocs/90001537/references/r_connectport_x2e.htm">SolarCity
ConnectPort X2e Zigbee device (referred to throughout as X2e
device). In href="/content/fireeye-www/en_US/blog/threat-research/2021/02/solarcity-exploitation-of-x2e-iot-device-part-one.html">Part
One, we discussed the X2e at a high level, performed initial
network-based attacks, then discussed the hardware techniques used to
gain a remote shell on the X2e device as a non-privileged system user.
In this segment, we’ll cover how we obtained a privileged shell on the
device locally using power glitching attacks, and explore
CVE-2020-12878, a vulnerability we discovered that permitted remote
privilege escalation to the root user.
Combined with CVE-2020-9306 (discussed in Part One), this would result
in a complete remote compromise of the X2e device.
Technical Analysis
Recap
Before we dive into next steps, let’s recap where we left off:
- The X2e has an exposed universal asynchronous transmit/receive
(UART) interface, which allows a physically connected user to view
(but not interrupt) the Das U-Boot (U-Boot) boot process, and given
proper credentials, authenticate to the Linux operating system.
Since we do not have root credentials, we
put this thread on the backburner. - We have a full NAND dump
of the Spansion raw flash, which includes boot configuration,
bootloader firmware, filesystems, and the Linux kernel image. This
was used previously in Part One to obtain the hardcoded credential
for the python user.
Knowing that UART is present and access to the bootloader would be
extremely valuable, we decided to revisit that thread.
Gaining Privileged Access Locally
Revisiting the Bootloader
Figure 1 shows the U-Boot boot process displayed while connected via
UART connection. In some cases, it is possible to send keyboard input
to the device during a set period (usually one to four seconds) when
the bootloader presents the message, “Hit any key to stop autoboot,”
which interrupts the boot process and drops the user into a U-Boot
shell. On the X2e, this feature has been disabled by setting the
U-Boot configuration parameter href="https://lists.denx.de/pipermail/u-boot/2015-October/229631.html"> class="code">CONFIG_BOOTDELAY to 0.
src="https://www.fireeye.com/content/dam/fireeye-www/blog/images/solarcity2/2fig1.png" alt="" />
Figure 1: Uninterruptable U-Boot
bootloader output
One attack that has been documented to be successful to disrupt
autoboot is to manipulate the bootloader’s ability to access the flash
storage during the boot process. In certain circumstances where the
U-Boot bootloader is unable to access its own configuration, it fails
into a default environment, which may be less restricted. We decided
to see if this would be possible on the X2e.
These attacks, known as glitch attacks (or more officially known as
fault-injection), are a type of side channel attack that attempts to
cause a microcontroller unit (MCU) to skip instructions, perform wrong
instructions, or fail to access href="https://blog.nviso.eu/2020/02/21/iot-hacking-field-notes-1-intro-to-glitching-attacks/">flash
memory. Various types of glitching attacks exist including
electrical, thermal, and radiation. Based on our objective, we opted
to try glitching the power between the MCU and the Spansion NAND
flash. Note that glitch attacks can often cause damage to the
components on a board or put the device in an unusable state. These
types of attacks should be tested as either a last resort or against a
secondary device you are comfortable with damaging.
Glitching the Bootloader
Based on previous research in href="https://www.brettlischalk.com/posts/nand-glitching-wink-hub-for-root">this
domain, we opted to target the data lines (I/O) between the MCU
and NAND flash. Recall from Part One that the NAND flash on the X2e
was the Spansion S34ML01G1, which was a 63-pin ball grid array (BGA)
package. This chip is capable of supporting both 8-bit and 16-bit bus
width, which corresponds to the number of I/O lines utilized. By using
the href="https://datasheetspdf.com/datasheet/S34ML01G1.html">datasheet
for the flash and then querying the ONFI Device ID of our chip,
we determined our chip was utilizing the 8-bit configuration, meaning
eight I/O lines were present between the NAND flash and the MCU. For
this attack, we focused on manipulating the power on the first ( class="code">I/O0) data line. Figure 2 shows the configuration
of the BGA-63 pins, with I/O0 highlighted.
src="https://www.fireeye.com/content/dam/fireeye-www/blog/images/solarcity2/2fig2.png" alt="" />
Figure 2: Identifying I/O0 for NAND chip
in the Spansion datasheet
Because the pins are actually underneath the flash package, we
needed to find an exposed lead that corresponded to class="code">I/O0 elsewhere on the PCB. One such method for
tracing connections across a PCB is a continuity test. A continuity
test (using a multimeter) sends a low current electrical signal across
two points and produces an audible beep if the points are connected.
Using this technique, we located an exposed test point (known as a
via) on the bottom of the PCB. Figure 3 shows the class="code">I/O0 pin on the top of the PCB (under the NAND
chip), and Figure 4 shows the I/O0 pin
exposed on the bottom of the PCB.
src="https://www.fireeye.com/content/dam/fireeye-www/blog/images/solarcity2/2fig3.png" alt="" />
Figure 3: I/O0 on top of PCB (under NAND chip)
src="https://www.fireeye.com/content/dam/fireeye-www/blog/images/solarcity2/2fig4.png" alt="" />
Figure 4: I/O0 on bottom of PCB
With exposed access to I/O0 located, we
experimented with connecting this pin directly to a known ground
(GND) pin at various points during the
boot process. Figure 5 shows the device powering on with the metal
tweezers connecting I/O0 to GND.
src="https://www.fireeye.com/content/dam/fireeye-www/blog/images/solarcity2/2fig5.jpg" alt="" />
Figure 5: Shorting I/O0 to GND
While connected to the UART interface, we noted several different
outcomes. When shorting the pin immediately after powering on, the
device failed to produce any output or boot. When shorting after the
bootloader finished loading (and handing off to the Linux kernel), the
device would also force reboot. However, when timed perfectly between
the bootloader loading and attempting to read its configuration, we
noted that the bootloader would present different output, and the
option to interrupt the boot process was possible with a four-second
delay. By pressing keyboard input, we were successfully able to drop
into a U-Boot shell, which is shown in Figure 6.
src="https://www.fireeye.com/content/dam/fireeye-www/blog/images/solarcity2/2fig6.jpg" alt="" />
Figure 6: Access to U-Boot bootloader shell
While this was great progress, we noted that the current failback
bootloader configuration was completely inoperable and certain NAND
blocks had been marked as bad (as expected). To get our device back to
a working state, we needed to revisit the NAND dump we generated in
Part One.
Repairing the Bootloader Configuration
While the current configuration provided us a working shell, we
needed to fix the damage we had done. This was performed in two steps:
fixing the mistakenly marked bad blocks and then rebuilding the
configuration. In our case, the nand utility
and its sub-commands read, class="code">write, and scrub allowed
us to inspect and manipulate pages and blocks of the NAND. The class="code">nand scrub command with a valid offset and size
could be used to completely reset a segment of the NAND, which removed
any bad block markers. The next challenge was determining what needed
to be replaced in the damaged blocks and rebuilding the configuration.
Since we had a valid NAND image, we revisited the sections read by
the bootloader to determine what changes were needed. The format did
not match a known format, so we wrote a simple parser in Python to
read the binary structure, shown in Figure 7.
src="https://www.fireeye.com/content/dam/fireeye-www/blog/images/solarcity2/2fig7.png" alt="" />
Figure 7: Parsing bootloader nvram
configuration from flash
With details of how the configuration should look, we used the class="code">nand write to rebuild this section, byte by byte
with the correct details. We also set the boot delay to be four
seconds, so that we could always interrupt the bootloader once the new
configuration was committed. Once we confirmed our changes were
stable, we saved the configuration to flash and could access the
bootloader without performing the aforementioned glitch attack.
Accessing Linux as root User
Now that we have unrestricted access to the bootloader, we can
finally influence the rest of the boot process and achieve a
privileged shell. We alluded to this in Part One, but the easiest way
to turn an unlocked U-Boot shell into a root
Linux shell is to adjust the boot arguments that U-Boot passes to the
Linux kernel. In our case, this was accomplished by using the class="code">setenv utility to change the class="code">std_bootarg environment variable to be class="code">init=/bin/sh and instructing U-Boot to resume the
standard boot process. Figure 8 shows the Linux shell presented over UART.
src="https://www.fireeye.com/content/dam/fireeye-www/blog/images/solarcity2/2fig8.jpg" alt="" />
Figure 8: root shell after bootloader
At this point, we’ve demonstrated a repeatable method for achieving
local privilege escalation. In the final segment, we’ll complete our
attack by exploring an avenue to remotely escalate privileges.
Gaining Privileged Access Remotely
Since the X2e has only two available listening network services, it
makes sense to reinvestigate these services. During Part One, we
identified hardcoded credentials for the limited user class="code">python. This was useful for initial probing of the
device while it was running, but where do we go from here?
Embedded devices typically only have a handful of users, with a
majority of functionality being performed by the class="code">root user. This presents an interesting
opportunity for us to abuse overlap between actions performed by the
root user on contents owned and controlled
by the python user.
By reviewing the boot process, we noted a large number of custom
init scripts in the class="code">/etc/init.d/ directory. These scripts are executed
at system start by the root user and were
responsible for starting daemons and ensuring directories or files
exist. One file in particular, class="code">/etc/init.d/S50dropbear.sh, was interesting to us,
as it appeared to perform a number of actions on files within the
directory specified by the $PYTHON_HOME
variable, which was /WEB/python/, shown in
Figure 9.
src="https://www.fireeye.com/content/dam/fireeye-www/blog/images/solarcity2/2fig9.png" alt="" />
Figure 9: Unsafe operations on
$PYTHON_HOME directory
At first glance this may seem benign but considering that the class="code">/WEB/python/ directory is controllable by the
python user, it means that we can
potentially control actions taken by root.
More specifically, the chown operation is
dangerous, as the previous mkdir command can
fail silently and result in an unsafe chown
operation. To weaponize this, we can use symbolic links to point the
/WEB/python/.ssh/ to other areas of the
filesystem and coerce the root process into
chown’ing these files to be owned by the
python user. The process we took to
exploit this was as follows:
- Authenticate over SSH using hardcoded class="code">python user credentials.
- Create a
symbolic link, /WEB/python/.ssh, that
points to /etc/init.d/. - Reboot
the X2e, forcing the system to re-execute class="code">/etc/init.d/S50dropbear.sh. - After boot
completes, create a malicious init script
in /etc/init.d/ as the class="code">python user. - Reboot the X2e, forcing
the system to execute the new init
script.
While not the cleanest approach (it requires two reboots), it
accomplishes the goal of achieving code execution as class="code">root. Figure 10 shows the output of our proof of
concept. In this case, our malicious init
script spawned a bind shell on TCP port 8080, so that we could connect
in as root.
src="https://www.fireeye.com/content/dam/fireeye-www/blog/images/solarcity2/2fig10.jpg" alt="" />
Figure 10: Exploiting chown vulnerability
to gain shell as user root
And there we have it: a remote connection as root, by abusing two
separate vulnerabilities. While not explored in this series, another
viable avenue of attack would be to explore potential vulnerabilities
in the web server listening on TCP ports 80 and 443; however, this was
not an approach that we took.
Conclusion
We covered a wide variety of topics in this two-part series, including:
- Physical device inspection
- Identifying and exploring
physical debugging interfaces (UART) - Chip-off techniques to
remove the NAND storage - Binary analysis of the filesystems
and bootloader configurations - Power glitch attacks against
the U-Boot bootloader - Linux user space privilege
escalation
We hope that readers were able to learn from our experiences with
the X2e and will be inspired to use these techniques in their own
analysis. Finally, Mandiant would like to thank both Tesla/SolarCity
and Digi International for their efforts to remediate these
vulnerabilities and for their cooperation with releasing this blog series.
Source:
Shining a Light on SolarCity: Practical Exploitation of the X2e IoT
Device (Part Two)
