The Path to the Cloud is Filled with Holes: Exploiting 4G Edge Routers

Executive Summary

• Team82 uncovered and disclosed critical vulnerabilities in ConnectedIO’s ER2000 edge routers

• These 3G/4G routers act as gateways, connecting IoT devices to the internet. 

• The vulnerabilities we found affected not only the edge routers, but also the cloud-based device management platform, and the communication protocol used between devices and the cloud

• An attacker could have leveraged these flaws to fully compromise the cloud infrastructure, remotely execute code, and leak all customer and device information

• ConnectedIO has provided firmware updates that address all of the vulnerabilities uncovered by Team82. Users are protected automatically as these updates were made to the cloud infrastructure and edge devices.

• Team82 presented this research at S4x23 earlier this year, see the video below.

Introduction

The sharp rise in the number of IoT devices has created a management challenge for companies with hundreds or thousands of connected things they need to secure. In many industrial settings, for example, it’s unfeasible to visit each device individually when security or feature updates must be applied, or when a device is down and needs attention. 

As a result, an increasing number of devices are being remotely accessed and managed through cloud-based applications from a centralized point, providing a single view of device status and configurations

In the IT realm, where almost any device is connected to the organization’s internal network, making sure every device is internet connected and has access to their cloud management platform is easy. In other environments however, connectivity isn’t so straightforward.

In real-world applications, companies might have hundreds of remote sites, each containing dozens of devices working together to enable the company's production. In many cases, connecting each site to a cable-based internet connection is not possible, either due to high costs or physical limitations. 

In order to allow these sites to be internet-enabled, a 3G/4G router is used, acting as a gateway between the remote site and the internet, and allowing the XIoT devices on that site to connect online. Through this connection, organizations are also able to manage their devices remotely. 

This use case of 3G/4G routers acting as gateways intrigued us and raised research questions: Can the same architecture be abused by attackers? Could attackers leverage this to access and control millions of remote sites around the world, disrupting thousands of processes at the same time?

This is the attack surface we wished to explore, uncovering vulnerabilities in 3G/4G routers that could expose thousands of internal networks to attackers. By leveraging vulnerabilities in the routers, attackers can compromise them and gain full control over networks. This could open many attack scenarios, enabling attackers to decommission a remote site, man-in-the-middle traffic, or attack internal XIoT devices. 

In this blog, we will zero in on our research and discovery of critical vulnerabilities in the ConnectedIO platform, mainly the 4G ER2000 edge router and cloud services. The vulnerabilities we uncovered affect all unpatched devices, enabling attackers to execute arbitrary code on these devices without requiring direct access to them, or exposing them to the internet. In addition, we also discovered vulnerabilities that put ConnectedIO’s cloud platform at risk. All of these vulnerabilities were disclosed to ConnectedIO, which has provided firmware updates that address all of the vulnerabilities. Users are protected automatically as these updates were made to the cloud infrastructure and edge devices. 

Technical Details

ConnectedIO offers a wide range of connectivity solutions, mainly focusing on 4G-capable routers aimed at enabling IoT/IIoT devices to stay connected to the Internet. In its portfolio, ConnectedIO offers a wide range of 4G routers and modems, each with different communication capabilities and specifications.

In addition to the hardware, ConnectedIO offers a cloud-based SaaS platform for managing its devices. This cloud platform allows asset owners to control and monitor their devices, and users to view data for their devices, change configurations, perform maintenance operations, and upgrade the device firmware automatically.

At the outset of our research, we needed to understand how users claim their devices and how devices register and identify themselves to the cloud.

While there are many methods through which devices can identify themselves to the cloud, ConnectedIO chose to rely on hardware identifiers burned into the device during its manufacturing: the device MAC address and IMEI number.

Using these two identifiers, the device connects to the cloud and informs that it wants to connect. ConnectedIO cloud then checks the parameters, making sure they match an actual device manufactured by ConnectedIO, and only if these two identifiers are correct will the cloud accept the connection.

After understanding how devices identify themselves in front of the cloud, our next step was to understand how users are able to claim their devices, taking ownership, and gaining full control over their devices.

In order to claim devices, a user must prove to the cloud they are indeed the rightful owner of the device. To do so, ConnectedIO’s cloud requires the user to supply “secrets” known only to the owner of the device: the serial number and IMEI of the device. Since this information is printed on the device label, located on the back of the device, it should only be known to people with physical access to the device itself.

After a user successfully claims their device, they are able to fully control it using ConnectedIO cloud. Then they are able to utilize the full functionality offered by the cloud to manage their device. 

While it is important to authenticate and validate the device, requiring it to supply “secrets” only known to it before accepting it into the cloud, we noticed a reliance of manufacturers and vendors on using hardware identifiers of MAC address and IMEI/serial number. IThese identifiers should not be used to authenticate/claim devices because they are not cryptographically secure and could be easily guessed by attackers.

A MAC address, for example, should not be relied on as an identifier because it is comprised of three random bytes, meaning that the entropy of the secret is very low. With regard to an IMEI/Serial number, usually these identifiers are sequential, meaning that by knowing the identifier of one device, it is fairly easy to guess other devices' identifiers.

Understanding Cloud-to-Device Communication

After understanding the device claiming process, our next goal was to understand how the device actually connects to the cloud and establishes a secure communication channel between itself and the cloud. 

We started by extracting the device’s firmware, which we managed to find inside ConnectedIO’s cloud platform. Extracting the firmware was straightforward because ConnectedIO did not implement a firmware packing/encryption mechanism. Instead, we simply used binwalk to extract the firmware contents, giving us access to the device’s file system.

Next, we looked for URLs and IP addresses that could be associated with ConnectedIO’s cloud. Our goal was to understand the exact communication protocol ConnectedIO chose to implement for their device-cloud communication. We discovered a URL pointing to ConnectedIO’s cloud inside a configuration file and an executable. Inside the configuration file, called redirectTo.sh, we discovered the router setting up the following configuration in the image below:

Inside this script, the router sets up the MQTT configuration for the device, including the URL, username and password. Then, it starts the cioClient binary, which handles the actual cloud communication. However, even before researching the cioClient binary, we already can infer that the device communicates with the cloud using MQTT.

MQTT is a publisher-subscriber protocol (pub-sub) aimed at allowing distributed remote communication. Two entities reside within the MQTT protocol: a client that sends and receives messages, and a broker that distributes received messages and routes them to appropriate clients.

In order to distribute the messages to the correct clients, the broker holds a list of topics, which are differently named communication channels that clients can use for communication. Clients can publish messages to those topics, or they can subscribe and receive messages sent to the topics they subscribed to. Whenever a message is sent to a specific topic, the broker distributes it to all the users who have subscribed to this topic.

This communication ranges from simple status updates the device sends periodically, to commands the cloud sends to the device and responses to those commands.

After understanding that the device communicates with the cloud using MQTT, our next goal was to understand which topics devices subscribe and publish messages to. To do so, we researched the cioClient binary, and discovered this topic scheme:

• cio/device/status: A topic used by devices to send periodical status updates to the cloud. These updates include the current state of the device, along with a discovery message of its neighbors.

• cio/device/DR{DEVICE_IMEI} (Device Receive): A topic group used by the cloud to issue commands to devices. Each device has its own unique topic, dictated by the device IMEI number.

• cio/device/DS/{DEVICE_IMEI} (Device Send): A topic group used by the devices to send responses to the cloud after it issues the device commands. Each device has its own unique topic, dictated by the device IMEI number.

Since the devices are scattered throughout the world, the MQTT broker must be internet-facing and accessible by all, see below. This made it an interesting target for our research.

Pwning Cloud Communication

After understanding the topic scheme ConnectedIO used, we wanted to see if the hardcoded credentials for the MQTT broker would actually work. After further researching the binary, we learned that it uses these credentials to authenticate to the broker, which gave us further hope we would be able to connect to it.

We then wrote a simple Python script that tries to authenticate to ConnectedIO’s MQTT broker using the hardcoded credentials we found, and to our surprise the broker accepted our connection, give us the ability to connect to and communicate with the MQTT broker just like a valid device However we wanted to fully understand our permissions; to what topics could we subscribe or publish?

Due to its business logic, devices are needed to publish to the status topic to send periodic status messages. In addition to the status topic, devices must also be able to subscribe to their own DR (device receive) topic, as well as send messages to their DS (device send) topic.

However, since all devices used the same hardcoded credentials, the broker could not differentiate between devices, meaning it had to allow all devices to subscribe to all DR topics, as well as publish to all DR topics. This meant that if we knew a device’s IMEI (which is needed to know the full topic name of a device), we could impersonate every device and receive/send messages on its behalf.

Our next goal was to subscribe to the status topic (which devices should not subscribe to, but only publish to), and when we tried doing so we managed to receive the status messages other devices sent. This meant the MQTT broker was misconfigured to allow devices to subscribe to the status topic, instead of only sending messages to it.

By subscribing to this topic, we saw tens-of-thousands of messages from devices all around the world. These messages contained simple heartbeat reports, along with more complicated configuration and sensitive information such as the device MAC address, WiFi settings, SSID and passwords.

While the information sent inside the status messages was sensitive on its own, one field in the message proved more crucial; the device sent its IMEI in order for the cloud to know which device sent the status report. This meant that by passively sniffing the status topic, we could leak all of the device’s IMEI, which is a “secret” component of the DS/DR topics, meaning we now had the ability to impersonate any device we choose. 

We now had a full list of all of the devices' IMEI identifiers, meaning we could impersonate all devices. However, while impersonating devices could prove critical to the platform, our real goal was interacting with devices on our own, meaning we wanted to issue commands to devices directly. 

To do so, we had to publish messages to the DR (device receive) topic of other devices, impersonate the cloud, and issue commands for devices to execute. Logically, the DR topic should not be publishable by devices, because it is used solely by the cloud to issue commands to devices. However, we wanted to see if a similar vulnerability to the one we found in the status topic exists in the DR topic, allowing devices to publish messages to the DR topic instead of just subscribing to it. As it turns out, this vulnerability affected the DR topic as well, giving us the ability to impersonate the cloud and issue commands to devices.

Using the ability to publish commands to the device to execute, we could force devices to execute our commands, change their configurations and much much more. This opened a vast range of possibilities for us to exploit devices.

Pwning Devices

After gaining the ability to issue commands to devices through the DR topic and leaked IMEIs, we wanted to explore what kind of commands the cloud may issue devices. In order to do so, we researched the cioClient binary, which handles the entire cloud communication. 

When researching the binary on the device, we identified the function that parses the received MQTT payload and executes the commands it receives.

The actual payload sent over the MQTT is ascii based, with parameters delimited using ”,”. When the program receives the message, it splits it by “,” and places each section in its corresponding parameter. Then, each parameter has a different purpose, for example the second parameter in the payload is an integer opcode dictating what type of command the device should execute.

One specific command caught our attention in particular: the bash command with the opcode “1116” which executes a remote command “as-is”.

This command, which does not require any other form of authentication other than being able to write it to the correct topic, allows us to execute arbitrary commands on all devices. It lacks validation that the sender of the commands is actually an authorized issuer.

Using this command opcode, we were able to generate a payload that will result in code execution whenever it is sent to a device.

Then, by sending this payload to our own device, supplying the opcode 1116 (bash command), we managed to achieve arbitrary code execution on our device.

This meant we had the ability to remotely execute code on any cloud-based ConnectedIO devices.

In addition to these vulnerabilities, we managed to identify four more vulnerabilities allowing attackers to execute remote code on all ConnectedIO devices. These vulnerabilities are tracked under the following CVEs: CVE-2023-33375, CVE-2023-33376, CVE-2023-33377, and CVE-2023-33378. 

Key Takeaways

We’re seeing more 3G and 4G routers act as gateways to the internet when devices cannot directly connect online. Locking down these routers and gateways is crucial to the integrity and availability of the IoT devices and backend services behind them. 

In this report, we researched ConnectedIO’s ER2000 edge router family firmware, and uncovered a number of critical vulnerabilities, not only on the vendor’s cloud-based management platform and the devices themselves, but also in the communication protocol implementation that supports device-to-cloud and cloud-to-device communication. 

To exploit ConnectedIO devices, we had to chain together multiple vulnerabilities, both in the devices as well as in the MQTT broker, giving us the ability to execute arbitrary code on all internet-connected devices.

First, through researching the router firmware, we found the hardcoded authentication credentials used by all routers to communicate with ConnectedIO’s cloud. We then used these credentials, along with a misconfiguration in the MQTT broker, to subscribe to the status topic and listen to messages, giving us knowledge of the IMEI identifier of all connected devices.

We then used the list we composed of all of the devices’ IMEIs to generate the DR (device receive) topic names for all of the routers, allowing us to issue these devices commands, impersonating ConnectedIO’s cloud. 

Lastly, by exploiting one of multiple vulnerabilities we identified in the routers themselves, ranging from a simple OS-command-as-a-service API, to a buffer overflow vulnerability, we had the ability to execute code on any internet-connected device.

Summarized, here is the exploitation chain:

1. Use hardcoded credentials to connect to the MQTT broker, impersonating a device.

2. Leak all of the devices’ IMEI through the status topic.

3. Generate the DR topic name for all devices (using the leaked IMEI).

4. Send devices a malicious payload invoking one of multiple RCE vulnerabilities, gaining full control over these devices.

These vulnerabilities, if exploited, could pose serious risk for thousands of companies around the world, allowing attackers to disrupt the companies’ business and production, along with giving them access to the companies’ internal networks.

To remediate these vulnerabilities, ConnectedIO has provided firmware updates that address all of the vulnerabilities uncovered by Team82. Users are protected automatically as these updates were made to the cloud infrastructure and edge devices.