Avaya's IP Phone Security Research - Part A

VoIP Phones: A Potential Attack Vector

A voice over IP (VoIP) phone enables you to place and transmit telephone calls over an IP network, like the Internet, instead of the traditional public switched telephone network (PSTN), using control protocols, such as the Session Initiation Protocol (SIP), Skinny Client Control Protocol (SCCP), and various other proprietary protocols. While convenient and cost effective, if an IP phone is compromised it can give an attacker a way into the network. Knowing this, we took a look at Avaya's IP phones, which are quite common in healthcare delivery organizations, to try to understand their vulnerabilities and the risks they may pose. This blog details the research we did into an Avaya 9608G IP Phone - the findings can be generally extrapolated to other Avaya models because they have a common base.

In this part, we'll show a research setup and preparation for the vulnerability research. The Target was to extract the firmware and allows dynamically debug and reverse engineer the system.

Preliminary Setup

First, we bought a couple of 9608G VoIP phones, powered them up, and connected them via a POE Adapter to our research network. Then, we set up  Asterisk as an operation SIP server, in order to emulate an operational VoIP network.

Firmware Exploration

We downloaded the latest SIP software update for the 9608G from the Avaya support website, called “96x1-IPT-SIP-R71100-062220.zip”. The software update contained several binaries. We checked the “96x1Supgrade.txt” file, which is the file the phone uses to select a software update, and based on its model, it pointed to the “S960811SALKRR71100r6.bin” file:

# 9608GBSW
SET APPNAME S960811SALKRR7110_0r6.bin
GOTO GETSET
We ran binwalk on this binary and found a UBIFS file system at offset 0xA8FB0:

692144 0xA8FB0 UBIFS filesystem superblock node, CRC: 0x3B14D669, flags: 0x4, min I/O unit size: 2048, erase block size: 126976, erase block count: 431, max erase blocks: 2047, format verson: 4, compression type: zlib
819120 0xC7FB0 UBIFS filesystem master node, CRC: 0x29AECE8E, highest inode: 1520, commit number: 0
946096 0xE6FB0 UBIFS filesystem master node, CRC: 0xAFB4FCE5, highest inode: 1520, commit number: 0
Then we extracted the FS with nlitsme / ubidump. We found the Firmware Update was NOT Encrypted, so we could start exploring the Binaries.

IP Phone Exploration

We started to look into the menu of the 9608G VoIP phones. We saw a boot menu that can be entered by pressing a password. The default password is 27238( 'C','R','A','F','T','#'). We noticed there was a Debug mode option on the menu. We enabled it. After we enabled it, we noticed (via a port scan) there was a SSH console enabled. Unfortunately, it required challenge-response authentication. So, we decided to look for a Serial Console that would hopefully give us root access to the files. If successful, we could disable the PAM authentication by editing the ssh_config file.

Serial Console

When we opened the plastic enclosure of the Serial Console, the PCB was revealed. We then found '3 test points' with a 'silk mark' beside them, which told us they are UART signals (Tx, Rx, GND). From some investigating on the Internet, we found  the Serial Console could be enabled by configuring the EEPROM properly. The EEPROM could be found near the UART TP's.

EEPROM

The Electrical Erasable Programmable Read-Only Memory (EEPROM) basically means its a read only memory (ROM) that can be erased electrically and reprogramed, in case needed. The EEPROM in the 9608G VoIP phone is M95080-W. It holds 1024 bytes (0x400 bytes) and can be accessed via a SPI connection.

We assumed there was a binary in the filesystem that we could extract to handle the EEPROM and reverse it, which would help us understand the structure of the data in the EEPROM. We ran "rgrep" on the FS to find any references to the string "EEPROM" and we found the following:

etc/init.d/log: # SIP controls log + console from BM/console EEPROM setting
etc/init.d/log: # BM/console EEPROM setting.
Binary file bin/LogControl matches
Binary file bin/Upgrade matches
Binary file bin/ft matches
Binary file sbin/ethtool matches

We noticed binaries with the string "EEPROM" were importing a dynamic library with the name "libavaya.so". We looked for the library and found it at "/usr/lib/libavaya.so" and imported the primary binaries:

The EepromGetEntry got element index (a1) and returned a pointer to an ASCII string that held the content for the specified index at "/bin/LogControl". We found the following:

You can see that index 0x26 holds the string "UART" and probably the relevant configuration. To understand the indexes, we took a dump of the EEPROM content, after we de-soldered the chip and read it with a memory programmer, and tried to find matches to strings that we knew were in use in the binaries:

We noticed that we couldn't find the string UART at all, so we looked for others we knew, such as “9608G”, which is the model number, at 0x280:

We could see the index for it was 0x23:

We had first thought that each 16 bytes (0x10 bytes) was raw, therefore the string for index 0x23 should have been at 0x230. But we found it at 0x280, so we realized we had a mismatch of the offsets. We wanted to understand the function EepromGetEntry to see how it fetches data. We noticed EepromGetEntry has access to an array of structs, and that each struct holds the real offset in the EEPROM, including the size it should read. That means each raw can be at a different length. The index for EepromGetEntry is the index of the array of structs with associated access to the right struct.

For example, index 0x23 holds the string "MODEL_NUM", which is probably an info field at the struct, and has the offset of 0x280 with the size we thought (0x10). So, when we went back to the UART string, we knew it should be at index 0x26, (info="P_R2"), so the offset was 0x2B0 and the size 0x10:

We saw the Value was "BM" instead of "UART", so we decided to change it to "UART". Before we patched the EEPROM, we noticed that the last byte was not zeros. We assumed it was a checksum of the raw, and found the below function:

We found it was a sum of all the values at the raw, except for the last one, which holds the checksum, and reversed it (not_8()). In this case, the patched raw became:

We programmed the patched content back to the EEPROM, but it did not seem to have any affect on the UART at all, so we had to go back to the drawing board.

Patching file via the NAND Flash

We decided to try to enable the SSH interface by editing the ssh_config and patching the NAND Flash content.

The goal was to:

1. Read the NAND Flash

2. Explore the structure

3. Understand the FileSystem

4. Mount it on our PC

5. Change the ssh_config

6. Unmount the patched FS

7. Combine the binary back to one blob

8. Write back to the flash

(Spoiler Alert: it was way more difficult than we thought)

NAND Flash

NAND Flash is non-volatile storage organized in blocks, where each block contains several pages of data. A page is the smallest object that can be read. If you want to write something, you need to erase a full block and then write the relevant pages.

The Flash assembled in the Avaya 9608G VoIP is S34ML02G100TF100:

• Each page contains 2048 bytes and 64 spare bytes (a.k.a. OOB bytes) for a total of 2112 bytes

• Each block contains 64 pages (204864 +6464) for a total of 132 kiloybytes, which is 128 kB without the OOB

• Each plain contains 1024 blocks

• The total amount of plains in the chip is two

Therefore, the total size of the Flash is 256 megabytes with 8 MB of OOB

Firmware Extraction

To extract the firmware we had to de-solder the Flash and read it with a Flash Programmer:

We used the "Peebrog2" programmer with a NAND-3(70-3081) adapter.

We read the flash content twice:

1. With the OOB

2. Without the OOB

Then we ran Binwalk on each blob:

binwalk ./2ndtrynosparebytes.bin
DECIMAL HEXADECIMAL DESCRIPTION
--------------------------------------------------------------------------------
16079 0x3ECF Unix path: /localdisk/a1/xml-data/build-dir/PHONEX6-COMMON2-JOB1/brcm-sla/bootstraps/util/lowlevel/nand_core.c
16340 0x3FD4 Unix path: /localdisk/a1/xml-data/build-dir/PHONEX6-COMMON2-JOB1/brcm-sla/bootstraps/util/lowlevel/nand_block.c
179108 0x2BBA4 Unix path: /localdisk/a1/xml-data/build-dir/PHONEX6-COMMON2-JOB1/brcm-sla/bootstraps/util/lowlevel/nand_core.c
3276800 0x320000 JFFS2 filesystem, little endian
3423220 0x343BF4 gzip compressed data, maximum compression, from Unix, last modified: 2014-05-13 15:31:14
6568948 0x643BF4 gzip compressed data, maximum compression, from Unix, last modified: 2014-07-18 13:55:48
9699328 0x940000 JFFS2 filesystem, little endian


We can see the binwalk finds two JFFS2 file systems, which are different from the Firmware update. The assumption is one of these file systems is the primary RootFS and the second one is the backup RootFS. We tried to mount them and extract them, but our attempts fail.

JFFS2 file system

A JFFS2 is a file system for embedded systems with Flash devices, which is often NAND Flash. The JFFS2 is a journalist file system, which means it holds logs of any changes to the file.  A JFFS2 contains a physical management layer and can manage the block writes by itself, therefore, it can access the OOB and change the values. To be able to manage the Flash itself, without any external physical management, it has to "live alone", in a partition. Inside the partition, JFFS2 can change the location of each block each time it needs to be written, in order to extend the life of the blocks.

Block management

Each block the JFFS2 writes is marked with a JFFS2 mark [OOB (0x1985)]. Each time a page is modified, the JFFS2 moves it to a "clean" block. When all the blocks in the partition are "dirty", the JFFS2 will erase it.

Finding the partition table

As noted above, the JFFS2 FS is in its own partition. If we were to merge two JFFS2 to one partition, when we try to build a tree with both blobs we will get node conflicts, so we need to find and figure out the partition table. When we explored the firmware update, we came across the name "AVayaDIr". We started to look for the string "Avaya" and found some sort of partition table at the very beginning of the binary blob. We then tried to arrange it to find Periodic behavior:

From the figure above, each row in the table holding 0x1b bytes and representing a specific partition, we see there are 9 different partitions. The structure of each raw is as below:

From the struct above, we can build the partition table of the binary:

Then we can split the blob into 9 partitions and parse them separately.

When we separated each partition, we found there were not just two JFFS2 partitions, but 4! They consisted of:

1. nvdata - some configuration files that probably change between different models or regions

2. root0 - RootFS - seems to be a backup

3. root1 - RootFS  - the main ROOTFS

4. AvayaFS - user modules and bash scripts that are related to the specific model.

To mount them separately, we used MTD-utils.

MTD-utils

Each Linux distribution can include MTD-utils, which contain several utilities to provide a generic interface between the hardware and the upper layers of the system. The last known utility is "nandsim". This utility is intended to be used for debugging purposes. We used it to simulate the real behavior of the Flash to mount the JFFS2 partitions. The nandsim is a kernel module, so we need to use "modprobe" with root privileges:

sudo modprobe nandsim firstidbyte=0x01 \
secondidbyte=0xDA \
thirdidbyte=0x90 \
fourthidbyte=0x95 \
parts=1,4,4,17,24,24,384,384
When the 4 ID bytes match the NAND Flash used in the VOIP (S34ML02G100TF100), the ID code (4 bytes) can be found in the datasheet of the relevant Flash. The "parts" argument represents the size of each partition in the number of blocks, where each block is 0x2000bytes. For example, the second partition is 0x8000, so we know it contains 4 blocks. The 9th block gets the rest of the Flash, so we don't need to do anything with it at this time.

After we load the nandsim module, we can use "mtdinfo" to see if the partition loaded:

Now that we know we have the right partitions divides, we can load the four JFFS2 binaries blob that we split before, using the "nandwrite" utility (part of the MTD-utils):

nandwrite -p /dev/mtd3 ./avaya/2nd_flash/partitions/nvdata.bin
nandwrite -p /dev/mtd6 ./avaya/2nd_flash/partitions/root0.bin
nandwrite -p /dev/mtd7 ./avaya/2nd_flash/partitions/root1.bin
nandwrite -p /dev/mtd8 ./avaya/2nd_flash/partitions/avayafs.bin

Then we load the mtdblock module by running:

sudo modprobe mtdblock

Finally, we mount it as JFFS2:

modprobe jffs2
mount -t jffs2 /dev/mtdblock6 /mnt/root0
mount -t jffs2 /dev/mtdblock7 /mnt/root1
mount -t jffs2 /dev/mtdblock8 /mnt/root1/AvayaDir
mount -t jffs2 /dev/mtdblock3 /mnt/root1/nvdata

We seem to have some success!

Loading partitions with OOB

Because we want to mount the FS and overwrite some files, we need to also load the OOB and let the JFFS2 module and nandsim module change it. This will make sure the Flash won't be corrupted when we write the modified content back. As explained above, each page has 64 bytes of spare bytes, which is meant for managing the Flash. The structure of the OOB for each page is:

Error correction code

The error correction code (ECC) is used to control errors in data to improve the reliability of the memory. In order to produce a chip for NAND Flash memory, manufacturers had to improve the yield, which meant they needed a way to use NAND Flash with a slight defect. This is where ECC came in. It was added to correct minor errors in the Read/Write process. In addition, NAND Flash can get "tired", which means after some number of writes some bits might stuck. Without ECC, the Flash management would mark all the blocks as a bad block because of those few bits. With ECC, the block is fine.

We used Autoplace to identify the ECC in the JFFS2 file system. The structure we depicted above is for Linux Hamming code. Other correcting codes may have a different structure.

We then read the Flash with the OOB, split it again into partitions, and loaded them:

flash_erase /dev/mtd3 0 0
flash_erase /dev/mtd6 0 0
flash_erase /dev/mtd7 0 0
flash_erase /dev/mtd8 0 0
nandwrite -o /dev/mtd3 ./partitionswithoob/nvdata.bin
nandwrite -o /dev/mtd6 ./partitionswithoob/root0.bin
nandwrite -o /dev/mtd7 ./partitionswithoob/root1.bin
nandwrite -o /dev/mtd8 ./partitionswithoob/avayafs.bin
modprobe mtdblock
modprobe jffs2
mount -t jffs2 /dev/mtdblock6 /mnt/root0
mount -t jffs2 /dev/mtdblock7 /mnt/root1
mount -t jffs2 /dev/mtdblock8 /mnt/root1/AvayaDir
mount -t jffs2 /dev/mtdblock3 /mnt/root1/nvdata

For some reason we got an ECC error.  We assumed it was because the ECC algorithm was different. We decided to look at OOB area in the binary:

We saw in the area reserved for the ECC (according to the Linux structure of the OOB), there was FF's with no dependency on the values on the current page. We tested it and saw the same thing for more than 10 different pages. When we disregarded the structure we expected, we saw that every 16 bytes (0x0F) start with 3 FF's. We also saw there were 13 LSB's with high entropy (looks like random). This allowed us to mark each row as some object.

After doing some Internet research, we found that Broadcom has its own way to calculate the ECC, using a hardware module that is part of each Read/Write operation (to improve efficiency). To make the hardware module generic, they support pages with the size of 512 bytes. If the page is larger, it can contain several sub-pages. For example, in our case, the page size is 2048 bytes, so each page contains 4 sub-pages with a size of 512 bytes and 16 OOB bytes. This means we have four groups of 16 bytes OOB. The ECC algorithm they use is based on the BCH algorithm. As we know, the first two bytes are used to mark for bad blocks, but we found some OOBs where only the first byte was marked as 0xFF (non-bad block), so we made the assumption that the Broadcom OOB structure is:

We then wanted to understand the BCH configuration that Broadcom used. We found a repo for fixing the ECC for Broadcom (https://github.com/StrayLightning/brcm-nand-bch), but that configuration didn't match ours, so we needed to reverse the current configuration. The parameters we needed to reverse were:

From the OOB structure we can place the sizes and offsets values at:

1. sector_sz - the size of the blob for which the ECC is calculated. In our case, this is the subpage size, which is sector_sz=512bytes

2. OOB_sz - the size of the OOB (used for validation purposes). In our case, this is the size of the OOB group, which is OOB_sz=16bytes 

3. sectorperpage - respectively to the sector_sz. The sector is actually a subpage, so we obviously have four subpages:  sectorperpage=4

4. OOBECCOFS - the assumed offset of the ECC from the start of the subpage OOB. As mentioned above, the 13 LSB's belong to the ECC: OOBECCOFS=3

5. OOBECCLEN - corresponds to the LSBs above: OOBECCLEN=13

This means we are left with: BCH_T and BCH_N.  Through documentation on the BCH algorithm, we know there is a relationship between them and the ECC_LEN:

We already know that OOBECCLEN=13.  In addition, we know there is a dependency of BCH_T that must be multiplication of 2 (2,4,8...). Broadcom usually uses BCH_N=13/14. From this information, we deduce the only combination is BCHN=13, BCHT=8, which means the final configuration is:

After we compiled the utility from the repo with the modification, we ran it:

cat inputfile.bin | reencode >outputfile.bin

Patching File sequence

Once we understood the ECC algorithm, we knew how to correct it, so we can ignore the error and do the following:

1. Mount the partitions with the OOB area

2. Patch the desired file

3. Unmount the modified partition (without resetting the MTD blocks)

4. Use "nandump" to read the modified partition

5. Use reencode utils to correct the ECC

6. Build a unified blob from all the partitions

7. Write back the blob to the Flash

Patching SSH files

At this point, we can patch the files we wanted to patch - the SSH configuration files. We decided to create a workaround for the authentication and patch the following files:

/etc/pam.d/sshd
/etc/ssh/shd_config

From the PAM configuration file, we comment out most of the "required" action. At:  shd_config, we allow root users to connect and disable the use of PAM authentication as default.

Final results

After patching the above files, unmounting the partitions, fixing the ECC in the root partition (where the SSH files are stored), combining all the partitions into one binary blob, and writing the modified blob back to the Flash, we were allowed to access the SSH interface with the credential : root:root