Understanding and Improving the XIoT Vulnerability Disclosure and Publishing Processes

Introduction

Thousands of new and exploitable vulnerabilities have been disclosed in recent years that make devices within the Extended Internet of Things (XIoT) a major target for threat actors. This puts new pressure on vendors to understand and connect with the research community finding and disclosing these vulnerabilities. It's also important for vendors to develop vulnerability remediation strategies to accept bug reports, and triage and remediate flaws within connected industrial, healthcare, and commercial devices before they're exploited in the wild.

For many industrial control system and operational technology vendors, this is new territory. Smaller providers may be just now taking their first steps toward a successful vulnerability remediation program, and understanding how and why researchers look for software and firmware vulnerabilities in products, and why it's crucial to update these systems in a timely coordinated fashion. The urgency to find and fix ICS and OT vulnerabilities is clear: No other technology impacts human safety more.

In this blog, we hope to bring some clarity to the vulnerability disclosure process from the researcher's point of view. We also believe this will help vendors and technology providers understand the motivations of research organizations such as Claroty's Team82 and independent white-hat hackers, and the benefits they can bring to improving product safety and reliability, as well as share examples of successful coordinated vulnerability disclosures.

Vulnerability Disclosure Process

Finding vulnerabilities is not trivial. It takes quite a bit of experience and investment to understand how software and firmware packages work, and how to trigger vulnerabilities that can be exploited by a threat actor. Team82 is no exception, and when we find a vulnerability, we initiate an important disclosure process and try to work closely with vendors to securely share details about the flaw and eventually get them patched. Many times Team82 writes extensive technical explanations of how we were able to trigger the vulnerability we're disclosing, and also adds proof-of-concept (PoC) exploits and video demonstrations of the exploits. Team82 may also advise the affected vendor how to fix the vulnerability in the most secure way possible, preventing future vulnerabilities in the code and the logical flow of the products researched.

Our Insights From Disclosing 300 XIoT Vulnerabilities

Team82's experience with disclosures is extensive. Since our inception, we've disclosed more than 300 vulnerabilities to more than 50 affected vendors within the XIoT domains. The chart below illustrates the percentage of vulnerabilities disclosed to each affected vendor.

After working with so many different vendors, we can draw numerous conclusions, some of which hearken back to the early days of IT vulnerability disclosures, including the distrust many vendors had of researchers, and the challenges in standing up a remediation program with XIoT:

We'd like to call out four:

• Some vendors we worked with were diligent about product security and care about their customers very much, but sometimes lacked the right tools and the understanding of cybersecurity and the risk to business and public safety in some cases.

• As a researcher, you may be considered the bad guy until the affected vendor  understands you are providing help.

• Effective communication between researchers and affected vendors remains a challenge, especially for some XIoT vendors who haven't established mature programs and lack secure communication mechanisms.

• Patience is a virtue, and disclosure is a journey; you have to treat the other party with utmost respect.

Researchers Contacted My Organization After Finding Vulnerabilities, What Should I Do?

Often, the first and only indication a vendor gets about a vulnerability is from a research organization such as Team82—or if the flaw has been exploited in the wild first. Each of the vendors Team82 has worked with on disclosures has varying degrees of maturity to their vulnerability remediation programs. Some are successful and established, while others are not as much.

Some programs start with a dedicated product security page with a secure email address clearly available for reporting vulnerabilities. This page should also make available a public PGP key for encrypted communication about the disclosure. A secure contact page is an indication that the vendor has a well-defined, dedicated process for accepting vulnerability reports and is intent on improving the overall security of their products.

On the other hand, Team82 has attempted disclosures to some affected vendors where it was very challenging to find a person to report the vulnerability to. A number of vendors, for example, do not provide a simple, secure-contact email address and PGP key. This forces researchers to hunt and peck for people within the organization to contact—sometimes through personal email addresses—to notify them about an issue without disclosing sensitive details over an unencrypted channel.

There have also been cases where  vendors refused to cooperate, either ignoring our disclosure reports or asking us to cease communication about a flaw they refused to acknowledge.

Below, you will find a breakdown of the affected vendors Team82 has disclosed with, and the available product security elements related to disclosures and vulnerability remediation:

Moreover, we're pleased to report that 11 vendors we disclosed to published a product security webpage, dedicated security email address, and PGP key after our disclosure process was initiated with our guidance.

Being Organized Is The Key To Better Product Security

An organized vendor will be able to receive a disclosure report, and maintain and patch their products in the most efficient, robust, polished way.

These are five key steps to a coordinated vulnerability disclosure process:

Vulnerability Discovery

Uncovering new vulnerabilities in one or more of a vendor's products often requires extensive examinations of software and firmware packages, understanding logic flows and code dependencies, and the use of manual and automated means to trigger conditions that would allow an attacker to either crash a device or execute code.

Report

Once a vulnerability has been confirmed, the researcher reaches out to the affected vendor and shares their findings. Here are three steps in this part of the process:

1. Contact the vendors using the dedicated email address provided on the product security page, encrypting the session with a PGP key, getting a ticket/issue-tracking number in response.

2. The researchers shall provide as much detailed technical information, PoCs, and perhaps videos demonstrating exploitation.

3. The ideal initial outcome is that the vendor then responds to the researchers, letting them know that their disclosure is being reviewed and verified.

For reference, you can read Team82's coordinated disclosure policy for reporting new vulnerabilities.

Triage

The vendor should then confirm that the vulnerability indeed exists and assesses its severity.

1. If a product computer emergency response team (PCERT) exists, it will try to do initial review and triage, and if needed, pass the ticket to the relevant body within the organization.

2. Using the information and PoCs provided by the researchers, the PCERT will try to reproduce the vulnerabilities internally, and also target a wider range of versions of the same product that may be affected, as well as other products and specific configurations.

3. Then the vendor will have a mapping of all vulnerable product versions that will be used to assess the risk level of this vulnerability.

Remediation

The vendor will try to patch or mitigate the vulnerabilities, and provide a secure solution to its users.

1. In some cases it might be beneficial for the vendor to consult with the researchers about the best way to patch or mitigate the vulnerability.

2. Before the team starts to fix the vulnerability, it should try to learn the root cause in order to prevent the same class of vulnerabilities from surfacing again. This can be done by micro-training developers on relevant security topics.

3. Successful vendors have included a cybersecurity expert in product architecture meetings in order to improve software and firmware coding practices.

4. The relevant development teams shall fix the vulnerabilities in as many products as possible.

5. Vendors should consider it a best practice to send the researchers a patched version of the product for evaluation to determine whether the vulnerabilities have been remediated as intended.

Vulnerability Disclosure

The affected vendor must next disclose to users—and the public—that a vulnerability has been reported and patched, either through an advisory or a post on their product security page

1. The affected vendor should assign each vulnerability a CVE ID through the relevant CVE number authority (CNA).

2. After a CVE ID has been assigned, a relevant advisory needs to be published through a dedicated cybersecurity advisories page or any other method chosen for publication.

3. Also a patch must be released and distributed to users.

The Product Security Page

Mature organizations with a PCERT should afford that team a public presence on the vendor's website. This page is the gateway for the PCERT to interact with the outside world. Researchers finding vulnerabilities in the vendor's products will interact with this page every time they find a new vulnerability or a security issue. This is why it should be easy to access and contain pages and functionalities that will make the new vulnerability submission process go smoothly for both sides.

We recommend that the PCERT/Product Security page contain the following:

• Dedicated email address for vulnerability reports and a PGP key for secure communications

• Cybersecurity advisories for publishing vulnerabilities and security notices

• Vulnerability disclosure policy

• Issue identification number for researchers and vendors to track whether vulnerabilities have been addressed and published

Why is a PGP key essential?

PGP (Pretty Good Privacy) is an encryption program for private data communications. It is often used for enhanced email security.

We recommend using PGP because it provides a private channel for disclosing vulnerabilities without the opportunity for a malicious actor to learn about them from the communication between the researcher and affected vendor.

As an example you can see Claroty's PGP Key.

Publishing Vulnerabilities

Vulnerability Announcement Process

Today, most of the vendors in the market use one or more of the following methods for sharing vulnerability information with users. We examine the pros and cons of these methods:

Our recommendation is to use these three disclosure tools in coordination, to harness all of their capabilities and provide the best experience for the wide range of users who may be interested in knowing which of their devices are vulnerable.

What Should A Good Security Advisory Contain

We recommend that each security advisory released should follow a consistent format while containing the following:

• CVE ID: Issued by the CNA

• CVSS: The Common Vulnerability Scoring System (CVSS) rates the severity of vulnerabilities and is used to prioritize patching.

• Attack Vector: The means and path an attacker may take to exploit a vulnerability

• CWE: Common Weakness Enumeration - A system for categorizing software and firmware weaknesses and vulnerabilities

• Affected Products: A of mapping programs/models/series to versions to vulnerability in order to identify which products are vulnerable.

This is an example:

Make sure that the way the models and versions are written is consistent between different advisories.

• Fixed Versions: Clearly identify which updated versions or programs/models/series contain a patch

• Mitigations/Countermeasures:In some instances, software patches or firmware updates are not available or yet possible, and instead a mitigation or countermeasure must be used. Some popular mitigations include: network segmentation and secure remote access.

• Document/API changes: A tracking method that identifies publication updates and advisory revision history

• Researcher Acknowledgments: Acknowledging researchers and organizations will produce a better relationship with them for the long run.

• Related advisories and links: If necessary.

Summary

The XIoT world needs a standard for publishing new vulnerabilities. As of today, the situation is that a large number of vendors are not prepared enough for when a researcher finds and reports a vulnerability in their products. We at Claroty recommend vendors to follow these guidelines to make their internal and external vulnerability disclosure processes better, safer, and faster, and in turn, make the XIoT world a safer place.