Turning Public Data into Public Knowledge: Behind the Scenes of Claroty’s Biannual Report

It’s been three years since I started examining industrial control system (ICS) vulnerabilities. What began with my boss saying: “Gather some data and let’s see what we find,” turned into an automated scraping infrastructure allowing us to inspect the bigger picture of industrial device vulnerabilities exposed in key industries. Sharing our research and knowledge was always the end goal, and with enough data and conclusions gathered, Claroty’s Biannual ICS Risk & Vulnerability Report was born.

I learned a lot over the course of this project, and now after writing four biannual reports, I figured it was time to share some behind-the-scenes information about the process we’ve created, hoping it will help others with their respective approaches to data collection and analysis, and emphasize the importance of sharing knowledge for the benefit of the community.

Let’s start from the beginning, and remind you that the next version of the report covering vulnerabilities from the first half of 2022 will be available in August.

Start By Making it Useful

Like I said above: it all started with my boss asking me to gather some data about ICS vulnerabilities and seeing what I have to say about it. But, truth be told, I wasn’t sure right from the start what data I was supposed to gather in order to have something meaningful to say. So the first question (to myself) was: What would one want to get from the data I’m collecting and analyzing? Having this as my guide from the get-go, I knew one thing for sure: I needed to give people something useful to read, otherwise what would be the point?

To achieve that, I had to create actionable information that will be relevant for the ICS community, regardless if the user is a vendor, CISO, researcher, analyst, or OT operator. And so the work wasn’t going to be about bringing new ideas and concepts into the world, but rather to paint as full a picture of the ICS vulnerability landscape as possible—based on data—and to provide recommendations. The full picture is meant to help understand the risk stemming from these vulnerabilities and steps to manage them, touching on subjects such as mitigations, remediations, severity of successful exploitation, and more.

With that in mind I focused my research on the following core questions:

1. What conclusions can be drawn from publicly available data about ICS vulnerabilities disclosed within a certain period of time?

2. What challenges do these vulnerabilities pose to those who have to patch or mitigate these systems?

3. What can be done to better manage security risks and vulnerabilities in ICS networks with respect to the conclusions and challenges found?

After having a better idea of what I was looking to do with this project, I started to look for data I could use. Starting with two public sources—the National Vulnerability Database (NVD) and ICS-CERT—I developed crawlers that scraped every ICS-related CVE and its relevant details, including CVSS severity scoring, any Common Weakness Enumerations (CWE) involved, and the researchers who disclosed the vulnerabilities. In the biannual reports that followed, new data sources were added, and the crawlers collected more data such as affected products, availability of remediation and mitigations, and much more.

Of course, the work was not complete at that point, because even though all that data is accessible publicly, that doesn’t mean it’s available to be used by all. Not everyone has the resources, ability, or time to collect it, inspect it as a whole, and analyze it. That leaves many security analysts to go over lists of vulnerability disclosures with no ability to benefit from it any further. That’s the gap that the biannual report was aiming to solve: Turn that publicly available data into public knowledge available to all.

An example of similar philosophy is the work of Dan Ricci, founder of the ICS Advisory Project. Ricci, formerly with a large consulting firm, conducted risk assessments for chemical manufacturers. He quickly learned that asset operators and network managers were having challenges maintaining awareness of control system vulnerabilities. Combine that lack of awareness with few trained cybersecurity staff—a common scenario in most smaller and medium-sized manufacturers—and companies were often at a disadvantage.

“I thought it would be great to be able to provide a leave behind artifact that could easily provide them with a way to filter by ICS vendor products for their environment,” Ricci explained.

The project hosts a number of interactive dashboards that users can filter by vendor, product, CVSS score, critical infrastructure sector, or Common Weakness Enumeration (CWE). The dashboard visualizes advisory information; advisories are culled from a number of sources and gives users a one-stop shop for vulnerability information.

Food For Thought and Steps For Action

When done collecting data from the various sources, the crawlers process, merge, and store it for future analysis. When the time came to write the biannual report, I would go over the volumes of data and start analyzing, looking for trends and useful information to highlight. 

When doing so it’s sometimes easy to forget that not everything is equally important to mention. Thinking otherwise usually creates a mess for the readers because it’s easy for anyone to get lost within all the data, and it becomes unclear as to what are the takeaways. So as a rule of thumb, it’s usually better to pick a few key trends and emphasize them, for example, within an executive summary, or a dedicated webinar.

In order to create meaningful knowledge and actionable insights, just counting vulnerabilities and identifying growing trends was not enough. Bringing context to these disclosures is important because otherwise we won’t be able to provide a deep understanding of the risks these vulnerabilities pose to ICS networks. 

One good example was the realization that addressing vulnerabilities only by CVSS scoring would not be good enough. The CVSS scoring model was created with IT networks in mind and not OT, where priorities are different (e.g. data integrity is not as important in OT as availability or safety). For that reason, we broke the CVSS scores into their different components and added Common Weakness and Enumeration (CWE) information, knowing the ramifications of exploits leading to denial of service attacks, or remote code execution, are more important than others and should be emphasized regardless of their criticality score. 

As we said earlier, just counting and sorting vulnerabilities by type and severity isn’t enough. What use is such information if it’s not complemented with actionable advice around mitigations or patching information. This is the type of information that’s crucial to a security operations center analyst, or OT operator when strategizing around acceptable downtime and system updates. 

We keep using the word mitigation separately from remediation, and with good reason. It became evidently clear from the data and from what we know in the industry that software patches and firmware updates aren’t always immediately applied. Patches must be regression tested to understand whether there are any compatibility issues with existing applications; we don’t want a patch breaking a smoothly running process! 

Firmware updates are another challenge. These take longer to develop, and once they’re available, are not trivial to implement across sites, many of which may be in different geographies, or in difficult-to-reach places. 

Clearly, updates within OT environments are a much different beast than in IT networks. More often than not, operators resort to a mitigation of sorts as a stopgap until an update window works enterprise-wide, or until a patch or firmware update is made available by an affected vendor. Sometimes, this aspect of vulnerability management is also hindered by incomplete information provided in a vendor advisory or from a CERT. This is the true value of our data exercise: showing you what successes your peers are having with mitigations, and how you can apply them in your environments. 

We understand that not everyone reading our report is a technical person behind a keyboard or with their eyes on an HMI all day. The biannual report provides an important mix of technical and strategic insight that shape not only day-to-day operational decisions but also long term strategic ones. It’s an honor to put this report together, and exciting to see new trends emerge, and to confirm that older ones still have some validity. Team82’s mission is to make the industrial, healthcare, and commercial sectors safer by improving the security, availability, and reliability of systems in critical industries. The biannual report is just one of our contributions, but it’s an important one, and we hope you keep reading.