Last week, Pfizer Head of Global Automation Engineering Jim LaBonty sat down with Claroty SVP of Global Customer Care Guilad Regev to share insight into Pfizer's industrial cybersecurity journey, securing COVID vaccine production while meeting unprecedented demand, and more. Edited for brevity and clarity, this blog shares some highlights from their conversation.
Q: What led to Pfizer's decision to adopt an operational technology (OT)-focused solution to secure your manufacturing sites?
In the biopharma arena, our wake-up call really happened in 2017, when [Pfizer competitor] Merck MSD was majorly impacted by NotPetya, affecting its manufacturing capability and supply chain across the globe. That was the turning point for Pfizer to really start thinking about the potential impact of a cyber attack on our operations. By the end of 2017, our board had issued a strong directive to better secure our production floor systems at manufacturing sites.
At that point, we had already started an industrial cybersecurity program a couple of years prior, roughly around 2015. We were already putting in place technologies to protect our manufacturing environments. But it wasn't until 2017, when our board gave us a strong message to really zero in specifically on securing the OT environment—the production floor and industrial control systems—from the potential impact of a cyber attack.
Q: What were the next steps after Pfizer's board of directors made the decision to put more resources into OT security?
The directive was a gift from leadership because it led to the IT and engineering organizations working closely together. We started a program, established what we needed to do to meet our objectives, determined which partners and cybersecurity consultants to work with, and so on.
This analysis took place in the spring of 2018, and our goal was to work out a first phase, or at least a pilot activity to help us understand what technology would eventually be covered under our industrial cybersecurity program. At that point, we had no idea what IT tools would work in the OT world and which tools wouldn't. We embarked upon a series of pilots, testing out various technologies, and after extensive studies and the analysis, we honed in on a couple of key technologies to the production floor environment.
We established some clear goals that we were trying to achieve as a program. We also started to make the case for securing a budget for our global program. We built a project management organization (PMO) structure, and we got underway.
Q: Why did you choose to work with Claroty in securing your manufacturing environment?
We knew that we needed to have a tool that gave us full visibility into our production environment. You can't protect what you don't know is actually out there. Once you identify it, then you can protect it. Having a complete OT inventory, having that visibility, and knowing what assets are communicating, is imperative to be able to understand and protect your environment.
The NIST Cybersecurity Framework has been the underlying guidance for our industrial cybersecurity strategy. It works quite well, and we continue to use it. We also use guidance from NIST 800.82, it was written in a way that makes it easy to understand how to go about securing an ICS environment.
Q: What milestones have you accomplished since 2017, and what lessons have you learned along the way? For those just starting their industrial cybersecurity journeys, what shortcuts would you advise them to take?
The first thing should be to specify the key objectives you're trying to accomplish. The guiding principles should be the goals you're trying to achieve with the program. For Pfizer, the six main objectives we set in the beginning haven't changed. One of those goals was IT-OT segmentation, and now, it's even more important how essential this is. In the past couple years, we've seen attackers be highly successful in their use of phishing as a conduit for delivering malware to the OT environments in networks that lack proper segmentation.
If your company's email system is not segmented from your production environment, that's a concern. If you have a flat network structure and you fall victim to phishing, malware can spread everywhere. At Pfizer, we recognize that segmentation is a good defense mechanism to still connect the production floor with IT systems. You're sending critical data flows back and forth, but it's a very limited set of traffic in terms of what assets are able to communicate between the production floor and IT. Just the critical business functions, and that's it.
Back in 2015, we had established a firewall for bridging between enterprise IT and manufacturing IT, and then we added another layer of industrial firewalls to segment out manufacturing IT assets from the OT on the production floor.
Q: Pfizer is a huge biopharma firm with a global footprint. Did regional or geographic perspective come into play?
That's a good question. When we started the program, we set up three project managers within the project management organization (PMO). The cultures, environments, and production capabilities of the regions differ between North America, Europe, Asia-Pacific, and Africa. When we started the overall program, many of the activities were very much the same across regions, but the speed at which they were carried out varied.
As an example of how regions can differ: most of our production operational assets in Africa are isolated from IT. They're air-gapped islands of automation, especially on the production floor. Since these OT sites lack interconnectivity, there isn't the same need for securing measures to compensate for IT-OT connectivity that are needed in other regions, since air-gapped environments are segmented by nature.
But even with these air-gapped environments in Africa, we realized we needed a tool that would scan the mobile devices, because there was still information being passed between our IT environment and the production floor. There's no silver bullet. We've had about five different technologies, with each of the various technologies providing an element of the overall security fabric. Each one of those technologies is providing a threat detection feed up into our global SOC.
Q: What came first: Segmentation of IT and OT, or the implementation of Claroty?
In 2017, segmentation made a lot of sense to separate out the communications interconnectivity between IT and OT. We already had some segmentation with the ICS and our distributed control systems, but we had learned that our sites that already had a very segmented layer were very quick to be able to implement firewall technologies. And we had lessons learned from three or four other production sites where we had put in a firewall in 2014. The number of issues with the firewalls at those production sites was minimal or zero, so it became pretty obvious that industrial firewalls made a lot of sense. We started putting in industrial firewalls in first and foremost, and once we got the firewalls in place, we implemented Claroty. We also put in a set of aggregate firewalls for North-South traffic, plus a set of core switches.
So, Claroty was implemented a close second after the firewalls going in, just a few months afterwards. Claroty was set up with passive monitoring in mind. Production is king, it will always be king in manufacturing. We wanted to ensure the solution we put in place could monitor traffic in the network, in the production environments, but we didn't want to impact operations in any way, shape, or form. Claroty's Passive Monitoring worked very well for us, and we left it in that mode until about this time last year, when we began using the Active Query capabilities to get richer data.
Q: Can you talk about the impact of team structure on IT-OT convergence?
I've been dealing with the word convergence for around 15 years. Convergence, in my mind at least, is not the two worlds of IT and OT becoming one world, but rather, it's the interconnection of the two worlds. It's not truly convergence, because they won't ever exactly become one. That being said, we do need the convergence of communications and the ability to work across the boundary between production manufacturing systems and the IT environment. Understanding, skill building, and cooperation are key to that.
A big part of the success of the Pfizer program is collaboration and willingness to work across teams. IT and OT personnel don't always have the same goals or mindsets, but they have the ability to collaborate and understand each other enough to be able to make collective decisions. Both parties can't always be happy with a decision, but they can agree that they are making the right decision for Pfizer in the long run.
Getting to this place of cooperation and understanding is a journey, and it takes time to get it right. It's not something that can be solved over a year. Cultural differences need to be melded and blended together. We aren't perfect at it yet, it's still a work in progress. But we've made huge strides, especially in the past six months, to ensure that our OT cybersecurity is being properly implemented at our manufacturing sites. I have to give a lot of credit to our suppliers and the personnel at our sites for getting the appropriate solutions in place and keeping our operations secure.
Q: Taking things to the next level in terms of team structure, do IT and OT have separate SOCs, or is it one converged SOC? And do you utilize an MSSP?
Right now, our SOC is on-premises, totally integrated within Pfizer. Our SOC looks at both our business functions and our manufacturing. So our data feeds from four or five different technologies are fed into the SIEM we're utilizing, and that data is fed into our SOC.
So our SOC is looking at all threats, from anything within the manufacturing side or any of our business functions across the globe. It's a fully integrated SOC, across IT and OT. We were thinking of having a separate SOC for the OT side, but it doesn't really make sense for a lot of reasons.
On the OT side, each manufacturing site is an entity in itself, and it needs to be able to look at threats specific to that location in order to defend themselves effectively. Think of it this way: you care about what happens in the town you live in, but your top priority is protecting the house you live in.
Q: Different sides are involved in such a big project. What about the pushback from the sites themselves, or about security not being a priority?
So obviously, the COVID-19 vaccine is our first priority right now, but a close second is securing the vaccine's production and supply chain. A year ago, we didn't have the production capabilities we have now. We didn't have mRNA, we didn't have the scale we have built up so quickly. But fortunately, we've been able to move at light speed, and actually, this program at Pfizer is called Lightspeed.
We were able to build up our production capability across the globe to be able to supply billions of doses. In the past, a typical vaccine for Pfizer has meant producing 100-200 million doses, so billions is definitely raising the bar very high.
I have to commend all my colleagues across Pfizer for the incredible job they have done, but we still have to secure our scaled-up production process and supply chain. We're doing assessments with our suppliers on the cybersecurity of the equipment they're using. Does it have any vulnerabilities? Does it have any weak spots? So far, we've done a really good job ensuring that the entire supply chain is secure, from what Pfizer handles in-house to aspects handled by third parties.
There is always pushback. You're changing. You're adding scale to a production site. Not everyone is looking at the big picture of the overall supply chain and how that priority translates at their production site. It takes a lot of communication with everyone involved to make sure they understand the importance of the overall program, to make sure they understand that yes, production is king, but we need strong cybersecurity to ensure we can maintain effective and reliable vaccine production. We can't afford to be impacted by a cyber attack right now, nor can our patients relying on us for our COVID-19 vaccine or other life-saving drugs. It's essential that our supply chain stays intact.
Q: With COVID, do you feel there are more regulations targeting biopharma?
We knew from the beginning that we couldn't do it alone. First, we needed a security consultancy, and we selected Booz Allen Hamilton to help us get our program moving. They did a great job helping our team here at Pfizer get structured and up to speed. The second part of that is we needed a global provider we could partner with that would aid us with cybersecurity and the infrastructure we needed to put in place. For that, we leveraged Rockwell Automation, because they have that global depth and work close to a lot of our clients sites. We've leveraged their scale, their technology and their people fairly well to implement the technologies at production sites.
One of the key goals that I had for program implementation was to have zero impact—zero downtime to production at any of our sites. A lot of the work we were doing was either implemented in parallel with production at a low risk, while in other cases, we found narrow windows of opportunity to implement technologies, verifying everything is set up properly with no impact, and changing the environment back to production.
In terms of regulatory changes, biopharma hasn't really been hit as hard by regulations as other sectors, but there are requirements to ensure you have command and control over your environment. So cybersecurity comes into play, because there is a question to be asked by regulatory boards how you're structuring it. There's currently no direct regulatory requirement for cybersecurity, but there are indirect questions concerning what we're doing to better secure production and ensure our operations haven't been impacted or adulterated by somebody else. So, with our technologies, with the tools we have in place, we have audit trails, and we inspect traffic coming in to ensure that our operations are secure.
Q: During implementation of your industrial cybersecurity program, what went according to plan, and what caught you by surprise?
In 2017, we all didn't really know what we're doing. There was no well-defined blueprint and roadmap you could follow step by step. We were formulating a plan and determining what to do next. I think rapid prototyping is very key to understanding how any kind of technology is going to work in your environment. As you embark, obviously leveraging from others that are taking the journey now, you can definitely tap into that insight.
It doesn't feel like we've reached the end of our industrial cybersecurity journey, because that journey never really ends. But we're getting to the end of the implementation phase. Now, we're working to fully leverage the technologies and capabilities we've put in place in securing our production capabilities.
Q: You've worked with Claroty to fine-tune our ability to help your SOC weed out false positives. Can you tell us about what's involved in those efforts?
I'm a big advocate for weeding out false positives, and I'm constantly pushing Claroty with new ideas and innovations to help SOC personnel focus on the alerts that matter most. One thing I've pushed for is adding context into knowing what's running in a production environment, and more specifically, what assets and processes in that environment should not be going through any change. During production, any new traffic or data flow to these specified areas should be high priority. On the other hand, during downtime periods when these assets are being updated, you don't want the SOC to be bombarded with false positives, especially while you're still in the process of making changes. There's a final verification phase before we go back into production after implementing updates, so the SOC needs to understand that intentional changes will still be occurring. Communication between IT and the production floor is really important to successfully weeding out false positives.
Q: What was the Claroty implementation process like for you?
I was pleasantly surprised by the quickness and ease of implementing Claroty. It was very straightforward, and we were done within a week. The level of administration required is very low, and it's providing rich data to people who need it on a timely basis. Claroty has taken a lot of the heavy lifting out of understanding what's in your production environment, which is important when you're not in a position to hire an army of people.
Q: How did you address the issue of awareness around industrial cybersecurity within Pfizer?
Good communication is essential. On a production side, for clarity purposes, no pun intended, you have the on-site leadership and other plant personnel, such as production-floor engineering and technical workforce, who are all pretty supportive. They understand the value of industrial cybersecurity, because they're responsible for maintaining that environment. At the same time, they don't want the implementation, verification, and use of cybersecurity technologies to impact their applications.
The more challenging factor is often leadership, because cybersecurity isn't always at the forefront of their mindset. They're worrying about other things. So, communicating the need for strong industrial cybersecurity is crucial. We set up a community of practice (COP), which provides a way to regularly communicate lessons learned, what's working well, and little snippets of knowledge at a time. It's a way of making Pfizer's global network feel small, because people are able to reach out and inform each other of what's happening.
Fortunately, our CISO here at Pfizer is fantastic. Early on in our industrial cybersecurity journey, I reached out to him, and that led to him coming to visit the production environment to walk in my shoes as an automation engineer, in order to better understand what it's like to be on the floor dealing with ICS environments. That really set the tone for us working together, and I give him a lot of credit for going above and beyond.
Q: Pfizer's industrial cybersecurity program was already well underway by the onset of the COVID-19 pandemic. Did COVID-19 have an impact on either expediting or delaying the program?
It was a combination of both. Some sites that were ahead and had just completed implementation were in a great shape to accept the COVID-19 challenge. On the other hand, large sites that were in the midst of the implementation and they're still in the midst of the implementation faced a bit of a slowdown.
Our manufacturing sites are a very scarce resource. And from a schedule standpoint, those larger sites house a lot of the key resources that enable COVID-19 vaccine production. You can only juggle so many things at a time. And producing the COVID vaccine was No. 1. Behind that, our No. 2 priority was ensuring that vaccine production was not impacted by malware.
