Sunday, November 15, 2020

How do we assess risks in the products we buy?


This was originally going to be the third part of a set of three posts about a great supply chain security presentation by Matt Wyckhouse of Finite State at the MRO (virtual) Security Conference last month. After the conference, Matt and I exchanged some emails and I decided to break the single post I was going to write into three. You can read the first two posts here and here.

This third post was always going to be different from the others – and it will live up to that billing. The first two are discussions of what Matt said (and my email exchange with him afterwards helped me better understand what he said – especially since I just had the slides to work with, not the recording). However, in that email exchange we traded views about a subject on which we have seemingly different opinions: the role of supplier assessments in managing product security risks.

For this third post, I was originally going to simply reproduce what we had said about supplier assessments in his emails to me. However, last week I noticed an article on the Finite State web site, which explained Matt’s position on this issue better than he’d been able to in our relatively short email exchange. So I decided to make this third post a response to his article.

Here’s my summary of Matt’s article, using liberal quotes from the article (although Matt, if I’ve missed something, send me an email and I’ll put it up as another post), along with my responses to some of his points in red:

1.      In the past, users of critical infrastructure have been able to rely on vendor questionnaires to determine what risks might be present in an OT device we purchase. “But the rapid adoption of connected devices—the internet of things (IoT), operational technology (OT), and other embedded systems—has completely changed the source of device risk.”

2.      Here’s the problem: “Whereas before, having a deep understanding of our vendors was the primary option for assessing risk, it is now clearer than ever that these third party risk assessments are falling short of where we need to be to prevent our networks and infrastructure from being compromised.”

3.      What’s the solution? “Both device manufacturers and asset owners need a scalable solution that takes inherent device risk into account. And while these assessments give us key information, we cannot see the full picture without looking directly into the devices themselves.”

4.      In the second section of the article, Matt discusses third party (supplier) risk assessments. He says, “Through lengthy questionnaires that evaluate a vendor’s security posture, reputation, privacy risk, and so on, these assessments attempt to answer the question, ‘should I trust this company with my data?’”

I totally agree with Matt that most supplier risk questionnaires ask questions about how the supplier protects customer data. That’s great if you’re assessing the risk of say a cloud provider or a payroll services company. In those cases, the whole question is how they will handle your data. However, that shouldn’t be the case with OT supplier assessments. With a couple notable exceptions[i], OT suppliers hold or process little or none of their customers’ data. Why should say a supplier of relays hold a lot of data on an electric utility’s operations? If they need to troubleshoot a problem, they can get all of the data they need at that time. When they no longer need the data, they should delete it.

However, I have yet to see a questionnaire that focuses solely on OT risks, other than the one I have developed with my clients. This summer, I wrote four posts about the questionnaire NATF introduced in a webinar in May, including this one that pointed out that a significant number of the questions in that questionnaire don’t address OT risks at all. This includes questions about data privacy and how the supplier assesses the companies with which they share data (both important questions for a data services supplier, yet close to irrelevant when it comes to BES Cyber Sytems).

While there are some data-oriented questions that do address legitimate OT risks (including all of the data risks listed in the NATF Criteria), it’s a huge waste of your time and your supplier’s time to ask them questions about how they handle data, when they will be handling very little of your data, if any. So I agree with Matt that asking suppliers about data is in most cases not useful. But I disagree with his implication that data questions are inevitable in a supplier questionnaire. It’s certainly possible to design a questionnaire without them (or more correctly, without the basic set of questions that are necessary, such as those in the NATF Criteria). In fact, I’ve done it!

5.      Matt continues: “A lack of hard data means that you must rely on trust. There’s something strange about asking a vendor questions about their own trustworthiness. If you’re worried that you can’t trust a company with your data, why should you trust their responses? This leads to situations where your team may have to rely on gut instinct to know when something just feels off.”

This is another example where I think Matt is starting from the wrong premise. What is the purpose of a supplier questionnaire? Is it to help your organization make the decision whether or not to purchase from this supplier? I’m sure that the great majority of cyber risk management professionals in any industry, including the power industry, will say that is the case. In fact, NATF says that the purpose of the supplier assessment process is to make a purchase decision.

This is almost certainly true for purchases of IT systems. However, I’m certain that, for OT systems, in at least 95% of procurements (really, asymptotically approaching 100%) the decision on who to purchase from was made a long time ago. One municipal utility estimated to me that for their OT systems, they consider changing a supplier no more than once every five years, and probably a lot less often than that.

Let’s say you don’t like your EMS vendor’s answer to a couple questions on the latest security questionnaire you sent to them. Are you immediately going to put out an RFP for a new EMS supplier? How about the supplier for the turbines in your generating plants? Or the relays in your substations? It just doesn’t happen. Consider: There are some organizations that have been buying products from GE for close to 100 years (i.e. since just a few decades after Thomas Edison founded the predecessor company). Are they going to tell GE to take a hike if they think there are a couple areas where their cybersecurity leaves something to be desired?

Of course not. So why do organizations send cybersecurity questionnaires to OT suppliers? In some cases, the answer may be “Because that’s our policy”. But the real answer should be “Because we have decided to purchase the product from supplier X, we want to make sure we understand their current posture with regard to the supply chain cybersecurity risks that we think are important. If their answers indicate that they may not have mitigated a particular risk to our satisfaction (for example, they haven’t implemented multi-factor authentication for their own remote access systems. DHS said in 2018 that over 200 suppliers to the power industry had been penetrated by the Russians, usually through their remote access systems), we will talk with them about how they can rectify this problem.”

So the question of supplier trust isn’t the issue here, as Matt seems to be saying. We don’t – or at least we shouldn’t – ask the supplier questions in order to form an opinion on whether they’re trustworthy or not. We should ask them questions about their policies and practices because we want to see them improve their security over time. After all, they’re our supplier and that’s not likely to change.

6.      Matt goes on “You must assume that the vendor being assessed understands their products, firmware, and supply chain. Even if you deem a vendor to be trustworthy, you are then relying on their processes and on every member of their team to know what they’re doing and to function flawlessly. The vendor in question may very well believe that their products are secure, but if they don’t understand their own supply chain risk, or if their suppliers don’t understand the security of their components, then how can you trust their answers?”

I’ll paraphrase this: “Risks that arise from the supplier’s own supplier chain are just as important as risks that are due to the supplier’s own policies and practices. A questionnaire can’t truly assess risks that come through the supplier’s own supply chain, since the supplier itself usually doesn’t understand those risks.”

Once again, I don’t disagree at all with the statement that suppliers don’t sufficiently understand their own supply chain risks, meaning the risks that arise from third-party and open source components they have included in their software and firmware products. Both suppliers and consumers need to understand those risks – the problem is they don’t normally have any information on the components of the products they purchase; if they did, they could research those risks. What’s the ultimate solution to this problem? SBOMs!

7.      Matt concludes this section thus: “Even if we do our best to minimize these issues, what are we actually measuring? Again, we are asking, “should I trust this company with my data?” but that doesn’t help us understand actual product risk. It doesn’t give asset owners the ability to see what vulnerabilities are inside their devices or what issues are introduced by their supply chains. What if in addition, we asked, “should I trust this device on my network?” How would we assess risk then?”

Again, I want to paraphrase what he said. I’ve already pointed out that questionnaires for OT products a) shouldn’t ask a lot of questions about data, and b) aren’t used to determine whether to trust the supplier – they’re already trusted, which is why they’ve been the RTU supplier for 30 years. The question we should be asking in OT supplier risk assessments is “What are the supply chain risks that could arise through this supplier, that could have an adverse impact on our ability to control and/or monitor the Bulk Electric System?” Even though a lot of NERC entities don’t explicitly have this question in mind when they send a questionnaire to a supplier, this is IMHO the real justification for doing so.

However, I’m in total agreement with the last two sentences of the above quotation. We do need to ask whether we should trust a new device on our network, and we need to assess the risks that may apply if we do connect the device to our network, before we connect it. The point is this is a completely different question from the question that we’re trying to answer with our questionnaire. The questionnaire is trying to identify risks that may be present in a supplier that can affect products they produce in the future. Mitigating them is important because we don’t want to have bad experiences with the supplier’s products in the future.

But getting the supplier to for example implement background checks for all software developers won’t make the product we buy today any more secure than it was yesterday. We also need to assess the product itself. The moral of this story: We need both to assess supplier risks through questionnaires, and assess product risks through testing, before we install a product.

The last section of the article describes what it means to assess risk in a device that you’re planning to install on your network. I don’t disagree with any of that, except to say that assessing risk must always itself be a risk-based decision. If you’re about to install the 50th HP server in your Control Center, you probably don’t have to conduct an in-depth assessment like Matt describes. On the other hand, if you’re installing the first of a new type of relay – and the supplier rushed development to get this into your hands by your deadline – you should definitely give it a thorough looking-over before you install it.

Now that I’ve finally finished discussing Matt’s MRO presentation, maybe one of these days I’ll get to what I consider the most interesting topic: How he came up with the name for his company, and why he made a puzzling (to me, anyway) choice in arriving at that name. But that post won’t help you at all with CIP-013 compliance (although it might help us all understand what “programmable” means).

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.



[i] E.g. a turbine manufacturer holding a lot of performance data for predictive maintenance purposes.

No comments:

Post a Comment