This was originally going to be
the third part of a set of three posts about a great supply chain security
presentation by Matt Wyckhouse of Finite State at the MRO (virtual) Security
Conference last month. After the conference, Matt and I exchanged some emails
and I decided to break the single post I was going to write into three. You can
read the first two posts here
and here.
This third post was always going
to be different from the others – and it will live up to that billing. The
first two are discussions of what Matt said (and my email exchange with him
afterwards helped me better understand what he said – especially since I just had
the slides to work with, not the recording). However, in that email exchange we
traded views about a subject on which we have seemingly different opinions: the
role of supplier assessments in managing product security risks.
For this third post, I was
originally going to simply reproduce what we had said about supplier
assessments in his emails to me. However, last week I noticed an article
on the Finite State web site, which explained Matt’s position on this issue
better than he’d been able to in our relatively short email exchange. So I
decided to make this third post a response to his article.
Here’s my summary of Matt’s article,
using liberal quotes from the article (although Matt, if I’ve missed something,
send me an email and I’ll put it up as another post), along with my responses
to some of his points in red:
1.
In the past, users of
critical infrastructure have been able to rely on vendor questionnaires to determine
what risks might be present in an OT device we purchase. “But the rapid
adoption of connected devices—the internet of things (IoT), operational
technology (OT), and other embedded systems—has completely changed the source
of device risk.”
2.
Here’s the problem: “Whereas
before, having a deep understanding of our vendors was the primary option for
assessing risk, it is now clearer than ever that these third party risk
assessments are falling short of where we need to be to prevent our networks
and infrastructure from being compromised.”
3.
What’s the solution? “Both
device manufacturers and asset owners need a scalable solution that takes
inherent device risk into account. And while these assessments give us key
information, we cannot see the full picture without looking directly into the
devices themselves.”
4.
In the second section
of the article, Matt discusses third party (supplier) risk assessments. He says,
“Through lengthy questionnaires that evaluate a vendor’s security posture,
reputation, privacy risk, and so on, these assessments attempt to answer the
question, ‘should I trust this company with my data?’”
I totally agree with Matt that most supplier
risk questionnaires ask questions about how the supplier protects customer data.
That’s great if you’re assessing the risk of say a cloud provider or a payroll
services company. In those cases, the whole question is how they will handle
your data. However, that shouldn’t be the case with OT supplier assessments. With
a couple notable exceptions[i], OT suppliers hold or
process little or none of their customers’ data. Why should say a supplier of
relays hold a lot of data on an electric utility’s operations? If they need to
troubleshoot a problem, they can get all of the data they need at that time.
When they no longer need the data, they should delete it.
However, I have yet to see a
questionnaire that focuses solely on OT risks, other than the one I have
developed with my clients. This summer, I wrote four posts about the
questionnaire NATF introduced in a webinar in May, including this
one that pointed out that a significant number of the questions in that
questionnaire don’t address OT risks at all. This includes questions about data
privacy and how the supplier assesses the companies with which they share data
(both important questions for a data services supplier, yet close to irrelevant
when it comes to BES Cyber Sytems).
While there are some data-oriented questions
that do address legitimate OT risks (including all of the data risks listed in
the NATF Criteria), it’s a huge waste of your time and your supplier’s time to
ask them questions about how they handle data, when they will be handling very
little of your data, if any. So I agree with Matt that asking suppliers about data
is in most cases not useful. But I disagree with his implication that data
questions are inevitable in a supplier questionnaire. It’s certainly possible
to design a questionnaire without them (or more correctly, without the basic set
of questions that are necessary, such as those in the NATF Criteria). In fact,
I’ve done it!
5.
Matt continues: “A lack of hard data means
that you must rely on trust. There’s something strange about asking a
vendor questions about their own trustworthiness. If you’re worried that you
can’t trust a company with your data, why should you trust their responses?
This leads to situations where your team may have to rely on gut instinct to
know when something just feels off.”
This is another example where I
think Matt is starting from the wrong premise. What is the purpose of a supplier
questionnaire? Is it to help your organization make the decision whether or not
to purchase from this supplier? I’m sure that the great majority of cyber risk
management professionals in any industry, including the power industry, will
say that is the case. In fact, NATF says that the purpose of the supplier
assessment process is to make a purchase
decision.
This is almost certainly true for
purchases of IT systems. However, I’m certain that, for OT systems, in at least
95% of procurements (really, asymptotically approaching 100%) the decision on
who to purchase from was made a long time ago. One municipal utility estimated
to me that for their OT systems, they consider changing a supplier no more than
once every five years, and probably a lot less often than that.
Let’s say you don’t like your EMS
vendor’s answer to a couple questions on the latest security questionnaire you
sent to them. Are you immediately going to put out an RFP for a new EMS
supplier? How about the supplier for the turbines in your generating plants? Or
the relays in your substations? It just doesn’t happen. Consider: There are
some organizations that have been buying products from GE for close to 100 years
(i.e. since just a few decades after Thomas Edison founded the predecessor
company). Are they going to tell GE to take a hike if they think there are a
couple areas where their cybersecurity leaves something to be desired?
Of course not. So why do organizations
send cybersecurity questionnaires to OT suppliers? In some cases, the answer
may be “Because that’s our policy”. But the real answer should be “Because we
have decided to purchase the product from supplier X, we want to make sure we understand
their current posture with regard to the supply chain cybersecurity risks that
we think are important. If their answers indicate that they may not have
mitigated a particular risk to our satisfaction (for example, they haven’t
implemented multi-factor authentication for their own remote access systems.
DHS said in 2018 that over 200 suppliers to the power industry had been
penetrated by the Russians, usually through their remote access systems), we
will talk with them about how they can rectify this problem.”
So the question of supplier trust
isn’t the issue here, as Matt seems to be saying. We don’t – or at least we
shouldn’t – ask the supplier questions in order to form an opinion on whether
they’re trustworthy or not. We should ask them questions about their policies
and practices because we want to see them improve their security over time. After
all, they’re our supplier and that’s not likely to change.
6.
Matt goes on “You must assume that the vendor
being assessed understands their products, firmware, and supply chain. Even
if you deem a vendor to be trustworthy, you are then relying on their processes
and on every member of their team to know what they’re doing and to function
flawlessly. The vendor in question may very well believe that their products
are secure, but if they don’t understand their own supply chain risk, or if
their suppliers don’t understand the security of their components, then how can
you trust their answers?”
I’ll paraphrase this: “Risks that
arise from the supplier’s own supplier chain are just as important as risks
that are due to the supplier’s own policies and practices. A questionnaire can’t
truly assess risks that come through the supplier’s own supply chain, since the
supplier itself usually doesn’t understand those risks.”
Once again, I don’t disagree at all
with the statement that suppliers don’t sufficiently understand their own
supply chain risks, meaning the risks that arise from third-party and open
source components they have included in their software and firmware products. Both
suppliers and consumers need to understand those risks – the problem is they don’t
normally have any information on the components of the products they purchase;
if they did, they could research those risks. What’s the ultimate solution to
this problem? SBOMs!
7.
Matt concludes this section thus: “Even if we do
our best to minimize these issues, what are we actually measuring? Again, we
are asking, “should I trust this company with my data?” but
that doesn’t help us understand actual product risk. It doesn’t give asset
owners the ability to see what vulnerabilities are inside their devices or what
issues are introduced by their supply chains. What if in addition, we
asked, “should I trust this device on my network?” How
would we assess risk then?”
Again, I want to paraphrase what he
said. I’ve already pointed out that questionnaires for OT products a) shouldn’t
ask a lot of questions about data, and b) aren’t used to determine whether to
trust the supplier – they’re already trusted, which is why they’ve been the RTU
supplier for 30 years. The question we should be asking in OT supplier
risk assessments is “What are the supply chain risks that could arise through
this supplier, that could have an adverse impact on our ability to control
and/or monitor the Bulk Electric System?” Even though a lot of NERC entities don’t
explicitly have this question in mind when they send a questionnaire to a
supplier, this is IMHO the real justification for doing so.
However, I’m in total agreement with
the last two sentences of the above quotation. We do need to ask whether we
should trust a new device on our network, and we need to assess the risks that
may apply if we do connect the device to our network, before we connect it. The
point is this is a completely different question from the question that we’re
trying to answer with our questionnaire. The questionnaire is trying to
identify risks that may be present in a supplier that can affect products they
produce in the future. Mitigating them is important because we don’t want to
have bad experiences with the supplier’s products in the future.
But getting the supplier to for
example implement background checks for all software developers won’t make the
product we buy today any more secure than it was yesterday. We also need to
assess the product itself. The moral of this story: We need both to assess supplier
risks through questionnaires, and assess product risks through testing, before
we install a product.
The last section of the article describes
what it means to assess risk in a device that you’re planning to install on
your network. I don’t disagree with any of that, except to say that assessing risk
must always itself be a risk-based decision. If you’re about to install the 50th
HP server in your Control Center, you probably don’t have to conduct an in-depth
assessment like Matt describes. On the other hand, if you’re installing the
first of a new type of relay – and the supplier rushed development to get this
into your hands by your deadline – you should definitely give it a thorough looking-over
before you install it.
Now that I’ve finally finished
discussing Matt’s MRO presentation, maybe one of these days I’ll get to what I
consider the most interesting topic: How he came up with the name for his
company, and why he made a puzzling (to me, anyway) choice in arriving at that
name. But that post won’t help you at all with CIP-013 compliance (although it
might help us all understand what “programmable” means).
Any opinions expressed in this
blog post are strictly mine and are not necessarily shared by any of the
clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would
love to hear from you. Please email me at tom@tomalrich.com.
[i] E.g.
a turbine manufacturer holding a lot of performance data for predictive
maintenance purposes.
No comments:
Post a Comment