Good for Schweitzer!

An important part of supply chain security risk management is assessing vendors to identify “holes” in their security posture that need to be addressed. That’s what this post is about, but first I want to discuss how assessment – and especially the use of questionnaires – fits into the overall supply chain security methodology that I and my CIP-013 clients have developed over the past year.

For CIP-013 compliance and supply chain security in general, NERC entities need to mitigate two types of procurement risks: those that come through Vendors and Suppliers (almost always risks that are due to inadequate security policies or procedures on the Vendor/Supplier’s part) and those that come through the entity’s own actions or lack thereof (these are also due to inadequate policies or procedures, but this time those of the NERC entity itself).

This post is about the first type of procurement risk: Vendor and Supplier risks (note that I – as well as my clients – think Vendors and Suppliers should be treated differently, since the risks that apply are very different in the two cases. I’ll use ‘Suppliers’ for most of the rest of this post, although I’ll always mean both). Vendor/Supplier risks are the ones that almost everyone thinks about when they think about supply chain risks, and indeed, probably 70-80% of the most important procurement risks do come through Suppliers or Vendors.

I call these risks Vulnerabilities, although I capitalize the term to distinguish it from the normal cybersecurity term vulnerability - which means a flaw in software or firmware that could lead to compromise of a system (a BCS in this case). In my sense, a Vulnerability means any situation that enables a Threat to be realized – and I define a Threat as something that could damage the BES. For example, if a software company doesn’t perform background checks on the developers that it hires, this Vulnerability could enable a terrorist or other person with malicious intent to plant malware or a backdoor in software or firmware in a BCS.

A Vulnerability can often be stated in this way: If the Supplier doesn’t do X, then it’s possible that the NERC entity (that would be you, Dear Reader) will procure and install one or more hardware or software components of BES Cyber Systems that have a vulnerability (lower-case, of course), leading to some impact on the BES. To continue with the above example, if one of your BCS Suppliers doesn’t perform background checks on their developers, it’s possible that your organization will procure and install a software or hardware component of a BCS that contains a vulnerability or backdoor, that could be exploited by a bad guy (or woman, of course!) to damage the BES.

The impact on the BES doesn’t need to be specified, and indeed there are close to an infinite number of impact types – what matters is simply whether or not there could be any BES impact at all. This is of course how the BES Cyber Asset definition works: If the compromise or loss of a Cyber Asset can impact the BES in any way, it’s a BCA. There’s no such thing as big or small BES impact.

Why does this matter? It’s no exaggeration to say that your most important steps to improving your supply chain cybersecurity (and achieving CIP-013 compliance, of course) are to a) “identify and assess” the most important Vulnerabilities that apply to Suppliers; b) determine, for each Supplier of BCS hardware or software components, which Vulnerabilities it has already mitigated and which it hasn’t; c) endeavor to get the Supplier to agree to mitigate whatever Vulnerabilities it hasn’t yet mitigated; and d) in cases where the Supplier can’t or won’t mitigate a Vulnerability, take steps on your own to mitigate it (although the steps you can take are usually not as effective as those the Supplier could take. But they’re better than nothing).

Of course, b) is the assessment step. There are three main way to assess a Supplier: audits, questionnaires and certifications. Audits are definitely the most expensive of these options. While there can certainly be times when an audit is necessary (e.g. when a Supplier just can’t be trusted to answer a questionnaire truthfully. But really…if they’re so untrustworthy, why are they still your Supplier?), it shouldn’t be the default option.

There are also definite reasons why you might want to use certifications to determine to what degree the Supplier has mitigated particular risks. The biggest is when a large company like Cisco or Microsoft clearly won’t respond to questionnaires (although it wouldn’t be a bad idea to try. Work through your VAR for this, since they’re most likely to know who to go to at the Supplier– and they’re definitely the most motivated party to help you find that person or group). Then you should look at the Supplier’s certifications, and try to find answers there to the security questions you would otherwise ask in a questionnaire.

But certifications have their limitations, too. The main limitation is that, for questions specific to OT, IT-focused certifications like ISO 27001, or SOC II audit reports, aren’t likely to provide answers. NERC has said they’re working on a certification specific to OT risks to the BES, so that will certainly be of interest when it’s finalized. However, that’s unlikely to occur anytime soon.

So I think the best way to determine to what degree a Vendor or Supplier has mitigated each of the threats or vulnerabilities that you’ve identified as important, is to send them a questionnaire. However, I’ve found people raise two objections when I say this:

1. The Vendor may lie in their answers; or
2. The Vendor won’t respond to the questionnaire at all.

 For the first question, it’s definitely true that Suppliers don’t always tell the truth, but Supplier relationships in this industry – especially in the OT environment – last for decades, if not centuries. Given that, it’s hard to believe a Supplier would put all that in jeopardy, just to score a short-term point or two by misrepresenting the truth in their questionnaire answers. And if you think you can’t trust the Supplier’s answers on a particular questionnaire, I’m sure your contract retains the right for you to audit them. Just about all such contracts have that clause (but again: If you’ve decided you really can’t trust the Supplier, why are you still buying their products at all?).

The second objection is a little harder to counter. I know that OT Suppliers definitely don’t like to answer questionnaires, especially if they think they’ll be forced to answer a large number of questions that are IT-focused (and any “information security” framework like ISO 27001/2 is inherently IT-focused), and have little if anything to do with risks to OT. Of course, it is OT risks, and especially risks to the Bulk Electric System, that are the concern of NERC entities subject to CIP-013.

One major OT software Supplier to the electric power industry put out a short white paper for their customers at the end of last year. Its purpose was to address two problems they’re having, related to CIP-013:

1.       Customers are sending them questionnaires that are both lengthy and mostly irrelevant to OT risks – asking how they will protect the customers’ intellectual property, personal information on employees, trade secrets, etc. While these are great questions to ask a Supplier of IT products, they’re mostly irrelevant for Suppliers of OT products. So this Supplier is being asked to spend a lot of time answering a lot of questions, when their answers won’t help the customer at all in understanding their OT supply chain security risks, and specifically risks applicable to CIP-013.

2.       Customers are asking them to sign addenda to contracts that are both lengthy and contain terms that are mostly irrelevant to OT risks – ordering them to take steps to protect the customers’ intellectual property, personal information on employees, trade secrets, etc. As with questionnaires, while these might be great terms to require of a Supplier of IT products, they’re mostly irrelevant for Suppliers of OT products. This is because your Suppliers should store or transmit little if any information on the configuration of your BCS (and how to locate them logically or physically, which I the definition of BCSI), and that is the only information whose misuse might lead to a BES impact. So this Supplier’s attorneys are being asked to spend a lot of time negotiating contract terms that in fact won’t help the customer at all in mitigating their OT supply chain security risks, and specifically mitigating risks applicable to CIP-013. Plus the customers (the electric utilities) are putting a big burden on their own lawyers, since you can be sure the Supplier’s lawyers will want to negotiate or remove almost every security term you ask them to include in a contract. This probably means hours of negotiations over every single term.

In this Supplier’s well-written letter, they didn’t say they weren’t going to respond to questionnaires, but they made it clear that a) they are going to be reluctant to respond to them, especially if they’re lengthy (and they specifically mention 100 questions, which I agree is too lengthy); and b) they now have policies and procedures in place to comply with ISO 27001/2 - they’ll be audited and certified on those in the near future. In other words, “Before you send us a security questionnaire, please look through ISO 27001 Appendix A (and perhaps their audit report, although they didn’t mention that at all - so I don’t know whether or not they’ll make it available to customers) to see if your questions are answered there. And by the way, don’t send us any long questionnaire at all.” 

This might be OK if a) the ISO 27001 certification will actually answer all, or even most, of the questions that NERC entities will need to answer for CIP-013 compliance purposes; b) this Supplier will actually help their customers find where in ISO 27001/2 their particular questions are answered (since Appendix A is a long document in itself, and ISO 27002 – which provides the real meat on Appendix A’s bones – is much longer, devoting an entire page to each control, rather than a sentence or two as in Appendix A); and c) the Supplier will provide the actual audit report, since it’s very possible that the Supplier might not be in compliance with every control, even if they have received the certification.

Of course, both b) and c) are up to the Supplier. However, I can legitimately take a stab at answering a). This is because I am in the process of developing a questionnaire with my clients, based on the set of 42 Supplier/Vendor Vulnerabilities that we have jointly identified, including those that are the basis for the required items listed in R1.2. Since this post is about questionnaires, not certifications (and since I hope to write a post on certifications in the near future), I can’t provide detail on this now. But I will say that almost none of the Vulnerabilities in my list is addressed directly in 27001 (and none of the items in R1.2 are fully addressed, either). So I feel quite safe in saying that, at least for the list of questions that my clients are likely to have, there are few that are directly answered by ISO 27001/2 (although I will reach out to this Supplier to see if they feel differently about this or anything else in this post. I’ll put their response – unedited! – in a future post, if they’ll allow that).

Item c) is actually very important. If this Supplier plans to just publish their overall score for the audit, that isn’t very helpful at all. In fact, any certification or service that just provides a single security score for a Supplier is close to useless for assessing OT Suppliers to the electric power industry. Why? Consider this: How often does your organization either 1) choose a completely new vendor on the OT side of the house or 2) conduct a thorough assessment of all of the competitive Suppliers to whoever is your current Supplier for a product or class of products?

One of my clients had a succinct answer to this: Once every five years at most. Even if it’s more frequent for you, my point is this: An overall security score for a Supplier is designed to aid a decision on whether or not you will buy from a particular Supplier, or perhaps to serve as an input to choosing among multiple suppliers of similar or identical products. It provides little or no help for the task that you perform much more frequently: procuring a product from an existing Supplier, from which you may or may not have previously procured the identical product.

And this is exactly the scenario that’s addressed by the CIP-013 section of the NERC Evidence Request Spreadsheet v. 3.0 (as well as the just-released v. 4.0, which seems completely unchanged, at first glance). For each procurement during the audit period, the NERC entity needs to comply with the wording of R1.1 and R1.2. In other words, you need to “identify and assess” the procurement and installation risks that apply to each procurement, as well as explicitly assess each of the risks addressed in R1.2.  You can’t do this with an overall risk score for the Supplier; you need some sort of score for each risk (Vulnerability) that you identify in R1.1, as well as the six items in R1.2 (and there are really eight risks in R1.2, since R1.2.5 and R1.2.6 address two risks each). This score will let you decide whether each particular risk (each Vulnerability, in my terminology) has already been mitigated, or whether you need to take additional steps during this procurement to mitigate it. This is the beating heart of CIP-013 compliance, since this is the entirety of the evidence that will be required up front by the auditors.

The upshot of this is that, since this Supplier doesn’t want to commit at the moment to answering all questionnaires (of reasonable length) from customers, and since I don’t yet know what they will say about questions b) and c) immediately above, I have to give the supplier an Incomplete grade at the moment. And I’d like to point out to them that they need to make sure that, whatever answer they give to my questions, they should make sure it will satisfy what is required by the Evidence Request Spreadsheet.

Note on Sunday evening: I put up this post earlier in the day and sent the link to a security person I know at the Supplier in question. He got back to me fairly quickly, and we're now engaged in an email dialogue that will I hope nail down what they're actually prepared to do, which may very well meet what I think is required. Rather than make an already-long post even longer, I'm going to write about that in another post, hopefully Monday or Tuesday night.

You may be wondering what the title of this post has to do with the content (not that the titles I provide for my posts ever tell you exactly what the content is). Well, I heard a few weeks ago – in response to a question I’d sent – from a different Supplier, one that just about every (or perhaps every) electric utility in the US uses: Schweitzer Engineering Labs.

I asked George Masters of SEL, whose title is Lead Product Engineer, Secure Engineering – and a good friend of mine from the NERC CIPC Supply Chain Working Group – what SEL’s position on customer security questionnaires was. His answer was unequivocal:

SEL uses security processes which combine best practices from a variety of standards and from our own experience inventing, designing, and building products and systems, then tracking their performance across decades of service life. We will always respond to questionnaires from our customers, as we understand that they are working to implement processes to meet CIP requirements as well as their own unique requirements.

We are watching for certifications or similar assurance mechanisms to emerge that can be broadly accepted by our customers as evidence that their requirements are being satisfied. We have taken that approach in other domains. As an example, our Quality Management System is certified to the ISO 9001 standard, and our manufacturing processes comply with the IPC-A-610 Class 3 workmanship standard for products requiring high reliability, such as those used in life-support and aerospace systems (see https://selinc.com/api/download/117615/).

There you have it. No qualifications, no ifs, ands or buts. This is another example of SEL’s commitment to supply chain security, which I experienced when I was moderator for the panel on that topic at last October’s GridSecCon. NERC had asked SEL to have someone join the panel, and I was expecting they would send George or one of his peers (which would have been fine with me). Instead, they sent Dave Whitehead, who was COO at the time and is now CEO.

Dave gave a good (five-minute, since that’s all that was allowed) presentation, and answered a couple questions very well. Plus he prepared a document for me to publish, when I requested afterwards that panelists do that. In the document, I suggested that the panelists repeat what they said during the panel (including the answers to the questions they received), and also go beyond that if they’d like. Dave prepared an excellent document summarizing their security and quality practices – in fact, on rereading it just now, I realized that this in itself answers many of the questions in my questionnaire, for SEL. You can find it here.