Good for OSI!

In my last post, I discussed the importance of supplier/vendor security questionnaires and described how two different suppliers – both major suppliers to electric utilities – have described their policies for responding (or not) to questionnaires submitted to them by their customers. I finished the post by praising one of them, Schweitzer Engineering Laboratories, for saying they would answer all questionnaires from customers, full stop.

However, I first described a “major OT software supplier” to the power industry, who had put out a paper at the end of last year - which described their policy toward customer security questionnaires. Their story was more nuanced. Based just on my reading of the letter, I summarized it in two points: a) They will be reluctant to respond to customer security questionnaires in general, especially lengthy ones; and b) They are now compliant with ISO 27001/2, and will be audited soon. They want customers to look through ISO 27001 first, to see if their certification will address some (and hopefully most) of their questions.

I said this would be OK (with me, at least), as long as

1. ISO 27001 will answer all or most of the security questions a customer is likely to have. However, I pointed out that, in a list of 42 questions based on 42 vulnerabilities that I and my CIP-013 clients have identified as important enough to require industry suppliers to mitigate, I don’t think a single one is addressed in any meaningful way in 27001. This isn’t hugely surprising, since ISO 27001/2 (and just about every other important cybersecurity standard) is designed to address IT threats, not OT ones. The NERC CIP standards are all about OT.
2. This supplier will help their customers find where in 27001 their questions are answered. Simply pointing them to the document isn’t enough.
3. The supplier will provide the actual audit report, since simply knowing that the supplier passed an audit (no matter what the standard) is just about useless for supply chain security risk management purposes. Both for CIP-013 compliance and overall supply chain security, it’s important to know if and whether the supplier has mitigated each of the risks you (the utility) have identified as important – in the case of my clients, it’s the 42 vulnerabilities that led to the 42 questions we want to ask each supplier. These vulnerabilities came from a) “identifying and assessing” supply chain cyber security risks to BCS, as required by CIP-013-1 R1.1; and b) assessing the six (actually, eight) “required” risks described in R1.2.

Because I didn’t have answers to these questions at the time, in my post I gave this supplier an “Incomplete” for questionnaire responsiveness, pending any answers I received from them. But I later heard from Ron Koziy, the Director of Cyber Security & Compliance of this organization – whom I know through the NERC CIPC Supply Chain Working Group. Through a couple of back-and-forth emails, I confirmed that this organization – and I can now reveal they are OSI International, which has a huge share of the US EMS market (although they have a lot of other software offerings a well) – will in fact answer questionnaires from customers as well. However, before the customer sends them a questionnaire, OSI wants them to first read the following on OSI’s secure website, which will presumably answer some of their questions:

1. The document detailing how OSI stands with respect to the NATF Criteria. I agree this is a good first step, since most of the Criteria address real risks to BES Cyber Systems. But as I explained in this post, these are far from being the only risks that NERC entities should consider for their CIP-013 plans. Risks not addressed in the Criteria include those due to a) vulnerabilities found in fourth-party software components of the software or firmware (whether compiled with the supplier’s own code or provided as standalone components); b) insecure software or firmware development practices; c) insecure shipment of hardware; d) inadequate protections for systems used for remote access to the utility's BCS; e) inadequate anti-phishing and anti-ransomware measures on the supplier’s part; f) vulnerabilities in open source software included in the product; and g) lack of two-factor authentication for the supplier’s own remote access systems (since DHS in 2018 said in a briefing that “over 200” suppliers to the electric power industry had been penetrated by the Russians, through their own remote access systems).
2. The document (available in March) that describes the steps OSI is taking specifically to help their customers comply with CIP-013 R1.2.1 – R1.2.6;
3. The report from OSI’s ISO 27001 audit, available in April. As I’ve said earlier, the majority of the questions in ISO 27001/2 have little or no relevance for control systems. I pointed out in my last post that posting the audit report will only help if OSI helps their customers find answers to particular questions, if they’re there. Rob says they will do this, although he also says (with my italicized notes) “Specific questions will more likely be found in OSI procedures or policies for ISO 27001 (found on the secure website), for which we can assist with directing customers to the specific section (of the audit report).”
4. But let’s say the above steps don’t answer all of your questions (and I can promise they won’t answer all of my clients’ questions, since we have already developed them – BTW, the majority of my CIP-013 clients are OSI customers); what’s plan B? Here’s Ron’s response to that question: “If entities have questions that have not been answered within the posted NATF criteria, please send them to OSI at any time (i.e. now).  OSI customer questions can be sent to: CIP13@osii.com or via their account manager, or to me directly (rob.koziy@osii.com).  OSI will provide responses to all customers directly and subsequently update our CIP-013 website with new questions and answers on a regular basis.”

Of course, answer 4 is the key for me, which is why I upgraded OSI’s “grade” for questionnaire responsiveness from Incomplete to Good (like Schweitzer’s – in fact, I would now say both of their policies for responding to security questions are very good). I also like OSI’s idea of compiling and posting a list of these questions and their answers. This will further reduce the likely number of questions that are still unanswered, after the customer has spent some quality time on the OSI secure site. In fact, I recommend that Schweitzer do the same thing.

Good for Schweitzer!

An important part of supply chain security risk management is assessing vendors to identify “holes” in their security posture that need to be addressed. That’s what this post is about, but first I want to discuss how assessment – and especially the use of questionnaires – fits into the overall supply chain security methodology that I and my CIP-013 clients have developed over the past year.

For CIP-013 compliance and supply chain security in general, NERC entities need to mitigate two types of procurement risks: those that come through Vendors and Suppliers (almost always risks that are due to inadequate security policies or procedures on the Vendor/Supplier’s part) and those that come through the entity’s own actions or lack thereof (these are also due to inadequate policies or procedures, but this time those of the NERC entity itself).

This post is about the first type of procurement risk: Vendor and Supplier risks (note that I – as well as my clients – think Vendors and Suppliers should be treated differently, since the risks that apply are very different in the two cases. I’ll use ‘Suppliers’ for most of the rest of this post, although I’ll always mean both). Vendor/Supplier risks are the ones that almost everyone thinks about when they think about supply chain risks, and indeed, probably 70-80% of the most important procurement risks do come through Suppliers or Vendors.

I call these risks Vulnerabilities, although I capitalize the term to distinguish it from the normal cybersecurity term vulnerability - which means a flaw in software or firmware that could lead to compromise of a system (a BCS in this case). In my sense, a Vulnerability means any situation that enables a Threat to be realized – and I define a Threat as something that could damage the BES. For example, if a software company doesn’t perform background checks on the developers that it hires, this Vulnerability could enable a terrorist or other person with malicious intent to plant malware or a backdoor in software or firmware in a BCS.

A Vulnerability can often be stated in this way: If the Supplier doesn’t do X, then it’s possible that the NERC entity (that would be you, Dear Reader) will procure and install one or more hardware or software components of BES Cyber Systems that have a vulnerability (lower-case, of course), leading to some impact on the BES. To continue with the above example, if one of your BCS Suppliers doesn’t perform background checks on their developers, it’s possible that your organization will procure and install a software or hardware component of a BCS that contains a vulnerability or backdoor, that could be exploited by a bad guy (or woman, of course!) to damage the BES.

The impact on the BES doesn’t need to be specified, and indeed there are close to an infinite number of impact types – what matters is simply whether or not there could be any BES impact at all. This is of course how the BES Cyber Asset definition works: If the compromise or loss of a Cyber Asset can impact the BES in any way, it’s a BCA. There’s no such thing as big or small BES impact.

Why does this matter? It’s no exaggeration to say that your most important steps to improving your supply chain cybersecurity (and achieving CIP-013 compliance, of course) are to a) “identify and assess” the most important Vulnerabilities that apply to Suppliers; b) determine, for each Supplier of BCS hardware or software components, which Vulnerabilities it has already mitigated and which it hasn’t; c) endeavor to get the Supplier to agree to mitigate whatever Vulnerabilities it hasn’t yet mitigated; and d) in cases where the Supplier can’t or won’t mitigate a Vulnerability, take steps on your own to mitigate it (although the steps you can take are usually not as effective as those the Supplier could take. But they’re better than nothing).

Of course, b) is the assessment step. There are three main way to assess a Supplier: audits, questionnaires and certifications. Audits are definitely the most expensive of these options. While there can certainly be times when an audit is necessary (e.g. when a Supplier just can’t be trusted to answer a questionnaire truthfully. But really…if they’re so untrustworthy, why are they still your Supplier?), it shouldn’t be the default option.

There are also definite reasons why you might want to use certifications to determine to what degree the Supplier has mitigated particular risks. The biggest is when a large company like Cisco or Microsoft clearly won’t respond to questionnaires (although it wouldn’t be a bad idea to try. Work through your VAR for this, since they’re most likely to know who to go to at the Supplier– and they’re definitely the most motivated party to help you find that person or group). Then you should look at the Supplier’s certifications, and try to find answers there to the security questions you would otherwise ask in a questionnaire.

But certifications have their limitations, too. The main limitation is that, for questions specific to OT, IT-focused certifications like ISO 27001, or SOC II audit reports, aren’t likely to provide answers. NERC has said they’re working on a certification specific to OT risks to the BES, so that will certainly be of interest when it’s finalized. However, that’s unlikely to occur anytime soon.

So I think the best way to determine to what degree a Vendor or Supplier has mitigated each of the threats or vulnerabilities that you’ve identified as important, is to send them a questionnaire. However, I’ve found people raise two objections when I say this:

1. The Vendor may lie in their answers; or
2. The Vendor won’t respond to the questionnaire at all.

 For the first question, it’s definitely true that Suppliers don’t always tell the truth, but Supplier relationships in this industry – especially in the OT environment – last for decades, if not centuries. Given that, it’s hard to believe a Supplier would put all that in jeopardy, just to score a short-term point or two by misrepresenting the truth in their questionnaire answers. And if you think you can’t trust the Supplier’s answers on a particular questionnaire, I’m sure your contract retains the right for you to audit them. Just about all such contracts have that clause (but again: If you’ve decided you really can’t trust the Supplier, why are you still buying their products at all?).

The second objection is a little harder to counter. I know that OT Suppliers definitely don’t like to answer questionnaires, especially if they think they’ll be forced to answer a large number of questions that are IT-focused (and any “information security” framework like ISO 27001/2 is inherently IT-focused), and have little if anything to do with risks to OT. Of course, it is OT risks, and especially risks to the Bulk Electric System, that are the concern of NERC entities subject to CIP-013.

One major OT software Supplier to the electric power industry put out a short white paper for their customers at the end of last year. Its purpose was to address two problems they’re having, related to CIP-013:

1.       Customers are sending them questionnaires that are both lengthy and mostly irrelevant to OT risks – asking how they will protect the customers’ intellectual property, personal information on employees, trade secrets, etc. While these are great questions to ask a Supplier of IT products, they’re mostly irrelevant for Suppliers of OT products. So this Supplier is being asked to spend a lot of time answering a lot of questions, when their answers won’t help the customer at all in understanding their OT supply chain security risks, and specifically risks applicable to CIP-013.

2.       Customers are asking them to sign addenda to contracts that are both lengthy and contain terms that are mostly irrelevant to OT risks – ordering them to take steps to protect the customers’ intellectual property, personal information on employees, trade secrets, etc. As with questionnaires, while these might be great terms to require of a Supplier of IT products, they’re mostly irrelevant for Suppliers of OT products. This is because your Suppliers should store or transmit little if any information on the configuration of your BCS (and how to locate them logically or physically, which I the definition of BCSI), and that is the only information whose misuse might lead to a BES impact. So this Supplier’s attorneys are being asked to spend a lot of time negotiating contract terms that in fact won’t help the customer at all in mitigating their OT supply chain security risks, and specifically mitigating risks applicable to CIP-013. Plus the customers (the electric utilities) are putting a big burden on their own lawyers, since you can be sure the Supplier’s lawyers will want to negotiate or remove almost every security term you ask them to include in a contract. This probably means hours of negotiations over every single term.

In this Supplier’s well-written letter, they didn’t say they weren’t going to respond to questionnaires, but they made it clear that a) they are going to be reluctant to respond to them, especially if they’re lengthy (and they specifically mention 100 questions, which I agree is too lengthy); and b) they now have policies and procedures in place to comply with ISO 27001/2 - they’ll be audited and certified on those in the near future. In other words, “Before you send us a security questionnaire, please look through ISO 27001 Appendix A (and perhaps their audit report, although they didn’t mention that at all - so I don’t know whether or not they’ll make it available to customers) to see if your questions are answered there. And by the way, don’t send us any long questionnaire at all.” 

This might be OK if a) the ISO 27001 certification will actually answer all, or even most, of the questions that NERC entities will need to answer for CIP-013 compliance purposes; b) this Supplier will actually help their customers find where in ISO 27001/2 their particular questions are answered (since Appendix A is a long document in itself, and ISO 27002 – which provides the real meat on Appendix A’s bones – is much longer, devoting an entire page to each control, rather than a sentence or two as in Appendix A); and c) the Supplier will provide the actual audit report, since it’s very possible that the Supplier might not be in compliance with every control, even if they have received the certification.

Of course, both b) and c) are up to the Supplier. However, I can legitimately take a stab at answering a). This is because I am in the process of developing a questionnaire with my clients, based on the set of 42 Supplier/Vendor Vulnerabilities that we have jointly identified, including those that are the basis for the required items listed in R1.2. Since this post is about questionnaires, not certifications (and since I hope to write a post on certifications in the near future), I can’t provide detail on this now. But I will say that almost none of the Vulnerabilities in my list is addressed directly in 27001 (and none of the items in R1.2 are fully addressed, either). So I feel quite safe in saying that, at least for the list of questions that my clients are likely to have, there are few that are directly answered by ISO 27001/2 (although I will reach out to this Supplier to see if they feel differently about this or anything else in this post. I’ll put their response – unedited! – in a future post, if they’ll allow that).

Item c) is actually very important. If this Supplier plans to just publish their overall score for the audit, that isn’t very helpful at all. In fact, any certification or service that just provides a single security score for a Supplier is close to useless for assessing OT Suppliers to the electric power industry. Why? Consider this: How often does your organization either 1) choose a completely new vendor on the OT side of the house or 2) conduct a thorough assessment of all of the competitive Suppliers to whoever is your current Supplier for a product or class of products?

One of my clients had a succinct answer to this: Once every five years at most. Even if it’s more frequent for you, my point is this: An overall security score for a Supplier is designed to aid a decision on whether or not you will buy from a particular Supplier, or perhaps to serve as an input to choosing among multiple suppliers of similar or identical products. It provides little or no help for the task that you perform much more frequently: procuring a product from an existing Supplier, from which you may or may not have previously procured the identical product.

And this is exactly the scenario that’s addressed by the CIP-013 section of the NERC Evidence Request Spreadsheet v. 3.0 (as well as the just-released v. 4.0, which seems completely unchanged, at first glance). For each procurement during the audit period, the NERC entity needs to comply with the wording of R1.1 and R1.2. In other words, you need to “identify and assess” the procurement and installation risks that apply to each procurement, as well as explicitly assess each of the risks addressed in R1.2.  You can’t do this with an overall risk score for the Supplier; you need some sort of score for each risk (Vulnerability) that you identify in R1.1, as well as the six items in R1.2 (and there are really eight risks in R1.2, since R1.2.5 and R1.2.6 address two risks each). This score will let you decide whether each particular risk (each Vulnerability, in my terminology) has already been mitigated, or whether you need to take additional steps during this procurement to mitigate it. This is the beating heart of CIP-013 compliance, since this is the entirety of the evidence that will be required up front by the auditors.

The upshot of this is that, since this Supplier doesn’t want to commit at the moment to answering all questionnaires (of reasonable length) from customers, and since I don’t yet know what they will say about questions b) and c) immediately above, I have to give the supplier an Incomplete grade at the moment. And I’d like to point out to them that they need to make sure that, whatever answer they give to my questions, they should make sure it will satisfy what is required by the Evidence Request Spreadsheet.

Note on Sunday evening: I put up this post earlier in the day and sent the link to a security person I know at the Supplier in question. He got back to me fairly quickly, and we're now engaged in an email dialogue that will I hope nail down what they're actually prepared to do, which may very well meet what I think is required. Rather than make an already-long post even longer, I'm going to write about that in another post, hopefully Monday or Tuesday night.

You may be wondering what the title of this post has to do with the content (not that the titles I provide for my posts ever tell you exactly what the content is). Well, I heard a few weeks ago – in response to a question I’d sent – from a different Supplier, one that just about every (or perhaps every) electric utility in the US uses: Schweitzer Engineering Labs.

I asked George Masters of SEL, whose title is Lead Product Engineer, Secure Engineering – and a good friend of mine from the NERC CIPC Supply Chain Working Group – what SEL’s position on customer security questionnaires was. His answer was unequivocal:

SEL uses security processes which combine best practices from a variety of standards and from our own experience inventing, designing, and building products and systems, then tracking their performance across decades of service life. We will always respond to questionnaires from our customers, as we understand that they are working to implement processes to meet CIP requirements as well as their own unique requirements.

We are watching for certifications or similar assurance mechanisms to emerge that can be broadly accepted by our customers as evidence that their requirements are being satisfied. We have taken that approach in other domains. As an example, our Quality Management System is certified to the ISO 9001 standard, and our manufacturing processes comply with the IPC-A-610 Class 3 workmanship standard for products requiring high reliability, such as those used in life-support and aerospace systems (see https://selinc.com/api/download/117615/).

There you have it. No qualifications, no ifs, ands or buts. This is another example of SEL’s commitment to supply chain security, which I experienced when I was moderator for the panel on that topic at last October’s GridSecCon. NERC had asked SEL to have someone join the panel, and I was expecting they would send George or one of his peers (which would have been fine with me). Instead, they sent Dave Whitehead, who was COO at the time and is now CEO.

Dave gave a good (five-minute, since that’s all that was allowed) presentation, and answered a couple questions very well. Plus he prepared a document for me to publish, when I requested afterwards that panelists do that. In the document, I suggested that the panelists repeat what they said during the panel (including the answers to the questions they received), and also go beyond that if they’d like. Dave prepared an excellent document summarizing their security and quality practices – in fact, on rereading it just now, I realized that this in itself answers many of the questions in my questionnaire, for SEL. You can find it here.

What, I can use my best judgment?

Since I consider CIP-013 compliance to be at heart the responsibility of the NERC entity, not mine, my consulting approach for helping a NERC entity come into compliance consists of a series of 1-3 weeklong workshops with cybersecurity, compliance, procurement and (sometimes) legal people. In these workshops, I go through what CIP-013 says (that’s an easy one: It says what’s in the requirements; nothing more, nothing less. Although the Evidence Request Spreadsheet for CIP-013 also provides some very good information on what audits will focus on), as well as the “crowdsourced” methodology and MS Excel™ workbooks that I and my clients have developed over the past year (the methodology and workbooks continue to develop, although the changes nowadays are more in the fine points, not fundamental concepts).

CIP-013 compliance, and certainly my methodology, involves some concepts that don’t come easily. I consider an initial workshop to be very successful if even two or three people get the ideas I’m talking about. And I find the people who are hardest to convince are the Walking Wounded – people who have been beaten into a submissive state by the prescriptive CIP requirements (most but fortunately not all of the existing requirements) for years (usually with PTSD from one or two bad audits). They have given up forever the idea that they can ever make sense of what CIP requires them to do and guide their actions by the criterion of what’s sensible. They pray that someone, somewhere will provide them the magic key that will unlock the true meaning of all the CIP requirements, so that they can implement their compliance programs in complete confidence that the auditors will love what they do.

And barring delivery of that magic key, they pray for an early death.

I bring this up because, in a conversation at one of the meetings during this week’s workshop at a medium-to-large-sized municipal utility, we got into a discussion of what would happen if, during a Procurement Risk Assessment, they decided that mitigating the remaining residual risk from a particular threat would just be too costly – so they decided to accept the risk. Would they have the book thrown at them at their next audit and the utility would be bankrupted by the fines?

I said no (and by the way, I’m paraphrasing this discussion, since I don’t remember the exact details). In CIP-013 compliance, if mitigating a particular risk will require an unreasonable amount of cost or effort, and if the risk isn’t one that might likely involve loss of human life or limb if realized, you can certainly accept the risk if that is the reasonable thing to do. At that point, a woman from procurement sitting next to me asked something like “What, you mean we can use our best judgment?”

If you think about it, this is fairly sad. In most cybersecurity compliance regimes – HIPAA, PCI, the NIST frameworks, etc. – the organization is allowed to use their judgment to determine what’s a reasonable action, including accepting risks that can’t be mitigated at a reasonable cost. But people who have worked in organizations where CIP compliance is a big deal (even if they haven’t worked directly in compliance, as was the case with this woman) have just come to accept that they have no choice but to do whatever they think is required, regardless of what’s reasonable.

So for all of those people, I have good news: With CIP-013, you’re free, free! Now all that’s required is to rewrite all of the other CIP standards as risk-based ones like CIP-013, and you’ll be truly free. CIP compliance people of the world, unite! You have nothing to lose but your chains!