Sunday, August 2, 2020

Guidance for vendors in NERC CIP environments Part I


If you're looking for my pandemic posts, go here.


Tim Conway of SANS last week published a guide[i] for vendors who sell their products or services to NERC entities like electric utilities, that have to comply with some or all of the NERC CIP standards[ii]. I was quite pleased to see this document, because there has been nothing published so far – by someone who understands NERC CIP well – that even tries to explain the relevant CIP requirements to vendors, let alone explain how they can proactively help their customers comply with those requirements. As opposed to coming to the first meeting and asking the customer to – as described in the report -  “explain this whole CIP thing”.

While there is nothing in the paper that I have big objections to, I do want to point out that it’s far from being a complete guide to what vendors should know about CIP, and especially to the ways that vendors can help their customers comply. There are good discussions of CIP-003, CIP-004 and CIP-011. However, there is no discussion of how vendors can help their customers comply with CIP-005 R1 and R2.1-R2.3; with CIP-006 R2; with CIP-007 (all requirements); with CIP-009 R1; and with CIP-010 R1 and R4. I don’t think the report is deficient in not addressing these requirements, but I do think that’s an opportunity for someone to address in the future.

However, my big concern is with the discussion of CIP-013. While the discussion (especially in Figure 7) is basically good as far as it goes, it also misses a large part of the story of what a vendor can do to help their customers comply with CIP-013. More importantly, it doesn’t describe how vendors can proactively help their customer understand what the risks are, and at the same time save themselves a lot of time answering questionnaires. I'll discuss this question in this post, and point out a few more deficiencies in the paper in my next post.

The fundamental problem in Tim’s discussion of CIP-013 is that it focuses 100% on R1.2, and not at all on R1.1 or R2 (I don’t hold it against Tim that he didn’t discuss R3, since that isn’t a big vendor concern at the moment, although the customer’s R1 plan needs to describe how they’ll comply with R3). Here’s what I mean:

R1.1 requires NERC entities to “identify and assess” supply chain cybersecurity Risks to the Bulk Electric System resulting from:

1.      Procurement of Products (hardware or software components of BES Cyber Systems) and Services for BCS;
2.      Installation of Products;
3.      Use of Services; and
4.      Transitions between vendors (although since I break the generic term “vendors” into Vendors and Suppliers, I read this as “Transitions between Vendors or Suppliers”).

The entity needs to identify Risks in each of these areas. While some of these Risks are due to the entity itself (e.g. a new BCS will be installed in an insecure manner), the great majority of them are due to Vendors or Suppliers. In fact, my clients and I have identified over 100 Risks that apply to Suppliers or Vendors (although some clients have decided to include some subset of those Risks in their R1 plans, which is fine); we have identified only about 15 Risks that apply to the entity itself.

Tim’s paper just discusses the eight “mandatory” Risks identified in R1.2 (R1.2.5 and R1.2.6 identify two Risks each). Of course, it’s important to address all of these, but if a NERC entity’s R1 plan just includes those eight Risks and nothing more, they’ll certainly receive a talking-to (if not a PNC) from their auditors. All of the guidance and guidelines that have been published by NERC make clear that there are many Risks beyond these eight. The others need to be identified while complying with R1.1.

I divide CIP-013 Risks into six categories:

1.      Risks due to the entity itself,
2.      Risks primarily due to Product Suppliers,
3.      Risks primarily due to Product Vendors,
4.      Risks primarily due to Service Vendors,
5.      Risks due to the Entity’s use of open source software, and
6.      Risks due to transitions between Suppliers or Vendors.

I also divide each of these categories of Risk into sub-categories. For example, I divide Product Supplier Risks (some of which also apply to Vendors) into:

a)      Risks due to a Supplier’s insecure development or manufacturing environment;
b)     Risks due to a Supplier’s mismanagement of product vulnerabilities;
c)      Risks due to a Supplier’s mismanagement of third-party or open source software code included in a Product;
d)     Risks due to inadequate Supplier and Vendor cybersecurity policies and processes, non-customer-facing;
e)     Risks due to inadequate Supplier or Vendor “customer-facing” policies and processes; and
f)       Risks due to inadequate Supplier or Vendor customer information protection policies and processes.

Note that the eight R1.2 Risks fall into categories 2-5 above. And the 60 version 1 NATF Criteria (which I’ve included entirely in my list of Risks, although I’ve combined groups of similar Criteria into a single Risk - for example, in the cases of Criteria dealing with incident response or with access control and management) fall into categories 2-5 as well.

So a Supplier or Vendor needs to think about a lot more than the Risks identified in R1.2. Of course, those eight Risks are mandatory, whereas any others aren’t. However, any NERC entity that tries to tell their auditor that they didn’t identify any Risks in R1.1 because the eight Risks in R1.2 are the only ones that apply to them is likely to face a lot of skepticism, if not also a PNC.

But since the R1.1 Risks aren’t defined in the requirement, how can a Vendor or Supplier possibly help a NERC entity comply with R1.1? Here’s where it gets interesting. Of course, identifying Risks is different from assessing Risks. Assessment is often done using questionnaires. The questionnaire that an entity sends to their Suppliers and Vendors needs to include at least one question for every Risk that the entity has identified as important – but it also shouldn’t include any questions that don’t address a Risk they’ve identified.

This means that, if an entity asks a question on a questionnaire that doesn’t correspond to a Risk they’ve already identified, they have in effect identified that as a new Risk, and they have to mitigate it like other Risks. Why do I say this? Because if an entity asks a question in a questionnaire and the Supplier or Vendor gives an answer that indicates this might be a Risk they haven’t mitigated, the entity must now take steps to mitigate that Risk or possibly be found non-compliant with R2, since there would be an element in their R1 plan that they didn’t implement when complying with R2.

Here’s an example of what I mean: Let’s say an entity uses a questionnaire that includes the question “Do you have cooling and fire suppression systems in all datacenters where utility data will reside?”[iii] Since Suppliers or Vendors of BES Cyber Systems aren’t likely to hold much sensitive data about actual BCS installed in your ESPs – if they hold any at all – this question doesn’t have much relevance for CIP-013 purposes, or for OT security purposes in general.

But if the question is asked and the Supplier answers No, what will the entity do at that point? If they follow up and try to get the Supplier to have these systems installed (perhaps using a term in the Supplier’s contract), it might end up costing the entity a lot of money (in higher costs from this Supplier), or at the minimum a lot of time – for just about zero benefit to the entity. But if the entity doesn’t follow up with the Supplier about their answer, they will be violating their R1 supply chain cyber risk management plan, since the plan – hopefully – says they will follow up on every question that may indicate a high level of risk. The moral of this story is that a NERC entity should never include questions in its questionnaire that don’t address Risks it has decided to mitigate[iv].

But the problem is that some NERC entities are planning to submit questionnaires that were developed by another organization and include a lot of questions like the above – that address extraneous Risks including those that apply to data services providers, which aren’t relevant for Suppliers or Vendors of BCS Products and Services. And of course, when the Supplier or Vendor receives a questionnaire like this, they need to answer every question on it, even though it might be an essay question like “Please provide a network diagram and relevant information of how the offering/service is delivered. Information should, at a minimum, include details about network layers, segmentation, secure VLANs, and firewalls.”[v]

Of course, it will take a lot of the Supplier’s or Vendor’s time to answer questions like this, but it’s also true that, if the NERC entity asked the question as part of their CIP-013 program, they did this without understanding the compliance risk it could lead to. So the Supplier will waste time answering questions that the NERC entity should never have asked in the first place, if they knew the compliance risk they were running.

This leads to what I think should be the Supplier’s or Vendor’s strategy for helping the NERC entity comply with CIP-013-1 R1.1: They should proactively identify the Risks that they think should apply to them, and develop – without prompting – answers to the questions based on those Risks. For example, one Risk that applies to many Suppliers is “A Supplier includes open source or third-party-developed module(s) in its software Product, but does not provide full support, including promptly downloading and testing patches before releasing them to the customer.”

What if the Supplier identifies this as a Risk and then proactively develops their answer to the question, something like “We fully support third-party or open source components in our software, including promptly downloading and testing patches for these components and forwarding them to our customers if not already included in the product”? Then the Supplier submits this list of Risks and responses to all of their NERC entity customers (or specifically those with High or Medium impact BES Cyber Systems).

In my opinion, this would not only save Suppliers or Vendors needless work in answering irrelevant questions, it would be a great selling point for their customers and prospects, since it would show they were proactively identifying and mitigating the supply chain cybersecurity Risks that apply to them, rather than hiding and trying to avoid big questionnaires in any way possible. And at the same time, the Suppliers were taking a big compliance load off their customers, since they were identifying and proactively mitigating[vi] the Risks that apply to them.

Sounds like a win-win to me!


Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Are you hot at work – or should be – on getting ready for CIP-013-1 compliance on October 1? Here is my summary of what you need to do between now and then.


[i] The document is available here. You need to be a member of the SANS ICS Community, but frankly you should be one anyway!

[ii] Development of the document was sponsored by Fortinet.

[iii] This is question CSTA-04 on the NATF Questionnaire, which is available here

[iv][iv] For a further discussion of this point, see this post.

[v] This is question CSTA-13 on the NATF Questionnaire.

[vi] Of course, just because a Supplier or Vendor thinks there is a Risk that applies to them, this doesn’t mean they have to mitigate it before they send this document out to their customers. For example, if they have identified a Risk that they haven’t been assessed according to a cyber framework like the NIST Cyber Security Framework (and this Risk applies to lots of Vendors and Suppliers, of course), they don’t need to get the assessment immediately – but they do need to be able to (truthfully) make a statement like “We will be assessed, and draw up a mitigation plan, within the next 12 months.”

No comments:

Post a Comment