You too can waste BIG money on CIP-013 compliance! (part one)

I’ve been working almost full time on CIP-013 compliance since the beginning of this year, and I’ve always been very concerned about making sure that clients carefully target their supply chain risk mitigation efforts, so they don’t waste money.

Believe it or not, wasting money on CIP-013 compliance is really a new concern in the NERC CIP world. When complying with the prescriptive CIP requirements (with CIP-007 R2 and CIP-010 R1 being the poster children for that), waste really isn’t the question. You are required to do certain things at certain times, and you need to invest whatever it costs to be assured that you’re doing the right things and you’re doing them at the right times. If you pass the audit for a particular requirement, not a single penny that you spent on compliance with that requirement was wasted. If you don’t pass, the whole amount you spent was wasted.

But CIP-013 is very different, of course. R1.1 requires you to “identify and assess” supply chain security risks, and then (implicitly) requires you to mitigate the most important of those risks (since there’s no way any entity could ever mitigate all, or even a significant fraction of its supply chain security risks). But, other than the fact that you need to include in your plan the six risks (really 8) mandated by R1.2.1-R1.2.6, CIP-013 provides no further guidance on:

1. What risks you should identify, or even how you should go about identifying risks;
2. How you should decide, of the risks you’ve identified, which ones to mitigate and which to accept; and
3. To what degree you should mitigate each risk. Should you apply so many mitigations that the risk level asymptotically approaches zero? Or should you just do a pro forma mitigation of each risk (such as simply sending an email to each vendor giving them a list of things you’d like them to do, with no effort later on to verify that they’re actually doing it), which will in fact mitigate almost no real risk?

But just because there isn’t any guidance from NERC on these questions[i] doesn’t mean you shouldn’t think about them. Granted, you can’t be held in violation if you don’t address them, but your goal should be more than compliance: It should be making sure that, no matter what amount of resources (staff time and funds) you have available to you for CIP-013 compliance, you are using them as efficiently and effectively as possible. Because after all, supply chain risks are probably the fastest growing set of risks in cybersecurity today (just ask the Russians, who are routing close to 100% - if not 100% - of their attacks on the US electric power grid through vendors). You (and the BES) can’t afford to waste resources in this effort, especially if you don’t have adequate resources to begin with (and raise your hand if you have all the resources you need for supply chain security…yup, I didn’t think I’d see any hands).

I see three main ways in which a NERC entity can waste resources on CIP-013 compliance:

1. Scatter their resources on partially mitigating a lot of supply chain security threats (probably without realizing they’re doing that), without actually mitigating any of them (and I define mitigation to be lowering the level of risk posed by a particular threat from medium or high to low, for a particular vendor, product or service).
2. Pour resources into over-mitigating a small number of threats, while barely addressing other threats at all, even though they may pose the same – or even a higher – level of risk.
3. Devote resources to mitigating risks that don’t even apply to BES Cyber Systems.

However, while I’ve been sure for a long time that it’s possible to waste serious money on CIP-013 compliance, until fairly recently I struggled to identify what was the most likely way this might happen. Then, a month or so ago, a lawyer with a very large IOU contacted me about a problem she had – and voila! In one document I found great examples of all three ways to waste money. Moreover, it’s a document that is likely to be put into practice by a number of large NERC entities.

The document is EEI’s “Model Procurement Contract Language Addressing Cybersecurity Supply Chain Risk”. EEI released it in March, and while I certainly looked at it at the time, I admit that it didn’t make much of an impression on me, good or bad. However, the lawyer said her team was working on procurement language for CIP-013 compliance, and she was being asked to base that procurement language on this document. She said she thought the document was kind of “over the top” and would require her to devote a lot of resources to CIP-013 contracts, as well as delay procurements by months. She wondered if I agreed with her opinion.

So for the first time, I went through the EEI document in detail, and I came to the realization: While some parts of the document are OK and might even be worthwhile examples to follow, others would require a huge effort and lots of resources to implement properly. In fact, the document provides great examples of all three wasteful practices listed above. Before I elaborate on that statement, here’s a little background on the document.

As I’ve already pointed out, CIP-013 R1.1 requires NERC entities to identify and assess supply chain cyber security risks, but it doesn’t provide direct guidance on what risks should be considered, or even where they can be found. However, R1.2.1 – R1.2.6 state six mitigations that must be included in your plan (and since R1.2.5 and R1.2.6 include two mitigations each, it’s really eight in total). With a little wordsmithing, you can “go behind” each of these mitigations, to identify the threats they’re intended to mitigate (I prefer the term threat to risk, since what is needed here is simply a statement of something bad that can happen, not referring to its likelihood and impact. When you later consider likelihood and impact to determine the risk posed by the threat, then it becomes a risk. This approach is consonant with NIST 800-30, although I’ll admit I didn’t develop it based on that).

For example, R1.2.1 reads “Notification by the vendor of vendor-identified incidents related to the products or services provided to the Responsible Entity that pose cyber security risk to the Responsible Entity.” The threat behind this mitigation can be found by inverting the wording: “A vendor won’t notify the Responsible Entity of an incident related to products or services it provides to the RE. Because the RE doesn’t know about this incident, it isn’t able to take adequate steps to mitigate the risk(s) to the BES that were revealed by the incident. This leads to a compromise of a BCS and damage to the BES.” Note that your statement of a threat for CIP-013 should always end with an adverse impact on the BES. If you can’t think of a BES impact for this threat, then you shouldn’t be considering it under CIP-013 (although you might consider it in your supply chain cyber security risk management plan for IT systems).

When you consider supply chain threats in general for CIP-013 plan, you need to include the eight R1.2 threats in the plan, regardless of whether you would do that if they weren’t mandatory. Fortunately, all of the R1.2 threats are important ones, so they would almost all probably make the cut anyway. This means that you need to describe in your plan how you will mitigate each of the R1.2 threats.

There are various tools you can use for supply chain risk mitigation. One tool is RFP language; you can include risk mitigation measures as deliverables of the RFP itself. Another is vendor questionnaires, in which you ask the vendor to describe how they have mitigated each of the vendor (or supplier) threats; you can then use their responses to develop a risk score for each threat, for each vendor. These let you precisely target your procurement mitigations, instead of basing them on an overall vendor risk score, which applies to all threats and requires the same mitigations for any vendor with the same overall score.

Of course, contract language is another tool for mitigating vendor risk, and the EEI document lays out what they think is appropriate language for mitigating the eight R1.2.1 – R1.2.6 threats. I do want to mention that simply getting the vendor to agree to contract language doesn’t in itself mitigate any risk; you need to take some steps to verify they’re doing what they promised to do – just like you would do if the vendor makes their promise in a letter, an email, or simply a phone call.

So how will adapting the EEI contract language as your own lead to your wasting money? Let’s start with the first way described in item A above: scattering your resources among a wide variety of supply chain risks, and not mitigating any of them very well.

You may notice a contradiction here: I’m saying the EEI document encourages you to waste your resources by addressing a wide variety of supply chain threats, yet I also said the document was developed to provide contract language specifically for the eight threats implied in R1.2.1 through R1.2.6. Which is it?

The problem is that the people who wrote the EEI document included a lot of language that addresses threats other than those included in R1.2. For example, R1.2.5 requires mitigation of two, and only two, threats: a) the threat that a malicious third party will intercept a patch while you're downloading it from a trusted site and insert a malware-laden substitute; and b) the threat that you will download inauthentic software or patches, either from a fake site that's a good facsimile of the trusted site or from the trusted site itself, which has been compromised - with inauthentic patches inserted – by a third party. 

However, EEI's language for R1.2.5 includes this:

Contractor shall identify the country (or countries) of origin of the procured product and its components (including hardware, software, and firmware). Contractor will identify the countries where the development, manufacturing, maintenance, and service for the product are provided. Contractor will notify Company of changes in the list of countries where product maintenance or other services are provided in support of the procured product. This notification shall occur 180 days prior to initiating a change in the list of countries.

This one paragraph (and there are many more like it throughout the document) identifies four threats that are different from the two threats that are the subject of R1.2.5. These threats are:

a)      A supplier will provide the entity with a product or product component that has been manufactured or developed in a suspect foreign country.

b)      A product that is shipped back to the supplier for service will in fact be serviced in a suspect country.

c)       Even though a supplier has informed the entity of the country(ies) where service is performed and they were acceptable to the NERC entity, the supplier will in fact send a product to a different country that is a suspect one (“Oh, we forgot to tell you. We normally ship our products to Japan for service, but because we got a great deal this one time, we sent your product to North Korea instead. Hope you don’t mind…”).

d)      The supplier will notify the entity of an upcoming change in service countries, from an acceptable country to an unacceptable one, close to the date when that change is effective, when it will be too late for the entity to do anything about it.

Now, I’m not disputing that all of these threats might be worth addressing in contract language (or with other mitigation tools). But the NERC entity needs to identify the threats it wants to mitigate at the beginning of the CIP-013 process, when they “identify and assess” risks in compliance with R1.1 - EEI shouldn't have made this choice for the entity. I’m sure that whoever wrote the EEI document didn’t understand that they were addressing a lot more threats than advertised, but that’s exactly what they did (I haven’t counted them, but I’m sure there are at least 50 threats addressed in EEI’s language, that don’t have anything to do with the R1.2 threats that are the ostensible object of this document).

You might ask “Why is it such a big deal if you have a bunch of other threats addressed in your contract? Maybe the supplier will actually take them seriously; maybe they won’t. But there’s no harm in just including them.” To be honest, I might have said something like that, before my email conversation with the lawyer. She said “What I don’t want to do is overburden the procurement process with unnecessary language, or have to evaluate the acceptability of each and every deviation proposed by a vendor. That might add months to the procurement process.”

From experience, she knows that the supplier’s lawyers will always try to minimize the compliance burden on their employer, by proposing deviations from the entity’s proposed language – after all, that’s what they’re paid to do. There will almost inevitably be a back-and-forth between the two sides, until they have arrived at an acceptable term (or the NERC entity has been convinced to remove the term altogether). This will have to be done for every term. Again, if the term is there to mitigate a threat that the entity has decided is important to mitigate, that’s fine. It’s worth the lawyers’ time (and the delay in the procurement process) to do this negotiation. But I really don’t believe your lawyers will be too happy when they’re asked to negotiate terms relating to threats that you don’t think are important enough to mitigate in the first place!

And the lawyer didn’t mention another big cost to having superfluous language in your contract: You need to take steps to verify that the supplier is actually keeping their promises. This doesn’t mean you have to do an expensive audit of each supplier. I think a questionnaire is fine for this purpose, as long as it was made clear in the RFP or contract that the supplier will need to fill out a security questionnaire at some regular interval.

But keep in mind that there is a cost to the supplier for every question they answer. Since it’s very unlikely that the person answering the questions will have all of the answers at their fingertips, they will have to spend a lot of time trying to find the right person to answer each question, asking the question multiple times if they’re very busy, etc. While it’s not likely that the supplier will tell you they’re just too busy to answer your questionnaire and they wish you good luck with the next sucker…er, supplier…that you choose to buy from, you can be sure that, especially if your questionnaire has lots of questions, you’ll sooner or later pay the price for this in one way or the other. Plus, you will need to evaluate the supplier’s response to every question in the questionnaire, and that requires a lot of valuable time on your end.

Similarly to the case with the lawyers, if you’re sure that every question in the questionnaire is precisely targeted to identify how the supplier has mitigated a threat that you’ve decided you need to mitigate, this cost should be acceptable to you, since the questionnaire is the best way to determine what steps the supplier has taken to mitigate each threat. But if you have a bunch of questions that are only there because of some language that you blindly copied from the EEI document into your contract, you’re once again wasting your resources.

In part 2 of this post, I’ll address even more exciting ways that the EEI document can help you waste money! Please try to contain your excitement.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. My offer of a free webinar on CIP-013, specifically for your organization, has received a great response, and remains open to NERC entities and vendors of hardware or software components of BES Cyber Systems. To discuss this, you can email me at the same address.

---

[i] One of the five white papers recently published on NERC’s web site by the Supply Chain Working Group of the NERC CIP Committee discusses these issues, but doesn’t constitute compliance guidance (as the disclaimer says at the beginning of the paper). It’s the one called “Supply Chain Cyber Security Risk Management Lifecycle”.