NERC, please postpone the CIP-013 compliance date

I think the CIP-013 compliance date, currently set for July 1, needs to be postponed. I think October 1, 2020 would be an appropriate target, although that might have to be re-thought if the Covid-19 epidemic in the US lasts more than a couple months. I have two reasons for saying this:

First: Obviously, the country is in a serious – and rapidly growing – health crisis. It will probably result in a number of CIP Exceptional Circumstances declarations within a week or two. For perspective on how fast the number of cases is growing, last Tuesday morning the reported number of cases in the US was 6. This morning (9 days later) it was over 1,500. It will definitely be over 2,000 this weekend, probably a lot more than that. And the big problem is that there are without a doubt thousands of people walking around now who are infected but don’t know it (or they may have some symptoms but for whatever reason – lack of health insurance, no sick leave, etc. – they don’t want to get tested, even if they could. Only a little more than 5,000 people have been tested in total, whereas in South Korea they’ve been able to test 10,000 people a day for 2 or 3 weeks).

One of my clients on the West coast has ordered all employees who aren’t absolutely essential to operations to work from home. Naturally, that includes just above everyone who is involved in getting ready for CIP-013 compliance. So that process has ground to a halt for now. It won’t be long before a lot of other utilities are in the same situation. In fact, I’m sure all onsite audits will be cancelled pretty soon, by all Regions (just like all of the spring compliance workshops are being cancelled. I got two notices this week). Even if the auditors are still willing to travel, the lawyers will ultimately tell management that the liability would be too huge if the auditor were infected beforehand, and infected everyone at the utility he was auditing, or conversely if the auditor got infected on an audit and became very sick or died.

Second: Even if Covid-19 hadn’t happened, NERC entities are in general way behind where they should be in terms of having a good supply chain cyber risk management program developed and implemented by July 1. Will they be compliant on that date after all? Probably, since all that’s required for CIP-013 compliance is that the entity have some sort of plan written, and that whatever’s in the plan – no matter how minimal – be implemented. But the whole idea of CIP-013 was not to give the industry a standard that they could all be compliant with if they’re minimally competent, but to – you know – help them meet the number two cybersecurity challenge worldwide (after ransomware), namely supply chain security. It will be quite a shame if the rush to the compliance date leaves utilities with a bunch of slap-dash plans, put in place because they decided they had to just write something at the last minute, rather than well thought-out plans that actually identify, assess and mitigate supply chain cyber risks.

And why are NERC entities not ready for CIP-013? Because they are hungry for guidance on how to comply, not just guidelines like the NERC FAQ, the SCWG’s white papers, the NATF documents, etc. Guidelines are deliberately designed not to provide guidance on compliance (in fact, it always took a lot of effort, while the SCWG was developing the white papers last year, to get people to understand that we couldn’t say anything at all about CIP-013. Even though I was one of the leading Nazis on that subject, I also forgot sometimes, like when someone pointed out that one of the white papers whose development I led included a couple references to BES Cyber Systems!).

Unfortunately, NERC is absolutely certain that to provide any guidance at all on compliance would be a violation of the hallowed principle of Auditor Independence – which basically says that, if the auditor (or auditing organization) gives an organization any sort of compliance guidance and then audits them, they’re just auditing themselves. This is because it’s just human nature that any person, auditor or not, will give a passing grade to someone that simply implemented the advice they provided.

In the long run-up to CIP v5 (mainly 2014 and 2015), when there were constant complaints about people not understanding how to comply with v5 - given the various definitions (like Programmable) that were absolutely essential to compliance but which were nowhere to be found in the NERC Glossary, as well as other ambiguities and requirements that are implicit but not actually stated – NERC made a lot of efforts to produce real guidance in the form of Lessons Learned, and finally the dreaded Memoranda. But literally every document NERC produced that provided real guidance (such as a very good Lesson Learned that defined Programmable) got shot down by the lawyers (or whoever) and was removed from the web site.

I had always accepted the sacredness of auditor independence, assuming that it was enshrined in the NERC Rules of Procedure or maybe GAGAS – but when I finally went to look for it there, I couldn’t find it. And I’ve talked to others, including some who are part of the NERC ERO, who have confirmed that auditor independence isn’t mentioned anywhere in the governing documents for NERC. So why does NERC make such a big thing of auditor independence?

I attribute it to the fact that NERC made the mistake of bringing in a Big Four financial auditing firm to train them (and the Regions) on auditing a number of years ago. With financial auditing, auditor independence is absolutely critical, so I’m sure this firm emphasized it constantly. After all, it’s much better to let a company make mistakes in their accounting than to compromise the auditing process, so that the auditors might deliberately overlook financial wrongdoing.

However, with cybersecurity there’s no such thing as “the letter of the law”. The goal of cybersecurity is to improve the odds that you won’t get hacked (or be infected with ransomware, etc); it’s not to comply with deterministic rules like the tax code or the laws of physics (which is ultimately what “enforces” the NERC Operations and Planning standards. You do or don’t do something important, and – if other conditions are right – you absolutely will cause some sort of BES event). For financial rules and the O&P standards, rigid prescriptive requirements are the only way to go, with the integrity of audits protected by strict auditor independence rules.

So with CIP, there’s no harm done if NERC or one of the Regions helps an entity become compliant. Sure, they’ll almost certainly pass their next audit, unlike an entity that didn’t receive any help. But they would be – you know – much more secure. And the last time I looked, securing the BES was the goal of the CIP standards, right? Please let me know if something has changed, but I believe that’s still the idea…

However, I somewhat doubt NERC is going to throw over the idea of auditor independence anytime soon, meaning it’s very likely that, come July 1, NERC entities will still be as confused about CIP-013 compliance as they are now. I might have advocated pushing the date back anyway, as I did repeatedly in the runup to CIP v5 compliance. But now with Covid-19 upending basically the entire world’s plans, I’d definitely say that a 3-month delay in the CIP-013 date wouldn’t be a huge problem in the grand scheme of things, and would lead to a much more secure BES at the same time.

Note to NERC: I realize you will need FERC's approval to push the date back, just as you did with CIP v5 in 2016. But my guess is they'll be glad to do this. Why wouldn't they?

Note to the Trade Associations: In the CIP v5 postponement, you folks took the lead in requesting FERC to postpone the date, and NERC actually filed a brief opposing the move. So you may have to take the lead again, although I hope not.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. My offer of a free webinar on CIP-013, specifically for your organization, remains open to NERC entities and vendors of hardware or software components for BES Cyber Systems. To discuss this, you can email me at the same address.

Advice on CIP-010-3 R1.6

I met Richard Brooks, formerly of ISO New England and now with his own company, Reliable Energy Analytics, at the IEEE Smart Grid Security conference in December in Atlanta (remember when there were these things called conferences that people attended without the slightest thought about health risks? Ah, those were the days…). He and I both spoke at the conference. He’s a pretty interesting guy and seems to really know his stuff, especially about software security.

Richard emailed me this week about a post of his on Energy Central (where he contributes readily) on verifying integrity and authenticity for patches, in compliance with the upcoming CIP-010-3 R1.6, which is kind of the doppelganger of CIP-013 R1.2.5. Compliance with both requirement parts comes due on July 1 (BTW, if you read the post, I want to let you know that I object to Richard’s characterization of me as a High Priest. Maybe a First Deputy Associate Priest – something like that).

If you look through Richard’s post (and let me know if you have trouble getting into it. You should be able to, even if you’re not an Energy Central member), you’ll see that he’s suggesting a number of means of verifying software integrity and authenticity, none of which will be at all easy for someone who isn’t a real software geek (e.g. me).

Fortunately, R1.6 includes the provision “..when the method to do so is available to the Responsible Entity from the software source..” This means that, if the software supplier doesn’t provide a means to fairly easily verify that a patch came from them and wasn’t interfered with in transit (i.e. digital signature and hash value), you’re not obligated to take extraordinary measures like the ones Richard describes.

But remember that you also have to comply with CIP-013 R1.2.5, which requires you to have a process for “Verification of software integrity and authenticity of all software and patches provided by the vendor for use in the BES Cyber System”. What’s the difference here? CIP-013 is a risk based standard; CIP-010 isn’t. In CIP-013, you need to consider the risk in every situation.

So let’s say you’re patching your EMS, and the supplier doesn’t provide a digital signature or hash value (not very realistic, I’ll admit. But bear with me). If you find something that looks like their patch on something that looks like their website, it certainly might be worthwhile taking some more extraordinary measures to verify it, given the importance that EMS poses for the BES.

Another difference between R1.2.5 and CIP-010 R1.6 is that R1.6 (like all of the other requirements in CIP-002 through CIP-011) applies to all BES Cyber Systems, not just ones that are being newly procured. CIP-013 just applies to BCS that are procured after 7/1/20 (as stated in NERC’s new CIP-013 FAQ).

And by the way, if you think all this verifying integrity and authenticity is a hoax, I’ll point out that one of my CIP-013 clients had an incident some years ago where a contractor downloaded a patch from what looked like the supplier’s site, but it turned out to be a fake site – and the patch had malware in it. Luckily, it got caught before it was installed (it was going on the IT network, not OT). But this can certainly happen.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. My offer of a free webinar on CIP-013, specifically for your organization, remains open to NERC entities and vendors of hardware or software components for BES Cyber Systems. To discuss this, you can email me at the same address.

NERC’s CIP-013 FAQ

As unaccustomed as I am to praising NERC, I have to say that the FAQ on CIP-013 that NERC released a few weeks ago (you can find it here) is spot on. I don’t have any major issues with what was said, and I think it shows the people who prepared the FAQ have a good overall understanding of the standard, which is quite different from the previous CIP standards. I’ll discuss some of NERC’s answers in this post, and hopefully later I’ll address some more in another post (although I won’t call this Part I, since I have a long history of writing Part I and never getting around to Part II).

One question I hear all the time is “Do we need to address procuring X under CIP-013?” One of the questions in the FAQ was of exactly that form: It asked “A registered entity buys equipment from a vendor with third-party software installed. What are your recommendations for showing evidence of due diligence?”

When I get a question like this, I always turn it around and ask “Could X pose any risk to the BES if procured and installed in one of our ESPs? In this case, the question would be “Could that third-party software pose any risk to the BES when you install the hardware it’s on?” Of course, the answer to that is yes, it could; there’s nothing special about buying pre-installed software that makes it not carry any risk at all. As with any procurement, you need to identify all of the risks that apply to the procurement (and installation), and determine whether they have all been sufficiently mitigated (either by the supplier or by your organization, in the case of risks that apply to you). If any of them aren’t fully mitigated and there is residual risk, you need to identify mitigations you can apply during the procurement and installation of this product.”

NERC’s answer to the above question was “The registered entity should use its SCRM plan to identify and assess the risks associated with the third party software installed. The results of this analysis would dictate what mitigations are appropriate to address the risks related to the third party software…”

Of course, you may well run your risk assessment and determine there are no significant risks that aren’t already mitigated – meaning there isn’t any residual risk to be mitigated in this procurement. In the above example, you should run through the same set of risks that you would consider for any other software procurement under CIP-013 – for example, the risk that the Supplier of the software won’t regularly patch (or otherwise remediate vulnerabilities found in) third-party and open source software that is included in their product. If you decide that the Supplier has already sufficiently mitigated all of the risks that apply to procuring the software (and also that your organization has sufficiently mitigated the risks that apply to you, such as the risk that your staff won’t install the system in a secure manner and it will be vulnerable to an attack), then you don’t have to take any particular mitigation measures with regard to the software, other than those you take with any procurement/installation of BES Cyber Systems.

The compliance risk here might come if you don’t consider all of the “normal” risks that you consider for a piece of software in other procurements, and you haven’t documented why you didn’t consider the others. For example, if you decided that the risk that a backdoor was planted in the software while it was being developed doesn’t need to be considered in this case – but you have considered this in other procurement risk assessments - you definitely need to document why you didn’t consider it here.

You might point to the fact that this system will be on a High impact Control Center network that is protected by all of the controls in CIP-005 and CIP-007; thus, it would be very unlikely that someone could even reach the system to exploit the backdoor. That’s a good reason, but you need to document it during the procurement risk assessment, not wait until you get audited three years later and have to rack your brain to remember why you didn’t consider it when the auditor asks you that question.

Another question in the FAQ follows along the same line. It reads “Is open source software in scope for CIP-013-1 and CIP-010-3?” NERC’s response is “The Supply Chain Standards are silent on terms and conditions for procured products or services that registered entities may install. A registered entity should implement its risk identification and assessment methodology for all procurements and installations of open-source software on applicable BES Cyber Systems.” 

Let me paraphrase NERC’s answer: “It doesn’t matter if it’s open source software or not. You still need to consider the risks attendant on procuring and installing it. Of course, with open source software, there are important risks you need to assess, that you don’t have with commercial software. A good guideline on these is one of the papers produced by the Supply Chain Working Group, available here.”

Another type of question I see a lot of is questions regarding contracts. A lot of people in the NERC CIP world have formed the idea that contracts have some sort of special status in CIP-013 compliance, apart from all other mitigations that a NERC entity might apply to supply chain cyber risk. One of the questions in the FAQ is “Can a registered entity provide redlined contracts, demonstrating contract negotiations, as a part of our evidence to prove compliance with CIP-013-1 R1 and R2?”

NERC’s answer reads in part “…the R1 plan should provide processes and procedures to indicate how the registered entity will meet the security objectives of CIP-013-1 and address each component of R1 Part 1.1 and Part1.2. While redlined contracts may serve as evidence of R2 implementations, the R1 plans should describe the registered entity’s methodology for identifying and assessing risks associated with applicable procurements. Contracts may be a component of the R1 plan, but the registered entity should ensure the procurement documents support the development of a contract that meets the CIP-013-1 security objectives.”

Here’s my translation of this (I took four years of NERC-speak in college, so I’m fluent in that arcane dialect): “To comply with R1 (both R1.1 and R1.2), you need to develop a plan that demonstrates how you will identify, assess and mitigate supply chain cybersecurity risks to the BES (the plan needs to include the six – really eight – risks in R1.2, but you can’t limit your plan to addressing just those risks). To comply with R2, you need to show how you implemented that plan. Contracts are one way to mitigate those risks, so the redlined contracts could serve as part of the evidence for R2 compliance.

“However, you should keep in mind that the important thing isn’t that you got a vendor or supplier to agree to mitigate particular risks, but that they actually did it. This means your plan must discuss not only how you will use contracts for risk mitigation where possible, but how you will verify that the supplier is actually following through on what they promised. And if they don’t follow through at some point, the plan needs to describe how you will endeavor to get them to forsake their wicked ways and keep their promise. And you’ll need to have some documentation that you did this, including copies of emails, memoranda of phone conversations, etc.”

Speaking of contracts, another question reads “What if a registered entity has a master agreement effective before the effective date of CIP-013-1 (July 1, 2020) which does not include terms associated with CIP-013-1 R1 Part 1.2 and its sub-parts, and purchase products or services after July 1, 2020; do I need to conduct a risk assessment on the vendor?”

NERC’s response reads “The risk assessment should be performed on the vendor, product, and/or service as dictated by the SCRM plan. The registered entity’s SCRM plan determines where and how the risk assessment is performed. Regarding R1 Part 1.2 and its sub-parts, while the action to renegotiate or abrogate existing contracts is not required, it is expected that mitigations are implemented to address the risks of these elements not being contractually binding on the vendor. All procurements of products or services applicable to high or medium impact BES Cyber Systems after July 1, 2020 would be applicable, under the R1 SCRM plan and R2 implementation.”

I interpret this to mean “First, you need to distinguish vendor, product or service risk assessments from the actual procurement risk assessment you need to perform for every procurement (and you should read the current NERC Evidence Request Spreadsheet section for CIP-013 for a description of the documentation you’ll need regarding the procurement risk assessment, since CIP-013 audit evidence requests will ask for this evidence and this evidence alone); you can perform the vendor risk assessment whenever you want, as long as you follow what you said you would do in your R1.1 plan (I’m recommending that my clients perform vendor risk assessments – using questionnaires – once a year).

“However, you do have to do something in order to be compliant, which is to perform a procurement risk assessment with every procurement. And since this purchase is a procurement, you need to do one of these here, and it needs to address all of R1.1 and R1.2 (which BTW is exactly what the Evidence Request Spreadsheet requires).”

I agree with all of NERC’s answer so far, but there’s one subject they’re avoiding: There’s no NERC definition of a “procurement”, meaning it’s up to the entity to define what it means. This is just like in the runup to CIP v5, when the entity needed to define for themselves what “programmable”, “affect the BES”, “reliability purposes” and other terms meant. And if you never did that, you still need to do it today, since compliance with the current CIP standards (including CIP-013) continues to be based on these undefined terms.

For example, if you don’t define these terms, you might be at the mercy of an auditor who says that you didn’t identify Cyber Assets properly, because you used the “wrong” definition of programmable. If you have your definition documented and you can show that you followed it when you identified your Cyber Assets, you will simply point to the definition and say “Since the term was never defined, this is how we defined it.” Of course, you should never get a violation if you’ve done that, but you might be open to it if you haven’t.

The same thing with “procurement”. It would be nice if it had been defined with CIP-013, but it wasn’t; and I know of no initiative on NERC’s part to draft a definition. So if you say that your procurement started with the master contract and all purchases since then are part of that procurement, then you could try to use that to justify not doing anything when you buy products after July 1 under that agreement – since they’re part of a procurement that started before the compliance date.

However, I don’t think this is a good idea. Since master agreements can go on for many years, you would essentially be saying that none of the risks are going to change at all during that time; and frankly, I think that’s a stretch. I think you’re safe in saying that, if you make two purchases three months apart under one master agreement, it’s not likely the risks would have changed much, so doing a procurement risk assessment for the first purchase will cover the second purchase as well. On the other hand, if you make two purchases under the same purchase agreement that are one year apart, I’d say you wouldn’t be justified in making the same assertion in that case.

But you shouldn’t be afraid of procurement risk assessments. As long as you have a well-defined procedure for doing them, and you have a way of generating the documentation you need during the assessment - so that you don’t have to try to recreate what you did years later in an audit – in general I think you’ll find that you will identify very few risks that weren’t mitigated before the procurement began, so that that you don’t have to mitigate many of them during the procurement itself (and perhaps none at all). At least, that’s been my preliminary conclusion as I’ve done “trial runs” of procurement risk assessments with clients.

Of course, everything I say here will depend on the auditors being in line with what NERC has said in the FAQ. Is that a good assumption? Let’s put it this way – I would feel a lot better if I knew NERC was planning a robust training program for the different Regions on CIP-013 auditing, since that will be hugely different from auditing the other CIP standards. But I haven’t heard that anything like that is in the offing. Have a nice day!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. My offer of a free webinar on CIP-013, specifically for your organization, remains open to NERC entities and vendors of hardware or software components for BES Cyber Systems. To discuss this, you can email me at the same address.

A Clarification from OSI

Rob Koziy of OSI, who had provided me the information on which I based my most recent post, sent me this clarification yesterday, regarding their policy regarding making their audit report available. This seems to be a confirmation that one or two guesses I made were correct. As they say, it's smarter to be lucky than it's lucky to be smart.

OSI anticipates that our ISO 27001/27002 independent audit report will be posted on our CIP-013 secure webpage in the April timeframe.   In addition we will provide copies of OSI policies-procedures that were reviewed during the audit and specifically relate to CIP-013 requirements for compliance to 27001/27002 controls.     

Good for OSI!

In my last post, I discussed the importance of supplier/vendor security questionnaires and described how two different suppliers – both major suppliers to electric utilities – have described their policies for responding (or not) to questionnaires submitted to them by their customers. I finished the post by praising one of them, Schweitzer Engineering Laboratories, for saying they would answer all questionnaires from customers, full stop.

However, I first described a “major OT software supplier” to the power industry, who had put out a paper at the end of last year - which described their policy toward customer security questionnaires. Their story was more nuanced. Based just on my reading of the letter, I summarized it in two points: a) They will be reluctant to respond to customer security questionnaires in general, especially lengthy ones; and b) They are now compliant with ISO 27001/2, and will be audited soon. They want customers to look through ISO 27001 first, to see if their certification will address some (and hopefully most) of their questions.

I said this would be OK (with me, at least), as long as

1. ISO 27001 will answer all or most of the security questions a customer is likely to have. However, I pointed out that, in a list of 42 questions based on 42 vulnerabilities that I and my CIP-013 clients have identified as important enough to require industry suppliers to mitigate, I don’t think a single one is addressed in any meaningful way in 27001. This isn’t hugely surprising, since ISO 27001/2 (and just about every other important cybersecurity standard) is designed to address IT threats, not OT ones. The NERC CIP standards are all about OT.
2. This supplier will help their customers find where in 27001 their questions are answered. Simply pointing them to the document isn’t enough.
3. The supplier will provide the actual audit report, since simply knowing that the supplier passed an audit (no matter what the standard) is just about useless for supply chain security risk management purposes. Both for CIP-013 compliance and overall supply chain security, it’s important to know if and whether the supplier has mitigated each of the risks you (the utility) have identified as important – in the case of my clients, it’s the 42 vulnerabilities that led to the 42 questions we want to ask each supplier. These vulnerabilities came from a) “identifying and assessing” supply chain cyber security risks to BCS, as required by CIP-013-1 R1.1; and b) assessing the six (actually, eight) “required” risks described in R1.2.

Because I didn’t have answers to these questions at the time, in my post I gave this supplier an “Incomplete” for questionnaire responsiveness, pending any answers I received from them. But I later heard from Ron Koziy, the Director of Cyber Security & Compliance of this organization – whom I know through the NERC CIPC Supply Chain Working Group. Through a couple of back-and-forth emails, I confirmed that this organization – and I can now reveal they are OSI International, which has a huge share of the US EMS market (although they have a lot of other software offerings a well) – will in fact answer questionnaires from customers as well. However, before the customer sends them a questionnaire, OSI wants them to first read the following on OSI’s secure website, which will presumably answer some of their questions:

1. The document detailing how OSI stands with respect to the NATF Criteria. I agree this is a good first step, since most of the Criteria address real risks to BES Cyber Systems. But as I explained in this post, these are far from being the only risks that NERC entities should consider for their CIP-013 plans. Risks not addressed in the Criteria include those due to a) vulnerabilities found in fourth-party software components of the software or firmware (whether compiled with the supplier’s own code or provided as standalone components); b) insecure software or firmware development practices; c) insecure shipment of hardware; d) inadequate protections for systems used for remote access to the utility's BCS; e) inadequate anti-phishing and anti-ransomware measures on the supplier’s part; f) vulnerabilities in open source software included in the product; and g) lack of two-factor authentication for the supplier’s own remote access systems (since DHS in 2018 said in a briefing that “over 200” suppliers to the electric power industry had been penetrated by the Russians, through their own remote access systems).
2. The document (available in March) that describes the steps OSI is taking specifically to help their customers comply with CIP-013 R1.2.1 – R1.2.6;
3. The report from OSI’s ISO 27001 audit, available in April. As I’ve said earlier, the majority of the questions in ISO 27001/2 have little or no relevance for control systems. I pointed out in my last post that posting the audit report will only help if OSI helps their customers find answers to particular questions, if they’re there. Rob says they will do this, although he also says (with my italicized notes) “Specific questions will more likely be found in OSI procedures or policies for ISO 27001 (found on the secure website), for which we can assist with directing customers to the specific section (of the audit report).”
4. But let’s say the above steps don’t answer all of your questions (and I can promise they won’t answer all of my clients’ questions, since we have already developed them – BTW, the majority of my CIP-013 clients are OSI customers); what’s plan B? Here’s Ron’s response to that question: “If entities have questions that have not been answered within the posted NATF criteria, please send them to OSI at any time (i.e. now).  OSI customer questions can be sent to: CIP13@osii.com or via their account manager, or to me directly (rob.koziy@osii.com).  OSI will provide responses to all customers directly and subsequently update our CIP-013 website with new questions and answers on a regular basis.”

Of course, answer 4 is the key for me, which is why I upgraded OSI’s “grade” for questionnaire responsiveness from Incomplete to Good (like Schweitzer’s – in fact, I would now say both of their policies for responding to security questions are very good). I also like OSI’s idea of compiling and posting a list of these questions and their answers. This will further reduce the likely number of questions that are still unanswered, after the customer has spent some quality time on the OSI secure site. In fact, I recommend that Schweitzer do the same thing.

Good for Schweitzer!

An important part of supply chain security risk management is assessing vendors to identify “holes” in their security posture that need to be addressed. That’s what this post is about, but first I want to discuss how assessment – and especially the use of questionnaires – fits into the overall supply chain security methodology that I and my CIP-013 clients have developed over the past year.

For CIP-013 compliance and supply chain security in general, NERC entities need to mitigate two types of procurement risks: those that come through Vendors and Suppliers (almost always risks that are due to inadequate security policies or procedures on the Vendor/Supplier’s part) and those that come through the entity’s own actions or lack thereof (these are also due to inadequate policies or procedures, but this time those of the NERC entity itself).

This post is about the first type of procurement risk: Vendor and Supplier risks (note that I – as well as my clients – think Vendors and Suppliers should be treated differently, since the risks that apply are very different in the two cases. I’ll use ‘Suppliers’ for most of the rest of this post, although I’ll always mean both). Vendor/Supplier risks are the ones that almost everyone thinks about when they think about supply chain risks, and indeed, probably 70-80% of the most important procurement risks do come through Suppliers or Vendors.

I call these risks Vulnerabilities, although I capitalize the term to distinguish it from the normal cybersecurity term vulnerability - which means a flaw in software or firmware that could lead to compromise of a system (a BCS in this case). In my sense, a Vulnerability means any situation that enables a Threat to be realized – and I define a Threat as something that could damage the BES. For example, if a software company doesn’t perform background checks on the developers that it hires, this Vulnerability could enable a terrorist or other person with malicious intent to plant malware or a backdoor in software or firmware in a BCS.

A Vulnerability can often be stated in this way: If the Supplier doesn’t do X, then it’s possible that the NERC entity (that would be you, Dear Reader) will procure and install one or more hardware or software components of BES Cyber Systems that have a vulnerability (lower-case, of course), leading to some impact on the BES. To continue with the above example, if one of your BCS Suppliers doesn’t perform background checks on their developers, it’s possible that your organization will procure and install a software or hardware component of a BCS that contains a vulnerability or backdoor, that could be exploited by a bad guy (or woman, of course!) to damage the BES.

The impact on the BES doesn’t need to be specified, and indeed there are close to an infinite number of impact types – what matters is simply whether or not there could be any BES impact at all. This is of course how the BES Cyber Asset definition works: If the compromise or loss of a Cyber Asset can impact the BES in any way, it’s a BCA. There’s no such thing as big or small BES impact.

Why does this matter? It’s no exaggeration to say that your most important steps to improving your supply chain cybersecurity (and achieving CIP-013 compliance, of course) are to a) “identify and assess” the most important Vulnerabilities that apply to Suppliers; b) determine, for each Supplier of BCS hardware or software components, which Vulnerabilities it has already mitigated and which it hasn’t; c) endeavor to get the Supplier to agree to mitigate whatever Vulnerabilities it hasn’t yet mitigated; and d) in cases where the Supplier can’t or won’t mitigate a Vulnerability, take steps on your own to mitigate it (although the steps you can take are usually not as effective as those the Supplier could take. But they’re better than nothing).

Of course, b) is the assessment step. There are three main way to assess a Supplier: audits, questionnaires and certifications. Audits are definitely the most expensive of these options. While there can certainly be times when an audit is necessary (e.g. when a Supplier just can’t be trusted to answer a questionnaire truthfully. But really…if they’re so untrustworthy, why are they still your Supplier?), it shouldn’t be the default option.

There are also definite reasons why you might want to use certifications to determine to what degree the Supplier has mitigated particular risks. The biggest is when a large company like Cisco or Microsoft clearly won’t respond to questionnaires (although it wouldn’t be a bad idea to try. Work through your VAR for this, since they’re most likely to know who to go to at the Supplier– and they’re definitely the most motivated party to help you find that person or group). Then you should look at the Supplier’s certifications, and try to find answers there to the security questions you would otherwise ask in a questionnaire.

But certifications have their limitations, too. The main limitation is that, for questions specific to OT, IT-focused certifications like ISO 27001, or SOC II audit reports, aren’t likely to provide answers. NERC has said they’re working on a certification specific to OT risks to the BES, so that will certainly be of interest when it’s finalized. However, that’s unlikely to occur anytime soon.

So I think the best way to determine to what degree a Vendor or Supplier has mitigated each of the threats or vulnerabilities that you’ve identified as important, is to send them a questionnaire. However, I’ve found people raise two objections when I say this:

1. The Vendor may lie in their answers; or
2. The Vendor won’t respond to the questionnaire at all.

 For the first question, it’s definitely true that Suppliers don’t always tell the truth, but Supplier relationships in this industry – especially in the OT environment – last for decades, if not centuries. Given that, it’s hard to believe a Supplier would put all that in jeopardy, just to score a short-term point or two by misrepresenting the truth in their questionnaire answers. And if you think you can’t trust the Supplier’s answers on a particular questionnaire, I’m sure your contract retains the right for you to audit them. Just about all such contracts have that clause (but again: If you’ve decided you really can’t trust the Supplier, why are you still buying their products at all?).

The second objection is a little harder to counter. I know that OT Suppliers definitely don’t like to answer questionnaires, especially if they think they’ll be forced to answer a large number of questions that are IT-focused (and any “information security” framework like ISO 27001/2 is inherently IT-focused), and have little if anything to do with risks to OT. Of course, it is OT risks, and especially risks to the Bulk Electric System, that are the concern of NERC entities subject to CIP-013.

One major OT software Supplier to the electric power industry put out a short white paper for their customers at the end of last year. Its purpose was to address two problems they’re having, related to CIP-013:

1.       Customers are sending them questionnaires that are both lengthy and mostly irrelevant to OT risks – asking how they will protect the customers’ intellectual property, personal information on employees, trade secrets, etc. While these are great questions to ask a Supplier of IT products, they’re mostly irrelevant for Suppliers of OT products. So this Supplier is being asked to spend a lot of time answering a lot of questions, when their answers won’t help the customer at all in understanding their OT supply chain security risks, and specifically risks applicable to CIP-013.

2.       Customers are asking them to sign addenda to contracts that are both lengthy and contain terms that are mostly irrelevant to OT risks – ordering them to take steps to protect the customers’ intellectual property, personal information on employees, trade secrets, etc. As with questionnaires, while these might be great terms to require of a Supplier of IT products, they’re mostly irrelevant for Suppliers of OT products. This is because your Suppliers should store or transmit little if any information on the configuration of your BCS (and how to locate them logically or physically, which I the definition of BCSI), and that is the only information whose misuse might lead to a BES impact. So this Supplier’s attorneys are being asked to spend a lot of time negotiating contract terms that in fact won’t help the customer at all in mitigating their OT supply chain security risks, and specifically mitigating risks applicable to CIP-013. Plus the customers (the electric utilities) are putting a big burden on their own lawyers, since you can be sure the Supplier’s lawyers will want to negotiate or remove almost every security term you ask them to include in a contract. This probably means hours of negotiations over every single term.

In this Supplier’s well-written letter, they didn’t say they weren’t going to respond to questionnaires, but they made it clear that a) they are going to be reluctant to respond to them, especially if they’re lengthy (and they specifically mention 100 questions, which I agree is too lengthy); and b) they now have policies and procedures in place to comply with ISO 27001/2 - they’ll be audited and certified on those in the near future. In other words, “Before you send us a security questionnaire, please look through ISO 27001 Appendix A (and perhaps their audit report, although they didn’t mention that at all - so I don’t know whether or not they’ll make it available to customers) to see if your questions are answered there. And by the way, don’t send us any long questionnaire at all.” 

This might be OK if a) the ISO 27001 certification will actually answer all, or even most, of the questions that NERC entities will need to answer for CIP-013 compliance purposes; b) this Supplier will actually help their customers find where in ISO 27001/2 their particular questions are answered (since Appendix A is a long document in itself, and ISO 27002 – which provides the real meat on Appendix A’s bones – is much longer, devoting an entire page to each control, rather than a sentence or two as in Appendix A); and c) the Supplier will provide the actual audit report, since it’s very possible that the Supplier might not be in compliance with every control, even if they have received the certification.

Of course, both b) and c) are up to the Supplier. However, I can legitimately take a stab at answering a). This is because I am in the process of developing a questionnaire with my clients, based on the set of 42 Supplier/Vendor Vulnerabilities that we have jointly identified, including those that are the basis for the required items listed in R1.2. Since this post is about questionnaires, not certifications (and since I hope to write a post on certifications in the near future), I can’t provide detail on this now. But I will say that almost none of the Vulnerabilities in my list is addressed directly in 27001 (and none of the items in R1.2 are fully addressed, either). So I feel quite safe in saying that, at least for the list of questions that my clients are likely to have, there are few that are directly answered by ISO 27001/2 (although I will reach out to this Supplier to see if they feel differently about this or anything else in this post. I’ll put their response – unedited! – in a future post, if they’ll allow that).

Item c) is actually very important. If this Supplier plans to just publish their overall score for the audit, that isn’t very helpful at all. In fact, any certification or service that just provides a single security score for a Supplier is close to useless for assessing OT Suppliers to the electric power industry. Why? Consider this: How often does your organization either 1) choose a completely new vendor on the OT side of the house or 2) conduct a thorough assessment of all of the competitive Suppliers to whoever is your current Supplier for a product or class of products?

One of my clients had a succinct answer to this: Once every five years at most. Even if it’s more frequent for you, my point is this: An overall security score for a Supplier is designed to aid a decision on whether or not you will buy from a particular Supplier, or perhaps to serve as an input to choosing among multiple suppliers of similar or identical products. It provides little or no help for the task that you perform much more frequently: procuring a product from an existing Supplier, from which you may or may not have previously procured the identical product.

And this is exactly the scenario that’s addressed by the CIP-013 section of the NERC Evidence Request Spreadsheet v. 3.0 (as well as the just-released v. 4.0, which seems completely unchanged, at first glance). For each procurement during the audit period, the NERC entity needs to comply with the wording of R1.1 and R1.2. In other words, you need to “identify and assess” the procurement and installation risks that apply to each procurement, as well as explicitly assess each of the risks addressed in R1.2.  You can’t do this with an overall risk score for the Supplier; you need some sort of score for each risk (Vulnerability) that you identify in R1.1, as well as the six items in R1.2 (and there are really eight risks in R1.2, since R1.2.5 and R1.2.6 address two risks each). This score will let you decide whether each particular risk (each Vulnerability, in my terminology) has already been mitigated, or whether you need to take additional steps during this procurement to mitigate it. This is the beating heart of CIP-013 compliance, since this is the entirety of the evidence that will be required up front by the auditors.

The upshot of this is that, since this Supplier doesn’t want to commit at the moment to answering all questionnaires (of reasonable length) from customers, and since I don’t yet know what they will say about questions b) and c) immediately above, I have to give the supplier an Incomplete grade at the moment. And I’d like to point out to them that they need to make sure that, whatever answer they give to my questions, they should make sure it will satisfy what is required by the Evidence Request Spreadsheet.

Note on Sunday evening: I put up this post earlier in the day and sent the link to a security person I know at the Supplier in question. He got back to me fairly quickly, and we're now engaged in an email dialogue that will I hope nail down what they're actually prepared to do, which may very well meet what I think is required. Rather than make an already-long post even longer, I'm going to write about that in another post, hopefully Monday or Tuesday night.

You may be wondering what the title of this post has to do with the content (not that the titles I provide for my posts ever tell you exactly what the content is). Well, I heard a few weeks ago – in response to a question I’d sent – from a different Supplier, one that just about every (or perhaps every) electric utility in the US uses: Schweitzer Engineering Labs.

I asked George Masters of SEL, whose title is Lead Product Engineer, Secure Engineering – and a good friend of mine from the NERC CIPC Supply Chain Working Group – what SEL’s position on customer security questionnaires was. His answer was unequivocal:

SEL uses security processes which combine best practices from a variety of standards and from our own experience inventing, designing, and building products and systems, then tracking their performance across decades of service life. We will always respond to questionnaires from our customers, as we understand that they are working to implement processes to meet CIP requirements as well as their own unique requirements.

We are watching for certifications or similar assurance mechanisms to emerge that can be broadly accepted by our customers as evidence that their requirements are being satisfied. We have taken that approach in other domains. As an example, our Quality Management System is certified to the ISO 9001 standard, and our manufacturing processes comply with the IPC-A-610 Class 3 workmanship standard for products requiring high reliability, such as those used in life-support and aerospace systems (see https://selinc.com/api/download/117615/).

There you have it. No qualifications, no ifs, ands or buts. This is another example of SEL’s commitment to supply chain security, which I experienced when I was moderator for the panel on that topic at last October’s GridSecCon. NERC had asked SEL to have someone join the panel, and I was expecting they would send George or one of his peers (which would have been fine with me). Instead, they sent Dave Whitehead, who was COO at the time and is now CEO.

Dave gave a good (five-minute, since that’s all that was allowed) presentation, and answered a couple questions very well. Plus he prepared a document for me to publish, when I requested afterwards that panelists do that. In the document, I suggested that the panelists repeat what they said during the panel (including the answers to the questions they received), and also go beyond that if they’d like. Dave prepared an excellent document summarizing their security and quality practices – in fact, on rereading it just now, I realized that this in itself answers many of the questions in my questionnaire, for SEL. You can find it here.

What, I can use my best judgment?

Since I consider CIP-013 compliance to be at heart the responsibility of the NERC entity, not mine, my consulting approach for helping a NERC entity come into compliance consists of a series of 1-3 weeklong workshops with cybersecurity, compliance, procurement and (sometimes) legal people. In these workshops, I go through what CIP-013 says (that’s an easy one: It says what’s in the requirements; nothing more, nothing less. Although the Evidence Request Spreadsheet for CIP-013 also provides some very good information on what audits will focus on), as well as the “crowdsourced” methodology and MS Excel™ workbooks that I and my clients have developed over the past year (the methodology and workbooks continue to develop, although the changes nowadays are more in the fine points, not fundamental concepts).

CIP-013 compliance, and certainly my methodology, involves some concepts that don’t come easily. I consider an initial workshop to be very successful if even two or three people get the ideas I’m talking about. And I find the people who are hardest to convince are the Walking Wounded – people who have been beaten into a submissive state by the prescriptive CIP requirements (most but fortunately not all of the existing requirements) for years (usually with PTSD from one or two bad audits). They have given up forever the idea that they can ever make sense of what CIP requires them to do and guide their actions by the criterion of what’s sensible. They pray that someone, somewhere will provide them the magic key that will unlock the true meaning of all the CIP requirements, so that they can implement their compliance programs in complete confidence that the auditors will love what they do.

And barring delivery of that magic key, they pray for an early death.

I bring this up because, in a conversation at one of the meetings during this week’s workshop at a medium-to-large-sized municipal utility, we got into a discussion of what would happen if, during a Procurement Risk Assessment, they decided that mitigating the remaining residual risk from a particular threat would just be too costly – so they decided to accept the risk. Would they have the book thrown at them at their next audit and the utility would be bankrupted by the fines?

I said no (and by the way, I’m paraphrasing this discussion, since I don’t remember the exact details). In CIP-013 compliance, if mitigating a particular risk will require an unreasonable amount of cost or effort, and if the risk isn’t one that might likely involve loss of human life or limb if realized, you can certainly accept the risk if that is the reasonable thing to do. At that point, a woman from procurement sitting next to me asked something like “What, you mean we can use our best judgment?”

If you think about it, this is fairly sad. In most cybersecurity compliance regimes – HIPAA, PCI, the NIST frameworks, etc. – the organization is allowed to use their judgment to determine what’s a reasonable action, including accepting risks that can’t be mitigated at a reasonable cost. But people who have worked in organizations where CIP compliance is a big deal (even if they haven’t worked directly in compliance, as was the case with this woman) have just come to accept that they have no choice but to do whatever they think is required, regardless of what’s reasonable.

So for all of those people, I have good news: With CIP-013, you’re free, free! Now all that’s required is to rewrite all of the other CIP standards as risk-based ones like CIP-013, and you’ll be truly free. CIP compliance people of the world, unite! You have nothing to lose but your chains!