NERC’s new CIP-013 FAQ part I

NERC released a new CIP-013 FAQ to the industry in December. It’s based on the CIP-013 Small Group Advisory Sessions that were held in the fall of 2019 BC (Before Covid). This is the second FAQ NERC has released based on those sessions; the first one came out last February. This new FAQ includes all of the questions and answers in the old FAQ, as well as others in an Addendum.

For this post, I went through about half of the questions (I skipped a few where I didn’t have much to add to what NERC said), providing the question, NERC’s response and my response. I will hopefully get through the rest of the questions in part II, coming soon to a blog near you. Note that almost everything I say in this post is shamelessly stolen from my upcoming book “Supply Chain Cybersecurity for Critical Infrastructure”, which I expect to have published in February, or March at the latest (I was really hoping to have it out by the end of January, but I decided I’d like to have a good discussion of lessons to be learned from the SolarWinds attacks, of which there are a number).

A registered entity buys equipment from a vendor with third-party software installed. What are your recommendations for showing evidence of due diligence?

NERC response: The registered entity should use its SCRM plan to identify and assess the risks associated with the third party software installed. The results of this analysis would dictate what mitigations are appropriate to address the risks related to the third party software. Some common forms of evidence include, but are not limited to, checklists or the contents of a change ticket that documents the due diligence performed.

Tom’s response: I consider this to be two procurements: one is from the hardware vendor and the other is from the software vendor. The hardware vendor should be assessed according to the risks applicable to hardware, while the third party software vendor should be assessed according to the risks applicable to software (which are much more extensive than hardware risks. If you don’t believe that, ask the folks at SolarWinds).

What about the OS? OS’s certainly have risks, so they need to be assessed as well. Usually, if you have standard servers, I would consider the hardware and the OS to be part of one procurement, if you always purchase them together.

I don’t think you should assess the supplier whenever you make a procurement from them, even if it’s a new product. I suggest you assess all of your suppliers (the entities that make the hardware or software you buy) and vendors (the entities that sell the product to you, although sometimes they’re the same as the entity that makes it) annually, and use the results from the most recent assessment when you make a procurement.

II. What should a registered entity do if a vendor is purchased by another vendor?

NERC’s response: One approach is to ensure the registered entity’s SCRM plan details the process to re-evaluate or reassess the vendor(s). This should include the controls the registered entity deploys to maintain awareness of possible vendor acquisitions.

Tom’s response: NERC’s response seems to me to be circular: If your vendor is purchased by another vendor, you should look at your plan and do what it tells you to do. Gee, that’s real helpful.

In my opinion, if a Supplier or Vendor is purchased by another, you need to assess the acquiring organization, just as you would any new Supplier or Vendor. But you need to do this right away, rather than wait for the next time you assess all your Suppliers and Vendors. And if there are any answers you receive that raise concerns, you should address these with the Supplier or Vendor immediately, and discuss how they can remediate whatever problems they may have. Follow-up is required for proper management of supply chain security risks – and therefore for CIP-013-1 compliance. You have to take steps to mitigate vendor risks, which includes making sure the vendor follows up on any promises they made (in a contract or otherwise) to mitigate cybersecurity risks.

Is open source software in scope for CIP-013-1 and CIP-010-3?

NERC’s response: The Supply Chain Standards are silent on terms and conditions for procured products or services that registered entities may install. A registered entity should implement its risk identification and assessment methodology for all procurements and installations of open-source software on applicable BES Cyber Systems.

Tom’s response: The first sentence of NERC’s response has nothing to do with the question. The second sentence affirms that open source software is in scope, although like any software and any hardware, what you need to do depends on your assessment of the risks.

What risks apply to open source software? Every risk that applies to non-open source software. The main difference is that there’s no supplier you can contract with and hold accountable for problems. But the lack of a supplier doesn’t mean the risks disappear. You just need to figure out how to mitigate the risk in the absence of a single throat to choke. See the next question, where I give an example of this.

In fact, there are risks that apply to open source software, that don’t apply to proprietary software. For example, one open source risk, discussed in the NERC Supply Chain Working Group’s guideline “Risk Considerations for Open Source Software”, is that the community that maintains the software will dwindle to the point where patches are no longer developed when security issues arise. How will you mitigate that risk?

What compliance documentation and evidence should a registered entity create and maintain to comply with CIP-013-1 R1 Part 1.2 and its sub-parts for software that has no associated vendor, such as open source software?

NERC’s response: The registered entity may address Part 1.2.1 and Part 1.2.4 by developing one or more internal processes to identify and monitor reputable third-party sources for assessments and reports of applicable open source software incidents or vulnerabilities. The registered entity may consider developing a modified Part 1.2.5 process for acquiring, verifying, and authenticating such software and applicable patches, as released by reputable sources (e.g., for software upgrades or security patches for identified vulnerabilities). An example of this could be a completed evaluation that specifically addresses open source technology.

Tom’s response: While there’s nothing terribly wrong about NERC’s response, I take a more fundamental approach: All of the sub-parts of R2.1 represent real risks that apply to software suppliers. The risks don’t magically disappear when there’s no supplier. All of these risks apply to open source communities, although R1.2.3 is unlikely to apply to them because those communities don’t (as far as I know) provide onsite support to users, as some proprietary software suppliers do.

But other than R1.2.3, each of the parts of R1.2 can be reworded to apply to open source software with two easy steps:

1.      Identify the risk behind the item (each of the six items in R1.2 is a mitigation of a particular risk. Your mission – should you decide to accept it – is to identify that risk), and

2.      Identify the mitigation for that risk.

For example, the risk behind R1.2.1 is that you won’t receive notification of an incident related to a product you have installed on a BCS (perhaps a new vulnerability that hasn’t been patched yet). Since open source communities don’t do this, where else can you go for such notifications? Of course, there are lots of sources, both commercial and free – such as the NVD. True, it takes more effort to follow those, rather than just rely on a supplier to tell you about them. But hey, open source software is free.

Can a registered entity provide redlined contracts, demonstrating contract negotiations, as a part of our evidence to prove compliance with CIP-013-1 R1 and R2? Is this a common way to show compliance and are there other considerations we should take into account?

NERC’s response: The audit team will sample all R2 implementations, so the initial evidence request will ask for a complete list of applicable procurement(s). The audit team will sample the list in accordance with the ERO Sampling Handbook3 and request complete implementation documentation for the sampled procurements. Keep in mind the R1 plan should provide processes and procedures to indicate how the registered entity will meet the security objectives of CIP-013-1 and address each component of R1 Part 1.1 and Part1.2. While redlined contracts may serve as evidence of R2 implementations, the R1 plans should describe the registered entity’s methodology for identifying and assessing risks associated with applicable procurements. Contracts may be a component of the R1 plan, but the registered entity should ensure the procurement documents support the development of a contract that meets the CIP-013-1 security objectives.

Tom’s response: CIP-013-1 R1.1 and R1.2 are about risk management. You need to develop a plan for managing supply chain cybersecurity risks to BES Cyber Systems. A contract alone doesn’t mitigate any risk. Only if the vendor does what they promised to do in the contract does risk get mitigated. It’s up to the NERC entity to follow up with the vendor – at least with annual questionnaires, but perhaps with phone calls and emails if the questionnaire raises questions about what they’ve actually done – to make sure they’re doing everything they agreed to do. And it doesn’t matter whether they agreed in a contract, a letter, an email, a tweet, or simply on the phone – you need to follow up on everything the vendor agreed to do.

If they did it, great. If they didn’t, you might give them another chance. If they still don’t do it, you can a) try to force them to do it, through legal action or threats to stop doing business with them (but don’t issue those threats unless you’re prepared to follow through); or b) take steps to at least partially mitigate the problem on your own. For example, the mitigation for R1.2.1 compliance, that I discussed above for open source software, can easily apply to the case where the vendor won’t take responsibility. Often, when this happens your organization won’t want to drop them (and this is probably what will happen in 95% of these cases. Firing a vendor in the OT world, including electric power, happens about as often as a sighting of Bigfoot). I and my clients have identified about 80 supplier or vendor risks, and there are only a few that simply can’t be at least partially mitigated by actions the NERC entity itself can take, although without doubt it’s better to get the vendor to mitigate them.

What if a registered entity has a master agreement effective before the effective date of CIP-013-1 (October 1, 2020) which does not include terms associated with CIP-013-1 R1 Part 1.2 and its sub-parts, and purchase products or services after October 1, 2020; do I need to conduct a risk assessment on the vendor?

NERC’s response: The risk assessment should be performed on the vendor, product, and/or service as dictated by the SCRM plan. The registered entity’s SCRM plan determines where and how the risk assessment is performed. Regarding R1 Part 1.2 and its sub-parts, while the action to renegotiate or abrogate existing contracts is not required, it is expected that mitigations are implemented to address the risks of these elements not being contractually binding on the vendor. All procurements of products or services applicable to high or medium impact BES Cyber Systems after July 1, 2020 would be applicable, under the R1 SCRM plan and R2 implementation.

Tom’s response: This is a question I’ve been asked repeatedly, and here NERC completely misses the point. What this entity is really asking is “If I have a master agreement that was in effect before 10/1/2020 which doesn’t take CIP-013-1 into account, do I need to do a risk assessment of the vendor?”

My answer is: The risk assessment has nothing to do with a contract, or whether you have to negotiate a new one. After 10/1/20, every procurement you do for BCS should include a procurement risk assessment, in which you consider all of the risks that apply to this procurement. For each of these risks that has not already been mitigated to your satisfaction, you should design a mitigation that can be applied during a) procurement of a BCS product or service, b) installation and use of a BCS product, or c) use of a BCS service. A contract is one way of mitigating procurement risks, but if you won’t be negotiating a new contract because a master contract is already in place, you still need to mitigate that risk using other measures (e.g. get the vendor to agree verbally to do what you need them to do. As I said above, you need to follow up to make sure they do this. That applies regardless of whether they agreed in a contract or just a phone call).

The real question isn’t when the contract was negotiated, but when was the last time you did a procurement risk assessment for this vendor and this product. If this is a product you buy every month, you certainly don’t have to do a PRA every month (yes, I use PRA to mean procurement risk assessment). On the other hand, if it’s been a year or more since you last procured this product from this vendor, you do need to conduct an assessment. Since I’m assuming you assess the vendor once a year and you’ll just use the most recent assessment (as long as it’s less than a year old), this doesn’t have to be a long, drawn-out procedure. In fact, my clients have told me that PRAs take very little time, since usually the great majority of the risks have already been mitigated.

One thing you do need to do is consider risks that might have arisen since the last time you assessed your vendors. For example, let’s say you’re procuring a new BCS product from a new software supplier today (January 11, 2021). Maybe you should ask them a few (at least!) questions you wouldn’t have asked before December 14 of last year, when the SolarWinds attacks were revealed. For example:

1.      Do you outsource a lot of your software development to Belarus, Poland and the Czech Republic like SolarWinds did?

2.      Have you separated your development network from your IT network, including separate authentication and perhaps even controls like those in CIP-005-6 R1 and R2? Of course SolarWinds almost certainly didn’t do that, and they’re hardly alone among software developers in that regard. Yet if they’d done that, it would have made it unlikely that the Russians, having first penetrated the IT network, would have quickly penetrated the development network (I’ll admit that, if they’re really serious, they’ll figure out a way to bridge the gap between networks. But that takes time, and increases the risk they’ll be discovered before they’ve bridged the gap. In the case of SolarWinds, there was no gap, so there was no delay).

3.      Do you require multi-factor authentication for access to your most sensitive development servers, as opposed to, for example, putting a patch server on GitHub with a password of SolarWinds123? This doesn’t seem to have led directly to the Russian penetration of SolarWinds, but it’s certainly indicative of poor practices.

Unless you updated your list of risks you want to ask suppliers after December 14, it’s probably guaranteed you won’t have asked these questions. But if you’re doing a procurement today, you shouldn’t wait until the next time you update your questionnaire; you ought to ask the supplier these questions as part of the procurement. And if you don’t like their answers, you should ask them to change their wicked ways, even if you’re not negotiating a new contract with the procurement.

Part II of this set of posts is here.

Is it time to review your CIP-013 R1 plan? Remember, you can change it at any time, as long as you document why you did that. If you would like me to give you suggestions on how the plan could be improved, please email me so we can set up a time to talk. Also, if you work for a supplier trying to figure out what you need to do to help your power industry customers comply with CIP-013-1 (without wasting money on unneeded certifications), please contact me.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.