Lew Folkerth: Everything you always wanted to know about CIP-013, but were afraid to ask! (Part 2)

I recently wrote the first of a two-part post about the latest article by Lew Folkerth of RF on supply chain security risk management / CIP-013 compliance. The first part of the first post deals with CIP-010 R1.6, but the second part (much longer) is devoted to answering the first of six questions on CIP-013 that NERC has received. I discussed Lew’s interesting answer to that question (and also my complementary perspective on the issue). Now I’ll discuss his answers to the other five questions.

The second question Lew addresses is “If I buy routers at Office Depot, does that constitute a contract or is that just a procurement?” Lew’s answer is “Any equipment, software, or services whose acquisition is begun on or after July 1, 2020, that will become or will be directly related to a high or medium impact BES Cyber System must be acquired in accordance with your supply chain cyber security risk management plan. The plan must be used whether or not a contract is involved. The only place in the enforceable language of CIP-013-1 where the term “contract” appears is in the note to Requirement R2. Risks incurred by acquisitions from vendors such as Walmart (yes, they do carry business-grade Cisco products) or sellers of new and used equipment on eBay are some of the risks this Standard is intended to mitigate. In particular, there could be an elevated risk of compromised or counterfeit hardware from such sources[i].”

I think Lew’s answer is spot on. A contract is definitely not required for a procurement to be in scope for CIP-013. A contract is simply one means (and an important one, don’t get me wrong) of mitigating procurement risk – but it obviously won’t apply if you buy something from Office Depot, Walmart or eBay. In each procurement, your entity needs to identify the supply chain cyber risks that apply to the procurement and take steps to mitigate the most important of those risks. As Lew points out, some risks are definitely higher from those sources, such as the risk that you’ll acquire compromised or counterfeit hardware.  You’ll need to take all of these into account when you conduct your risk assessment at the start of the procurement (and for more on what constitutes the start of a procurement, see below).

I also want to point out that NERC’s most recent Evidence Request Spreadsheet makes it clear that a big part of the CIP-013 audit will be focused on individual procurements. The auditors will require you to provide a list of the procurements that you started during the audit period (whether or not they’re completed), and they’ll randomly sample some of these. For each procurement that’s chosen, they’ll ask you to “provide evidence of the identification and assessment of cyber security risk(s) to the Bulk Electric System from vendor products or services resulting from: (i) procuring and installing vendor equipment and software; and (ii) transitions from one vendor(s) to another vendor(s).” You will need to conduct a risk assessment for every procurement, and document you’ve identified and assessed the risks described.

The risks will vary by the product (or service) being purchased (and installed, since installation is explicitly included in R1.1). In some cases, you may have a contract with the supplier or vendor, but in some cases you won’t – the risks will be somewhat different in each case. In some cases, you’ll be buying from an authorized vendor (e.g. a Cisco dealer) and hopefully you won’t have to worry about the risks of counterfeit or compromised hardware, whereas in other cases you’ll buy on eBay and you will definitely need to worry about those risks. Your assessment needs to take into account the risks that apply to that procurement, and you need to have a defined methodology for identifying those risks (rather than just relying on whoever is doing the assessment to decide for him or herself what the risks are).

The third question Lew addresses is “Will a Responsible Entity be expected to perform and document initial cyber security risk assessments on all its existing vendors that provide their BES Cyber System products and services prior to the compliance effective date?” Lew responds: “No, CIP-013-1 affects only new procurements. This answer is supported by the General Considerations section of the Implementation Plan: ‘In implementing CIP-013-1, responsible entities are expected to use their Supply Chain Cyber Security Risk Management Plans in procurement processes (e.g., Request for Proposal, requests to entities negotiating on behalf of the responsible entity in the case of cooperative purchase agreements, master agreements that the responsible entity negotiates after the effective date, or direct procurements covered under the responsible entity’s plan) that begin on or after the effective date of CIP-013-1. Contract effective date, commencement date, or other activation dates specified in a contract do not determine whether the procurement action is within scope of CIP-013-1.’”

Lew continues “In order to determine the begin date of a procurement, you must document that date in a manner suitable for use as audit evidence. Without such documentation, audit teams will use the earliest date that provides reasonable assurance of the beginning of the procurement process.”

The fourth question is closely related to the third: “If I procured hardware or software from a vendor prior to 7/1/2020, but installed that hardware or software after that date, must I perform a risk assessment of that vendor?”

Lew responds “Risk assessments of vendors that provided equipment, software, or services prior to the CIP-013-1 effective date of July 1, 2020, are not required. Any procurements for high or medium impact BES Cyber Systems equipment, software, or services begun after July 1, 2020, must be performed in accordance with your documented CIP-013-1 R1 supply chain cyber security risk management plan. Any software installed on or after July 1, 2020, must have its identity and integrity verified, regardless of when the software was obtained.”

Of course, Lew’s last sentence refers to R1.2.5. And this bears emphasizing: The risks addressed in R1.2.1 through R1.2.6 all have to do not with the actual procurement of products or services, but the vendor’s actions after your organization has purchased and installed them. So these apply to every Medium or High impact BES Cyber System in your environment, not just those purchased after 7/1/20.

The fifth question is “Contracts for procurement that are in place prior to July 1, 2020, are not in scope for CIP-013. What about contract renewals?” This is a very good question. Lew answers “CIP-013-1 applies to any procurements begun after July 1, 2020, regardless of the existence of a standing contract, and regardless of any revisions to such a contract. You are not required to invalidate or renegotiate any contract, but you must demonstrate that any procurement begun after July 1, 2020, has been performed in accordance with your supply chain cyber security risk management plan. You will need to establish a beginning date for the procurement. The effective date of a contract is not necessarily the beginning of a procurement. The beginning date might be the date of an expenditure authorization or a request for bid, quote, etc. You will then need to show how you followed your risk management plan throughout the acquisition.”

Lew is speaking very carefully here, since he’s not allowed to go beyond what is stated in the standard (or the SDT’s CIP-013 Implementation Guidance, which is the only official guidance on CIP-013). What I think he’s saying is “In the case of standing contracts in which you can buy products at widely separated intervals, you need to determine when you’re actually starting a new procurement, or just taking another delivery as part of an existing procurement.”

This might sound pretty arbitrary to you, and it’s certainly not clearly defined (nor is it meant to be, either). In my opinion, the question should be “Have the risks changed since the last time I ordered this product?” If you just ordered the product three months ago, I doubt the risks have changed very much in that time, and I’d say this isn’t a new procurement. On the other hand, if it’s been two years since you last ordered the product, I would say it’s definitely a new procurement.

Remember, every 15 months you’re obliged to reconsider your whole R1.1 plan, which means re-identifying the risks you will mitigate. Some risks will drop out, but other risks will be added (or at least they should be. If you simply mitigate the same risks year after year, I’d say you’re doing something wrong – you have to “identify and assess” risks every 15 months, which means it’s unlikely you’ll end up with exactly the same set of risks year after year).

Another reason why you might want to declare a new procurement is if the vendor has made some substantial changes in the product since the last time you ordered it. You need to examine if any of those changes have resulted in changes in the risks posed by the product. Or perhaps your own environment has changed, and that has introduced new risks and/or (hopefully) effectively mitigated other risks. You need to consider all of these things in deciding whether this is a new procurement or not.

The last question is interesting. “My source for equipment says that they are not a ‘vendor’, but rather a ‘supplier’, and so they are not subject to CIP-013-1. How do I answer this?”

Lew answers “Any organization or person that supplies equipment, software, or services to your entity must be considered a ‘vendor’ in the meaning of CIP-013-1. Your ‘supplier’ is quite correct to say that they are not subject to CIP-013. Only NERC Registered Entities that are procuring hardware, software, or services that will become or that will directly affect high or medium impact BES Cyber Systems are subject to CIP-013-1. It is your relationship with each vendor, supplier, etc. that is subject to CIP-013-1, not the vendor itself. In managing that relationship you may use many tools, including purchase or acquisition contracts, existing vendor practices such as incident notification, existing or emerging security practices, such as software verification, vendor web site features such as digital certificates and digital signatures, and so forth. Although you may choose to manage your vendors through contracts, CIP-013-1 does not explicitly require this. If your vendor will provide a feature or a service as part of its ongoing security practices, there may be no requirement for a contract for such matters. And you may show that the implementation of your risk management plan accomplishes its goal of reducing supply chain risk by means other than contracts.”

Of course, Lew is very right on all of this. He makes these two important points:

1. It doesn’t matter what the organization calls itself. If you’re purchasing BCS hardware, software or services from them (that will be used at Medium or High impact BES assets), or if they develop software or manufacture hardware that you buy through another entity (like a dealer),  that procurement is in scope for CIP-013. I would go further to point out that the risks posed by a supplier (the developer or manufacturer) are very different than those posed by the vendor (the dealer, distributor, systems integrator, etc). It’s actually very advantageous to distinguish suppliers from vendors, since doing so let’s you address supply chain risks with a surgical knife, rather than a chain saw.
2. You shouldn’t get hung up on contracts with CIP-013 (which a lot of people do). Contracts are just one tool for managing risk, but there are a number of other tools you can use for that. I’d go further to say that you’re making a big mistake if you’re planning on making contract language your primary means for mitigating supply chain risks for CIP-013. You’re overusing the most expensive (and contentious) tool in your toolbox. Try to mitigate as much risk as you can with other means, before relying on contract language to solve your problems. And remember: Just getting some language in a contract doesn’t itself show you’ve mitigated any risk. You need to show you’re taking steps to verify the vendor/supplier is actually doing what they promised to do, whether they promised it in contract language, in a letter to you, in a phone call, in a message in a bottle, etc.

This concludes Lew’s article. I asked him what readers who want to ask other questions should do to get answers. He replied “If readers have additional questions they should send them to me. I can’t promise to answer all questions, but I’ll try. RF entities will have first priority. Entities from other regions will either be referred to that region or the answer will be coordinated with their region.” So just email your questions to Lew; the address is in the box at the end of the article. And you should probably mention what Region you’re part of.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. My offer of a free webinar on CIP-013, specifically for your organization, has received a great response, and remains open to NERC entities and vendors of hardware or software components of BES Cyber Systems. To discuss this, you can email me at the same address.

---

[i] Lew continues this answer for another paragraph, but I don’t think that’s terribly important to discuss here. You can read it for yourself!

The SCWG white papers are up!

I started talking about the NERC CIP Committee's Supply Chain Working Group in this blog back in March, including in this post. In that post, I described the five white papers the SCWG was working on, and I confidently stated that they would be published in June. Well, the wheels of NERC grind slowly, but they grind fine (most of the time). Here it is the last day of September, and they were posted today on NERC’s web site. To find them, go here and drop down “Current/Approved Security Guidelines” under “CIPC Security Guidelines”. Then drop down the “Supply Chain” item.

You’ll see a list of ten PDF files. Each of the five white papers (which are called Guidelines now, of course) has both the paper itself and slides describing the paper, which we presented before the CIPC meeting in Orlando in June.

I led the development of two of the papers: Risk Management Lifecycle and Vendor Risk Management Lifecycle. However, all of the papers are worth reading, whether your organization has to comply with CIP-013 or not; they were all put together in many meetings among SCWG members, including NERC entities and vendors. There are two more papers in the approval process, regarding risks related to cloud computing and vendor cyber incident response plans. These have both been finalized by the SCWG sub-groups that drafted them; however, given the normal approval process (including a comment period for CIPC members), I’d say those probably won’t be out until the end of the year.

The SCWG does plan to create more papers, but they definitely won’t be published this year. In fact, since these papers took six months from start of development to posting, this means any further papers might not be posted before July 1, 2020, when CIP-013 compliance is due.

However, this isn’t the tragedy that I once would have thought it to be. Since CIP-013 R1 just requires the entity to develop a supply chain security risk management plan (which includes the six mitigations in R1.2), almost any plan you develop by 7/1/20 will be compliant. However, every NERC entity should be thinking about more than just CIP-013 compliance, since IMHO supply chain security is the number one cybersecurity problem in the world today, especially for the electric power industry (look at Target, Stuxnet, NotPetya, Delta Airlines, and now Airbus. Look at the Russian attacks on the grid through the supply chain, as detailed by DHS and the Wall Street Journal. Look at last week’s article in the Times by Bruce Schneier).

Even if you think it’s too late to change your course for compliance on 7/1/20, you can still make changes to your CIP-013 program after that date; you just need to document why and what you’re changing. The point is that you want to make your plan as efficient (i.e. it mitigates the most supply chain cyber risk possible, given your available resources) and effective (it mitigates the risks that are most important for your organization’s BES Cyber Systems, which will usually be different from another utility’s) as possible; it’s never too late to do that, and the auditors certainly aren’t going to complain if you make a midcourse correction after 7/1/20 to improve your plan’s efficiency and effectiveness.

I’ve heard that one or two major utility organizations are planning on investing huge sums and lots of hours in CIP-013 compliance.  If that’s true, I suspect that they’re doing something wrong. Everything I’ve seen so far with my CIP-013 clients leads me to believe that mitigating the risks in CIP-013 is almost entirely a matter of policies and procedures, both for your organization and for your vendors/suppliers. Nothing requires you to install expensive systems or deploy legions of consultants. This is very different from the CIP v5 rollout, since CIP-013 is very different from CIP-002 through CIP-011.

A disclaimer on each paper says (with my correction of two improper verbs, which I hope will be corrected soon on NERC’s web site) “Reliability guidelines are not binding norms or parameters to the level that compliance to NERC’s Reliability Standards is monitored or enforced. Rather, their incorporation into industry practices is strictly voluntary.”

I’ll translate this from NERC-speak (and I’m a fluent translator of NERC-speak, although I’ve so far resisted trying to speak it on my own. I’ll leave that to the experts at NERC): It says there’s nothing in these guidelines (or ones to come) that constitutes some sort of binding guidance on NERC entities or auditors as they comply with/audit CIP-013; in fact, the only document that is officially designated as guidance for CIP-013, the Implementation Guidance prepared by the Standards Drafting Team, doesn’t provide anything that’s binding either.

The SCWG’s papers are guidelines to help you develop your plan; they’re not there to tell you how to comply with CIP-013. But guess what? Complying with CIP-013 requires developing and implementing a good supply chain cyber security risk management plan. If you develop a good plan and implement it properly, you’re compliant. Period, end of story. With CIP-013, unlike with any of the current CIP standards, compliance = security. And if that doesn’t make you happy, I don’t know what will.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. My offer of a free webinar on CIP-013, specifically for your organization, has received a great response, and remains open to NERC entities and vendors of hardware or software components of BES Cyber Systems. To discuss this, you can email me at the same address.

Another big supply chain breach

Kevin Perry, retired Chief CIP Auditor of the SPP Regional Entity, sent me today a link to this article about a breach at Airbus in which a lot of technical data was exfiltrated, presumably to China. By the way, I love this wonderful formula for success: stealing your way to technical competence. I don’t know why I didn’t think of that earlier in life, when it might have done me some good.

Of course, this breach came through the supply chain, in fact through multiple vendors. The attackers were able to penetrate the suppliers’ networks (evidently not too hard to do), and were able to get remote access to VPNs connecting the suppliers to Airbus (of course, it’s highly unlikely the VPNs were also protected with multi-factor authentication).

And in case you might be inclined to dismiss this attack as not too relevant to critical infrastructure - since of course it was data, not operations, that were affected - I want to point you to the last paragraph of the article, which noted that a debilitating malware outbreak at a key supplier this year had an impact on Airbus’ production (this is one downside of just-in-time procurement, of course).

Whenever a NERC entity (or really any organization) hears about an attack on another company, the question to ask is: What is the threat that’s the basis of this attack, and how can we mitigate it ourselves? This question applies especially to CIP-013 entities, since R1.1 requires you to “identify and assess” supply chain security risks (although I prefer “threats”, following NIST 800-30) to the BES? Here’s one you should identify!

The threat is that a supplier’s network will be penetrated by a malicious third party (or by a rogue insider), and they will be able to gain access to your network through vendor communications channels to your environment. Moreover, it doesn’t matter that much whether it’s your IT or your OT network that’s penetrated. If your IT network is owned by someone like the Chinese, they will ultimately figure out a way to get into the OT network - as in the first Ukraine attack, when the Russians were in the Ukrainian utilities’ IT networks for many months before they realized the HMIs for their substations were placed on the IT network. This was of course a gift to the Russians due to poor security practices by the utilities, but if a nation-state attacker is in your IT network long enough, they will very probably find a way into your OT network.

Of course, this is the second case in two posts where there’s a threat to utilities (or any organization) that results from penetration of a supplier’s network by the bad guys. In my last post, the threat was that the bad guys will penetrate the development or manufacturing environment of a supplier and substitute a hardware or software component that contains malware. This time, it’s that they’ll utilize the supplier’s access to your network to penetrate your ESP and cause havoc.

In my opinion, NERC entities should strongly consider mitigating threats that result from penetration of suppliers’ (and vendors’) networks as part of their CIP-013 plans. These aren’t idle threats, of course. Keep in mind that last summer DHS raised the alarm in a webinar that at least 200 suppliers to the power industry had been penetrated by the Russians, and that the attackers had success in penetrating power industry players that way (although that part was walked back by DHS two days later. But that walkback was contradicted by statements that had been made in a second DHS webinar that very morning. Moreover, DHS walked back the first walkback a week later - at a briefing in front of Mike Pence, Rick Perry and Kirstjen Nielsen, at the time Secretary of DHS - with a new story that was incompatible with the previous ones. I heard a third walkback at a meeting in McLean, VA last September. And to top it off, when I questioned a DHS official during a meeting at the RSA Conference this March, he started stammering out a bunch of strange excuses for this and for not investigating the Worldwide Threat Assessment. I haven’t had time to discuss this odd evening yet, but a reporter from Dark Reading did write about it, without knowing my name. Does it seem there might be something strange going on at DHS?).

And don’t forget the Wall Street Journal article from January, which detailed how the Russians are using phishing emails to penetrate supplier networks, and through them utilities. So it’s very important to take steps to make sure your suppliers are themselves following good security practices. How do you do this? There are lots of ways. Contract language is one, but hardly the only one. RFP terms and questions are another.

I think vendor questionnaires are an excellent tool, since they not only provide you with information on what your supplier is doing, but they let the supplier know what steps you think they should be taking. And having annual “security reviews” with important suppliers allows you and the supplier to freely exchange information about your security practices, as well as your mutual security objectives. The best thing about these is they shift the discussion from an adversarial one to a partnership, where you’re working with them to secure both of your environments.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. My offer of a free webinar on CIP-013, specifically for your organization, has received a great response, and remains open to NERC entities and vendors of hardware or software components of BES Cyber Systems. To discuss this, you can email me at the same address.

Bruce Schneier on supply chain risk

The Times ran a very readworthy article by Bruce Schneier today. In case you don’t know, Bruce is one of the great figures in cybersecurity. I’ve sometimes felt his articles were a little overblown in the past, but this one is spot on. He isn’t saying that all computing or telecomm equipment that we buy nowadays is riddled with back doors, but rather that there is currently no purely technical solution to this problem – so even if everything we buy has a backdoor, it’s very unlikely we’d ever know it, until of course bad things start happening.

While he doesn’t take the next step, I will for him: Organizations need to start assuming there are back doors in almost all hardware and software they buy, and perform a risk assessment before installing any hardware or software that might perform a critical function, such as – need I say it? – any system that could impact the Bulk Electric System.

A lot of the discussion on CIP-013 compliance has focused on contract language, and specifically language implementing requirements R1.2.1 – R1.2.6. This is understandable, since the path to successful NERC CIP compliance up until CIP-013 (with a few exceptions like CIP-014 and CIP-003-7/8) has been to be laser-focused on the exact wording of the requirements.

But let’s be clear: R1.2.1-R1.2.6 don’t do anything to mitigate the backdoor problem (R1.2.5, covering authenticity and integrity of patches and other software obtained electronically, makes sure that what you download is exactly what the supplier developed. But it doesn’t cover the case that the supplier’s development process itself was compromised, as happened to Juniper Networks in 2015 and to Delta Airlines last year – and which is of course what Bruce is writing about in this article).

R1.1 requires you to “identify and assess” supply chain risks to BCS. Since there are at least thousands of those risks, your job (Mr/Ms NERC entity) is to determine what are the most critical risks and mitigate those. The risk of backdoors in hardware and software will hopefully be among the risks that you choose to mitigate.

The tools for protecting your BES Cyber Systems (or any system, of course) from backdoors are all procedural: Don’t buy from unauthorized or unknown vendors; make sure hardware products are shipped securely to you (tamper tape, etc); perform a risk assessment before installing or upgrading any BCS hardware or software; etc. And not only should you follow those procedures, you should require your suppliers and vendors to do the same, through contract language and other means. In fact, you should also require your suppliers and vendors to have a supply chain cyber security risk management program, so that they require these same procedures of their suppliers.

Even though Bruce’s article isn’t aimed at the power industry, it’s one of the two industries he specifically mentions in his concluding sentence: “The risk from Chinese back doors into our networks and computers isn’t that their government will listen in on our conversations; it’s that they’ll turn the power off or make all the cars crash into one another.”

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. My offer of a free webinar on CIP-013, specifically for your organization, has received a great response, and remains open to NERC entities and vendors of hardware or software components of BES Cyber Systems. To discuss this, you can email me at the same address.

Lew Folkerth: Everything you always wanted to know about CIP-013, but were afraid to ask! (part 1)

I must admit this post is a little late. Lew Folkerth of RF published his most recent (and perhaps, for the moment at least, his last) article on CIP-013 in the RF newsletter almost two months ago. Usually, I put up a post on anything he says between 3 nanoseconds and a day after the newsletter comes out (OK, the posts been averaging longer than that – but why spoil a good story?). But what can I say? I enjoyed the summer while it lasted, and I’ve been busy with…what else?...CIP-013 clients.

You can find the article here. In fact, when you download that PDF you’ll also receive – absolutely free! – the full set of five articles that Lew has written on CIP-013 and supply chain security (such a deal!). I’ve asked Lew to autograph a set of those PDF’s, which I could sell for $39.95 on late-night TV, but I haven’t heard back from him on that attractive proposition. Probably some silly NERC ethics rule or something like that. Or perhaps he still hasn’t figured out a way to autograph a PDF file.

Lew’s article was in theory going to be about CIP-010-3 R1.6, which will come into effect at the same time as CIP-013-1 itself and essentially repeats CIP-013 R1.2.5 – although with a big (and quite unfortunate, IMHO) difference. However, upon reading this article, I realized that, while it leads with the discussion of R1.6, it doesn’t say anything notable about it, other than to repeat the requirement language.

Of course, this isn’t necessarily a bad thing; perhaps there’s nothing too important to say about CIP-010-3 R1.6. However, the rest of the article provides a lot of important information. In it, Lew responds to a number of questions that NERC has received on CIP-013 (and if you aren’t clear on this, RF and the other five Regions – yes, with the recent dissolution of FRCC, there are now a total of six Regions, vs. eight two years ago - are part of NERC). And his answers are very much worth paying attention to, if your organization will have to comply with CIP-013. I’ll discuss each one of them.

The first question he answers is “How many levels (tiers) of vendors must an entity consider for CIP-013-1 Compliance?” Of course, this refers to the fact that each of your organization’s immediate vendors (and if you haven’t yet decided to distinguish between vendors and suppliers, I highly recommend you consider it. It will somewhat complicate procedure writing – as I’ve found with my clients – but it provides a huge advantage, in letting you target your mitigations much more efficiently and effectively, as I pointed out in this post) has their own vendors that contribute to the product delivered to you. And those vendors have their vendors, etc. Where do you draw the line? Do you have to go all the way back to the company that mined the sand that was used to make the silicon chips?

Lew correctly begins his answer by pointing out that CIP-013 doesn’t provide any guidance on this question – or just about any other question you might have about it, of course. But he goes on to provide what I found to be a very surprising but provocative answer. Before I tell you his answer to this, I want to tell you what I’ve been telling my clients, since it provides a good contrast with Lew’s approach. Since the question of third-party software risk is one I’ve looked at a lot as part of the NERC CIPC Supply Chain Working Group (SCWG), for which I led the development of a white paper on Vendor Risk Management (which I’m told should be posted on NERC’s website soon, along with four other papers we developed earlier this year), and since it’s one that I’ve discussed with my CIP-013 clients as an important supply chain risk they should consider for mitigation in their R1.1 plans, I hope you’ll excuse me if I delve into this in  some detail. I find this to be a really interesting topic.

Tom’s approach to third party software risk

I’ve been telling my clients that they shouldn’t even try to reach out directly to any entities other than their current suppliers. What they need to do is make sure they’ve identified the important risks that arise through third-party suppliers, and work with (or lean on, since that will often be necessary) their direct suppliers to mitigate those risks.

For example, consider the risk that a third-party software developer, which provides code that your supplier incorporates into their software product, identifies a serious vulnerability in their code and issues a patch for it – yet your supplier doesn’t apply the patch. This means the software product they ship you (or already have shipped you) will contain that vulnerability, just as if it were in your supplier’s own code. Clearly, when the third party issues their patch, your supplier should quickly incorporate it into their product. How can you ensure this will happen?

One possible step is to include in an RFP and/or contract language the provision that your supplier must patch their own code within say a month of receiving a patch from the third party. Of course, there’s no question that, if the vulnerability were in your supplier’s own code, you would want a patch much more quickly than that! So does this mitigate the risk?

It is certainly advisable to include such a provision in your contract language, but there’s one problem: how will you verify your supplier is doing this (of course, you always need to take some steps to verify that your supplier is honoring the contract terms they agreed to. This is a matter of both good security practice and CIP-013 compliance)? You certainly know who your supplier is, but do you know all of their suppliers – specifically, all of the third parties from which they purchase code to include in their product?

Wouldn’t it be nice if the developers of the software that runs your BES Cyber Systems disclosed their ingredients in the same way that every jar of applesauce you buy does? Yet how often do you see this list of ingredients on a developer’s web site (and from everything I know, the practice of a software developer purchasing code developed by third parties and compiling it into their own product is widespread. The considerations are very similar for pieces of open source software that end up being included – there are hopefully patches for those as well)? The answer is probably never.

Does this mean you simply have to trust your supplier to always apply third party patches – i.e. you can’t do anything more than that? No, you can ask for something that’s being discussed a lot nowadays (there were at least four or five sessions on the concept at the RSA Conference this year): a software bill of materials (SBoM). This is just like it sounds: The supplier tells you the name, supplier and version number (and perhaps other information as well) of every piece of third-party code in their product. Armed with that information, you can research vulnerabilities or other issues in each piece of third party code in the product you just bought (or are considering buying). Even more importantly, you can get on the third party’s email list for information on new patches they have issued – and send a friendly email to your supplier asking them when they plan to apply that patch to their product.

But there’s another risk associated with third-party software suppliers: What if there’s a vulnerability in their software (especially one that becomes publicly known) that they don’t patch? How do you mitigate that risk? You will of course want to have contract language with your supplier saying they will develop a mitigation for any unpatched vulnerabilities in third-party code, just as they should for any unpatched vulnerabilities in their own code.

But once again, you can’t simply count on contract language to mitigate this risk. To do this, you will also need an SBoM, and you will need to be notified when there is such an unpatched vulnerability (through one of the services that does this, since the supplier itself isn’t likely to do it). And you will also need to send an email to your supplier (or even call them. Does anybody here remember when the main way you communicated with somebody in business was to use your phone? I’m told it used to happen in the dark period sometime after the Spanish-American War, but of course I’m way too young to remember anything but email or texts J) asking when they’ll have a mitigation available to address this vulnerability.

So it’s all pretty simple, right? Just get the SBoM from your supplier – that’s the key to mitigating these two important supply chain risks. I thought it would be simple, too, until I started having discussions with suppliers in the SCWG.  When I did that, I was surprised by some of the pushback I received from a couple very respected vendors (whose dedication to good cybersecurity is unquestioned in the industry). Of course, it’s hard to know their motive in opposing this, but I believe it’s out of a belief that incorporating this third party code in their products gives them a competitive advantage, and fear that disclosing information about this will allow their competitors to overcome that advantage.

Of course, there’s no way that an individual utility (except perhaps one of the very largest utility organizations) could force a supplier to provide them an SBoM. I do think this should be included as one element of the certification that NERC has said they would like to develop for vendors in the next year. The vendors wouldn’t be required to provide SBoMs, but they might find their score on the certification – if it comes to be – somewhat lower if they don’t do that. It will then be their choice whether or not to start offering one.

Of course, there isn’t any sort of near-term fix for this issue. I suggest that, if a supplier refuses to provide an SBoM, your contract terms on this issue make it clear that you consider this an important risk (assuming you do, once you’ve assessed all of your supply chain security risks to the BES and identified the ones that are important enough for you to mitigate), and that you perhaps require some sort of indemnification if a third party vulnerability goes both unpatched and unmitigated (but here I’m playing Tom Alrich, Boy Lawyer. I don’t provide legal advice! Or at least legal advice worth heeding).

Lew’s approach to third party supplier risk

Lew says the following about this topic:

My recommendation is that you should know as much about your equipment, software, and services as possible. I suggest that you document as much as you can about your BES Cyber Systems and their makeup, using your CIP-010 baselines and expanding on each baseline with as much detail as you can gather. From this information you can compose a list of hardware, software, and services that are used in your systems.

You can then assess your hardware, software, and service list based on risk. For example, you would probably assess the cyber security risk of a server power supply as very low. You would probably assess the cyber security risk of a network-connected out-of-band server management device as high or severe.

You should then be able to create a list of vendors of your devices, software, and services, and prioritize that list based on the assessed risk of each component a vendor supplies.

Of course, Lew is focusing more on third-party hardware in this discussion, whereas I’ve been focusing on third-party software. As such, our two approaches are very much complementary. I must admit it didn’t occur to me that a NERC entity would identify some of the higher-risk hardware components of say a server, and decide the risk posed by each component. But it’s possible to identify third party hardware components, since you can see them in a server or workstation, and hopefully identify their suppliers as well.

Once you’ve identified important third-party hardware components, you can reach out to their suppliers and ask to be notified of new patches to software or firmware, as well as other defects or unpatched vulnerabilities. And again, you should inform your supplier of the components that you consider most important, putting them on notice (perhaps through contract language, but perhaps through other means as well. Of course, contracts aren’t renegotiated every time a new vulnerability is identified) that you expect them to quickly address any issues that arise in these components.

When I started this post, I was thinking I could discuss all five or six questions that Lew answers in his article in one post. Looks like I was wrong about that. The remaining questions (most of them having to do with what is in scope for CIP-013 as of 7/1/20, and what isn’t) won’t require nearly as much time as this one, so I hope to discuss all of them in my next post (assuming the Russians or the cloud don’t kidnap me, as they’ve been doing recently).

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. My offer of a free webinar on CIP-013, specifically for your organization, has received a great response, and remains open to NERC entities and vendors of hardware or software components of BES Cyber Systems. To discuss this, you can email me at the same address.

My panel at GridSecCon

The E-ISAC has chosen the panelists for the panel that I’ll be moderating at GridSecCon on October 23, and it’s a very interesting one. It includes:

Howard Gugel, Director of Standards Development at NERC

Francois Lemay of Hydro Quebec

Bryan Owen of OSIsoft

Dave Whitehead, Chief Operating Officer of Schweitzer Engineering Labs

Ginger Wright of Idaho National Labs

Of course, I (and the E-ISAC) am very pleased that SEL’s COO will be joining us. This is clearly an indication of the importance that SEL places on getting their message across to the electric power industry. The initial focus of the panel was going to be on utilities themselves, but someone made the decision to instead focus on the vendor side, since vendors haven’t so far had a forum like this to answer questions on how they will work with their customers to implement their supply chain security risk management programs.

It will be a good discussion!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. My offer of a free webinar on CIP-013, specifically for your organization, has received a great response, and remains open to NERC entities and vendors of hardware or software components of BES Cyber Systems. To discuss this, you can email me at the same address.

A new dimension to the second Ukraine attack

The folks at Dragos are once again finding some really interesting stuff, this time about what could have been the real intention of the Russian attackers in the second Ukraine attack. And since this is in Wired, it's a very well-written article.