Lew Folkerth: Everything you always wanted to know about CIP-013, but were afraid to ask! (part 1)

I must admit this post is a little late. Lew Folkerth of RF published his most recent (and perhaps, for the moment at least, his last) article on CIP-013 in the RF newsletter almost two months ago. Usually, I put up a post on anything he says between 3 nanoseconds and a day after the newsletter comes out (OK, the posts been averaging longer than that – but why spoil a good story?). But what can I say? I enjoyed the summer while it lasted, and I’ve been busy with…what else?...CIP-013 clients.

You can find the article here. In fact, when you download that PDF you’ll also receive – absolutely free! – the full set of five articles that Lew has written on CIP-013 and supply chain security (such a deal!). I’ve asked Lew to autograph a set of those PDF’s, which I could sell for $39.95 on late-night TV, but I haven’t heard back from him on that attractive proposition. Probably some silly NERC ethics rule or something like that. Or perhaps he still hasn’t figured out a way to autograph a PDF file.

Lew’s article was in theory going to be about CIP-010-3 R1.6, which will come into effect at the same time as CIP-013-1 itself and essentially repeats CIP-013 R1.2.5 – although with a big (and quite unfortunate, IMHO) difference. However, upon reading this article, I realized that, while it leads with the discussion of R1.6, it doesn’t say anything notable about it, other than to repeat the requirement language.

Of course, this isn’t necessarily a bad thing; perhaps there’s nothing too important to say about CIP-010-3 R1.6. However, the rest of the article provides a lot of important information. In it, Lew responds to a number of questions that NERC has received on CIP-013 (and if you aren’t clear on this, RF and the other five Regions – yes, with the recent dissolution of FRCC, there are now a total of six Regions, vs. eight two years ago - are part of NERC). And his answers are very much worth paying attention to, if your organization will have to comply with CIP-013. I’ll discuss each one of them.

The first question he answers is “How many levels (tiers) of vendors must an entity consider for CIP-013-1 Compliance?” Of course, this refers to the fact that each of your organization’s immediate vendors (and if you haven’t yet decided to distinguish between vendors and suppliers, I highly recommend you consider it. It will somewhat complicate procedure writing – as I’ve found with my clients – but it provides a huge advantage, in letting you target your mitigations much more efficiently and effectively, as I pointed out in this post) has their own vendors that contribute to the product delivered to you. And those vendors have their vendors, etc. Where do you draw the line? Do you have to go all the way back to the company that mined the sand that was used to make the silicon chips?

Lew correctly begins his answer by pointing out that CIP-013 doesn’t provide any guidance on this question – or just about any other question you might have about it, of course. But he goes on to provide what I found to be a very surprising but provocative answer. Before I tell you his answer to this, I want to tell you what I’ve been telling my clients, since it provides a good contrast with Lew’s approach. Since the question of third-party software risk is one I’ve looked at a lot as part of the NERC CIPC Supply Chain Working Group (SCWG), for which I led the development of a white paper on Vendor Risk Management (which I’m told should be posted on NERC’s website soon, along with four other papers we developed earlier this year), and since it’s one that I’ve discussed with my CIP-013 clients as an important supply chain risk they should consider for mitigation in their R1.1 plans, I hope you’ll excuse me if I delve into this in  some detail. I find this to be a really interesting topic.

Tom’s approach to third party software risk

I’ve been telling my clients that they shouldn’t even try to reach out directly to any entities other than their current suppliers. What they need to do is make sure they’ve identified the important risks that arise through third-party suppliers, and work with (or lean on, since that will often be necessary) their direct suppliers to mitigate those risks.

For example, consider the risk that a third-party software developer, which provides code that your supplier incorporates into their software product, identifies a serious vulnerability in their code and issues a patch for it – yet your supplier doesn’t apply the patch. This means the software product they ship you (or already have shipped you) will contain that vulnerability, just as if it were in your supplier’s own code. Clearly, when the third party issues their patch, your supplier should quickly incorporate it into their product. How can you ensure this will happen?

One possible step is to include in an RFP and/or contract language the provision that your supplier must patch their own code within say a month of receiving a patch from the third party. Of course, there’s no question that, if the vulnerability were in your supplier’s own code, you would want a patch much more quickly than that! So does this mitigate the risk?

It is certainly advisable to include such a provision in your contract language, but there’s one problem: how will you verify your supplier is doing this (of course, you always need to take some steps to verify that your supplier is honoring the contract terms they agreed to. This is a matter of both good security practice and CIP-013 compliance)? You certainly know who your supplier is, but do you know all of their suppliers – specifically, all of the third parties from which they purchase code to include in their product?

Wouldn’t it be nice if the developers of the software that runs your BES Cyber Systems disclosed their ingredients in the same way that every jar of applesauce you buy does? Yet how often do you see this list of ingredients on a developer’s web site (and from everything I know, the practice of a software developer purchasing code developed by third parties and compiling it into their own product is widespread. The considerations are very similar for pieces of open source software that end up being included – there are hopefully patches for those as well)? The answer is probably never.

Does this mean you simply have to trust your supplier to always apply third party patches – i.e. you can’t do anything more than that? No, you can ask for something that’s being discussed a lot nowadays (there were at least four or five sessions on the concept at the RSA Conference this year): a software bill of materials (SBoM). This is just like it sounds: The supplier tells you the name, supplier and version number (and perhaps other information as well) of every piece of third-party code in their product. Armed with that information, you can research vulnerabilities or other issues in each piece of third party code in the product you just bought (or are considering buying). Even more importantly, you can get on the third party’s email list for information on new patches they have issued – and send a friendly email to your supplier asking them when they plan to apply that patch to their product.

But there’s another risk associated with third-party software suppliers: What if there’s a vulnerability in their software (especially one that becomes publicly known) that they don’t patch? How do you mitigate that risk? You will of course want to have contract language with your supplier saying they will develop a mitigation for any unpatched vulnerabilities in third-party code, just as they should for any unpatched vulnerabilities in their own code.

But once again, you can’t simply count on contract language to mitigate this risk. To do this, you will also need an SBoM, and you will need to be notified when there is such an unpatched vulnerability (through one of the services that does this, since the supplier itself isn’t likely to do it). And you will also need to send an email to your supplier (or even call them. Does anybody here remember when the main way you communicated with somebody in business was to use your phone? I’m told it used to happen in the dark period sometime after the Spanish-American War, but of course I’m way too young to remember anything but email or texts J) asking when they’ll have a mitigation available to address this vulnerability.

So it’s all pretty simple, right? Just get the SBoM from your supplier – that’s the key to mitigating these two important supply chain risks. I thought it would be simple, too, until I started having discussions with suppliers in the SCWG.  When I did that, I was surprised by some of the pushback I received from a couple very respected vendors (whose dedication to good cybersecurity is unquestioned in the industry). Of course, it’s hard to know their motive in opposing this, but I believe it’s out of a belief that incorporating this third party code in their products gives them a competitive advantage, and fear that disclosing information about this will allow their competitors to overcome that advantage.

Of course, there’s no way that an individual utility (except perhaps one of the very largest utility organizations) could force a supplier to provide them an SBoM. I do think this should be included as one element of the certification that NERC has said they would like to develop for vendors in the next year. The vendors wouldn’t be required to provide SBoMs, but they might find their score on the certification – if it comes to be – somewhat lower if they don’t do that. It will then be their choice whether or not to start offering one.

Of course, there isn’t any sort of near-term fix for this issue. I suggest that, if a supplier refuses to provide an SBoM, your contract terms on this issue make it clear that you consider this an important risk (assuming you do, once you’ve assessed all of your supply chain security risks to the BES and identified the ones that are important enough for you to mitigate), and that you perhaps require some sort of indemnification if a third party vulnerability goes both unpatched and unmitigated (but here I’m playing Tom Alrich, Boy Lawyer. I don’t provide legal advice! Or at least legal advice worth heeding).

Lew’s approach to third party supplier risk

Lew says the following about this topic:

My recommendation is that you should know as much about your equipment, software, and services as possible. I suggest that you document as much as you can about your BES Cyber Systems and their makeup, using your CIP-010 baselines and expanding on each baseline with as much detail as you can gather. From this information you can compose a list of hardware, software, and services that are used in your systems.

You can then assess your hardware, software, and service list based on risk. For example, you would probably assess the cyber security risk of a server power supply as very low. You would probably assess the cyber security risk of a network-connected out-of-band server management device as high or severe.

You should then be able to create a list of vendors of your devices, software, and services, and prioritize that list based on the assessed risk of each component a vendor supplies.

Of course, Lew is focusing more on third-party hardware in this discussion, whereas I’ve been focusing on third-party software. As such, our two approaches are very much complementary. I must admit it didn’t occur to me that a NERC entity would identify some of the higher-risk hardware components of say a server, and decide the risk posed by each component. But it’s possible to identify third party hardware components, since you can see them in a server or workstation, and hopefully identify their suppliers as well.

Once you’ve identified important third-party hardware components, you can reach out to their suppliers and ask to be notified of new patches to software or firmware, as well as other defects or unpatched vulnerabilities. And again, you should inform your supplier of the components that you consider most important, putting them on notice (perhaps through contract language, but perhaps through other means as well. Of course, contracts aren’t renegotiated every time a new vulnerability is identified) that you expect them to quickly address any issues that arise in these components.

When I started this post, I was thinking I could discuss all five or six questions that Lew answers in his article in one post. Looks like I was wrong about that. The remaining questions (most of them having to do with what is in scope for CIP-013 as of 7/1/20, and what isn’t) won’t require nearly as much time as this one, so I hope to discuss all of them in my next post (assuming the Russians or the cloud don’t kidnap me, as they’ve been doing recently).

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. My offer of a free webinar on CIP-013, specifically for your organization, has received a great response, and remains open to NERC entities and vendors of hardware or software components of BES Cyber Systems. To discuss this, you can email me at the same address.