The two most important supply chain cybersecurity risks

If you're looking for my pandemic posts, go here.

I’ve been saying for quite a long time that the biggest problem with CIP-013-1 is that it doesn’t provide you with guidance on the risks that should be addressed in your supply chain cybersecurity risk management plan, which is required by R1. Along with my clients, I’ve identified about 110 supply chain cybersecurity risks. I think every NERC entity, in their CIP-013 R1.1 plan, should consider a wide range of risks, then identify the most important of those to mitigate.

Here are what I consider the two most important supply chain cybersecurity risks to the Bulk Electric System. They both are due to a software supplier’s mismanagement of risks due to third-party or open source components of their products. I’ve been writing about software component risks a lot lately. As I mentioned in this recent post, about 80% of Web applications include an average of 73 third-party or open source components, yet half of the suppliers of those applications don’t apply security patches for those components.

My number one risk is that a software supplier won’t patch vulnerabilities in third-party or open source components of its software, even when a patch is available. Of course in some cases, they may have a legitimate reason for doing this, since sometimes the supplier has only implemented part of a software component and the vulnerability is in a different part, meaning applying the patch will do no good.

However, this explanation certainly doesn’t account for anywhere near the number of component patches that are never applied. The two more important reasons are:

a)      The supplier doesn’t even know a patch is available for a vulnerability in a component because they’re not proactively keeping in touch with the third-party supplier or following the open source community that wrote the component and is providing patches. Of course, you wouldn’t dream of not regularly reaching out to your software suppliers every 35 days (as required by CIP-007 R2.2) to find out if they have a new security patch available for the products that you have installed. Why shouldn’t your software suppliers do the same with their software suppliers?

b)     The supplier has lost track of the components of its own software, meaning the supplier no longer knows what is in its own software. Of course, this is an even more serious problem, since it means they’ll probably never know of vulnerabilities in the components that comprise up to 80% of the code in its software. If the supplier can’t give you any list at all of the components in its software, you need to start asking why you’re still buying that supplier’s products.

What’s the mitigation for this risk? A software bill of materials. For every software package that plays an important role in your BES environment (which might be just a few, of course, depending on the size of your BES footprint), you should ask the supplier for a first level software bill of materials (SBoM) – that is, a list of the components in their software, but not necessarily a list of components of those components, even though these also pose a risk (as do components of components of components, etc). For a discussion of how not having good SBoMs can play havoc with your effort – and that of just about every other organization with IoT or IIoT systems - to mitigate risks due to vulnerabilities in third-party software components, see this post).

But what steps can you take to mitigate these risks, since only the supplier knows what components are in their software? This is where the idea of a software bill of materials (SBoM) comes in. You should ask every software supplier to provide you at least a one-level SBoM (i.e. a list of the components in their software, but not necessarily of components of those components). And as I described in my most recent post, just asking the question lets the supplier know you’re concerned about this problem, and makes it more likely they’ll take it seriously, if they weren’t already.

There are two possibilities for the supplier’s response: They can give you at least a partial SBoM or they can give you nothing at all. If they give you some list, even if incomplete, you can try to get in contact with the third-party component suppliers on that list, as well as the open source communities in the case of open source components – and simply request to be on their mailing list for notices of new vulnerabilities and patches.

When you learn of a new patch for a component that is on the SBoM you received, you can contact your supplier and ask when they’ll apply the patch. If they say they can’t apply the patch for some technical reason, you should ask what mitigation they will apply, especially if the vulnerability the patch addresses is a serious one (after all, you have to develop a mitigation plan if you can’t apply a patch, per CIP-007 R2.3! They should as well). And in the case of a serious vulnerability, you should follow up to make sure they do something about it.

My second most important risk also has to do with software components, and also requires an SBoM for mitigation. This is the risk that a supplier will, inadvertently or otherwise, allow a third-party or open source component in their software to remain there after it has gone end of life, so that newly discovered vulnerabilities will not be patched. Again, there are two main reasons why this could happen:

a)      Your supplier hasn’t been in touch with the third party or open source community that wrote a component and didn’t realize they’d either discontinued support for that product or gone out of business.

b)     Your supplier did know this but decided not to do anything about it. They were hoping you wouldn’t notice this either, or even better that you wouldn’t ever ask to see an SBoM so you could track this information yourself.

Of course, without an SBoM you have no way of mitigating this risk. This is another reason to ask for one from any important software supplier. Whenever you learn a component supplier on this list has discontinued a component that’s in the software you have deployed, or if you learn that supplier might soon cease operations, you need to have a frank talk with your supplier to discuss how they will address this new risk. Will they

1.      Replace or upgrade the component?

2.      Write their own code to replace the component? After all, it’s their software – they should certainly be able to do that.

3.      Whenever a vulnerability is discovered in the component from now on, will your supplier write the patch themselves? Of course, they can only do this if they possess the source code for the component. Ideally, they would do what SEL does, namely – whenever they incorporate a third-party component into their software – obtain the source code from the supplier up front. But even if your supplier doesn’t do that, if the component supplier is going out of business or discontinuing support, your supplier should be able to buy the source code from them at that point (of course, for open source components this isn’t a problem, since the source code is available for free).

4.      Will your supplier watch the situation carefully, and if an important vulnerability is discovered in the component – for which there will be no patch – will they immediately develop a workaround so that you and other customers will be protected? This isn’t the best solution, but it’s better than nothing. You should still press them to replace the component, though.

5.      Will they do nothing at all? If they really say this, then you need to sit down with your lawyers and examine your options.

As you’re perhaps beginning to see, a software bill of materials is a very powerful tool. While you won’t always be able to get it, you should at least make the effort – with your most important software suppliers.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Are you hot at work – or should be – on getting ready for CIP-013-1 compliance on October 1? Here is my summary of what you need to do between now and then.