Is there such a thing as a product risk?

Note from Tom: If you’re only looking for today’s pandemic post, please go to my Pandemic Blog. If you’re looking for my cyber/NERC CIP post, you’ve come to the right place.

I’m currently working on a book on CIP-013 with Steven Briggs, a cybersecurity and compliance professional who works for a large electric utility and has been very involved with CIP-013 since the drafting phase. One of the issues that we’ve debated (and not necessarily resolved) – and which I’ve debated with others as well – is whether there are true product risks. Or are there only supplier risks, some of which apply to products?

At first, this seems like an easy question to answer: Of course there are product risks! Some products come with insecure configurations, such as a backdoor intentionally planted in the code to make the developers’ lives a little easier, which the supplier never removed before the product was shipped to you. Others come with unpatched software or firmware vulnerabilities, possibly because the product was sitting in a warehouse for a couple months before it was shipped and the latest patches weren’t applied before shipment. There are many other possibilities as well.

But what’s the best way to mitigate these risks? Both of the risks just mentioned are primarily due to poor practices on the part of the software supplier. The best mitigation for these risks is to get the supplier to implement proper procedures (such as applying the latest patches before shipping any product) so that incidents like these never happen. If the best way to mitigate a risk is to get the supplier to change their policies or procedures, it’s hard to imagine these are anything but supplier risks.

But here’s a risk that calls that conclusion into question: the risk that a hardware product will allow firmware upgrades and patches to be applied remotely without any authentication. This vulnerability was exploited by the Russians to “brick” RTUs in the first Ukraine attacks in 2015, so it’s definitely a real risk. However, it’s been five years since those attacks and many (most?) control systems purchased for field use (especially in substations) still don’t require authentication for firmware upgrades.

Is this because the suppliers of these devices are nefarious criminals who just care for the almighty buck, not for the security of their customers, let alone the Bulk Electric System? No. It’s hard to imagine that anyone is buying these products nowadays without being fully aware that this vulnerability is present in them, and we hope that suppliers aren’t hiding that fact. There are probably three reasons why nothing much has changed since the Russian attacks:

1. There are lots of legacy devices in the field that have this vulnerability. Since replacing them would be very expensive, especially if it would require changing suppliers, there’s not much incentive to replace them before they fail or become obsolete for other reasons.
2. As with anything that users would like to have, there’s a cost to it. A lot of control system suppliers might not be interested in being the first kid on the block to implement this, since it might make them non-competitive.
3. Probably the most important reason for non-action is that Medium impact substations all have good external security now, due to having to comply with CIP-005-5 R1 (Electronic Security Perimeters); it’s hard to imagine how an external attacker would be able even to ping the device in the first place. Of course, this doesn’t apply to Low impact substations, although with CIP-003-8 in place now, you might also consider that a mitigation. 

So is this a supplier or a product risk? Let’s say the supplier now designs authentication (or digital signature verification) of firmware upgrades into all new devices. Since their customers know that the supplier’s legacy products still have this vulnerability, while at the same time all future products won’t have it, the supplier has done all they can. If the user can live without authentication, then they can keep buying the legacy products. If they have to have authentication, they can either buy new products or else change suppliers (and I don’t know whether any suppliers are offering authentication even now). It’s up to the utility to mitigate this risk from here on out, at least as far as procurements from this supplier are concerned.

Does that make this a product risk? No, it’s still a supplier risk, but it turns out that the primary mitigation – having the supplier bake authentication of updates into the firmware – can’t be applied in this case, so you need to mitigate this risk through your own measures. This is why there always needs to be at least one “backup” mitigation for each supplier/vendor risk – i.e. a mitigation that your organization takes on its own (the NERC FAQ on CIP-013-1 made this point very clearly). As mentioned above, if this is a Medium impact substation the risk is probably already mitigated, due to the fact that your organization is compliant with CIP-005-5 R1 (although this needs to be documented).

But there’s one other consideration here: People have stated to me that the fact that product X doesn’t offer a particular capability – say it doesn’t allow use of more than two types of special characters in passwords – constitutes a product risk. No, it isn’t a risk. It’s a feature, or more accurately a lack of one. For example, if you were designing a public kiosk, you wouldn’t consider it a risk if the product didn’t require passwords at all, let alone if it didn’t require complex ones. But if this were a sensitive military system, I’m sure it would have to have all sorts of capabilities that aren’t available at all in the commercial market.

Whenever any organization buys any product of any type (even pens), they should always inquire about the features they’re interested in having. And if the feature isn’t there, they shouldn’t buy it. Organizations have been doing this since…perhaps the founding of Rome.

So if you didn’t inquire about password complexity up front when it was a feature you absolutely needed, shame on you if you’re blindsided when you discover it’s not there. If you get a product that doesn’t allow complex passwords and you wanted to have them, you haven’t been a victim of a supply chain cyberattack – you’ve just been stupid (of course, stupidity itself is a risk, and if you believe there’s an abundance of stupidity in your organization, you need to consider this risk in your Procurement Risk Assessments – and take steps to mitigate it, like only allowing smart people to be involved in the procurement).

Now, there is a risk that a supplier, after having promised that an important feature will be included in a product you are going to purchase, doesn’t in fact include it. This is a supply chain cybersecurity risk. The best mitigation for it would be to have a term in the contract (if there will be a contract for this Procurement) stating that the feature will be included, and if it’s not included you have the right to return the product and maybe even charge the supplier a penalty.

In fact, I can’t think of any “product risk” that isn’t actually a question about whether a particular feature is present. And this is why I believe that all real risks having to do with products are actually supplier risks; but I’d certainly like to hear from you if you disagree with me.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Are you working on your CIP-013 plan and you would like some help on it? Or would you like me to review what you’ve written so far and let you know what could be improved? Just drop me an email!