This is (probably not) my last post on product risks!

In this post last Tuesday, I reported on a call I’d received from a friend who works as an engineer at an important supplier of hardware products to the electric power industry. He’d called me after my Monday post made the point that I don’t believe there are true product risks, just vendor risks that apply to products.

My friend brought up a case that his company encountered. To quote my post:

There was a particular piece of third-party software that the supplier (i.e. my friend’s company) provided to their customers to use for collaboration with the supplier (so it wasn’t incorporated into one of their products). There was a vulnerability in this software that might have allowed an insider to gain control and do bad things. Despite this vulnerability, the supplier felt this product was still worth using, and they made a point of telling their customers about the vulnerability and how to mitigate it. Was this a product risk?...

I pointed out the difference between a vulnerability and a risk (in my way of thinking, of course). Software and firmware vulnerabilities occur all the time, but in my opinion they don’t constitute an actual risk unless the supplier a) doesn’t develop a patch in a timely manner, and b) doesn’t inform their customers and work with them to mitigate those vulnerabilities until a patch is developed, which is of course the best mitigation for a vulnerability…

…the risk is that a supplier will distribute third-party software that has an unpatched vulnerability that they know about but can’t patch (in this case, it’s because this software wasn’t part of one of their products – so they have no access to the code to develop a patch. And the third-party supplier wasn’t providing their own patch, for some reason), yet they also don’t inform their customers of the vulnerability, or how to mitigate it. Had this supplier not informed their customers of this vulnerability, it would have posed a serious risk for their customers. But of course this supplier did tell their customers and did suggest mitigating steps they could take… this supplier mitigated the risk as they should have done.

The next day, I got an email from Kevin Perry, bringing up an example he thought was a product risk:

…you cite the supplier as a risk because it knew there was a vulnerability and chose to tell their customers.  The key part being the supplier knew about the vulnerability.  Whether it chooses to inform simply defines the risk.  But what if it is a zero-day vulnerability, either as a result of an unknown software vulnerability or a hardware issue?  No one knows the vulnerability exists until one day you get taken down.  Does that change anything?  I look at this with the side channel vulnerability that exists in almost all modern computer processors.  Yes, the feature was designed into the CPU for processor speed.  The unintended consequence is that a well crafted exploit can take advantage of the feature to cause an exception and access the contents of the memory look-ahead cache before the exception handler clears the cache.  It is possible to read memory outside of your process boundaries.  Is this a supplier risk because you bought your computer from Dell or HP and it came with an Intel or AMD processor?  Is it Intel’s or AMD’s “supplier” risk?  Or is it truly a product risk.

Yes, once identified, there have been work-arounds developed.  And as soon as a work-around is developed, a new exploit path is found.  In the end, only a complete redesign of the CPU and replacement of any computer containing the old processors will resolve the issue.  I am having problems characterizing this as a supplier risk.  In my mind, a mitigation of a supplier risk could be to change suppliers.  In this instance, there is no supplier that does not have the vulnerability in their processor’s module

In other words, Kevin is pointing to an unpatched (and unpatchable, in this case) vulnerability in a processor as an example of a true product vulnerability. My reply was:

It seems you're actually addressing three separate potential risks. The first is the risk that the supplier will know of an unpatched vulnerability but won't tell you about it. That's what I discussed in the post, and I think you agree with me that this supplier mitigated that risk by telling their users about the vulnerability. But not all suppliers can be counted on to do this! That's why CIP-013 R1.2.4 is there.

The second risk is that there will be a true zero-day hardware or software vulnerability in the product. Of course, the supplier can't tell you about this because they don't know about it either. But I don't call this a risk. Literally every digital hardware or software product in the world could have an unknown vulnerability in it. The only thing a supplier can do is be as vigilant as possible in looking for vulnerabilities (but even if they find a true zero-day vulnerability in their product, they will then know about it and it won't be zero-day any more. So risk no. 1 applies). And the user can be as vigilant as possible in testing the product, etc. 

The third potential risk is that a hardware or software supplier will have a vulnerability in their product, which they can't fix at all - absent a complete redesign, as in the case of the Intel/AMD side channel vulnerability. This seems to me to be just another example of risk 1. The supplier can't patch the vulnerability, but they can at least tell you about it and let you decide whether or not you want to continue using the product. Of course, AMD and Intel have been quite open about this vulnerability, so I'd say they've mitigated the risk, just like the supplier in yesterday's post mitigated the risk by telling their customers about it. The customers then have the choice of either accepting the residual risk or not using the product at all (I'm assuming here that there's no good software mitigation or workaround for this vulnerability). In the case of the supplier in my post, it would be fairly easy not to use the vulnerable software, since it isn't required for the operation of the product itself. In the case of Intel/AMD, it would be very hard to simply move away from that architecture, so my guess is almost all users have formally or informally accepted this risk

At that point, our discussion split into three separate threads, but at least two of them led to the same conclusion as the thread below. Kevin defended the idea that the processor vulnerability actually is a product risk by saying:

Any time I acquire hardware or software, there is a risk that (1) the purchase will not perform as expected or intended, (2) I will not get exactly what I thought I was (product substitution), or (3) there will be failure points or vulnerabilities that I need to address in order to protect my networking and computing environment.  Just because I do not know that a specific vulnerability exists at the time I purchase the product does not mean that risk is absent and that I need to do nothing.  Depending on what the product does and where it is placed in my network will drive the risks that I need to mitigate to the extent possible.

I replied:

This is an installation risk, which is an entity risk. You need to make sure you take into account the risks that apply to the environment in which the product is being installed and apply controls that are appropriate in that environment. The risk is mitigated by training, mentoring, etc. Of course, installation risks are specifically addressed in CIP 13 R1.1.

So I continue to believe: There are no product risks, just supplier risks and entity risks. In my first post, I just said there were supplier risks, but there’s actually another type of supply chain risk – risks posed by your own organization. I didn’t bring entity risks up earlier because I didn’t think they were relevant to this product risk discussion. But the next day (last Wednesday), another friend wrote in, and that led me to realize that entity risks might masquerade as product risks also. I discussed that in this post.

Do I think this is the end of the discussion? Not by a long shot.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Are you working on your CIP-013 plan and you would like some help on it? Or would you like me to review what you’ve written so far and let you know what could be improved? Just drop me an email!