More on “product” risks

Note from Tom: If you’re only looking for today’s pandemic post, please go to my Pandemic Blog. If you’re looking for my cyber/NERC CIP post, you’ve come to the right place.

Today I received a call from a friend I’ve known for a while, an engineer with a major supplier of electronic devices to the electric power industry. The call was about yesterday’s post, in which I explained why I didn’t think there was such a thing as a product risk – just supplier risks that can be realized in their products (although I didn’t use exactly those words).

He brought up to me two cases having to do with third-party software that this supplier either incorporates into its products or provides with them. In both cases the third-party products contained unpatched vulnerabilities; he was wondering if these might be examples of true product risks.

The first case was in a protocol stack that the supplier licenses from a third party. It seems there were a few unpatched vulnerabilities in the software. Because the supplier hadn’t implemented the parts of the stack that included those vulnerabilities, they felt they had no obligation to try to incorporate their own mitigations for those vulnerabilities; moreover, they weren’t obligated to inform their customers of them. Did this situation constitute a product risk?

The second case was of a particular piece of third-party software that the supplier provided to their customers to use for collaboration with the supplier (so it wasn’t incorporated into one of their products, as in the first case). There was a vulnerability in this software that might have allowed an insider to gain control and do bad things. Despite this vulnerability, the supplier felt this product was still worth using, and they made a point of telling their customers about the vulnerability and how to mitigate it. Was this a product risk?

As we discussed these two cases, I pointed out the difference between a vulnerability and a risk (in my way of thinking, of course). Software and firmware vulnerabilities occur all the time, but in my opinion they don’t constitute an actual risk unless the supplier a) doesn’t develop a patch in a timely manner, and b) doesn’t inform their customers and work with them to mitigate those vulnerabilities (until a patch is developed, which is of course the best mitigation for a vulnerability).  

In both of these cases, there is a risk, but it’s a supplier risk. In the first case, the risk is that a vulnerability in third party software – which is incorporated into a product that the supplier sells for use with BES Cyber Systems – won’t be patched by the supplier (although in most cases the third party would have to develop the patch, and the supplier would just need to apply it or distribute it, e.g. in a new version of a software library). But I pointed out that, since the supplier wasn’t using the portions of the third party code that contained the vulnerabilities, that code wasn’t part of their product at all – and therefore this risk doesn’t even apply in this case.

In the second case, the risk is that a supplier will distribute third-party software that has an unpatched vulnerability that they know about but can’t patch (in this case, it’s because this software wasn’t part of one of their products – so they have no access to the code to develop a patch. And the third-party supplier wasn’t providing their own patch, for some reason), yet they also don’t inform their customers of the vulnerability, or how to mitigate it. Had this supplier not informed their customers of this vulnerability, it would have posed a serious risk for their customers. But of course this supplier did tell their customers and did suggest mitigating steps they could take.

So was there a product risk in either case? In the first case there was no risk at all, since the vulnerabilities were in code that wasn’t incorporated in the supplier’s products. In the second case, there was a risk, but it was a supplier risk, not a product risk. And this supplier mitigated the risk as they should have done.

So I continue to believe there are no product risks, just supplier or vendor risks. But I could be wrong – I’d love to hear if you have any counterexamples.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Are you working on your CIP-013 plan and you would like some help on it? Or would you like me to review what you’ve written so far and let you know what could be improved? Just drop me an email!

Is there such a thing as a product risk?

Note from Tom: If you’re only looking for today’s pandemic post, please go to my Pandemic Blog. If you’re looking for my cyber/NERC CIP post, you’ve come to the right place.

I’m currently working on a book on CIP-013 with Steven Briggs, a cybersecurity and compliance professional who works for a large electric utility and has been very involved with CIP-013 since the drafting phase. One of the issues that we’ve debated (and not necessarily resolved) – and which I’ve debated with others as well – is whether there are true product risks. Or are there only supplier risks, some of which apply to products?

At first, this seems like an easy question to answer: Of course there are product risks! Some products come with insecure configurations, such as a backdoor intentionally planted in the code to make the developers’ lives a little easier, which the supplier never removed before the product was shipped to you. Others come with unpatched software or firmware vulnerabilities, possibly because the product was sitting in a warehouse for a couple months before it was shipped and the latest patches weren’t applied before shipment. There are many other possibilities as well.

But what’s the best way to mitigate these risks? Both of the risks just mentioned are primarily due to poor practices on the part of the software supplier. The best mitigation for these risks is to get the supplier to implement proper procedures (such as applying the latest patches before shipping any product) so that incidents like these never happen. If the best way to mitigate a risk is to get the supplier to change their policies or procedures, it’s hard to imagine these are anything but supplier risks.

But here’s a risk that calls that conclusion into question: the risk that a hardware product will allow firmware upgrades and patches to be applied remotely without any authentication. This vulnerability was exploited by the Russians to “brick” RTUs in the first Ukraine attacks in 2015, so it’s definitely a real risk. However, it’s been five years since those attacks and many (most?) control systems purchased for field use (especially in substations) still don’t require authentication for firmware upgrades.

Is this because the suppliers of these devices are nefarious criminals who just care for the almighty buck, not for the security of their customers, let alone the Bulk Electric System? No. It’s hard to imagine that anyone is buying these products nowadays without being fully aware that this vulnerability is present in them, and we hope that suppliers aren’t hiding that fact. There are probably three reasons why nothing much has changed since the Russian attacks:

1. There are lots of legacy devices in the field that have this vulnerability. Since replacing them would be very expensive, especially if it would require changing suppliers, there’s not much incentive to replace them before they fail or become obsolete for other reasons.
2. As with anything that users would like to have, there’s a cost to it. A lot of control system suppliers might not be interested in being the first kid on the block to implement this, since it might make them non-competitive.
3. Probably the most important reason for non-action is that Medium impact substations all have good external security now, due to having to comply with CIP-005-5 R1 (Electronic Security Perimeters); it’s hard to imagine how an external attacker would be able even to ping the device in the first place. Of course, this doesn’t apply to Low impact substations, although with CIP-003-8 in place now, you might also consider that a mitigation. 

So is this a supplier or a product risk? Let’s say the supplier now designs authentication (or digital signature verification) of firmware upgrades into all new devices. Since their customers know that the supplier’s legacy products still have this vulnerability, while at the same time all future products won’t have it, the supplier has done all they can. If the user can live without authentication, then they can keep buying the legacy products. If they have to have authentication, they can either buy new products or else change suppliers (and I don’t know whether any suppliers are offering authentication even now). It’s up to the utility to mitigate this risk from here on out, at least as far as procurements from this supplier are concerned.

Does that make this a product risk? No, it’s still a supplier risk, but it turns out that the primary mitigation – having the supplier bake authentication of updates into the firmware – can’t be applied in this case, so you need to mitigate this risk through your own measures. This is why there always needs to be at least one “backup” mitigation for each supplier/vendor risk – i.e. a mitigation that your organization takes on its own (the NERC FAQ on CIP-013-1 made this point very clearly). As mentioned above, if this is a Medium impact substation the risk is probably already mitigated, due to the fact that your organization is compliant with CIP-005-5 R1 (although this needs to be documented).

But there’s one other consideration here: People have stated to me that the fact that product X doesn’t offer a particular capability – say it doesn’t allow use of more than two types of special characters in passwords – constitutes a product risk. No, it isn’t a risk. It’s a feature, or more accurately a lack of one. For example, if you were designing a public kiosk, you wouldn’t consider it a risk if the product didn’t require passwords at all, let alone if it didn’t require complex ones. But if this were a sensitive military system, I’m sure it would have to have all sorts of capabilities that aren’t available at all in the commercial market.

Whenever any organization buys any product of any type (even pens), they should always inquire about the features they’re interested in having. And if the feature isn’t there, they shouldn’t buy it. Organizations have been doing this since…perhaps the founding of Rome.

So if you didn’t inquire about password complexity up front when it was a feature you absolutely needed, shame on you if you’re blindsided when you discover it’s not there. If you get a product that doesn’t allow complex passwords and you wanted to have them, you haven’t been a victim of a supply chain cyberattack – you’ve just been stupid (of course, stupidity itself is a risk, and if you believe there’s an abundance of stupidity in your organization, you need to consider this risk in your Procurement Risk Assessments – and take steps to mitigate it, like only allowing smart people to be involved in the procurement).

Now, there is a risk that a supplier, after having promised that an important feature will be included in a product you are going to purchase, doesn’t in fact include it. This is a supply chain cybersecurity risk. The best mitigation for it would be to have a term in the contract (if there will be a contract for this Procurement) stating that the feature will be included, and if it’s not included you have the right to return the product and maybe even charge the supplier a penalty.

In fact, I can’t think of any “product risk” that isn’t actually a question about whether a particular feature is present. And this is why I believe that all real risks having to do with products are actually supplier risks; but I’d certainly like to hear from you if you disagree with me.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Are you working on your CIP-013 plan and you would like some help on it? Or would you like me to review what you’ve written so far and let you know what could be improved? Just drop me an email!

A dispatch from the front lines of the Covid-19 crisis

Note from Tom: Jude Gamel, a recently retired critical care nurse who lives in Kentucky, started corresponding with me a few weeks ago about my pandemic blog. We hadn’t discussed his own experiences (and those of his former colleagues, whom he keeps in touch with regularly) treating Covid-19 patients previously. But yesterday, he was inspired by that morning’s post, which focused entirely on deaths from Covid-19, to describe, in two emails, what “successful” outcomes are – i.e. when the patient survives. In many cases, the patient might well wonder whether the physical, emotional and financial toll on them were actually better than dying. And there’s also a huge physical and emotional toll on the health care workers. His account is gripping.

Jude’s first email
I just finished reading you post for today. I would like to add or at least to comment on something that too often gets lost or goes missing in conversations about Covid-19 and I think it is particularly relevant to what we see happening now: there are some outcomes that are almost as bad as dying and Covid-19 is a perfect example. Let me see if I can articulate this properly in order to make this clear.

Take for instance ICU beds. Most patients admitted to an ICU are admitted because they require specialized care and monitoring that cannot be done on another unit or floor. It is usually just to get a patient through a critical point. A clear example of this is a patient recovering from Open Heart Surgery. They generally spend 24-48 hours in a Cardiovascular Intensive Care Unit and are then discharged to a Step-Down Unit and then home or to a re-hab facility for recovery (roughly 7-10 days). A patient who has a Myocardial Infarction (Heart Attack) goes to a Catheterization Lab and gets stented and spends, again, 24 hours or so in an ICU (if even that). Now more complicated situations like Sepsis, Diabetic Ketoacidosis, COPD Exacerbation or a bevy of other diseases both chronic and acute result in longer stays but rarely more than 5 - 7 days. Now remember (I know this should be self evident) once a patient is admitted to an ICU bed you cannot admit another patient to that bed until that patient is discharged or dies (a celestial discharge).

Critically ill Covid-19 patients occupy ICU beds for WEEKS and require incredible resources to keep them alive. This is a drag on the system and the morale of physicians, nurses, respiratory therapists and ancillary staff. The care these patients require and the steps healthcare workers must take to protect themselves are incomprehensible to the average person. If you have not done it or experienced it you just do not know. Once these patients are finally well enough to be discharged from the ICU they spend weeks on Step-Down Units or Floors and then spend even more time in Re-hab. The costs incurred by their families are insurmountable; only the very richest in our society can manage even with gold-plated health insurance policies! This is why I say ‘worse than death.’ Many of those we ’save’ would have foregone the saving if they knew what it would do to their family’s financial future and after all the suffering they endure. It is sad.

Well, Tom, I could go on but the point I am trying to make is that we need to make distinctions about this disease that talking about Infection and Mortality just do not get at. Covid-19 is a devastating disease and you do not want to become infected. Yes, most cases are mild (thankfully) and some people may not even know they were infected. But new research is now indicating that even Asymptomatic people may have unrecognized sequelae from the disease whose after effects may not be known for years.

Jude’s second email

Let me give you more of a flavor of what I am talking about. Critically ill Covid-19 patients require 24/7 one on one nursing care. If they are on a ventilator they require frequent ’suctioning’ of their breathing tube, if their kidneys have shut down they are on bedside dialysis (CRRT…Continuous Renal Replacement Therapy), they require turning side to side every 2 hours minimum to prevent skin breakdown, they are on multiple vasoactive IV medications running through continuous pumps, they have a feeding tube threaded through their nose into their small bowel and are receiving nutrition via the tube, they have large bore IV lines in their neck, the feedings lead to diarrhea so you are constantly dealing with loose stool until you put a containment device in their rectum…it goes on and on, so I am sure you get the picture. Now, remember in order for the nurse managing this patient to protect themselves they are wearing an N-95 mask or a helmet like CAPR device, they are wearing a plastic gown, they have a face shield on, they have booties on their shoes, the rooms are hot, they cannot leave the room sometimes for hours and when they do they are soaked in sweat head to toe and they do this for 12-14 hours day in and day out.

Now, if this patient is on ECMO (Extracorporeal Membrane Oxygenation) [think of it as ‘bedside’ cardiopulmonary bypass], then they require two on one nursing care!

Oh, and I forgot to mention, the patient is a 5 ft 5 in woman who weighs 275 lbs or a 6 ft 5 in man who weighs 350 lbs. Their girth extends side to side in the bed and requires at least two sometimes 3 or 4 people to turn them…and, God forbid you are ‘proning’ the patient; that might require 6 or 7 people.

Now, imagine doing that for the foreseeable future. It is grinding, overwhelming work that will take you to your knees. That is what is coming to the nurses at hospitals in FL, TX, AZ.

For these patients, well in order to keep them comfortable some of them require continuous Neuromuscular Blockade (chemical paralysis), continuous pain and sedation medications for weeks. Some of them do not wake up for days when you try to wean them from the ventilator. The neuromuscular blockade leads to something called ‘critical illness polyneuropathy’ which means they initially have little motor movement or strength and take a long time to just sit up much less raise their arms to feed themselves or walk.

And then you have the bills…

So, yes, I do believe there are some things worse than dying. When we put all of this in that box that says ‘Infected’ or ‘Recovered’ or ‘Died’ this is the type of thing you are talking about. 

Covid-19 is a horrid disease and you want to do everything you can to avoid it because you do not know if you have that chink in armor that lets it take you down. 

New cases jumped from 41,000 to 49,000 yesterday. Given that there were four days in mid-June during which new cases were under 25,000, this is a stunning increase. Unfortunately, this probably isn’t the end of it. My biggest question is when and how much it will increase the deaths numbers. Yesterday, the 7-day average of new deaths jumped to 5%, and that didn’t change today. But if we have more deaths numbers soon that come anywhere near 2,500 (yesterday’s level), then it’s inevitable deaths will increase at an even greater rate soon. This is something to watch closely.

The numbers

These numbers are updated every day, based on reported US Covid-19 deaths the day before (taken from the Worldometers.info site, where I’ve been getting my numbers all along). No other variables go into the projected numbers – they are all projections based on yesterday’s 7-day rate of increase in total Covid-19 deaths, which was 5%.

Note that the “accuracy” of the projected numbers diminishes greatly after 3-4 weeks. This is because, up until 3-4 weeks, deaths could in theory be predicted very accurately, if one knew the real number of cases. In other words, the people who are going to die in the next 3-4 weeks of Covid-19 are already sick with the disease, even though they may not know it yet. But this means that the trend in deaths should be some indicator of the level of infection 3-4 weeks previous.

However, once we get beyond 3-4 weeks, deaths become more and more dependent on policies and practices that are put in place – or removed, as is more the case nowadays - after today (as well as other factors like the widespread availability of an effective treatment, if not a real “cure”). Yet I still think there’s value in just trending out the current rate of increase in deaths, since it gives some indication of what will happen in the near term if there are no significant intervening changes.

Week ending	Deaths reported during week/month	Avg. deaths per day during week/month	Deaths as percentage of previous month’s
March 7	18	3
March 14	38	5
March 21	244	35
March 28	1,928	275
Month of March	4,058	131
April 4	6,225	889
April 11	12,126	1,732
April 18	18,434	2,633
April 25	15,251	2,179
Month of April	59,812	1,994	1,474%
May 2	13,183	1,883
May 9	12,592	1,799
May 16	10,073	1,439
May 23	8,570	1,224
May 30	6,874	982
Month of May	42,327	1,365	71%
June 6	6,544	935
June 13	5,427	775
June 20	4,457	637
June 27	6,272	896
Month of June	23,626	788	56%
July 4	6,594	942
July 11	6,933	990
July 18	7,290	1,041
July 25	7,664	1.095
Month of July	34,191	1,103	145%
Total March – July	164,015

Red = projected numbers

I. Total deaths

Total US deaths as of yesterday: 127,649

Increase in deaths since previous day: 864

Yesterday’s 7-day rate of increase in total deaths: 5% (This number is used to project deaths in the table above; it was 5% yesterday. There is a 7-day cycle in the reported deaths numbers, caused by lack of reporting over the weekends from closed state offices. So this is the only reliable indicator of a trend in deaths, not the three-day percent increase I used to focus on, and certainly not the one-day percent increase, which mainly reflects where we are in the 7-day cycle).

II. Total reported cases

Total US reported cases: 2,504,676

Increase in reported cases since previous day: 49,010

Percent increase in reported cases since yesterday: 2%

Percent increase in reported cases since 7 days previous: 11%

III. Deaths as a percentage of closed cases so far in the US:

Total Recoveries in US as of yesterday: 1,068,768

Total Deaths as of yesterday: 127,649

Deaths so far as percentage of closed cases (=deaths + recoveries): 11% (vs. 11% yesterday)

For a discussion of what this number means – and why it’s so important – see this post.

I would love to hear any comments or questions you have on this post. Drop me an email at tom@tomalrich.com

EEI’s second try at procurement language

Note from Tom: If you’re only looking for today’s pandemic post, please go to my Pandemic Blog. If you’re looking for my cyber/NERC CIP post, you’ve come to the right place.

Last fall, I wrote a post that pointed out something I’d recently realized: In contrast to the other CIP standards, the risk-based approach of CIP-013 provides the NERC entity complete freedom to do something they would have a hard time doing while complying with the other CIP standards – waste money.

I said this because the requirements in the other CIP standards are almost all prescriptive – they tell you what to do; you have to do it, period. There’s officially no consideration of what you have to spend to comply with the requirement. You need to spend whatever time and money is required for your organization to comply.

On the other hand, CIP-013 doesn’t tell you what specific actions you need to perform. It simply says you need to a) develop a plan to “identify” cyber risks to the BES arising from your supply chain; b) “assess” those risks to identify the most important ones; and c) mitigate those risks (although the word mitigate was left out of the requirement, there’s no question the standard should be read as requiring mitigation. Just look at the Purpose statement at the beginning of CIP-013-1). You do need to address the six risks listed in R1.2.1 through R1.2.6, but you have a lot of freedom in how you do even that.

This means that the scope of the work you need to do for compliance with CIP-013 is very much under your control. In the post, I listed three ways in which it would be possible for you to waste a lot of money on CIP-013 compliance. The last was the most important, and I’ll rephrase it now to say: You can waste money by inadvertently requiring your organization to mitigate risks that you never identified in the first place.

I went on in that post, and the follow-up post, to point out that the biggest problem with the then-current version of the EEI language was that it threw in so much other language that doesn’t address the six risks in R1.2.1 through R1.2.6 (and there are really eight risks, since R1.2.5 and R1.2.6 address two risks each). So it’s essentially adding a bunch of risks to the ones you’ve decided to mitigate in your CIP-013-1 plan.

As I stated above, the first step in CIP-013 compliance is to identify BES supply chain cyber security risks and identify the most important ones for mitigation. The eight risks in R1.2 have kind of a special status: They need to be mitigated, regardless of whether or not you think they’re important (of course, they are all important risks, so this shouldn’t be a big issue for anybody). But the others you identify are up to you, although R1.1 lists (or implies) five types of risk that should all be addressed in the plan: risks from hardware or software procurement, hardware or software installation, procurement of services, use of services, and transitions between vendors.

But it’s very important to keep your list of risks under control; what you don’t want to do is inadvertently add to that list, because for every risk on the list, you will need to show that you a) took steps to assess the likelihood that the risk is present in the vendor’s environment, and b) took steps to mitigate the risk (which might include contract language, although that’s never sufficient in itself). I see most of the compliance risk in CIP-013-1 occurring as entities comply with R2: Whatever plan you drew up in R1 (and you have a lot of freedom to do that), you need to implement it exactly in R2. If you have said you’ll mitigate a particular risk, you have to both assess your vendors for that risk and take steps to get them to mitigate it if the assessment indicates they haven’t already done so.

So how could you “inadvertently” add a risk to your list of risks? One way to do that is to ask your vendors a questionnaire question that goes beyond the list of risks that you’ve identified. As I recently pointed out in one of my posts on the NATF questionnaire, asking questions that don’t correspond to risks you identified in your list means you are inadvertently adding those risks to your list. If a vendor’s answer indicates they haven’t mitigated one of those risks, you need to follow up with the vendor to get them to mitigate it. And if you don’t do that, you may be found non-compliant with R2.

The same thing applies to contract language. In fact, I think you need to have a one-to-one relationship between each risk and one contract term, just as you should have a one-to-one relationship between each risk and one questionnaire question. You shouldn’t require contract terms that don’t address risks on your list, just like you shouldn’t ask questions that don’t address risks on your list.

The EEI language is supposed to address the risks in R1.2.1 through R1.2.6. In my posts on the original EEI document last year, I pointed to a number of examples where they included language that goes beyond the eight risks in R1.2. I reminded my readers (as an attorney for an IOU had reminded me) that each “extra” term a) requires a lot of additional legal effort, since the vendor is almost always guaranteed to want to negotiate any contract term that you require of them; b) requires you to take steps to verify the vendor is in fact complying with the term; and c) puts you at compliance risk if you don’t follow up to get the vendor to comply with any term that they haven’t complied with so far.

I was hoping that EEI’s new version 2.0 would fix these problems, but I regret to say they’re still there. Let’s just go through what just one paragraph requires. Paragraph (a) under R1.2.5 on page 10 requires the vendor to address the following risks that have nothing to do with R1.2.5:

a)      chain of custody practices,

b)     inventory management program,

c)      information protection practices (and since it doesn’t specify what information, the vendor needs to document their entire program for protecting all of their information),

d)     integrity management program for components provided by sub-suppliers, and

e)     “instructions on how to request replacement parts, and commitments to ensure that for [negotiated time period] spare parts shall be made available by Contractor”.

The point is that, by including just this one paragraph in your required contract language, you’re committing your organization to consider each of these five items to be a risk that must be mitigated, just like the risks you explicitly identified in R1.1. Of course, maybe you think these all are important risks and you understand what you’re committing yourself to – fine, that’s great. But be sure to add them to your overall list of R1.1 risks, so you can track their mitigation, and so that you can get “credit”, come audit time for all of the risks you’ve mitigated.

And the same thing goes for any of the other EEI terms that go beyond what’s required by R1.2.1-R1.2.6: If you include these in contracts, you need to be prepared to address them the same way you address all of your other identified risks (including the eight R1.2 risks themselves). I haven’t even tried to identify them all, but I would think it’s in the hundreds, especially in cases where the language requires a vendor to “follow” particular sections of for example NIST 800-53 – just one of those probably includes 10-20 separate risks that need to be mitigated.

Of course, if you have lots of time and lots of money available for CIP-013 compliance (and you all do, I’m sure…), you’ll at least know what you’re committing to. But if you don’t, you may want to rethink the idea of implementing the whole of EEI’s latest contract language document. In general, I think their language that just applies to the R1.2 requirements is good. But you need to be very careful in using the other language in the document.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Are you working on your CIP-013 plan and you would like some help on it? Or would you like me to review what you’ve written so far and let you know what could be improved? Just drop me an email!