Have we had the first grid cyber attack?

A very interesting story by the always-interesting Blake Sobczak came out in E&E News this morning (and thanks to Blake for forwarding the free link), pointing out that there was an even more interesting OE-417 disturbance report filed with DoE about an incident on March 5. The report concerned a “Cyber event that causes interruptions of electrical system operations”, which is one of the categories of disturbances that needs to be reported using form OE-417.

The event occurred over a ten-hour period in four counties in the Western Interconnect: two in California, one in Wyoming and one in Utah (perhaps significantly, it was Salt Lake County). While there was an interruption of electrical system operations, there was no loss of load.

Of course, this is described as a cyber event, not a cyber attack (in fact, I doubt there is a category for cyber attacks in the OE-417 report). A longtime industry observer pointed out to me that a Control Center temporarily losing SCADA happens frequently. However, it really doesn’t seem to be that, for several reasons:

1. For an event to be happening simultaneously in four separate locations in three non-contiguous states, there would have to be a single common operating entity involved – and the only common entity for those four locations is Peak Reliability, the Reliability Coordinator for 14 Western states (which is based in Salt Lake City). Is it possible they are the entity that lost SCADA?
2. It’s possible, but not likely, since there is a different OE-417 classification for SCADA loss: "Complete loss of monitoring or control capability at its staffed Bulk Electric System control center for 30 continuous minutes or more." In fact, since the beginning of the year, this seems to have happened more than 10 times in the US. But could some other event – not a cyber attack – that occurred in Peak’s systems have caused the incidents in the other three locations?
3. That’s not known, but I do believe that the NERC E-ISAC is still investigating this event – almost two months later. If the event clearly had another cause than a cyber attack, I would guess the E-ISAC would have quickly wound up its investigation. If it’s true that they haven’t, at the least this means there’s still a strong possibility it was a cyber attack.

This – and what was reported in E&E News – is all the information I have now, and I’m not going to speculate about this. However, I hope that whoever investigates this also decides to investigate the Russian penetration of the US grid, reported in the Office of National Intelligence Worldwide Threat Assessment. After all, I’ve heard that at least some of those penetrations were in the West. The investigators might save some air fares by looking for those as well.  

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. To discuss this, you can email me at the same address.

A threat I didn’t know about

Blake Sobczak and Pete Behr of E&E News published a really excellent story today about the large transformer issue. I was expecting the story would focus on the big problems with replacing them – long lead times, difficult transportation, etc. (which wouldn’t be new news, of course). Instead, they focused on the potential problems posed by China becoming a big source of large transformers in the US.

They talk about two general problems. One is counterfeit – or deliberately tampered with – parts that could fail at an important time and cause big problems at key substations. But then they talk about cyber security problems. I never thought transformers could have cyber security problems, but they point out that transformers now all have sensors for maintenance purposes. Generating false readings on a sensor could lead to control centers taking actions they shouldn’t take, or not taking actions they should. If an adversary were able to fry a lot of large transformers at a time, this could cause severe problems for power flows.

So does this mean that the transformers (or perhaps the sensor monitoring systems) are BES Cyber Systems, and the industry should start applying all the CIP protections (including CIP-013) to them? I don’t think so. While these systems would definitely be Cyber Assets, I don’t think they’d be BES Cyber Assets. Yes, an adversary might cause a 15-minute BES impact if the monitoring system were didn’t tell a control center that the oil had drained out of the transformer. But it wouldn’t inevitably cause an impact. Maybe the control center would figure out that there was something wrong with the sensors and send someone to physically inspect the transformer – then there wouldn’t be an impact; there are many other such scenarios. I’ve always thought that the word “inevitably” is the key to understanding when a Cyber Asset is a BCA and when it isn’t – although don’t ask me to show you where I said it. I just looked and couldn’t find where I said it. But I know I did.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. To discuss this, you can email me at the same address.

EPRI's new Cyber Security Procurement Methodology

In my post yeseterday, I mentioned that Tobias Whitney had spoken to my sub-group of the NERC CIPC Supply Chain Working Group, and discussed a new vendor questionnaire that EPRI is working on. I'm sure he said it, but I must have missed the fact that this is actually just one part of Revision 2 of a document called "Cyber Security in the Supply Chain: Cyber Security Procurement Methodology".

Tobias emailed me today to point out this document is now published, and is available here. It sounds like it's pretty meaty, and it does carry a price tag of $300 for organizations that aren't EPRI members. So before you pull out your credit card and order it, see if your organization is a member! In fact, I've asked the owner of Tom Alrich LLC to check into this so I don't have to pay for it...Oh, wait...

Supply chain security lessons from the medical device industry – part I

Nowadays, I try to avoid multi-part posts, since it seems that most of the time I never deliver all of the parts I promised. But I think I might actually do that this time, since I want to tell you about three documents that should be quite interesting to anybody trying to figure out how they can put together a good supply chain cyber security risk management plan – either to prepare for CIP-013 compliance or just because they know their organization needs to have one anyway

I often say nowadays that threats through the supply chain are the number one source of cyber security threats, for just about any industry – I say it because I believe it’s true. And it’s especially true for the electric power industry, since our good friends the Russians have figured out that the supply chain is the soft underbelly of the US grid. They may have actually gotten into the grid itself this way (as the CIA and FBI have said), although we haven’t yet gotten up the nerve to find out whether or not this actually happened. It seems we don’t want to be impolite, or something like that.

And of course, another reason why supply chain security is very important for the power industry is that the NERC CIP-013 standard will come into effect in about 15 months, which isn’t very long considering what has to be done (especially by larger utilities). Besides currently working full time on CIP 13 in my day job, I’m also very involved with the Supply Chain Working Group of the NERC CIPC, which has formed five sub-groups to write white papers on various aspects of supply chain security/CIP 13 compliance (and believe it or not, with CIP 13, security literally equals compliance and vice versa. As everyone in the industry knows, this is very far from being the case with the other CIP standards).

I’m in charge of one of those sub-groups that’s looking at the vendor risk management lifecycle. We’ve been ranging far afield, looking for ideas that we can bring back to the industry in our white paper. I’ve known for­ two or three years that the medical device industry already had mandatory supply chain cyber regulations in place, so I invited a former colleague of mine, Nick Sikorski of Deloitte, to talk to our group a couple weeks ago about what hospitals due to secure their medical device supply chain.

Nick has been working in this area for maybe four or five years and is quite knowledgeable about it, so he didn’t disappoint. Everyone in the meeting found what he said to be quite interesting, and the questions went on for half an hour or so[i]. Plus Nick provided us with three really interesting documents, which seem to hold a lot of ideas for what the power industry could be doing in supply chain security. I was thinking I might discuss all of this in one longish post, but after going through the documents I realized that each one could be the subject of its own post.

First some background: For five or six years, the US Food and Drug Administration (FDA) has published mandatory guidance for cyber security of medical devices sold to hospitals (and sometimes provided to patients by the hospitals, such as pacemakers). You might think that “mandatory guidance” is an oxymoron, like “British cuisine” or “jumbo shrimp”. Either it’s mandatory or it’s guidance, but it can’t be both. However, the FDA has a unique position among regulators; it needs to approve medicines and medical devices for sale in the US. A device maker that wants to be able to sell their product here would be very well advised to follow the FDA’s “voluntary” guidance (and of course the FDA publishes guidance in many areas besides cyber security, and not just for medical devices).

The biggest difference between the FDA’s guidance and CIP-013 is that the former applies to the vendors, not to the end users (which in this case are hospitals). It would be nice if there could be direct regulation of power industry vendors, but since neither NERC nor FERC has any jurisdiction over vendors, this isn’t possible. The industry is left with the situation where a large part of the effort required for CIP-013 compliance depends for its success on getting vendors to do certain things, but in the end the electric utilities are on the hook for compliance, not the vendors. The hospitals aren’t in the same situation.

But this doesn’t mean utilities can’t learn a lot from what hospitals have done to help them secure their supply chains. Perhaps the poster child for this is the Manufacturer Disclosure Statement for Medical Device Security[ii] or MDS2. It’s been around since 2013 (although a successor is being developed now). It was drawn up by the hospitals (not the FDA), and I believe it is filled out religiously by most (all?) medical device makers. In fact, if you Google the name of the document, you’ll find some filled-out forms from vendors like GE Healthcare.

The document is an Excel spreadsheet with about 150 questions divided among 20 categories. Some sections don’t have relevance to CIP 13, since they deal with data privacy (and privacy isn’t a concern for control systems. Of course, data privacy is different from confidentiality of BES Cyber System Information, which is quite definitely a concern of the CIP standards, although not so much of CIP 13). But other sections are quite relevant, such as “System and Application Hardening” and “Security Capabilities”.

I think anyone (almost) involved in CIP 13 compliance should find these questions interesting. Through my work on CIP-013 compliance so far, I have come to see a vendor questionnaire as a key component of a good SCCSRMP (I’ll let you guess what that stands for, although if you go to CIP 13 R1, I think you’ll find out). This is because it can be a great way to assess a vendor’s cyber program, without sending out a team of inspectors to each of say 50 vendors. Of course, it would be nice if some industry body would draw up an “official” questionnaire (as we’ve been discussing in my group) – and maybe that will happen, since at the same meeting where Nick spoke, Tobias Whitney of EPRI discussed a vendor questionnaire they’re currently drawing up, although he can’t provide a draft of it now.

But for the time being I would recommend that each entity draw up its own questionnaire. It should be based on the set of supply chain threats that you have “identified and assessed”, as called for in R1.1, then decided you would mitigate in your SCCSRMP; but it shouldn’t go beyond those.  You shouldn’t just throw a bunch of questions in that sound like they’re good ones. If they don’t address threats you’ve chosen to mitigate in your plan, then you’re going to make vendors jump through hoops to answer a lot of questions, when you don’t really care about those answers.

Of course, your vendor might complain that they’re getting a bunch of different questionnaires thrown at them, and they might point you to some standardized description of their cyber security program for your answers (I know some vendors are doing this now. It’s certainly a good idea, but it probably isn’t going to answer all of the questions you need to ask, and more importantly it may be hard to actually get the answers that you need out of the general verbiage). It’s up to you to decide whether whatever answers you get from the vendor are adequate – and if you don’t think they’re being responsive with some of the questions, then you may want to consider the vendor to be high risk for those threats, and hence use appropriate mitigations. Because that’s really what the questionnaire is about – assessing the risk your vendor poses to BES security, as personified in your own little “corner” of the BES.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. To discuss this, you can email me at the same address.

---

[i] This is just one example of the great discussions that my group – and the other four sub-groups of the SCWG – have been having on different aspects of supply chain security. If you want to join the SCWG, you should send me an email and I’ll forward it to the right people.

[ii] I’d like to provide a direct link to the document, but as soon as you click on the Google result, the document downloads – so I can’t capture the URL. You can get it by searching on “mds2 form hn 1-2013”, then clicking on the link that starts “HIMSS/NEMA Standard…”. You can find a “guidance” document on it at this link.

CIP-013: Risk Management vs. Best Practices

I hope it’s obvious by now that I firmly believe CIP-013 is a standard for risk management, as stated in the Purpose (found in Section 3 of the standard): “To mitigate cyber security risks to the reliable operation of the Bulk Electric System (BES) by implementing security controls for supply chain risk management of BES Cyber Systems.” In other words, CIP-013 requires NERC entities to mitigate supply chain risks to the BES by managing them.

You manage the risks by developing a plan to identify the most important risks in your environment, and then mitigating those. As I mentioned in my last post, this inherently admits that there are a lot of less-important risks that you simply won’t mitigate at all, since nobody has an infinite budget for this. But as I’ll discuss below, taking this approach in theory guarantees that you will mitigate the most supply chain security risk possible.

However (and I realize this will come as shocking news to you), I have to say there isn’t 100% agreement with my opinions in the NERC community, even though nobody has ever come up to me on the street and said “Hey, Alrich! You know, you’re full of s___ on CIP 13, or even sent me an email to that effect. However, nobody has ever provided me with a full description of an alternative way to comply with CIP-013, one that follows the wording (I’ve heard and/or read a couple methodologies that pretty much pretended R1.1 doesn’t exist).

So I know that many and probably most NERC entities won’t follow what I’m saying about CIP-013 compliance (which I flatter myself to say is very close to what Lew Folkerth of RF is saying. BTW, you can now get Lew’s two articles on CIP 13 – without having to download the whole 13 MB newsletter – by going here and here. Lew says there will be a third article in this month’s newsletter, which should be posted any day on RF’s website. You can sign up to get notification of those newsletters by going to RF’s home page and scrolling to the bottom).

But I have thought about how most entities will probably comply with CIP-013, and I admit it certainly won’t be a disaster for the grid: They’ll follow best practices. And where will they find those? Well…everywhere. You can think of NIST 800-161, NIST 800-171, NIST 800-53, the white papers produced by APPA/NRECA, NAGF, UTC and others…as all best practices. But since even just one of the NIST documents would provide more than enough best practices for even the largest utility to ever be able to implement, how does the utility decide which ones it will adopt and which ones it will ignore? Of course, the reason for this question is what I mentioned in the second paragraph above: No utility (or any organization, for that matter) has an unlimited budget available for implementing best practices for supply chain risk management, or for that matter any other worthy goal.

And here’s the difference between risk management and best practices: If you decide to address supply chain security (and/or CIP-013 compliance, since in the case of CIP-013, security and compliance are literally the same thing) using a risk management approach, you will in theory ensure that every dollar or hour of staff time that you spend on supply chain security will yield the maximum possible return, which means reduction in supply chain security risk. In other words, if you follow the risk management approach, you are sure to achieve the most bang for the buck. This is because you will rank your risks by their importance (i.e. their degree of risk, which I define as likelihood plus impact) and only mitigate the most important ones.

If you take the best practices approach, you have no way of being sure that you are getting the greatest possible risk reduction for the resources you expend. This is because the return from mitigation (and best practices are of course mitigations for particular threats, although the threats aren’t usually explicitly stated in documents like NIST 800-53) depends entirely on the degree of risk posed by the threats that you’re mitigating. You could spend exactly the same amount of time and money mitigating a very serious threat (say one that has a very high impact and a moderate likelihood) as you would on a much less serious threat (one which also has a very high impact, but which is very unlikely to be realized), yet in the first case you would be reducing a lot of risk, while in the second case you would be reducing very little risk. And the problem with taking the best practices approach is that you don’t have any structured way to distinguish between the two cases, because you’re just applying mitigations that you like for one reason or another; you aren’t explicitly considering the degree of risk posed by the two threats that you’re mitigating.

For example, let’s take two very serious supply chain security threats:

1. The threat that you will install a software product on a BES Cyber System that includes a piece of third party code containing an undisclosed back door. A malicious third party learns of this back door and exploits it to cause damage to the BES.
2. The threat that you will purchase a BCS with a motherboard, into which a chip containing a back door has been inserted. A malicious third party learns of this back door and exploits it to cause damage to the BES.

What is the degree of risk posed by each of these threats? The impact of the threat if realized is the same in each case: high. It doesn’t matter whether an attacker gains control of a particular BCS using a hardware back door or a software back door. What they can do once they gain control is exactly the same.

But what about the likelihood? I’d say it’s high in the case of the first threat, since it’s happened multiple times. There have been a number of cases of software back doors. But how about the second threat? It has certainly been talked about a lot, and was the subject of a big Bloomberg article at the end of last year. But the article has been widely doubted, as well as denied by a couple of the “victims” mentioned in it, including Apple and Amazon. There may have been a successful attack that I haven’t heard of, but in any case I think this threat has a low likelihood of being realized.

Since I believe the risk of a security threat being realized is the sum of the likelihood and impact, and if we assign values of 1/2/3 to low/medium/high respectively, this means the risk score of the first threat is 6, while the risk score of the second threat is 4. Clearly, the first threat poses a significantly higher risk than the second.

But what does it cost to mitigate each of those threats? For the first threat, my guess is most utilities will just use the mitigation of only buying BCS software from a trusted vendor; they will trust their vendor to only incorporate third-party software from sources they trust, who won’t plant back doors in their products. Of course, there are more expensive mitigation steps they can take, like purchasing various software vendor risk services, doing penetration testing to find back doors or perhaps signing up for aDolus, a really interesting service I first learned about last week.[i] These all have a fairly moderate cost.

On the other hand, for the second threat, the sky’s the limit when you talk about mitigation cost. You can install an electron microscope and look for traces on the board or changed features of the chip that might give away that it’s different from the normal one (and of course, you have to do this for every chip on the motherboard, since there’s no way of knowing beforehand which one might be a counterfeit). You could also have a team fly to the country of origin of the motherboard, examine component inventories in the factory, inspect the factories where the components are made, etc. And if this were a very high-likelihood threat as well as very high-impact, this might be justified.

My point in discussing cost is that it’s very unlikely that the cost of mitigating the second threat is any less than the cost of mitigating the first threat, and it’s very likely to be far higher. But even if the costs are the same, the fact that the risk being mitigated is higher in the first threat than it is in the second means that the risk reduction achieved will be greater. And this means the return on the investment of mitigating the first threat is higher than the return on the second threat.

Yet I have had discussions with people who think that mitigating the second threat is as important as mitigating the first one; if you asked them for best practices, they would probably recommend you invest as much time and/or money in finding rogue chips as you do in verifying that your software vendor vets their vendors of software components carefully.

Of course, it’s unlikely that you will throw a lot of resources at mitigating a threat that has low likelihood or low impact or both. But think about it: This shows you’re at least on some level doing the risk analysis anyway! There is simply no way you could determine, merely from examining the mitigation itself, whether it mitigates a high-risk or a low-risk threat. The mitigation itself has no risk score. You might get lucky and subconsciously perform the risk analysis, then decide that your “gut feel” was that it was much better to spend your money on verifying that your software vendor has a good handle on their supply chain, than it was in verifying that no chips had been substituted on a motherboard. But your gut feel might very well have told you just the opposite, especially if you’d just finished reading the Bloomberg article.

To sum up, I really don’t think just following a list of supply chain security best practices (if you can find a realistic list targeted to supply chains for control systems for electric utilities, which I haven’t seen yet) is going to lead you astray. It’s certainly much better than not doing anything at all for supply chain security (or CIP-013 compliance). But you’ll never be able to get as good a return on your investment in risk mitigation as you would if you explicitly considered risk from the start. This is why I think the risk management approach is much better than the best practices approach. And it’s also why FERC ordered NERC to develop a risk-based standard in 2016.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. To discuss this, you can email me at the same address.

---

[i] Full disclosure: After he showed me the product and I became very interested in it, the founder of this company – a longtime friend - mentioned the idea of my providing consulting services to them. I’m not exactly sure now if there’s anything I can really do to help them, and I don’t want to spoil a friendship by taking money and not providing value, so I’m not sure what will come of this. But I don’t want to hide this fact.

CIP-013 and acceptance of risk

A longtime principle of risk management - for financial risks, general business risks, security risks, etc. – is acceptance of risk. To accept a risk is to acknowledge that it exists, but at the same time to decide that the costs of mitigating the risk outweigh the expected loss if the risk is realized. This is simply an acknowledgement that there are too many risks to make it worthwhile even to attempt to mitigate them all. Of course, this is especially true in cyber security, where there are a very large number of risks, if not an infinite number.

The team that drafted NERC CIP version 1 was composed of cyber security professionals who understood this principle very well. So they made sure that the draft of CIP v1 that was submitted to FERC made liberal use of acceptance of risk. Many requirements were written so that the entity could either perform what was required or “accept the risk” and not do anything. This seemed to the team to be an eminently reasonable approach.

However, it didn’t seem at all reasonable to FERC. They ordered the “acceptance of risk” language be removed from the standards; this was done in CIP v2. Their reasoning was very simple: There is no way a single entity can accept risk on behalf of all the entities that are part of the Bulk Electric System.

FERC’s reasoning made sense in the context of all of the standards that had so far been developed by NERC (the so-called “693” or “Operations and Planning” standards). These standards are almost all based on the laws of physics: You do or don’t do something (like trimming trees regularly under transmission lines), and the next thing you know, there’s a cascading outage that blacks out a large swath of the US and Canada. Or you make one wrong move in a key substation, and the next moment there’s a disturbance that takes out most of south Florida and is felt in less than a second in Canada. By not doing what they should have been doing, the utilities involved were essentially accepting risk on behalf of the entire North American power grid – but the rest of the grid had no chance to weigh in on whether they should do this or not. They just felt the consequences.

So FERC applied that reasoning to CIP v1, probably the first mandatory cyber security standard for the power grid of any country or region of the world. And why shouldn’t they? As everyone involved with developing CIP said at the time (and as many in the NERC community still say), the CIP standards are just basic best practices. Any NERC entity that doesn’t follow them leaves a hole that’s just waiting for some adversary to walk through.

But the big problem with taking a “best practices” approach to cyber security is it assumes that the set of cyber threats that need to be mitigated is the same for all entities and for all times. Specifically, it assumes that a) there will be no significant new risks that appear over time, and that all entities face the same threat landscape. And if this assumption doesn’t prove to be true, then b) the standards will be flexible enough to incorporate these new or variable threats.

Well, guess what? Assumption a) isn’t true for cyber threats. New threats appear all the time, and different NERC entities face different threats and to different degrees. As for assumption b), we all know too well that the NERC standards development framework makes it close to impossible to incorporate new cyber threats into the standards in anything less than a few years. For example, the phishing and ransomware threats have been in place for many years, yet there isn’t even discussion of developing new requirements to deal with these. And the threat posed by malware-infected laptops was well known since the late 1990’s, yet when did a requirement come into effect that addressed this threat? 2017.

This situation has strained the current NERC CIP standards to close to the breaking point, with entities spending huge amounts of time and money complying with certain very prescriptive CIP requirements, but then being starved of resources to deal with other cyber threats that aren’t addressed in CIP at all.[i] FERC knew this when they wrote Order 829 in 2016, which ordered NERC to develop a supply chain cyber security standard that was risk-based and not “one size fits all”. While they didn’t say it this way, this seems to me to be the reasoning they followed:

1. It was clear to FERC, as it’s clear now, that supply chain is by far the biggest area of cyber risk faced by electric utilities, or almost any other industry, today. Think Target, Stuxnet, NotPetya, and the current Russian attacks on the US power industry – all came, or are coming, through the supply chain. FERC felt it was necessary for CIP to address supply chain risks as soon as possible and as thoroughly as possible.
2. At the same time, FERC knew that the US power industry was struggling mightily just to comply with the existing CIP standards, in large part because they aren’t risk-based and only allow for one type of variation among individual entities: the impact level of particular assets on the BES (and even then, only in three very broad categories). Asking the industry to take on a huge new burden for supply chain security was impossible.
3. The only way that the burden of the new standard would be manageable would be if it were risk-based, with the entity itself determining the most important risks for it to mitigate, as well as how to mitigate them.
4. If the entity does this, it will be able to allocate its limited risk mitigation budget (and who has an unlimited budget?) in a way that every dollar or hour spent on supply chain cyber risk mitigation reduces the maximum possible amount of cyber risk. The alternative is a set of mostly prescriptive CIP requirements (as in CIP-002 through -011) that decide for the entity what are the risks it needs to address, as well as exactly what it needs to do to mitigate each of them. There is no room for variation among entities or over time. This will inevitably result in a lower amount of total risk being mitigated, than if the entity is in charge of deciding for itself what risks it will mitigate, and how it will mitigate them.

This is why CIP-013 requires the entity to do just three things: a) Develop a supply chain cyber security risk management plan (R1[ii]); b) Implement the plan (R2); and c) Review the plan every 15 months (R3). If the entity follows the wording of R1 (and it isn’t that easy!), it will be sure to get the most “bang for the buck” in allocating its limited budget to address supply chain cyber risk. And if the entity follows the wording of R3, it will be sure to make adjustments to its plan over time, so that it continues to address the most important supply chain cyber risks, using the most current mitigations for those risks.

Where does acceptance of risk come in here? It’s very simple: There’s no way that risk management can work, unless the entity can accept some risks without mitigating them. The entity has to decide what are the biggest threats that it faces, determine the risk posed by each of those threats, and line the threats up in a big spreadsheet, ranked by their degree of risk. Then the entity needs to start at the top and decide which threats (risks) it can mitigate, then draw a line under the last one of these. The entity will mitigate all of the risks above the line, and it will accept all of the risks below the line.[iii]

Of course, FERC never said that the supply chain standard should allow acceptance of risk, and CIP-013 doesn’t use those words at all. But I contend that it doesn’t have to. It makes no sense to talk about risk management if you aren’t accepting some risks – since, in the cyber security domain and even more so in the supply chain cyber security domain, there are close to an infinite number of risks. In complying with CIP-013, the entity is not only allowed to accept some risks, it is required to do so; otherwise, it’s literally impossible to comply with the standard without bankrupting the utility. And that doesn’t exactly do a whole lot for grid reliability.

If you’re wondering whether I’m the only person saying this, I’m not. Lew Folkerth of RF, the only person within the wider NERC organization who has written about how to comply with CIP-013, has written two great articles for RF’s bi-monthly newsletter (which you can get in PDF form if you want to email me. Otherwise, you have to download two 13 MB newsletter files). I’ve written about these in four posts, starting with this one (these posts haven’t ended, since Lew is promising a third article, to appear in the newsletter that will be released this month. And you thought the Muller report was going to cause the most excitement when it’s released this month? Not in the Alrich household, I can tell you that!).

In the first of Lew’s two articles, he says, regarding auditing of CIP-013 R1.1: “You will need to be able to show an audit team that you have identified possible supply chain risks to your high and medium impact BES Cyber Systems, assessed those risks, and put processes and controls in place to address those risks that pose the highest risk to the BES.” (my emphasis) If you think about it you’ll realize that, in saying that the auditors will look to see that you’ve addressed the highest supply chain cyber risks, Lew’s also saying that the auditors aren’t going to expect you to mitigate risks that aren’t among the highest.

In his second article, Lew makes this more explicit when he says: “You can’t address all risks, so you will need to prioritize the risks you will address.” He goes on to describe a process almost exactly like the one I described above, including identifying threats (risks), assigning risk scores to each one, then planning to mitigate the threats with the highest scores. Amen.

Most sermons open with a reading of Scripture. But it seems this one has ended with it.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. To discuss this, you can email me at the same address.

---

[i] I would love to refer to a post for more information on what I’ve just said, but it’s scattered around lots of posts. However, I did write an article on this topic for a UK security journal (print only), that I’m allowed to distribute in a PDF file. If you’d like me to send that to you, send me an email at the address above.

[ii] R1.2.1 through R1.2.6 list six specific risks that must be mitigated in the plan. They are there because FERC specifically ordered each of those to be included, when they wrote Order 829. You can think of these as the most important risks to be mitigated, but certainly not the only ones. The entity is in charge of deciding what other risks it will mitigate, consistent with its budget.

[iii] This is a simplified description, since  it may be possible for the entity to mitigate the same amount of risk, or even more, by only partially mitigating each of the risks at the top of the spreadsheet - yet mitigating more of them than if it required either total or no mitigation. But this is too complicated an idea to discuss in this post.

The best resource for CIP-013 compliance

I want to repeat again (and I’m not being redundant. I mean this is the third post where I’ve said it) that anyone seriously involved in CIP-013 compliance should run, not walk, to join the Supply Chain Working Group of the NERC CIPC. Membership is open to anyone with an interest in CIP-013 or supply chain security, or both (and I know there are at least a few members who aren’t involved with the electric utility industry at all, but are very interested in supply chain cyber security). Just drop me an email at the address below, and I’ll forward your name to the two people in charge of the group, Tony Eddleman of NPPD and Tom Hofstetter of NERC. I know that as of late last week there were over 120 members, and there are more now, I’m sure. To put this in perspective, my guess is most subcommittees of the NERC CIPC (which is what the SCWG is) don’t have more than 10-20 members, so this is certainly some sort of record.

And why am I so interested in getting more members, when there are already so many? Two reasons. First, because I would like to see the whole industry’s awareness of supply chain risk management (SCRM) best practices increase. I feel that will especially help establish a consensus among NERC entities and auditors as to what the requirements mean and how to comply with them. This is badly needed, since right now interpretations of CIP-013 are all over the map. A consensus will allow entities to focus primarily on supply chain security and only secondarily on worrying how the standard will be audited.

The focus of the SCWG now is the sub-groups. There are five sub-groups, each of which is preparing a short white paper (or three short white papers, as the group I’m leading will probably do) on a different aspect of supply chain cyber security. Three of those groups had meetings last week and this week, and they’re all planning on weekly meetings for the near future; the other two groups will start to have meetings next week, I believe. I’ve attended two of these meetings (and will attend another tomorrow), and led two of them as head of the supply chain risk management lifecycle group.

I can attest that all of these meetings have been really excellent, with lots of people volunteering what they know about the topic. For example, there’s a group led by George Masters of SEL, called “Considerations for unsupported or open-source technology”, which is addressing the questions of third-party (and fourth-, fifth-, sixth-, etc. party) components of commercial software platforms, as well as open-source components. It is really good, with a lot of contribution by people who know different aspects of the subject (and nobody knows it all, of course, even George).

My group isn’t too shabby, either. Our two meetings have both included about 25 people, including (at least in this week’s meeting), four vendors – and I can see that having vendors in a supply chain discussion is really helpful.  We’ve decided that we will write three white papers (all 2-3 pages), not just one, since we realized there are at least three lifecycles we wish to discuss:

1. The supply chain security risk management lifecycle itself. I believe this paper will follow somewhat along the lines of what Lew Folkerth of RF has been writing (and Lew has been in both of our meetings. I hope he can continue, since he has been a big help).
2. The vendor risk management lifecycle. We had a really good discussion this week on topics like vendor questionnaires and contract language, and we’ll continue that (with other topics) next week.
3. The product risk management lifecycle. We haven’t discussed this very much so far, but I’m sure it will be much more defined in a month or so.

All of the groups will present and discuss their white papers at a meeting held before the NERC CIPC meeting in Orlando (although location is still officially TBD) in early June (I think it will be Tuesday morning, June 4). I believe the meeting will be webcast by NERC, but you might want to come onsite and also attend the CIPC meeting (you have to register for that, of course. Registration isn’t available yet) – and those have been getting better all the time.

The topics of the five sub-groups are:

1. Considerations for secure hardware delivery
2. Considerations for establishing provenance of systems and components
3. Considerations for threat-informed procurement language
4. Considerations for supply chain risk management lifecycle
5. Considerations for unsupported or open-source technology

The white papers for all the groups except mine address a particular area of supply chain security risk (and they’re all meaty topics, with lots of nuances to understand). While I’m not sure that the leaders would all agree with me now, I think it’s inevitable that each group will end up providing two types of compliance “guidelines” for their particular risk area. One is ideas for risks that you should consider in your supply chain cyber security risk management plan; the other is possible mitigations for those risks.

But let’s make it clear: None of these groups, including mine, is providing “interpretation” – nor will the SCWG as a whole. The standard doesn’t need interpretation, since it’s very simple: Develop a supply chain cyber security risk management plan (R1). Implement the plan (R2). Review the plan every 15 months (R3). The question really is, what should a good plan contain? Specifically, what are risks that an entity should consider including in its plan, and what are possible ways of mitigating those risks? The white papers will provide suggestions on those topics, based on our discussions in these meetings.

At 2-3 pages each, the papers will have to treat one idea very well, and other ideas much less or not at all. Even though I wasn’t at the meeting where the SCWG decided to write small white papers, I think it was a good decision. I’d much rather read three short papers discussing one idea each, than one longer paper discussing three ideas. The author will be able to write much more clearly, and the reader will be able to retain that better.

But I can assure you that the meetings are each covering 5-10 ideas at least, and those ideas will probably change every week. In my group at least, I’m deliberately not worrying about the white papers yet. I’d much rather see what comes out of the weekly discussions for the next month, and then look through the notes for all that (my notes for last week’s meeting were four pages long, and this week’s will probably be at least that). My feeling is that after all this discussion, the ideas that should be addressed in the papers will emerge on their own, and the papers will pretty much write themselves.

This is why you should join the meetings now, rather than just read the white papers when they’re published in June. You will only hear the ideas that are raised in the meetings, but don’t make it into the papers themselves, if you’re at the meeting. Even the minutes will at best be a pale reflection of the meeting itself (trust the man who’s written one!).

And now I’ll get to the second reason why I’m so interested in getting new members: There’s a real virtuous cycle here. The more participants there are, the better the conversation is. And the better the conversation, the more participants there will be.

So send me an email at the address below if you’d like to join us. I’ll consolidate these and forward them to Tom Hofstetter and Tony Eddleman, who have proven more than willing to bring new sheep into the fold.

  

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. To discuss this, you can email me at the same address.

Alrich awarded Russian International Medal of Freedom!

  

I was very surprised to learn from a news feed this morning that the Ministry for Hybrid Warfare of the Russian Federation has awarded me the ironically-named International Medal of Freedom. Since I hadn’t heard of this, I looked it up online. It seems that this coveted award is bestowed annually on the foreign national who has done the most to advance the interests of Russia in his or her own country.

I was very puzzled by this news, since I certainly don’t think it can be said that I’ve provided any aid or comfort to Russia in my blog posts, especially recently. I was still wondering about this when I received a Skype call from a Russian reporter that I’d met at this year’s RSA Conference a few weeks ago. Of course, when he called me it was the later afternoon in Russia.

“I guess you heard the news?”

I did. What do you know about this?

“I was as surprised as you are when I heard it this morning. Fortunately, I have a very good friend who occupies a fairly high position in this Ministry. I called him about this. He said he did know the story behind this, but he wanted me to come in to talk to him, since he didn’t want to discuss this on the phone. I didn’t have other commitments this morning, so I drove in to talk with him.”

What did he say?

“First let me set the scene. Even though he’s been a friend since high school, I’ve never actually been in his office. I was very impressed – wood panels, high windows. He’s obviously somebody important there. What I found especially striking was the quotation from Nikita Khrushchev in gold lettering behind his desk: ‘We will bury you.’ I asked him why this quote from the height of the Cold War was still there. He answered that he’d regularly asked for the money to have it removed, but he was always turned down. Given how important he obviously is, I found this a little suspicious.”

Whatever.

“Then I got right to the point. I said to him, ‘I know you probably had never heard of Tom Alrich before today; it was probably some low-level guy who suggested him for the award. I actually met him in San Francisco just a few weeks ago, and I can certainly say he’s very nice. But he’s written a number of blog posts that have been very critical of our recent – ahem! – activities with respect to the US power grid. In fact, he’s been calling repeatedly of late for an investigation of the report in the 2019 Worldwide Threat Assessment, which implied that we’d actually penetrated control centers or substations of multiple electric utilities in the US, and had planted malware so that we could cause outages. Now, I know you’re going to deny this is true, since it’s never been officially acknowledged here, but…’

“At this point, I looked up at him, expecting him to be angry that I hadn’t taken the official line that the reports of Russians attacking the US grid were fantasy. Instead he said ‘Don’t worry, I know exactly who Tom Alrich is. I as well as a lot of my people have been reading his blog on certain occasions with great interest, even when he’s not talking about Russia. In fact, giving him the medal was my idea!’”

“Of course, now I was speechless. I almost blurted out ‘You know, I always thought you were smart. But giving a medal to someone who is doing everything he can to thwart what we’re doing with the US power grid is just about the stupidest thing I’ve ever heard. Have you been to the doctor lately?’”

“Fortunately, I didn’t have to say this, since he continued ‘I know this may seem strange to you. But consider: Has he been at all successful in his campaign to get an investigation?’ I admitted that he hadn’t, but I blurted out that this still isn’t a reason to give him a (Russian expletive) medal.

“He went on, ‘I first heard about him last summer when he started writing  about us, the day the big article appeared in the Wall Street Journal. This was a report on the first of four briefings that the Department of Homeland Security scheduled in late July. That article quoted DHS people at the briefing as saying that ‘Russian hackers’ had penetrated control networks at utilities, and had most likely planted malware that could cause outages (this story was immediately widely reported, of course).

I’ll admit that at the time I told others he was just an excitable fellow, and he wasn’t really a danger to us. After all, in that first post he asked a few pointed questions that implied the DHS people didn’t know what they were talking about. And when DHS issued a story two days after the briefing that just about completely discounted what had been said at the first briefing, Mr. Alrich expressed amazement that DHS could have been so wrong at first. But he clearly believed the walk back by DHS, not the briefings themselves. After that, people congratulated me that I had been right about his not being a danger, and I and my department pretty much forgot about him.

“But in January, the Wall Street Journal came out with another article – this one the product of its own research, not just repeating what was in a briefing – that provided a lot more detail about our campaign, including the statement by Vikram Thakur of Symantec that at least eight utilities had been penetrated at the control system level, and malware might have been planted. And shortly after that, the Worldwide Threat Assessment came out.

“At this point, Mr. Alrich completely changed his tune and – while he still indicated skepticism about the reports, he said an investigation was needed, and he started a one-man campaign for this (which he continues today). He made the point that a lot of experts had flown from the US to the Ukraine to investigate after our successful attacks there (both the 2015 and 2016 attacks), and had produced some very good papers on them. Plus DHS had held multiple briefings for the utilities, both classified and unclassified – so that they could be on the lookout for similar attacks. He started asking very loudly why so much had been done when the Ukraine was attacked, whereas when the FBI and CIA reported that the US was subjected to attacks that might be far more damaging, nobody felt motivated enough to look into why they said this – even though the general consensus in the utility industry was that those agencies must have been wrong.’

“At this point, I was really exasperated and got sarcastic. ‘So you decided the right way to neutralize Mr. Alrich as a threat was to give him a medal!? Whatever happened to the Skripal solution?’ Of course, I really like you so I certainly don’t want what happened to Skripal to happen to you, but I assumed the office was bugged and I began to think this was some sort of test of my loyalty. And you probably know that Russian journalists who show insufficient loyalty to President Putin sometimes die of acute lead poisoning, if not some other type of poisoning.”

Don’t worry, I’m sure you wouldn’t seriously advocate the Skripal solution for me. And even if you did, given how badly the Russians botched that one, I wouldn’t be too concerned about it.

“You’re right about that! We really blew it. At this point, my friend picked up his story. He said ‘Believe me, I started to advocate for something like Skripal to be applied to Mr. Alrich. But then I decided to talk to a deeply embedded asset we have in the US, who is really on top of all this electric utility stuff. I told him we were considering taking action to neutralize Mr. Alrich, and he laughed. He asked if I was crazy, and he said we should give Mr. Alrich a medal, not try to kill him.

“Now I was really confused, and I asked if he was joking with me. He said ‘Certainly not. The best thing you can do for yourself is let Mr. Alrich continue his campaign for an investigation. In fact, you should do everything you can to encourage him in it – maybe get some of our friends in the US government to egg him on, whatever. And do you know why I say this? It’s because that guy has been going on crusades for years - complaining about wording problems in CIP version 5 and trying to get them fixed, trying to get NERC to postpone the CIP v5 compliance date, trying to get all of the current CIP standards rewritten in a risk-based format, and on and on.

“Yet guess how much success he’s had? Zero. People read him, but nobody in a position to make anything happen ever agrees with him. In fact, what makes him really valuable to you is that, because he complains so much, I think the people in power deliberately don’t do what he’s demanding. So the best thing that ever happened to our attacks on the US power grid was when Tom Alrich started calling for an investigation. That probably guaranteed they’ll never be seriously investigated!’

“At this point, our asset in the US stopped talking, to let this sink in with me. I thought a bit, then asked him if he’d heard the phrase ‘useful idiot’. He said he had indeed, and that while it was often attributed to Lenin it was actually much older. I said ‘Well, it seems like Mr. Alrich may be our useful idiot. I absolutely agree we should give him a medal, not kill him. I’ll get that done right away.’’’

“And now you know why they gave you that medal. I have to admit I was in awe at the subtlety of this reasoning; and I was also very relieved that they’d rejected the idea of killing you – which means, by the way, that you shouldn’t hesitate to come here to St. Petersburg to accept the award. By the way, have you ever been to Russia?”

I haven’t, although I’ve always wanted to visit St. Petersburg. I’ve read a lot about the Bolshevik revolution and I’ve always wanted to see the Hermitage. However, I saw that the ceremony is during the next NERC CIPC meeting and I very much want to attend that because of the CIP-013 discussions beforehand – so I’ll have to decline.

“Too bad, I would have given you a personal tour of the Hermitage. I spend most of my Sundays there! Well, at least you now know why you got the medal.”

I thanked my friend for his invitation and said I’d hope to come another time. But I probably won’t, for fear there will be a change of heart at the Ministry, and they’ll go back to planning the Skripal solution for me. I’m safer here in the USA!

Plus I can’t say I really appreciate being called a useful idiot. I’ve been called an idiot many times before, but useful???

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. To discuss this, you can email me at the same address.