Monday, August 13, 2018

Is CIP-013 R1.1 auditable? No. Does this mean you’re off the hook? No.

In this recent story about the Russian hacking from E&E News last week, I was quoted as saying “'s not clear whether the federal rules on supply chain vulnerabilities can be effective..” Of course, this was referring to CIP-013, which came up in this story since the Russian attacks were (and are) all coming through the supply chain.

I was referring here to something I brought up in this post from April, when I pointed out that R1.1 is probably not auditable because it simply requires that the entity develop a supply chain cyber security risk management plan - the requirement doesn’t provide any information about the risks that should be addressed in that plan. I pointed to CIP-010 R4 as an example (definitely the best so far) of a plan-based requirement that does provide high-level criteria for what should be addressed in the plan (these are provided in Attachment 1, which is called out in the requirement itself and is therefore part of the requirement. That is important – Attachment 1 isn’t just some sort of guidance, but is part of the requirement).

In the April post, I noted that R1.1 simply requires the entity to develop a supply chain cyber security risk management plan; it says nothing about what that plan should contain[i]. I originally thought this was a good idea because of its purity: After all, cyber security is about risk management. The best way to deal with cyber threats is to put together a risk management plan, since there is no way anybody could ever write a set of prescriptive requirements (whether or not they’re mandatory) that would make the entity perfectly secure. The best that can be done is for the entity to assess the risks and develop a plan to mitigate the highest risks[ii] (this is what R1.1 requires the entity to do, although unfortunately the SDT left out the word “mitigate”. But the whole standard makes no sense if that word isn’t assumed to be in R1.1).

However, I later came to realize that, given NERC’s prescriptive auditing process, requiring an entity just to develop a plan, without saying what has to be in it, is a recipe for having a non-auditable requirement. Either a) the auditors will decide what they think should be in your plan and then try to hold you in violation if your plan doesn’t agree with their ideas, or b) the auditors will simply give everyone a pass as long as the plan is at least halfway credible. This is why R1.1 is unauditable.

I think b) is a much more likely scenario for what will happen with CIP-013 R1.1. So this leaves the entity (that would be you, Dear Reader) with two choices:

  1. You can develop a minimal R1.1 plan, perhaps just addressing the six items in R1.2 (since we already know they have to be in the plan - for a recipe on how to do this, go to my April post). This will make your CIP-013 compliance job much easier. And even though it’s likely your auditor will berate you – and most likely issue an Area of Concern - for not having developed much of a plan, you can still sleep at night, knowing that he or she won’t be able to give you a PNC for this (and if they do, it won’t hold up); or
  2. You can Do the Right Thing (to quote the title of a great Spike Lee movie) and actually develop a real supply chain cyber security risk management plan. This will probably put you at greater compliance risk, since if you list a risk in the plan, you will have to take steps to mitigate it. And if you don’t do a good job of mitigation, you can probably still be held in violation of R2, even though you wouldn’t be in violation of R1.1 (i.e., NERC can’t audit the plan itself, but it can audit whether or not you actually did what you said you’d do in the plan).

So which course do I recommend? Door Number 1, the easier path which may allow you to leave at 5:00 now and then? Or Door Number 2, the hard path, where you’ll have to really sit down and think about what your supply chain cyber risks are and how you will mitigate the most important risks - and then, if you don’t mitigate them to the auditor’s taste, you might well receive a PNC for violating R2?

I’m sure you can guess which door I’m advocating you should take: It’s Door Number 2. Why do I say this? All you have to do is read this post on the Russian attacks. Even though it turns out DHS greatly exaggerated the success of those attacks, that doesn’t change the most important lesson to be learned from them: Supply chain security is the number one problem for the electric power industry (and probably for most other industries as well). The attacks described by DHS (both in their briefings, and in their excellent Alert from March) were all supply chain attacks. They’ve been going on for a couple years and will most likely continue, despite the increased scrutiny after DHS’ briefings. And if you want to see the damage that a supply chain attack can cause, you just need to look at two: the Target breach of 2013 and last year’s NotPetya malware.

In almost any other question of CIP compliance, I will always take the position that the entity’s job is to design procedures and policies that provide minimal compliance with the requirements. Most of the currently-enforced CIP requirements are prescriptive, and of course all CIP requirements – as all NERC requirements in general – are audited in a very prescriptive, did-they-do-it-or-didn’t-they fashion. Even if your organization might feel that good security practice is to go beyond what a particular requirement mandates, you definitely don’t want to design CIP compliance procedures that go beyond the requirement. If you do, you’re simply inviting compliance risk.[iii]

However, for a plan-based requirement, and especially one that explicitly allows the entity to consider risk, as is the case with CIP-013, this position doesn’t apply. The whole idea of developing a plan to manage risk is that you need to allocate the resources you have (staff time and money) in a way that will mitigate the most risk possible – i.e. you need to allocate your resources so that they get the most bang for the buck.

This requires considering all the major threats (which in the case of CIP-013 are supply chain cyber threats), then ranking them by the degree of risk they pose to the BES (remember, that is what risk means in any NERC standard. It’s always risk to the BES, not to the individual entity). Then you need to go through the list, starting at the top, and decide how much in the way or resources to allocate to mitigating each risk. When you feel you have mitigated the important risks, you stop.[iv] In my opinion, that is how you develop a risk management plan.

I hope to start doing some posts in the near future that elaborate on – at a high level – the steps you need to take to develop a plan for CIP-013 R1.1. If you are with a NERC entity or a vendor that is looking for a more in-depth discussion in order to start preparing for CIP-013 compliance, ask me about my free workshop offer, described in this post.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. And if you’re a security vendor to the power industry, TALLC can help you by developing marketing materials, delivering webinars, etc. To discuss any of this, you can email me at the same address.                   

[i] R1.2 lists six items – they are risk mitigations, rather than risks themselves – that should be included in the plan. That isn’t because these are the six actions that the SDT decided did the most to protect supply chain security. The six items are there because FERC specifically called for them in Order 829, which ordered NERC to develop the standard in the first place. The R1.1 supply chain cyber security risk management plan needs to include these six items, but only including them doesn’t give you a good plan.

[ii] If you’re wondering how a small utility might have the resources and know-how to conduct this whole risk-management exercise by themselves, so am I! Of course, since CIP-013-1 only applies to High and Medium impact assets – and since most of the organizations that own these assets probably do have at least some resources and know-how in this area – I don’t see this as an immediate problem for CIP-013. But for the future when Lows are included in CIP-013 in some way (and FERC might order this when they approve CIP-013-1), this will be a big issue. I would hope NRECA, EPSA, EEI and APPA could step up and help their smaller members in this process.

[iii] Of course, I’m not saying that you should limit the steps you actually take in any particular area of cyber security to the strict wording of the CIP requirement. For example, suppose you think that CIP-010 R1 doesn’t do a good enough job of capturing what an organization like yours should be doing for configuration management of BES Cyber Systems. You should definitely do whatever more you think is necessary; but just make sure not to include that in your actual compliance procedures for CIP-010 R1.

[iv] Of course, I’m glossing over the fact that it’s possible you may run out of budget before you have sufficiently mitigated the most important risks. When you see that is happening (and hopefully you’ll see it during the planning phase, not at the end of the implementation phase), you should try to get the additional resources needed to mitigate all the important risks. But if you don’t get those resources and you have to leave some important risk unmitigated, you will at least know that you mitigated the most risk possible with the resources you had - since you mitigated the different supply chain threats in the order of the risk they posed.

Wednesday, August 8, 2018

What should DHS do?

I have had a number of email conversations brought on by my recent posts on DHS’ briefings on the Russian hacking campaign against the power industry, and on some very misleading statements made in the briefings – as well as wildly exaggerated press reports afterwards. They have all come down to DHS. Here is the problem:

  • The Russians have obviously been conducting – for a couple years, it seems – a large-scale, sustained cyber attack on US utilities and IPPs; that attack is ongoing.
  • DHS has done a great job of thoroughly investigating what is going on, and explaining it all in great detail. In doing so, they have made it very clear that the power industry needs to focus on supply chain security much more heavily now, since these attacks are currently coming primarily through that vector.
  • However, some of the speakers at their recent briefings gave very misleading information about the results of this hacking, implying that it’s possible and even likely that the Russians have a lasting presence inside networks in utility control centers, where they’re just waiting for the signal to start messing with the US power grid and cause a major outage.
  • After the first of these briefings, a reporter from the Wall Street Journal wrote an article that said that about 200 “utility control rooms” had been penetrated by the Russians. Of course, if that were really the case, it would literally constitute a national emergency, not just because we all might be in the dark for a while, but because we might then be forced to consider a military response.
  • The same week as the first briefing, two DHS spokespeople clarified in meetings that no, it was just one very small generating asset whose control network had been penetrated – and then it turned out that even that was an exaggeration, since it was really two turbines in a wind farm with probably hundreds of turbines. Yet there was no effort to counter the news reports – these walk backs were heard only by a small group of industry people.
  • Even worse, the same WSJ reporter came out with another story on Tuesday, which seemed to indicate that she hadn’t heard either of the walk backs. And it seemed from her story that one person at DHS was still peddling the idea that there had been widespread penetration of the US grid. I was charitable and thought that she and the DHS person both simply didn’t understand the terms that were being used, as well as some particular facts about the structure of the US power industry. My post yesterday tried to explicate these mysteries, in my usual mind-numbing detail.

So the fact is that we have a major national news source (actually two, since the New York Times put out their own article on Friday, which I discussed in this post. The sentence that I quote toward the beginning of that post is even more alarming than anything the WSJ report said) saying there is a true national emergency, and still DHS isn’t stepping up with something like a press release - or even better a press conference - to calm things down. They need to explain what really happened, while at the same time pointing out that there is a real supply chain threat to the grid – and I will be fine if they say that the industry isn’t doing enough to counter supply chain threats, as well as that the new CIP standard for supply chain security will likely prove pretty ineffective, unless NERC or somebody steps up and tries to fix this situation (this is the topic of what I hope will be my next blog post, although I won’t rule out some new development that will require a new post on the Russian story).

DHS needs to do something. Now.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. And if you’re a security vendor to the power industry, TALLC can help you by developing marketing materials, delivering webinars, etc. To discuss any of this, you can email me at the same address.         

Tuesday, August 7, 2018

Obviously, this one isn't going to go away very soon

After writing four posts in a row on the Russian hacking campaign against the US power grid – three of which were really about how it was characterized and reported by DHS and the press – I thought I was finished with this issue, and I could get back to writing about what I think is really of national importance, like CIP-013 (and I’m not kidding about CIP-013 being of national importance, since all the Russian attacks were supply chain attacks, and they continue to this day).

Specifically, I thought that, after a spokesperson for DHS admitted that the only control network that was penetrated was that of a “very small” generating plant, and after a high level DHS official further qualified that statement by saying - at a meeting where the Secretaries of DHS and DoE were in the room, as well as the US Vice President – that just two wind turbines on a wind farm were compromised (not even the whole wind farm), everyone involved in the misleading statements, and the erroneous reporting of them, would have felt properly shamed and would be more careful in the future.

Thus, I was surprised – to say the least – to read a front-page article in the Wall Street Journal today entitled “U.S. Steps up Grid Defense”,[i] which indicated a) at least one DHS official continues to put out deliberately misleading statements, which contradict the statements of other supposedly official spokespersons for DHS; and b) the same reporter who wrote the original WSJ article that set off this firestorm about two weeks ago doesn’t seem to have changed her narrative of what happened at all, despite DHS’ attempts to walk this back.

I find both conclusions quite disturbing, but I also find b) to be very puzzling. The four possible explanations I can think of are:

  1. The reporter has been living in an inaccessible cave since she wrote that article, and therefore missed DHS’ walk backs of the story; or
  2. She didn’t understand what the other official DHS spokespeople said when they issued the walk backs; or
  3. She was deliberately misled again by the DHS official who made the misleading statements quoted in her first article; or (finally)
  4. That DHS official – Jonathan Homer, whose title is Chief of Industrial Control Systems Group, Hunt and Incident Response Team – doesn’t himself understand the walk backs, because of his continued misunderstanding of a few power industry terms and facts.

I’m a fairly charitable person, so I prefer either explanation 2 or 4; of course they could both be true at the same time. So this is hopefully mostly a case of two people not understanding some important facts about, and terms used by, the US utility industry (although some DHS statements were still either deliberately or recklessly misleading). I’m also a very helpful person, so I will try to lay out those facts and terms using language that all can understand (which I didn’t do when I discussed them in previous posts).

1. Who owns the stuff, anyway?
First, most generation assets in the US aren’t owned by utilities, but by independent power producers.[ii] So it was very misleading that DHS’ statements all referred to “utilities” being penetrated. But there were only two assets that they specifically said were “penetrated” by the attackers. One was the wind farm where the control network was penetrated. The other was a combustion turbine plant. DHS didn’t specifically say that a CT was penetrated, but they did display a schematic drawing (which they said in the briefing was a screen shot of a Human-Machine Interface computer, or HMI) of a CT that they said had been obtained by the attackers. It is very unlikely that the wind farm was owned by a utility. It is possible that the CT (presumably a small one, not subject to NERC CIP – which explains the ease with which the attackers obtained the screen shot) was owned by a small municipal or cooperative utility.

Control rooms vs. control centers
But if the CT was owned by a small muni or coop, then this points to another problem with DHS’ statements: If a small generating plant was penetrated and it was owned by a utility, even if the control room of the plant was penetrated by the attackers, this is very far from saying that the control center of the utility itself was penetrated. A control room controls a single plant, period. A control center can control multiple plants, but more often it is much more comprehensive. At utilities that are designated Balancing Authorities by NERC, the control centers balance load (demand for power) and supply (generating assets as well as power generated elsewhere that is “imported” on transmission lines) in  real time – if they aren’t balanced, then bad things happen and some of the lights may go out. So whether or not a generation asset is owned by a utility, even if it is so owned and even if the utility’s control room is penetrated, that doesn’t mean there is any higher likelihood that the attackers would be able to get into the utility’s control center, than if the control room hadn’t been penetrated in the first place.

But some of DHS’ statements, quoted by the WSJ, deliberately imply that control centers were compromised. In the first article (published July 24), the following appears: “’They got to the point where they could have thrown switches’ and disrupted power flows, said Jonathan Homer, chief of industrial-control-system analysis for DHS.” You can’t disrupt power flows in the control room of a generating plant; the only thing you can do there is affect the generator(s) itself, possibly shutting it down. Only in a utility’s control center can you disrupt power flows.

DHS went even further in today’s WSJ article, saying:

In March, Homeland Security and the FBI pinned responsibility on a Russian group, often called Dragonfly or Energetic Bear, for intrusions into utilities that gave attackers remote access to critical industrial-control systems, called SCADA. These systems govern power flows and keep electricity supplies balanced with demand and thus prevent blackouts.

“They’ve had access to the button but they haven’t pushed it,” said Jonathan Homer, Homeland Security’s chief of industrial control system analysis.

SCADA systems aren’t found in power plants or wind farms. In the electric power industry, SCADA systems are only found in utility control centers, although they are usually called Energy Management Systems (EMS) there. So today, DHS - and specifically Mr. Homer - has stated that at least two utility control centers were compromised (penetrated, accessed, whatever). Of course, this means that the control networks were compromised (since SCADA systems are always on a separate control network, at least in the power industry). And Mr. Homer adds a nice little flourish by implying that the Russians have placed malware in those SCADA systems, ready to throw the US into darkness on a single word from Vladimir Putin.

Now that I think of it, this is the most depressing quote of all from DHS. After two deliberate repudiations of this idea by DHS spokespeople (see the second paragraph above), Mr. Homer is still saying the sky is falling; we should all head for the country with our guns and appropriate some property, where we can practice subsistence farming.

2. A penetrating analysis
And now there’s the word “penetrate”. Improper use of this word has gotten the US government in trouble before.[iii] Here, the problem is that DHS talked of “utilities” being “penetrated”, without saying what was penetrated. Putting aside the fact that true utilities probably weren’t penetrated in any way, the fact is that most power assets (and all utility main offices) have separate IT and OT networks. Penetration of the IT network at a generating plant is of course unfortunate, but in all but perhaps the smallest generating plants and wind farms (and in all utility offices), there is strict separation between the IT and OT networks, and it would be very difficult, although not impossible, for an attacker who had penetrated the IT network to then pivot to penetrate the OT network.

Yet DHS says that three or four “utilities” were “accessed”, although they’re saying that in only one case (the wind farm) was the control network (which is the OT network) accessed. This means that a few utility IT networks were penetrated by the attackers. Of course, this is a bad thing, but it certainly doesn’t justify the alarming statements by Mr. Homer in today’s article. IT networks don’t control power flows.

3. Who were the “victims”?
DHS uses the word “victims” very carelessly in their statements (at least I hope it was careless. If it wasn’t, we’re all victims of fraud). In the first WSJ article, the DHS briefers were quoted as saying there were “hundreds of victims”. They obviously weren’t referring to the two wind turbines that had their control systems penetrated. They also weren’t referring to the three or four “utilities” (which probably means generating plants owned by IPPs) whose IT networks were compromised. So what did they mean?

In the DHS webinar that I attended on July 24, they tried to make clear that a “victim” was an organization that was targeted or compromised. So that makes around 200 or more organizations that the Russians tried to break into but didn’t. Let’s stop here for a moment. DHS is saying that hundreds of organizations were targeted, but at most 3 or 4 were compromised, meaning that the campaign had a two percent success rate, at the very best. Is this going to set the vodka glasses clinking in St. Petersburg and Moscow? I don’t really think so; I think some official is going to get a phone call from his or her irritated boss, asking “Just how much did you say this whole thing is costing us, anyway?” My guess is there’s almost no American industry that you could target with an intensive two-year hacking campaign, that wouldn’t yield at least a two percent success rate.

But I digress. We were asking who these “hundreds” of victims are. We know they were almost all just targeted, not penetrated. But what kind of organizations were they? Were they power market participants, as again DHS implies more than once[iv]? That is highly unlikely, given a number of other things DHS said. They must mean that hundreds of vendors and “utilities” were targeted. True, the three or four organizations that were penetrated were all “utilities”. But the majority of the organizations that were targeted were almost surely vendors (including probably IT services vendors), and probably the majority of the rest were IT networks of utilities. But even calling vendors “targets” is very problematic. The Russians were aiming to obtain the ability to control assets that are essential to the US power grid, not a bunch of vendors. They decided that vendors were the best way to get into these assets (and I would agree with them in that judgment, since utilities and most IPPs have very good security for their own networks, but of course their vendors are another story).

I’d like to emphasize something else: It is very likely that even the three or four generation assets that were compromised (three just on the IT network side) were very small. This means that, even if all of their OT networks were compromised and all of the were plants taken down by the Russians simultaneously - and even if they all were very close to one another - there would have been zero impact on the grid, since the Independent System Operators and Regional Transmission Operators that actually run the grid[v] would easily be able to make up for these power losses from other sources - if they even noticed them in the first place.

Not only would there have been no immediate grid impact, but there would have been close to zero chance of the simultaneous loss of these four plants leading to a cascading outage, even if all four were actually 2500-megawatt behemoths. This is why I said previously that I see no possibility of a cyber attack that is purely focused on generation causing a major grid outage, cascading or not (for that matter, I see close to zero possibility that any purely cyber attack could cause a major outage).

I’d like to add one postscript to this post (as well as my previous three posts on this subject): There are at least two journalists on the energy cyber beat who actually believe in waiting until they have gathered and understand all the facts before they publish anything, even though government officials might be encouraging them to rush to print with a horror story. I’m referring to Blake Sobczak and Peter Behr of the online publication Energy and Environment News.

At least Blake had attended the original DHS briefing on Monday, July 23, and after the first WSJ article came out the next day, he and I talked for about an hour on this topic. I thought I was disappointing him because I spent so much time talking about the many areas of uncertainty that still needed to be resolved, before we drew any conclusions about the import of these briefings.

As it turns out, he was as skeptical as I was, and he and Blake doggedly talked to a number of people over the rest of that week and early last week. They read DHS’ first walk back attempt, which said that only a small generation plant had been compromised. They also checked with Congressional staffers, who confirmed that DHS’ briefings to them had also emphasized the walk back. And they finally published their first article on the whole affair last Tuesday, a whole week after the first WSJ story. They followed it up the next day with an article on the briefing in New York, which Blake attended. Both articles emphasized the large scale of the Russian threat and the fact that it’s continuing, but they also both emphasized that the Russians haven’t achieved their goal of gaining a foothold in U.S. grid control centers. They haven’t even come close.

I hope you don’t think I’m trying to be easy on the Russians in any of these comments. I think it’s outrageous that they undertook – and continue to undertake – these attacks. And I think it’s even more outrageous that a certain individual at the top of the U.S. government, who clearly has a good relationship with Vladimir Putin, hasn’t taken it upon himself to tell the latter person that both the grid and electoral system attacks need to stop today – because there are certainly a lot of good non-military weapons still left in the U.S. arsenal to punish any further attacks.

But it’s also reprehensible that DHS officials and staff members have both misrepresented the Russian threat to the grid and allowed much wilder misrepresentations to be published, without any public statement specifically repudiating them. I am sure they think that they’re serving the greater good with these exaggerations (and their very impressive and dogged investigations are the only reason we’re having this conversation in the first place), but I can assure them that their statements and inactions are only harming the cause of grid security, not helping it.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. And if you’re a security vendor to the power industry, TALLC can help you by developing marketing materials, delivering webinars, etc. To discuss any of this, you can email me at the same address.                    

[i] Because the WSJ’s web site is behind a paywall, you might have a problem reading this link. Since I have the article in hard copy, send me an email if you would like to see it in scanned form.

[ii] Although there are some generating plants (including some wind farms) that are owned by holding companies that also own utilities. But because of deregulation of generation, it is very rare that a utility itself owns generation assets nowadays.

[iii] The use of American military forces in Viet Nam was “sanctioned” by the 1964 Gulf of Tonkin Resolution, which was occasioned by the Gulf of Tonkin Incident. In that incident, North Vietnamese patrol boats were alleged to have fired torpedoes at a US warship in international waters, while the North Vietnamese said the ship had actually penetrated their waters. In the official Navy report on the incident, the words were used (and I just read this a few years later in some magazine. I haven’t been able to verify it through an online search) “Penetration, no matter how slight…is sufficient to constitute an offense.”  Supposedly, these words were copied verbatim from the US military’s definition of rape.

[iv] And if they didn’t mean this at all, why didn’t they try to correct the press reports – including the WSJ’s, of course – that implied that hundreds of “utilities” had been compromised?

[v] And of course, when I have talked of “the grid” in this post – as well as many other posts – I should more correctly say the grids, since there are four Interconnects in North America: Eastern, Western, Texas and Quebec. You could completely take down any one of these and have zero direct impact on any of the others.

Wednesday, August 1, 2018

NATF’s CIP-013 guidance

Recently, the North American Transmission Forum released their “CIP-013-1 Implementation Guidance”, which they have submitted for possible NERC approval as official guidance (although of course NERC won’t approve the content of any guidance document, only say that it was properly drafted, or something like that). While I think this is an interesting (and useful) document, it isn’t in any way a complete guide to how to comply with CIP-013, which it seems to aspire to being.

The document first describes how a NERC entity can a) ask vendors to get an “independent assessment…from a certified auditor” to evaluate the vendor’s controls to meet the criteria found in CIP-013 R1.2 (what I call the “six things”. These were originally mandated by FERC in their Order 829); b) assess the results; and c) document any mitigating actions that need to be performed by the vendor.

Fair enough. I agree that this would be a good thing to do, and would help with compliance with R1.2 and R2. But the document goes on to say “A Responsible Entity could choose to include this process as the only R1 process in its overall R1 plan or it could be one of several processes that meet the R1 requirements in its overall R1 plan.” So it is saying that implementing this process alone will constitute compliance with R1 in its entirety. Is that really the case?

Let’s look at what R1 requires, sticking just to what the words say. First, R1 itself says “Each Responsible Entity shall develop one or more documented supply chain cyber security risk management plan(s) for high and medium impact BES Cyber Systems. The plan(s) shall include:” (this is followed by R1.1 and R1.2). Let’s stop there and see what this says about what we should do. It says we should develop a plan that needs to include R1.1 and R1.2. Fair enough, what should this plan contain?

R1.1 reads “One or more process(es) used in planning for the procurement of BES Cyber Systems to identify and assess cyber security risk(s) to the Bulk Electric System from vendor products or services resulting from: (i) procuring and installing vendor equipment and software; and (ii) transitions from one vendor(s) to another vendor(s).” In other words, your plan should a) identify and assess[i] cyber security risks to the BES from procuring vendor equipment and software; b) identify and assess cyber security risks to the BES from installing vendor equipment and software; and c) identify and assess cyber security risks to the BES from transitions between vendors.

NATF is saying the entity can choose to designate the third-party vendor assessment process as the only R1 process in its plan – meaning that the plan could consist entirely of this process. Since that’s the case, then this process should also be enough to comply with R1.1. How well does it do this, for each of the three plan elements just identified from the R1.1 wording?

Let’s start with element a) above. Will implementing the vendor assessment process help the entity identify and assess risks from procuring vendor equipment and software? How could it possibly do that? A process to get vendors to have third-party assessments isn’t a process to identify and assess risks from procurement. This might be part of the mitigation plan for some of the risks identified, but that’s it. In itself, this process neither identifies nor assesses any procurement risks. The same thing can be said for element b); that is, the process neither identifies nor assesses any installation risks (and I can’t think of how it would even be part of the mitigation plan for installation risks). And as for c)…well, the vendor assessment process has nothing to do with identifying, assessing or mitigating risks of transition between vendors. So we can safely say that including the third-party vendor assessment process in your plan won’t in any significant way help with compliance with R1.1.

R1.2 reads “One or more process(es) used in procuring BES Cyber Systems that address the following..” Of course, we should preface this with “The plan(s) shall include…” from R1. Let’s ask the same question here. Does the third-party vendor assessment process help the entity comply with R1.2?

It does that partially, but not entirely. To understand what I mean, you need to understand why the six items in R1.2 are there: It’s because FERC explicitly ordered that these six things be mandated by the new supply chain security standard. But these six things aren’t included in CIP-013 as prescriptive requirements; rather, they’re supposed to be included in the overall plan mandated by R1 (of course, this is a good thing, since having six prescriptive requirements in a standard that is supposed to be entirely based on having a plan would create a very uncomfortable mix, both for entities that need to comply and for auditors).

And since the overall plan is a plan for risk management, the six items need to all be considered as mitigations of particular risks. Therefore, what R1.2 is really saying is “The risks of which these six items are mitigations must be included in your plan”. Of course, most of the risks in the plan will be those that the entity itself identifies in R1.1; but the six risks behind the six R1.2 items have to be included in the plan as well – and therefore have to be mitigated.

For example, R1.2.3 reads “Notification by vendors when remote or onsite access should no longer be granted to vendor representatives”. This is a mitigation of a risk[ii] which could be (fairly roughly) said to be “the risk that a terminated vendor employee will take out his unhappiness on your BES Cyber Systems”. So we will list this as the risk “behind” R1.2.3; and we’ll do the same for each of the other five items in R1.2.

Let’s say we’ve listed the risks behind each of the items in R1.2. Of course, the plan needs to include mitigation of each risk. Can the third-party vendor assessment discussed by NATF serve as the mitigation for these six risks? It can certainly provide part of the mitigation, but it isn’t the entire mitigation process by any means.

To go back to our example, if the assessor determines that the vendor does in fact endeavor to provide notice within 24 hours – and sooner if possible – whenever an employee who had access to a customer’s systems leaves, this would certainly be a big step toward mitigating the “risk that a terminated vendor employee will take out his unhappiness on your BES Cyber Systems”. But is this all that’s required? What happens if there’s a big management change at the vendor soon after the assessment, and the new management doesn’t care at all about the notification policy? If this happens, what if anything does the entity need to do to make sure this risk is mitigated?

There are a lot of things the entity can then do: When they first realize the policy has changed, they can call up the vendor and threaten to terminate their business – and then make good on that threat if the vendor doesn’t change their policy. And if it isn’t possible to terminate this vendor (which it often isn’t), the entity can at least terminate all remote access for this vendor and require that representatives who want physical access be accompanied at all times by an employee of the entity. These aren’t easy steps to take, but the point is that the entity has to have a plan to mitigate each risk, and simply getting a good assessment won’t always be enough to mitigate a risk.

At this point, someone might point out that “vendor performance and adherence to a contract” is explicitly stated to be out of scope in CIP-013 R2 (and implicitly in R1 as well). So if the entity had not only had the vendor get the third-party assessment, but they had also had them agree in their contract to implement whatever mitigations were indicated in the assessment, the entity wouldn’t be found in violation if the vendor didn’t perform.

This is very true. But this is very different from saying that, if the vendor doesn’t perform an obligation in their contract, the entity doesn’t need to take any more steps to mitigate the risk in question. The goal of CIP-013 is risk management, not vendor assessment and not contract language. Those are both useful tools for mitigating risk, but the risk must be mitigated regardless of what the vendor does or doesn’t do.

So is NATF’s statement that “A Responsible Entity could choose to include this process (i.e. the third-party vendor assessment) as the only R1 process in its overall R1 plan” accurate? No. This process can contribute very little to R1.1 compliance; for R1.2, it can contribute maybe half of the mitigation required. This process is far from being the “only R1 process” in an entity’s CIP-013 R1 plan. However, the clause that follows this says “or it could be one of several processes that meet the R1 requirements in its overall R1 plan.” I certainly agree with that, although I think there will usually be a good number of processes in an entity’s CIP-013 R1 plan – not just “several”. I think there will need to be a process to mitigate each of the risks the entity identifies in R1.1; and I would think that any entity that wanted to put together a credible plan would need to mitigate more than “several” risks.

Now let’s move on to R2. This requirement reads in its entirety “Each Responsible Entity shall implement its supply chain cyber security risk management plan(s) specified in Requirement R1.” And since NATF has stated that the third-party vendor assessment process by itself meets an entity’s compliance obligations under R1, it’s not hugely surprising that they say that implementing that process is all that’s needed to comply with R2, specifically “The documentation maintained by the entity would demonstrate that the entity used its supply chain cyber security risk management plan required in R1 and, therefore, met its requirements under R2.”

So is NATF’s statement true? Oddly, I’d say it is. There are two cases here. First, suppose that the entity’s auditor has already decided that just describing the third-party vendor assessment process constitutes compliance with R1. If the R1 plan has been found compliant, then the R2 implementation of that plan will also be found compliant (assuming the plan was actually implemented). Second, suppose the entity’s R1 plan hasn’t been found compliant. Even then, the R2 implementation might still be found compliant, since R2 doesn’t require that the entity rethink its R1 plan, but simply implement it.

So the important question is whether the R1 plan will be found to be compliant, if the entity has simply followed NATF’s advice and based their whole plan on a third-party vendor assessment. That is a real question, but in my opinion there’s an even more important question that needs to be asked. Since this post is already pretty long, I will save it for the next[iii] post. 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. To discuss any of this, you can email me at the same address.

[i] As I first noticed when I wrote a post that analyzed R1.1 in excruciating detail, the drafting team seems to have left out the word “mitigate” here. There’s certainly no point in developing a plan to just identify and assess risks, without doing anything to mitigate them! I don’t particularly blame the SDT for this error, since they were under a really tight deadline from FERC to develop and approve this standard in one year; and CIP-013 is probably the first non-military mandatory supply chain cyber security standard in the world. Unfortunately, this also seems to have happened with CIP-014, which faced a three-month deadline. The result is that both standards are in some way un-auditable. I’ll elaborate this more in a post soon on CIP-014, as well as in my next post on CIP-013.

[ii] I have a problem with R1, in that it uses the word “risk” where I would use “threat”. Forgetting about CIP for the moment and just thinking of the process a CISO goes through in deciding how to allocate her budget for the year, she must first identify a threat like malware, then assess the degree of risk it poses. Once she has identified all of the major threats and decided their degree of risk, then she can allocate her resources to particular threats based on their individual risks. If you substitute the word risk for threat in the above two sentences, you’ll see that “risk” will be used with two entirely different meanings – which of course is very unhelpful. But CIP-013 uses the word risk, and that’s because FERC used that word in Order 829. I’m not going to fight that here, but it will become important when an entity actually sets out to comply with CIP-013; then they will have to have these terms straight.

[iii] Even though I believe this will be the topic of my next post, there have been a number of other times when I’ve been sure I knew what my next post would be, only to have some other event intervene that I thought was more important to address. So if my next post is on a completely different topic, you can’t sue me.

Monday, July 30, 2018

A smoking gun

A long-time colleague wrote in to me last Friday regarding Thursday’s post. He pointed out to me that, not only were the statements from DHS staff members in the briefings on the Russian hacking of the grid misleading, but at least two slides they showed had text that directly conflicted with the statement from a DHS spokesperson, which I had quoted in Thursday’s post: “While hundreds of energy and non-energy companies were targeted, the incident where they gained access to the industrial control system was a very small generation asset that would not have had any impact on the larger grid if taken offline.”

Yet here are the statements from slides 18 and 19 of the presentation at the Wednesday briefing:

  • (slide 18) “Used initial compromised vendor to access several U.S. energy utilities and IT service providers”
  • (slide 19) “Leveraged early victim to gain entry to two previously accessed utilities and one new victim”

The combination of these two statements leads to the conclusion that a minimum of three “energy utilities” were “accessed”, as opposed to the one small generating plant (which most likely wasn’t owned by an electric utility at all) in the DHS spokesperson’s statement.[i] If DHS wants to come out and say the spokesperson’s statement was wrong and three utilities were actually accessed, so be it. But I certainly haven’t heard of that happening (Kristjen N, if that is in fact the case, please email me at the address below).

If even three electric utilities had their control centers (and presumably their EMS systems) compromised, that would be a bad thing, since a simultaneous attack on all three could possibly lead to three widespread outages, although probably not a cascading outage (like in 2003); there would then be justification for raising the alarm flags. But here we’re talking about the control room of a single very small generating plant that by DHS’ own admission doesn’t have any real impact on the “larger grid”. In my opinion, this fact, combined with the fact that hundreds of “utilities” were attacked by the Russians, leads me to believe that the industry’s defenses are in pretty good shape, not the exact opposite. This is a wakeup call, but not to cyber weakness in general at utilities. Rather, it’s a call for all utilities and IPPs to beef up defenses against supply chain attacks (as I pointed out in the first post in this series).

Yet the idea that the exact opposite is indeed the case seems to be spreading very rapidly. I had two new articles called to my attention today, including this one contributed by John Hargrove of Sam Houston Electric Coop, and this one contributed by another friend. I’m sure there will be others. Both of these articles include a quote (in fact the same one, even though it was delivered by email) by Robert Lee of Dragos. Taking DHS at their word that “utilities” had had their “control rooms” penetrated[ii], Robert points out that the activities in question – purely reconnaissance – wouldn’t be enough to be able to cause an outage.

However, Robert didn’t need to go this far.  It turns out no utility control centers were penetrated, period. And even if the generating plant whose control room was penetrated was a very large one, and even if several similar generating plants were also penetrated, this would be far from a danger to the grid itself, as discussed in this post.

In other words, even though the DHS people who put together the briefings (and didn’t provide any immediate corrections when the alarming news stories started flying) were only trying to call attention to a problem, by exaggerating what had happened they have damaged their credibility for future advisories. I hope it isn’t fatally damaged, because they (specifically, the ICS-Cert) do a lot of really excellent work! 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. And if you’re a security vendor to the power industry, TALLC can help you by developing marketing materials, delivering webinars, etc. To discuss any of this, you can email me at the same address.         

[i] I suppose you could interpret “accessed” to mean the attackers got into the IT network of the utility, but not the OT network; but of course this doesn’t mean they’re any closer to achieving their goal of being able to manipulate control systems (which are on the OT network, or should be) to cause an outage. In any case, if this is what DHS meant by “access”, they certainly have never stated that.

[ii] As I pointed out in my first post about this problem, to speak of an electric utility’s “control room” is essentially a non-sequitur, such as speaking of the Pope’s yarmulke. A control room controls an individual generating plant or substation, and is usually located at that asset. Utilities have control centers, which control many assets that generate and transmit power, as well as the assets like distribution substations that deliver that power to customers. But the single small generating plant that was actually penetrated is almost certainly not owned by a utility (most plants are owned by independent power producers, especially the small ones), and in any case its control room doesn’t control anything more than the plant itself.

Saturday, July 28, 2018

The Russians are coming! The Russians are coming!

The above is the title of a really hilarious film I remember from my childhood, in which – at the height of the Cold War – a Russian sub runs aground near a small island off New England. Crew members head into town to find a boat to pull them off, and in the process some of the townspeople become convinced they are the spearhead of an invasion, and almost ignite World War III. It seems we have a modern-day version of this film playing out with DHS, since their exaggeration of the success of Russian hackers in penetrating the US power grid is unfortunately becoming a fast-spreading meme that may be unstoppable.[i]

Here is an excerpt from an article that appeared on the New York Times website on Friday:

This week, the Department of Homeland Security reported that over the last year, Russia’s military intelligence agency had infiltrated the control rooms of power plants across the United States. In theory, that could enable it to take control of parts of the grid by remote control.[ii]

Yesterday evening, after seeing this article, I sent the following letter to the news editors of the Times (which by the way I think is a great paper, very dedicated to finding the truth. But being dedicated to the truth doesn’t mean you can’t be misled by people in government who have more information than you do, and have exaggerated an already-serious situation, for whatever reason):

Please stop promoting the story that the Russians have substantially penetrated the US power grid. While that was the tenor of DHS' initial briefing, it turns out DHS was wildly exaggerating. While hundreds of power plants (not utilities per se) were targeted by the Russians, they succeeded in penetrating the control systems of exactly one very small generating plant, which by DHS' own admission would have no significant impact on the power grid:

"While hundreds of energy and non-energy companies were targeted, the incident where they gained access to the industrial control system was a very small generation asset that would not have had any impact on the larger grid if taken offline." (this is a quote from DHS spokesperson Lesley Fulop, which appeared in an article on Power magazine's website on July 24)

Of course, it is true that the Russians are targeting the power grid constantly, and as your article points out, this has stepped up lately as election hacking seems to have fallen out of fashion in Russia. However, so far they have made no significant headway. Electric utilities in the US have invested very heavily in cyber security and continue to do so. While the utilities need to step up their efforts even further - and they are doing so - there is no need for Americans to lose sleep worrying whether a major cyber attack will bring down the US power grid. It isn't going to happen.

I sincerely doubt we’ve heard the end of this story.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. And if you’re a security vendor to the power industry, TALLC can help you by developing marketing materials, delivering webinars, etc. To discuss any of this, you can email me at the same address.         

[i] The biggest difference between the film and the current situation is that the Russian hackers are actually malign – or at least they’re being paid to be such. The Russian sailors in the film had nothing but good will toward the Americans, and the film had a very happy ending.

One reason the film was so funny is that one of its stars was Jonathan Winters, perhaps the funniest man that ever lived. He could have read the phone book and had you in stitches.

[ii] While the article does go on to point out that the hackers made no attempt to actually take control of the plants (which is also what DHS said), it repeats the canard that a large number of “control rooms” were penetrated – leaving open the possibility that malware has been implanted, so that just a single future signal would bring down scores of generating plants. This is simply not true. One very small plant was penetrated, and I’m sure it’s probably been made one of the most secure power plants in the world after this incident was discovered.