Obviously, this one isn't going to go away very soon

After writing four posts in a row on the Russian hacking campaign against the US power grid – three of which were really about how it was characterized and reported by DHS and the press – I thought I was finished with this issue, and I could get back to writing about what I think is really of national importance, like CIP-013 (and I’m not kidding about CIP-013 being of national importance, since all the Russian attacks were supply chain attacks, and they continue to this day).

Specifically, I thought that, after a spokesperson for DHS admitted that the only control network that was penetrated was that of a “very small” generating plant, and after a high level DHS official further qualified that statement by saying - at a meeting where the Secretaries of DHS and DoE were in the room, as well as the US Vice President – that just two wind turbines on a wind farm were compromised (not even the whole wind farm), everyone involved in the misleading statements, and the erroneous reporting of them, would have felt properly shamed and would be more careful in the future.

Thus, I was surprised – to say the least – to read a front-page article in the Wall Street Journal today entitled “U.S. Steps up Grid Defense”,[i] which indicated a) at least one DHS official continues to put out deliberately misleading statements, which contradict the statements of other supposedly official spokespersons for DHS; and b) the same reporter who wrote the original WSJ article that set off this firestorm about two weeks ago doesn’t seem to have changed her narrative of what happened at all, despite DHS’ attempts to walk this back.

I find both conclusions quite disturbing, but I also find b) to be very puzzling. The four possible explanations I can think of are:

1. The reporter has been living in an inaccessible cave since she wrote that article, and therefore missed DHS’ walk backs of the story; or
2. She didn’t understand what the other official DHS spokespeople said when they issued the walk backs; or
3. She was deliberately misled again by the DHS official who made the misleading statements quoted in her first article; or (finally)
4. That DHS official – Jonathan Homer, whose title is Chief of Industrial Control Systems Group, Hunt and Incident Response Team – doesn’t himself understand the walk backs, because of his continued misunderstanding of a few power industry terms and facts.

I’m a fairly charitable person, so I prefer either explanation 2 or 4; of course they could both be true at the same time. So this is hopefully mostly a case of two people not understanding some important facts about, and terms used by, the US utility industry (although some DHS statements were still either deliberately or recklessly misleading). I’m also a very helpful person, so I will try to lay out those facts and terms using language that all can understand (which I didn’t do when I discussed them in previous posts).

1. Who owns the stuff, anyway?

First, most generation assets in the US aren’t owned by utilities, but by independent power producers.[ii] So it was very misleading that DHS’ statements all referred to “utilities” being penetrated. But there were only two assets that they specifically said were “penetrated” by the attackers. One was the wind farm where the control network was penetrated. The other was a combustion turbine plant. DHS didn’t specifically say that a CT was penetrated, but they did display a schematic drawing (which they said in the briefing was a screen shot of a Human-Machine Interface computer, or HMI) of a CT that they said had been obtained by the attackers. It is very unlikely that the wind farm was owned by a utility. It is possible that the CT (presumably a small one, not subject to NERC CIP – which explains the ease with which the attackers obtained the screen shot) was owned by a small municipal or cooperative utility.

2. Control rooms vs. control centers

But if the CT was owned by a small muni or coop, then this points to another problem with DHS’ statements: If a small generating plant was penetrated and it was owned by a utility, even if the control room of the plant was penetrated by the attackers, this is very far from saying that the control center of the utility itself was penetrated. A control room controls a single plant, period. A control center can control multiple plants, but more often it is much more comprehensive. At utilities that are designated Balancing Authorities by NERC, the control centers balance load (demand for power) and supply (generating assets as well as power generated elsewhere that is “imported” on transmission lines) in  real time – if they aren’t balanced, then bad things happen and some of the lights may go out. So whether or not a generation asset is owned by a utility, even if it is so owned and even if the utility’s control room is penetrated, that doesn’t mean there is any higher likelihood that the attackers would be able to get into the utility’s control center, than if the control room hadn’t been penetrated in the first place.

But some of DHS’ statements, quoted by the WSJ, deliberately imply that control centers were compromised. In the first article (published July 24), the following appears: “’They got to the point where they could have thrown switches’ and disrupted power flows, said Jonathan Homer, chief of industrial-control-system analysis for DHS.” You can’t disrupt power flows in the control room of a generating plant; the only thing you can do there is affect the generator(s) itself, possibly shutting it down. Only in a utility’s control center can you disrupt power flows.

DHS went even further in today’s WSJ article, saying:

In March, Homeland Security and the FBI pinned responsibility on a Russian group, often called Dragonfly or Energetic Bear, for intrusions into utilities that gave attackers remote access to critical industrial-control systems, called SCADA. These systems govern power flows and keep electricity supplies balanced with demand and thus prevent blackouts.

“They’ve had access to the button but they haven’t pushed it,” said Jonathan Homer, Homeland Security’s chief of industrial control system analysis.

SCADA systems aren’t found in power plants or wind farms. In the electric power industry, SCADA systems are only found in utility control centers, although they are usually called Energy Management Systems (EMS) there. So today, DHS - and specifically Mr. Homer - has stated that at least two utility control centers were compromised (penetrated, accessed, whatever). Of course, this means that the control networks were compromised (since SCADA systems are always on a separate control network, at least in the power industry). And Mr. Homer adds a nice little flourish by implying that the Russians have placed malware in those SCADA systems, ready to throw the US into darkness on a single word from Vladimir Putin.

Now that I think of it, this is the most depressing quote of all from DHS. After two deliberate repudiations of this idea by DHS spokespeople (see the second paragraph above), Mr. Homer is still saying the sky is falling; we should all head for the country with our guns and appropriate some property, where we can practice subsistence farming.

3. A penetrating analysis

And now there’s the word “penetrate”. Improper use of this word has gotten the US government in trouble before.[iii] Here, the problem is that DHS talked of “utilities” being “penetrated”, without saying what was penetrated. Putting aside the fact that true utilities probably weren’t penetrated in any way, the fact is that most power assets (and all utility main offices) have separate IT and OT networks. Penetration of the IT network at a generating plant is of course unfortunate, but in all but perhaps the smallest generating plants and wind farms (and in all utility offices), there is strict separation between the IT and OT networks, and it would be very difficult, although not impossible, for an attacker who had penetrated the IT network to then pivot to penetrate the OT network.

Yet DHS says that three or four “utilities” were “accessed”, although they’re saying that in only one case (the wind farm) was the control network (which is the OT network) accessed. This means that a few utility IT networks were penetrated by the attackers. Of course, this is a bad thing, but it certainly doesn’t justify the alarming statements by Mr. Homer in today’s article. IT networks don’t control power flows.

4. Who were the “victims”?

DHS uses the word “victims” very carelessly in their statements (at least I hope it was careless. If it wasn’t, we’re all victims of fraud). In the first WSJ article, the DHS briefers were quoted as saying there were “hundreds of victims”. They obviously weren’t referring to the two wind turbines that had their control systems penetrated. They also weren’t referring to the three or four “utilities” (which probably means generating plants owned by IPPs) whose IT networks were compromised. So what did they mean?

In the DHS webinar that I attended on July 24, they tried to make clear that a “victim” was an organization that was targeted or compromised. So that makes around 200 or more organizations that the Russians tried to break into but didn’t. Let’s stop here for a moment. DHS is saying that hundreds of organizations were targeted, but at most 3 or 4 were compromised, meaning that the campaign had a two percent success rate, at the very best. Is this going to set the vodka glasses clinking in St. Petersburg and Moscow? I don’t really think so; I think some official is going to get a phone call from his or her irritated boss, asking “Just how much did you say this whole thing is costing us, anyway?” My guess is there’s almost no American industry that you could target with an intensive two-year hacking campaign, that wouldn’t yield at least a two percent success rate.

But I digress. We were asking who these “hundreds” of victims are. We know they were almost all just targeted, not penetrated. But what kind of organizations were they? Were they power market participants, as again DHS implies more than once[iv]? That is highly unlikely, given a number of other things DHS said. They must mean that hundreds of vendors and “utilities” were targeted. True, the three or four organizations that were penetrated were all “utilities”. But the majority of the organizations that were targeted were almost surely vendors (including probably IT services vendors), and probably the majority of the rest were IT networks of utilities. But even calling vendors “targets” is very problematic. The Russians were aiming to obtain the ability to control assets that are essential to the US power grid, not a bunch of vendors. They decided that vendors were the best way to get into these assets (and I would agree with them in that judgment, since utilities and most IPPs have very good security for their own networks, but of course their vendors are another story).

I’d like to emphasize something else: It is very likely that even the three or four generation assets that were compromised (three just on the IT network side) were very small. This means that, even if all of their OT networks were compromised and all of the plants were taken down by the Russians simultaneously (and even if they all were very close to one another), there would have been zero impact on the grid, since the Independent System Operators and Regional Transmission Operators that actually run the grid[v] would easily be able to make up for these power losses from other sources - if they even noticed them in the first place.

Not only would there have been no immediate grid impact, but there would have been close to zero chance of the simultaneous loss of these four plants leading to a cascading outage, even if all four were actually 2500-megawatt behemoths. This is why I said previously that I see no possibility of a cyber attack that is purely focused on generation causing a major grid outage, cascading or not (for that matter, I see close to zero possibility that any purely cyber attack could cause a major outage).

P.S.

I’d like to add one postscript to this post (as well as my previous three posts on this subject): There are at least two journalists on the energy cyber beat who actually believe in waiting until they have gathered and understand all the facts before they publish anything, even though government officials might be encouraging them to rush to print with a horror story. I’m referring to Blake Sobczak and Peter Behr of the online publication Energy and Environment News.

At least Peter had attended the original DHS briefing on Monday, July 23, and after the first WSJ article came out the next day, he and I talked for about an hour on this topic. I thought I was disappointing him because I spent so much time talking about the many areas of uncertainty that still needed to be resolved, before we drew any conclusions about the import of these briefings.

As it turns out, he was as skeptical as I was, and he and Blake doggedly talked to a number of people over the rest of that week and early last week. They read DHS’ first walk back attempt, which said that only a small generation plant had been compromised. They also checked with Congressional staffers, who confirmed that DHS’ briefings to them had also emphasized the walk back. And they finally published their first article on the whole affair last Tuesday, a whole week after the first WSJ story. They followed it up the next day with an article on the briefing in New York, which Blake attended. Both articles emphasized the large scale of the Russian threat and the fact that it’s continuing, but they also both emphasized that the Russians haven’t achieved their goal of gaining a foothold in U.S. grid control centers. They haven’t even come close.

P.P.S.

I hope you don’t think I’m trying to be easy on the Russians in any of these comments. I think it’s outrageous that they undertook – and continue to undertake – these attacks. And I think it’s even more outrageous that a certain individual at the top of the U.S. government, who clearly has a good relationship with Vladimir Putin, hasn’t taken it upon himself to tell the latter person that both the grid and electoral system attacks need to stop today – because there are certainly a lot of good non-military weapons still left in the U.S. arsenal to punish any further attacks.

But it’s also reprehensible that DHS officials and staff members have both misrepresented the Russian threat to the grid and allowed much wilder misrepresentations to be published, without any public statement specifically repudiating them. I am sure they think they’re serving the greater good with these exaggerations (and their very impressive and dogged investigations are the only reason we’re having this conversation in the first place), but I can assure them that their statements and inaction are only harming the cause of grid security, not helping it.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. And if you’re a security vendor to the power industry, TALLC can help you by developing marketing materials, delivering webinars, etc. To discuss any of this, you can email me at the same address.                    

---

[i] Because the WSJ’s web site is behind a paywall, you might have a problem reading this link. Since I have the article in hard copy, send me an email if you would like to see it in scanned form.

[ii] Although there are some generating plants (including some wind farms) that are owned by holding companies that also own utilities. But because of deregulation of generation, it is very rare that a utility itself owns generation assets nowadays.

[iii] The use of American military forces in Viet Nam was “sanctioned” by the 1964 Gulf of Tonkin Resolution, which was occasioned by the Gulf of Tonkin Incident. In that incident, North Vietnamese patrol boats were alleged to have fired torpedoes at a US warship in international waters, while the North Vietnamese said the ship had actually penetrated their waters. In the official Navy report on the incident, the words were used (and I just read this a few years later in some magazine. I haven’t been able to verify it through an online search) “Penetration, no matter how slight…is sufficient to constitute an offense.”  Supposedly, these words were copied verbatim from the US military’s definition of rape.

[iv] And if they didn’t mean this at all, why didn’t they try to correct the press reports – including the WSJ’s, of course – that implied that hundreds of “utilities” had been compromised?

[v] And of course, when I have talked of “the grid” in this post – as well as many other posts – I should more correctly say the grids, since there are four Interconnects in North America: Eastern, Western, Texas and Quebec. You could completely take down any one of these and have zero direct impact on any of the others.