After
writing four posts in a row on the Russian hacking campaign against the US
power grid – three of which were really about how it was characterized and
reported by DHS and the press – I thought I was finished with this issue, and I
could get back to writing about what I think is really of national importance,
like CIP-013 (and I’m not kidding about CIP-013 being of national importance,
since all the Russian attacks were supply chain attacks, and they continue to
this day).
Specifically,
I thought that, after a spokesperson for DHS
admitted that the
only control network that was penetrated was that of a “very small” generating
plant, and after a high level DHS official further qualified that statement by
saying - at a
meeting
where the Secretaries of DHS and DoE were in the room, as well as the US Vice
President – that just two wind turbines on a wind farm were compromised (not
even the whole wind farm), everyone involved in the misleading statements, and
the erroneous reporting of them, would have felt properly shamed and would be
more careful in the future.
Thus, I was
surprised – to say the least – to read a front-page article in the
Wall Street Journal today entitled “
U.S.
Steps up Grid Defense”,
[i] which
indicated a) at least one DHS official continues to put out deliberately
misleading statements, which contradict the statements of other supposedly
official spokespersons for DHS; and b) the same reporter who wrote the
original
WSJ article that set off this firestorm about two weeks ago doesn’t seem to
have changed her narrative of what happened at all, despite DHS’ attempts to
walk this back.
I find both
conclusions quite disturbing, but I also find b) to be very puzzling. The four
possible explanations I can think of are:
- The reporter has been living in an inaccessible cave since
she wrote that article, and therefore missed DHS’ walk backs of the story;
or
- She didn’t understand what the other official DHS
spokespeople said when they issued the walk backs; or
- She was deliberately misled again by the DHS official who
made the misleading statements quoted in her first article; or (finally)
- That DHS official – Jonathan Homer, whose title is Chief
of Industrial Control Systems Group, Hunt and Incident Response Team – doesn’t
himself understand the walk backs, because of his continued
misunderstanding of a few power industry terms and facts.
I’m a fairly
charitable person, so I prefer either explanation 2 or 4; of course they could
both be true at the same time. So this is hopefully mostly a case of two people
not understanding some important facts about, and terms used by, the US utility
industry (although some DHS statements were still either deliberately or
recklessly misleading). I’m also a very helpful person, so I will try to lay
out those facts and terms using language that all can understand (which I didn’t
do when I discussed them in previous posts).
1. Who owns the stuff, anyway?
First, most
generation assets in the US aren’t owned by utilities, but by independent power
producers.
[ii] So it
was very misleading that DHS’ statements all referred to “utilities” being
penetrated. But there were only two assets that they specifically said were
“penetrated” by the attackers. One was the wind farm where the control network
was penetrated. The other was a combustion turbine plant. DHS didn’t
specifically say that a CT was penetrated, but they did display a schematic
drawing (which they said in the briefing was a screen shot of a Human-Machine
Interface computer, or HMI) of a CT that they said had been obtained by the
attackers. It is very unlikely that the wind farm was owned by a utility. It is
possible that the CT (presumably a small one, not subject to NERC CIP – which
explains the ease with which the attackers obtained the screen shot) was owned
by a small municipal or cooperative utility.
2. Control rooms vs. control centers
But if the
CT was owned by a small muni or coop, then this points to another problem with
DHS’ statements: If a small generating plant was penetrated and it was owned by
a utility, even if the control room of the plant was penetrated by the
attackers, this is very far from saying that the control center of the utility itself was penetrated. A control room
controls a single plant, period. A control center can control multiple plants,
but more often it is much more comprehensive. At utilities that are designated
Balancing Authorities by NERC, the control centers balance load (demand for
power) and supply (generating assets as well as power generated elsewhere that
is “imported” on transmission lines) in
real time – if they aren’t balanced, then bad things happen and some of
the lights may go out. So whether or not a generation asset is owned by a
utility, even if it is so owned and even if the utility’s control room is
penetrated, that doesn’t mean there is any higher likelihood that the attackers
would be able to get into the utility’s control center, than if the control
room hadn’t been penetrated in the first place.
But some of
DHS’ statements, quoted by the WSJ, deliberately imply that control centers
were compromised. In the first article (published July 24), the following
appears: “’They got to the point where they could have thrown switches’ and
disrupted power flows, said Jonathan Homer, chief of industrial-control-system
analysis for DHS.” You can’t disrupt power flows in the control room of a
generating plant; the only thing you can do there is affect the generator(s)
itself, possibly shutting it down. Only in a utility’s control center can you
disrupt power flows.
DHS went
even further in today’s WSJ article, saying:
In March, Homeland Security and the FBI
pinned responsibility on a Russian group, often called Dragonfly or Energetic
Bear, for intrusions into utilities that gave attackers remote access to
critical industrial-control systems, called SCADA. These systems govern power
flows and keep electricity supplies balanced with demand and thus prevent
blackouts.
“They’ve had access to the button but
they haven’t pushed it,” said Jonathan Homer, Homeland Security’s chief of
industrial control system analysis.
SCADA
systems aren’t found in power plants or wind farms. In the electric power
industry, SCADA systems are only found in utility control centers, although
they are usually called Energy Management Systems (EMS) there. So today, DHS - and
specifically Mr. Homer - has stated that at least two utility control centers
were compromised (penetrated, accessed, whatever). Of course, this means that
the control networks were compromised (since SCADA systems are always on a
separate control network, at least in the power industry). And Mr. Homer adds a
nice little flourish by implying that the Russians have placed malware in those
SCADA systems, ready to throw the US into darkness on a single word from
Vladimir Putin.
Now that I
think of it, this is the most depressing quote of all from DHS. After two
deliberate repudiations of this idea by DHS spokespeople (see the second
paragraph above), Mr. Homer is still saying the sky is falling; we should all
head for the country with our guns and appropriate some property, where we can
practice subsistence farming.
3. A penetrating analysis
And now
there’s the word “penetrate”. Improper use of this word has gotten the US
government in trouble before.
[iii] Here,
the problem is that DHS talked of “utilities” being “penetrated”, without
saying what was penetrated. Putting aside the fact that true utilities probably
weren’t penetrated in any way, the fact is that most power assets (and all
utility main offices) have separate IT and OT networks. Penetration of the IT
network at a generating plant is of course unfortunate, but in all but perhaps
the smallest generating plants and wind farms (and in all utility offices),
there is strict separation between the IT and OT networks, and it would be very
difficult, although not
impossible,
for an attacker who had penetrated the IT network to then pivot to penetrate
the OT network.
Yet DHS says
that three or four “utilities” were “accessed”, although they’re saying that in
only one case (the wind farm) was the control network (which is the OT network)
accessed. This means that a few utility IT networks were penetrated by the
attackers. Of course, this is a bad thing, but it certainly doesn’t justify the
alarming statements by Mr. Homer in today’s article. IT networks don’t control
power flows.
4. Who were the “victims”?
DHS uses the
word “victims” very carelessly in their statements (at least I hope it was
careless. If it wasn’t, we’re all victims of fraud). In the first WSJ article,
the DHS briefers were quoted as saying there were “hundreds of victims”. They
obviously weren’t referring to the two wind turbines that had their control
systems penetrated. They also weren’t referring to the three or four
“utilities” (which probably means generating plants owned by IPPs) whose IT
networks were compromised. So what did they mean?
In the DHS
webinar that I attended on July 24, they tried to make clear that a “victim”
was an organization that was targeted or
compromised. So that makes around 200 or more organizations that the Russians
tried to break into but didn’t. Let’s stop here for a moment. DHS is saying
that hundreds of organizations were targeted, but at most 3 or 4 were
compromised, meaning that the campaign had a two percent success rate, at the
very best. Is this going to set the vodka glasses clinking in St. Petersburg
and Moscow? I don’t really think so; I think some official is going to get a
phone call from his or her irritated boss, asking “Just how much did you say
this whole thing is costing us, anyway?” My guess is there’s almost no American
industry that you could target with an intensive two-year hacking campaign,
that wouldn’t yield at least a two percent success rate.
But I
digress. We were asking who these “hundreds” of victims are. We know they were
almost all just targeted, not penetrated. But what kind of organizations were
they? Were they power market participants, as again DHS implies more than once
[iv]? That
is highly unlikely, given a number of other things DHS said. They must mean
that hundreds of
vendors and
“utilities” were targeted. True, the three or four organizations that were
penetrated were all “utilities”. But the majority of the organizations that
were targeted were almost surely vendors (including probably IT services
vendors), and probably the majority of the rest were IT networks of utilities.
But even calling vendors “targets” is very problematic. The Russians were
aiming to obtain the ability to control assets that are essential to the US
power grid, not a bunch of vendors. They decided that vendors were the best way
to get into these assets (and I would agree with them in that judgment, since
utilities and most IPPs have very good security for their own networks, but of
course their vendors are another story).
I’d like to emphasize
something else: It is very likely that even the three or four generation assets
that were compromised (three just on the IT network side) were very small. This
means that, even if all of their OT networks were compromised and all of the plants were taken down by the Russians simultaneously (and even if they all
were very close to one another), there would have been zero impact on the grid,
since the Independent System Operators and Regional Transmission Operators that
actually run the grid
[v] would
easily be able to make up for these power losses from other sources - if they
even noticed them in the first place.
Not only
would there have been no immediate grid impact, but there would have been close
to zero chance of the simultaneous loss of these four plants leading to a
cascading outage, even if all four were actually 2500-megawatt behemoths. This
is why I said
previously
that I see no possibility of a cyber attack that is purely focused on
generation causing a major grid outage, cascading or not (for that matter, I
see close to zero possibility that
any
purely cyber attack could cause a major outage).
P.S.
I’d like to
add one postscript to this post (as well as my previous three posts on this
subject): There are at least two journalists on the energy cyber beat who
actually believe in waiting until they have gathered and understand all the
facts before they publish anything, even though government officials might be encouraging
them to rush to print with a horror story. I’m referring to Blake Sobczak and
Peter Behr of the online publication Energy
and Environment News.
At least Peter had attended the original DHS briefing on Monday, July 23, and after the
first WSJ article came out the next day, he and I talked for about an hour on
this topic. I thought I was disappointing him because I spent so much time
talking about the many areas of uncertainty that still needed to be resolved,
before we drew any conclusions about the import of these briefings.
As it turns
out, he was as skeptical as I was, and he and Blake doggedly talked to a number
of people over the rest of that week and early last week. They read DHS’ first
walk back attempt, which said that only a small generation plant had been
compromised. They also checked with Congressional staffers, who confirmed that
DHS’ briefings to them had also emphasized the walk back. And they finally
published their
first article
on the whole affair last Tuesday, a whole week after the first WSJ story. They
followed it up the next day with an
article on the briefing in
New York, which Blake attended. Both articles emphasized the large scale of the
Russian threat and the fact that it’s continuing, but they also both emphasized
that the Russians haven’t achieved their goal of gaining a foothold in U.S.
grid control centers. They haven’t even come close.
P.P.S.
I hope you
don’t think I’m trying to be easy on the Russians in any of these comments. I
think it’s outrageous that they undertook – and continue to undertake – these
attacks. And I think it’s even more outrageous that a certain individual at the
top of the U.S. government, who clearly has a good relationship with Vladimir
Putin, hasn’t taken it upon himself to tell the latter person that both the
grid and electoral system attacks need to stop today – because there are certainly a lot of good non-military
weapons still left in the U.S. arsenal to punish any further attacks.
But it’s
also reprehensible that DHS officials and staff members have both
misrepresented the Russian threat to the grid and allowed much wilder
misrepresentations to be published, without any public statement specifically
repudiating them. I am sure they think they’re serving the greater good
with these exaggerations (and their very impressive and dogged investigations
are the only reason we’re having this conversation in the first place), but I
can assure them that their statements and inaction are only harming the cause
of grid security, not helping it.
Any opinions expressed in this blog post are strictly mine
and are not necessarily shared by any of the clients of Tom Alrich LLC.
If you would like to comment on what you have read here, I
would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that
if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or
challenges like what is discussed in this post – especially on compliance with
CIP-013. And if you’re a security vendor to the power industry, TALLC can help
you by developing marketing materials, delivering webinars, etc. To discuss any
of this, you can email me at the same address.
[i] Because
the WSJ’s web site is behind a paywall, you might have a problem reading this
link. Since I have the article in hard copy, send me an email if you would like
to see it in scanned form.
[ii]
Although there are some generating plants (including some wind farms) that are
owned by holding companies that also own utilities. But because of deregulation
of generation, it is very rare that a utility itself owns generation assets
nowadays.
[iii]
The use of American military forces in Viet Nam was “sanctioned” by the 1964
Gulf of Tonkin Resolution, which was occasioned by the
Gulf of Tonkin
Incident. In that incident, North Vietnamese patrol boats were alleged to
have fired torpedoes at a US warship in international waters, while the North
Vietnamese said the ship had actually penetrated their waters. In the official
Navy report on the incident, the words were used (and I just read this a few
years later in some magazine. I haven’t been able to verify it through an
online search) “Penetration, no matter how slight…is sufficient to constitute
an offense.”
Supposedly, these words were
copied verbatim from the US military’s definition of rape.
[iv]
And if they didn’t mean this at all, why didn’t they try to correct the press
reports – including the WSJ’s, of course – that implied that hundreds of
“utilities” had been compromised?
[v]
And of course, when I have talked of “the grid” in this post – as well as many
other posts – I should more correctly say the grids, since there are four
Interconnects in North America: Eastern, Western, Texas and Quebec. You could
completely take down any one of these and have zero direct impact on any of the
others.