I have
closed a couple recent posts by saying something to the effect of “I wish this
were the last time I had to write about what DHS did or didn’t do regarding the
story of the Russian cyberattacks on the US power grid, but I fear it may not
be”. I have good news and bad news regarding this sentence. The good news is
that I don’t plan to have to use it any more in the near future. The bad news
is that I don’t need to use it, because there’s no longer any doubt that I’ll
be writing more posts on this subject – the story has a long way to go before
it’s put to bed.
To be
honest, up until last Friday, I thought I had this pretty well figured out.
Here’s the timeline:
- On Monday July 23, DHS gave a briefing on the Russian
cyberattacks. The next day, a number of articles appeared, all saying that
the Russians had penetrated “control rooms” for the grid –and usually
hundreds of those. All the articles assumed that it was likely that
malware had been planted that would allow the Russians to cause outages.
As Jonathan Homer of DHS said in a quote in the Wall Street Journal’s article on the 24th, “’They
got to the point where they could have thrown switches’ and disrupted
power flows…” Not to be outdone, Michael Carpenter of DoD was quoted in
the same article as saying “They’ve been intruding into our networks and
are positioning themselves for a limited or widespread attack…They are
waging a covert war on the West”.
- These two statements, and others, both had only one
interpretation, for people knowledgeable about the electric power
industry: Control centers (not
“control rooms”, the term DHS erroneously used repeatedly) of US utilities
had been penetrated by the Russians, and were probably at that minute
harboring malware which could be activated at any point by the Russians to
cause widespread outages or worse (a much-larger-scale version of the
attacks that happened in the Ukraine).[i]
- DHS repeated the briefing on Wednesday, July 25. I was
able to attend this one, and I thought the overall tenor of this briefing
was little different from what had been reported about the Monday
briefing. I wrote my first post
about this subject that day. In that post, I pointed out that obviously
the biggest lesson of these attacks was that supply chain security should
be the primary cyber concern of all utilities (this remains my position).
However, beyond that, I thought there was too much missing information to
be able to draw more conclusions. Finally, I proceeded to violate my own
statement by drawing the conclusion that it was very likely that just
generation was penetrated, and I guessed it was under 25 generation
facilities.
- For me, the most compelling part of that briefing was when
the presenters displayed a screen shot of an HMI display[ii],
which they said had been uploaded from the target system by the Russians.
Of course, the only way the Russians could have taken the screen shot was
by penetrating the control system itself – where they could presumably
have planted malware. I didn’t pay close attention to the screen shot
since the presentation was moving on, but a longtime industry observer
emailed me later to point out that the screen shot was of a display of a
combustion turbine in a natural gas power plant. Keep this in mind; we’ll
come back to it later.
- On Thursday the 26th, that same industry
observer sent me a link to an article on Power Magazine’s web site, that quoted spokesperson Leslie
Fulop of DHS saying “While hundreds of energy and non-energy companies
were targeted, the incident where they gained access to the industrial
control system was a very small generation asset that would not have had
any impact on the larger grid if taken offline.” Hold this thought too; it
is also important.
- That day, I wrote a pretty indignant post,
pointing out the ways this statement contradicted various statements that
had been made in the previous day’s briefing. My last statement was
addressed to DHS: “…I can’t understand why you would want to pretend that
a lot of assets had been penetrated, when it was only one small one. By
doing so, you raised this threat from one that all power industry asset
owners should be aware of and should be taking steps to prevent, to
something approaching an imminent threat to our national security. And it
just isn’t that.”
- On Saturday July 28, I put up a post that quoted from the
New York Times article on this
story, published the day before, to the effect that “This week, the
Department of Homeland Security reported that over the last year, Russia’s
military intelligence agency had infiltrated the control rooms of power
plants across the United States. In theory, that could enable it to take
control of parts of the grid by remote control.” Despite the big
contradiction between these two sentences (if power plants were
infiltrated, this wouldn’t enable “taking control of parts of the grid”),
the most amazing aspect of this quote was that it was talking about
“…power plants across the United States”. It’s hard to reconcile this with
Leslie Fulop’s statement. Of course, I know of no effort by DHS to correct
the Times story, any more than the other stories.
- The following Monday July 30, I wrote a post
pointing to two slides (sent to me by a former colleague) that were shown
at the briefing on the 25th, that directly contradicted the
idea that only one small generator was penetrated. One of the slides said
the Russians “Leveraged early victim to gain entry to two previously
accessed utilities and one new victim”. This definitely says at least two
“utilities” were “accessed”, and perhaps four. Other statements led to the
conclusion that at least three “utilities” were accessed. I concluded the
post by saying “…even though the DHS people who put together the briefings
(and didn’t provide any immediate corrections when the alarming news
stories started flying) were only trying to call attention to a problem,
by exaggerating what had happened they have damaged their credibility for
future advisories.”
- On Wednesday August 1 (i.e. a week later), DHS conducted a
high-level briefing for utility CEOs in New York City, attended by no less
than the US Vice President, the Secretary of Energy and the Secretary of
Homeland Security. It was reported by the indefatigable Blake Sobczak of E&E News, although he chose the
(in hindsight) unfortunate title of “Grid leaders clear the air around Russian hacking”. In the article,
he quoted Christopher Krebs, undersecretary for DHS's National Protection
and Programs Directorate. Mr. Krebs started by saying (in what probably
qualifies as the Understatement of the Year) “In the initial webinar, I
think there was some context that was lacking…” He went on to say that the
Russians had taken control of "a renewable source of energy that
would not disrupt the grid." This was later clarified to mean two
wind turbines.
- A week later on
August 8, I put out what would be my last post
on this subject for three weeks. In the post, I strongly suggested that
DHS needed to finally step up and try to take control of this story,
hopefully by having a press conference to say that, while a few
individuals at DHS may have exaggerated what the Russians had achieved,
this was intended to be for the good purpose of alerting the power
industry to the serious situation they face (and there’s no dispute
anywhere that I know of – although I haven’t talked with Vladimir Putin
lately – that this is very serious, and the utilities need to step up
their cyber defense efforts, especially in supply chain security). Despite
my earnest appeal (or perhaps because of it), DHS has yet to take me up on
the suggestion.
- My next post
on this subject was three weeks later on Thursday August 30, when I
reported on a new link the same industry observer had sent to me,
describing Senator Markey’s announcement that he was sending queries to 14
utilities and four agencies asking what measures they were taking to
combat the Russian threats. The announcement repeated the same erroneous
ideas that had been promulgated in the press earlier. I very helpfully
wrote out for DHS how they could respond to the Senator – and, although I
didn’t mention it in the post, this could form the basis for a statement
that they would provide to the entire US population (or at least those
that are paying attention to issues like this in the last days of summer).
Senator Markey hasn’t called me to say that DHS took up my suggestion, so
I assume that once again they’ve ignored my advice (don’t worry, DHS. I’ve
given lots of advice to NERC and FERC that they’ve ignored as well. You’re
in good company).
- Please note that,
as of that post last Thursday, I still believed that the whole problem was
due to a few DHS employees that got carried away (with good intentions, of
course) and greatly exaggerated what the Russians had accomplished;
moreover, the press took what they said as gospel and ran with it (which
the press tends to do, of course. If you don’t want them to write
something, don’t say it! And don’t be cute and try to imply something that
you know isn’t true, without out saying it directly!). Some of the
higher-ups at DHS had tried to correct the record, but their efforts had
been very limited and hadn’t gone very far, to the extent that three weeks
later a US Senator still didn’t know that the news stories had been walked
back.
- Last Friday, the
plot thickened with two events. One was a long phone conversation I had with
a party I won’t identify, in which that party suggested two things that
really floored me: First, the people at DHS who made the statements at the
briefings might have really meant what they were trying to say
(notwithstanding the fact that they confused control centers with control
rooms): That the Russians had actually penetrated more than one utility control center, where they actually
could control the flow of power
on the grid itself. That meant to me that they had penetrated the Energy
Management System (EMS) which forms the core of the mission of most
utilities, and really does allow them to control power flows in a certain
domain.
- The second thing
this person said was that there may be two warring camps at DHS. One camp
is the people putting out the story that the Russians penetrated utility
control centers. The other camp is the people who are trying to walk this
story back by saying that only one small generation asset was penetrated.
This is why the two contradictory stories have both continued in a kind of
state of quantum coherence, just like an electron can be in two positions
at the same time. While I was quite surprised to hear this, I continued to
believe in the scenario I outlined in paragraph 12.[iii]
- The second event
last Friday – and this did cause
me to question the position I held on Thursday – was when the same
longtime industry observer emailed me to point out a contradiction he’d
seen: As I pointed out in paragraph 4, the HMI screen shot shown in the
webinar was of a combustion turbine (CT), and it could only have been
obtained by the Russians penetrating the control systems that controlled
that turbine – and by implication the plant itself.[iv]
So the screen shot was taken when the Russians penetrated a gas CT plant.
This person had previously pointed this out
to me. But now he said something I hadn’t thought of:
a) As described in paragraph 5, Leslie Fulop of
DHS said that in fact only one small generating plant was compromised. Since
this was right after the second DHS briefing where the HMI screen shot of the
CT had been shown, it would be logical to conclude that the small plant she was
referring to was a gas CT plant.
b) Yet a
week later in New York, Christopher Krebs of DHS said only two wind
turbines had been compromised. Since an HMI screen shot of those turbines would
have looked completely different from what was shown in the briefing, there is
no possibility that the single small plant referred to by Ms. Fulop was the wind
farm referred to by Mr. Krebs (and if it had really just been two wind turbines
that were compromised, Ms. Fulop would certainly have said that, not used the
words she did).
So it appears that either Ms. Fulop was wrong
when she said that one small plant was compromised, or Mr. Krebs was wrong when
he said that just two wind turbines were compromised. At the minimum, it seems
two plants were compromised – one wind farm and one gas CT plant – unless Ms.
Fulop or Mr. Krebs is simply lying. Had these two statements been made the same
day or one day apart, one could attribute this discrepancy to the likelihood
that DHS people were rushing to walk back the initial story and two different
people ended up with two different stories. This isn’t a good thing, but it
wouldn’t be unprecedented in the annals of government. But the fact that there
was a week (exactly) between the two statements and they still were
contradictory, is disturbing. And it’s especially disturbing that, more than a
month after all this started to happen, DHS hasn’t put out any statement or
held any press conference to explain all of these discrepancies.
As I said, these contradictory statements –
and the fact that they were separated by a whole week – made me reconsider the
position I held last Thursday. While I still believe that the initial
statements in the briefings were deliberate exaggerations made with good
intentions (and that no utility control centers were penetrated), I am now
suspicious of the two walk back attempts. I find it hard to believe that it’s
just the normal fog of war that led to these contradictions. In any case, a
single definitive statement of what happened – issued by someone presumably
above the fray and the factions – could settle this and the other question. I
hope that comes soon. This will be the fourth story that DHS has put out about
the Russian attacks. I sincerely hope it will be the last!
Of course, as I just said, I continue to
believe that no utility control centers were penetrated, even though the DHS
briefings strongly implied that they had been. But if they were – even if only
one small utility control center in West Texas was penetrated – this raises the
seriousness of the situation to a higher level, and puts official DHS actions
in a very different light from being mere bumbling. I will discuss this
scenario in (hopefully) my next post, coming soon.
Any opinions expressed in this blog post are strictly mine
and are not necessarily shared by any of the clients of Tom Alrich LLC.
If you would like to comment on what you have read here, I
would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that
if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or
challenges like what is discussed in this post – especially on compliance with
CIP-013. And if you’re a security vendor to the power industry, TALLC can help
you by developing marketing materials, delivering webinars, etc. To discuss any
of this, you can email me at the same address.
[i]
For an in-depth attempt to clarify the language that DHS used in their
briefings – which was repeated by the various news outlets – and determine what
they really meant to say, see this
post.
[ii]
HMI stands for human-machine interface. HMIs are probably the most important
component of any control system, since they gather all the data being put out
by the various devices in the system and impose it on a real-time schematic
drawing of the system; this is what enables operators to understand what is
going on, as well as intervene to make changes.
[iii]
To continue my quantum metaphor, I had made my observation that caused the wave
function to collapse into one of the two contradictory possibilities, just as
an observation of an electron causes its wave function to collapse to one
position or the other – or at least that’s what the Copenhagen Interpretation
of quantum mechanics says. But then, as Richard Feynman famously said, “Anyone
who says they understand quantum mechanics doesn’t understand quantum
mechanics.”
[iv]
Power plants are often controlled by a Distributed Control System (DCS). This
controls the combustion turbines, if they’re present. And the DCS usually
resides in a single control room. If systems in the room control more than one
plant, then the room is actually a control center, and subject to much more
stringent NERC CIP controls, as well as other NERC standards. No generation
operator in their right mind would ever describe a true control room as a
control center.
No comments:
Post a Comment