Note: there are multiple links to Wall Street Journal articles in this post, most of which are behind a paywall. If you would like to read any of these articles, please email me and I’ll send you a PDF of them.
On July 23, 2018, the Wall Street Journal published
an article[i]
by a reporter named Rebecca Smith (whom I had never heard of at the time)
titled “Russian Hackers Reach U.S. Utility Control Rooms, Homeland Security
Officials Say”. The article described an online web presentation that had been
given by Jonathan Homer of DHS that day. That presentation was repeated three
more times within two weeks.
The first four paragraphs of the article were quite
startling:
Hackers working for Russia claimed “hundreds of victims” last
year in a giant and long-running campaign that put them inside the control rooms of U.S. electric utilities where
they could have caused blackouts, federal officials said. They said the
campaign likely is continuing.
The Russian hackers, who worked for a shadowy state-sponsored
group previously identified as Dragonfly or Energetic Bear, broke into
supposedly secure, “air-gapped” or isolated networks owned by utilities with
relative ease by first penetrating the networks of key vendors who had trusted
relationships with the power companies, said officials at the Department of
Homeland Security.
“They got to the point where they could have thrown switches”
and disrupted power flows, said Jonathan Homer, chief of
industrial-control-system analysis for DHS.
DHS has been warning utility executives with security
clearances about the Russian group’s threat to critical infrastructure since
2014. But the briefing on Monday was the first time that DHS has given out
information in an unclassified setting with so much detail. It continues to
withhold the names of victims, but it now says there were hundreds of victims,
not a few dozen, as had been stated previously.
The article went on to describe how the attackers fairly
easily penetrated vendors to the utility industry and points out that “It was a
relatively easy process, in many cases, for them to steal credentials from
vendors and gain direct access to utility networks.” While the article makes it
abundantly clear that attackers penetrated the control networks (“control
rooms”, presumably meaning Control Centers) of electric utilities, it isn’t completely
clear that the “hundreds of victims” were all utilities or whether some of them
were vendors. In any case, it does seem that many utility control networks were
penetrated and that DHS thought the attackers were in a position to cause
outages if they wanted to.
The next day, Tuesday July 23, Rebecca’s article appeared in
the WSJ’s print edition. It is no exaggeration to say that it caused a
firestorm of reactions in the media, both in the US and abroad; there was
general agreement that Rebecca had described a huge threat to US national
security. My initial reaction was in a post
on Wednesday. I expressed skepticism about the article, although my main point
was that these attacks showed the vital importance of CIP-013, the supply chain
security standard that FERC had ordered in 2016. By that time, it had been drafted
and approved by NERC, but not yet by FERC.
But there was another reaction on Wednesday. I described it
in my post on
Thursday: “I learned today, from an article on Power Magazine’s
web site, and confirmed with a source who knew the contents of Congressional
briefings by DHS, that the true number of assets compromised was…. envelope,
please….one. And by the way, it was a very insignificant generating plant whose
loss would have no impact on the grid.”
This was interesting. It seems that DHS was suddenly trying
to take back the presentation they had just given on Monday. However, despite
their efforts, the same presentation was repeated unchanged by Mr. Homer
once that week and twice the following week. I won’t go through everything that
happened in the next few weeks, but I will point out that DHS, now that they
had walked back what Homer said once, decided that wasn’t enough.
The next week, DHS held a meeting
in New York, attended by Vice President Pence, Secretary Nielsen of DHS,
Secretary of Energy Perry, CEO Tom Fanning of Southern Company, and Chris Krebs
of DHS (a year before he put together CISA). In that meeting, Mr. Krebs went
one better than the story DHS had put out the previous week: He said it wasn’t
even an insignificant generating plant that was compromised. It was just two
wind turbines!
Despite this contradiction, I believed the general tenor of
both the DHS and Krebs statements was true. On August 7, I published this
post. It asserted that either Jonathan Homer had been lying in his
presentation, or Rebecca had completely misunderstood what he was talking
about. In the post, I made some unfair, and – frankly - misogynistic,
statements about Rebecca that I regret to this day. My next two posts on this
topic continued to drink the DHS Kool-Aid and deprecate what Rebecca had
written.
However, on September 4 I changed my tune in this
post. I did this partly because the contradictions in the DHS story had
become impossible to reconcile. However, the main reason I changed my position was
that Rebecca had emailed me the previous Friday (it turns out she had been a
longtime reader of my blog). I apologized to her for my statements in the
August 7 post and she accepted my apology. Being one of the two WSJ reporters that
disclosed the Enron scandal and having written the definitive
book on Enron’s collapse (of which her reporting was one of the main
causes), I imagine she’d seen a lot worse statements written about her. I
described what she said on the phone in my September 4 post, although she
wouldn’t let me reveal her name.
In that call, Rebecca made it clear that she didn’t
misunderstand Jonathan Homer’s statements (in fact, she had listened to the two
repeat webinars that Homer presented the following week. She said all three
webinars were close to identical, although the one she wrote about in her
article had some technical problems). In the post, I paraphrased what she said:
…the people at DHS who made the statements at the briefings…really
meant what they were trying to say (notwithstanding the fact that they confused
control centers with control rooms): that the Russians had penetrated more than
one utility control center, where they actually could control
the flow of power on the grid itself. That meant to me that they had penetrated
the Energy Management System (EMS). This system forms the core of the mission
of most electric utilities, since it allows them to control power flows over
their network.
The same day that I talked with Rebecca, a well-respected
CIP auditor (who also had to remain anonymous) pointed out to me that a screen
shot, that was captured by the Russian attackers and displayed in Jonathan
Homer’s webinar, showed that the Russians had penetrated a combustion turbine gas
plant. CT plants are usually fairly large, so most industry observers wouldn’t
consider one to be just an “insignificant plant” – the phrase DHS had used to describe
the single plant that was penetrated by the Russians. Of course, a CT plant
would also never be mistaken for “just two wind turbines”.
In other words, the one piece of documentary evidence
contradicted both of the stories that DHS had told so far, as they struggled
mightily (but unsuccessfully) to walk back the main content of Jonathan Homer’s
presentation.
From that day on, I believed Rebecca and we became good
friends, although we only met once in person - at the 2019 RSA Conference. I
was devastated to read in the WSJ just before Christmas of 2023 that she had
passed away. I strongly recommend you read the WSJ obituary
on her. She had a very remarkable career and took the time to really understand
the electric power industry. The last piece of hers that I read was one
of a series in 2021 about the disastrous
pricing decisions made by the Texas
Public Utilities Commission and ERCOT (the Texas grid operator) at the height
of the crisis early in the morning of February 15, 2021. As always, it was well
researched and thought out.
There is a lot more to the story of the Russians and the US
power grid in the next couple of years (2019 and 2020). While I hope to write a
full blog post (or even two) about those events soon, I want to point out three
highlights, as well as the lesson that I think can be learned from all of this:
1. On January 10, 2019, Rebecca and Rob Barry of the
WSJ published a journalistic tour de force: a very well-researched
article on how the Russians had penetrated a number of electric utilities (five
were named, but there were clearly more victims) using supply chain attacks
(you can read my post about the article here).
What was quite interesting about this was that the Russians didn’t seem to have
been trying to cause the Big One: a cascading outage like the one that caused
the 2003 Northeast Blackout (I had always assumed this was their Holy Grail). Thus,
they didn’t have to limit themselves to trying to penetrate vendors of Energy
Management Systems (EMS), which are presumably the only vendors that would have
access to these systems.
Instead, the Russians penetrated an excavating company, a
technical magazine publisher, a small construction company, some small power
generation companies, and other small vendors. Their ultimate targets will
still control systems operated by electric utilities, but now the utilities targeted
were small ones. None of these utilities would by themselves have been able to
cause a cascading grid outage if compromised, but most of them served military
bases. Clearly, the Russians were positioning themselves so they could disrupt
a US military response in case of war.
2. A while after the July/August 2018 blowup caused
by Rebecca’s article, it became clear to me that the various statements by DHS
trying to walk back Jonathan Homer’s presentation were probably not just the
result of individuals trying to “defend DHS” as best they could (although Mr.
Homer remained in the employ of DHS for at least another year). Instead, an
order must have come down from the highest level of the federal government to
the effect that a) Homer’s story needed to be discredited, and b) DHS should
stop investigating Russian cyber attackers, as they had been doing for several
years. It’s very likely the penalty for disobeying the order was being immediately
fired.
Indeed, it seems that order was very successful. Before July
2018, I had read a lot of news stories about what DHS had learned about Russian
cyberattacks on the grid; after July 2018, those stories ceased. In fact, at a
meeting in conjunction with the 2019 RSA Conference, I asked a Director in DHS,
who had been discussing their cyber capabilities, about Jonathan Homer’s presentation
and the multiple walkbacks that DHS had attempted. When I brought this up, he
turned white as a sheet and stammered off some incoherent statements, including
"We don't do technical investigations." Really, DHS (this was still
before CISA was formed) isn’t capable of doing technical investigations?
3. However, there were a huge number of news articles
and blog posts (including mine) starting in December 2020 about by far the most
successful targeted Russian attacks on the US federal government (as well as many
other organizations, both US-based and international): the SolarWinds supply
chain attacks.
What fascinated me most about those attacks was the amazing
degree of organization and planning that went into the attack on the SolarWinds
development environment; here is my post
on that topic. In fact, Microsoft stated
that probably 1,000 Russian engineers worked on the attack (which took a year
and a half and only ended when the attack was discovered by chance by FireEye).
But here’s the big question: With such a massive effort underway in the Russian
hacker community, why didn’t the US have any clue that this was going on until
the Russians had been helping themselves to secrets they found inside US
government and private networks for more than six months?
Although it took me a while to realize it, I now see that the answer to that question is simple: If you tell your investigators not to investigate one country anymore on pain of losing their job, you’ll end up with what you wanted – no investigations. Given how destructive the SolarWinds attacks were and the fact that we’ll never know how many secrets walked out the door into Putin’s hands during those attacks, we have all paid (and will continue to pay) a big price for that policy. I wish we all, including me, had listened to Rebecca in 2018.
My blog is more popular than
ever, but I need more than popularity to keep it going. I’ve been told I should
either accept advertising or charge a subscription fee, or both. However,
neither of those options appeals to me. It would be great if everyone who appreciates
my posts could donate
a $20-$25 “subscription fee” once a year (of course, I welcome larger amounts
as well!). Will you do that today?
If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.
[i] If
you run into a paywall on this or any other WSJ link in this post, send me an
email and I’ll send the PDF of the article to you.
No comments:
Post a Comment