I have closed a couple recent posts by saying something to the effect of “I wish this were the last time I had to write about what DHS did or didn’t do regarding the story of the Russian cyberattacks on the US power grid, but I fear it may not be”. I have good news and bad news regarding this sentence. The good news is that I don’t plan to have to use it any more in the near future. The bad news is that I don’t need to use it, because there’s no longer any doubt that I’ll be writing more posts on this subject – the story has a long way to go before it’s put to bed.
To be honest, up until last Friday, I thought I had this pretty well figured out. Here’s the timeline:
- On Monday July 23, DHS gave a briefing on the Russian cyberattacks. The next day, a number of articles appeared, all saying that the Russians had penetrated “control rooms” for the grid –and usually hundreds of those. All the articles assumed that it was likely that malware had been planted that would allow the Russians to cause outages. As Jonathan Homer of DHS said in a quote in the Wall Street Journal’s article on the 24th, “’They got to the point where they could have thrown switches’ and disrupted power flows…” Not to be outdone, Michael Carpenter of DoD was quoted in the same article as saying “They’ve been intruding into our networks and are positioning themselves for a limited or widespread attack…They are waging a covert war on the West”.
- These two statements, and others, both had only one interpretation, for people knowledgeable about the electric power industry: Control centers (not “control rooms”, the term DHS erroneously used repeatedly) of US utilities had been penetrated by the Russians, and were probably at that minute harboring malware which could be activated at any point by the Russians to cause widespread outages or worse (a much-larger-scale version of the attacks that happened in the Ukraine).[i]
- DHS repeated the briefing on Wednesday, July 25. I was able to attend this one, and I thought the overall tenor of this briefing was little different from what had been reported about the Monday briefing. I wrote my first post about this subject that day. In that post, I pointed out that obviously the biggest lesson of these attacks was that supply chain security should be the primary cyber concern of all utilities (this remains my position). However, beyond that, I thought there was too much missing information to be able to draw more conclusions. Finally, I proceeded to violate my own statement by drawing the conclusion that it was very likely that just generation was penetrated, and I guessed it was under 25 generation facilities.
- For me, the most compelling part of that briefing was when the presenters displayed a screen shot of an HMI display[ii], which they said had been uploaded from the target system by the Russians. Of course, the only way the Russians could have taken the screen shot was by penetrating the control system itself – where they could presumably have planted malware. I didn’t pay close attention to the screen shot since the presentation was moving on, but a longtime industry observer emailed me later to point out that the screen shot was of a display of a combustion turbine in a natural gas power plant. Keep this in mind; we’ll come back to it later.
- On Thursday the 26th, that same industry observer sent me a link to an article on Power Magazine’s web site, that quoted spokesperson Leslie Fulop of DHS saying “While hundreds of energy and non-energy companies were targeted, the incident where they gained access to the industrial control system was a very small generation asset that would not have had any impact on the larger grid if taken offline.” Hold this thought too; it is also important.
- That day, I wrote a pretty indignant post, pointing out the ways this statement contradicted various statements that had been made in the previous day’s briefing. My last statement was addressed to DHS: “…I can’t understand why you would want to pretend that a lot of assets had been penetrated, when it was only one small one. By doing so, you raised this threat from one that all power industry asset owners should be aware of and should be taking steps to prevent, to something approaching an imminent threat to our national security. And it just isn’t that.”
- On Saturday July 28, I put up a post that quoted from the New York Times article on this story, published the day before, to the effect that “This week, the Department of Homeland Security reported that over the last year, Russia’s military intelligence agency had infiltrated the control rooms of power plants across the United States. In theory, that could enable it to take control of parts of the grid by remote control.” Despite the big contradiction between these two sentences (if power plants were infiltrated, this wouldn’t enable “taking control of parts of the grid”), the most amazing aspect of this quote was that it was talking about “…power plants across the United States”. It’s hard to reconcile this with Leslie Fulop’s statement. Of course, I know of no effort by DHS to correct the Times story, any more than the other stories.
- The following Monday July 30, I wrote a post pointing to two slides (sent to me by a former colleague) that were shown at the briefing on the 25th, that directly contradicted the idea that only one small generator was penetrated. One of the slides said the Russians “Leveraged early victim to gain entry to two previously accessed utilities and one new victim”. This definitely says at least two “utilities” were “accessed”, and perhaps four. Other statements led to the conclusion that at least three “utilities” were accessed. I concluded the post by saying “…even though the DHS people who put together the briefings (and didn’t provide any immediate corrections when the alarming news stories started flying) were only trying to call attention to a problem, by exaggerating what had happened they have damaged their credibility for future advisories.”
- On Wednesday August 1 (i.e. a week later), DHS conducted a high-level briefing for utility CEOs in New York City, attended by no less than the US Vice President, the Secretary of Energy and the Secretary of Homeland Security. It was reported by the indefatigable Blake Sobczak of E&E News, although he chose the (in hindsight) unfortunate title of “Grid leaders clear the air around Russian hacking”. In the article, he quoted Christopher Krebs, undersecretary for DHS's National Protection and Programs Directorate. Mr. Krebs started by saying (in what probably qualifies as the Understatement of the Year) “In the initial webinar, I think there was some context that was lacking…” He went on to say that the Russians had taken control of "a renewable source of energy that would not disrupt the grid." This was later clarified to mean two wind turbines.
- A week later on August 8, I put out what would be my last post on this subject for three weeks. In the post, I strongly suggested that DHS needed to finally step up and try to take control of this story, hopefully by having a press conference to say that, while a few individuals at DHS may have exaggerated what the Russians had achieved, this was intended to be for the good purpose of alerting the power industry to the serious situation they face (and there’s no dispute anywhere that I know of – although I haven’t talked with Vladimir Putin lately – that this is very serious, and the utilities need to step up their cyber defense efforts, especially in supply chain security). Despite my earnest appeal (or perhaps because of it), DHS has yet to take me up on the suggestion.
- My next post on this subject was three weeks later on Thursday August 30, when I reported on a new link the same industry observer had sent to me, describing Senator Markey’s announcement that he was sending queries to 14 utilities and four agencies asking what measures they were taking to combat the Russian threats. The announcement repeated the same erroneous ideas that had been promulgated in the press earlier. I very helpfully wrote out for DHS how they could respond to the Senator – and, although I didn’t mention it in the post, this could form the basis for a statement that they would provide to the entire US population (or at least those that are paying attention to issues like this in the last days of summer). Senator Markey hasn’t called me to say that DHS took up my suggestion, so I assume that once again they’ve ignored my advice (don’t worry, DHS. I’ve given lots of advice to NERC and FERC that they’ve ignored as well. You’re in good company).
- Please note that, as of that post last Thursday, I still believed that the whole problem was due to a few DHS employees that got carried away (with good intentions, of course) and greatly exaggerated what the Russians had accomplished; moreover, the press took what they said as gospel and ran with it (which the press tends to do, of course. If you don’t want them to write something, don’t say it! And don’t be cute and try to imply something that you know isn’t true, without out saying it directly!). Some of the higher-ups at DHS had tried to correct the record, but their efforts had been very limited and hadn’t gone very far, to the extent that three weeks later a US Senator still didn’t know that the news stories had been walked back.
- Last Friday, the plot thickened with two events. One was a long phone conversation I had with a party I won’t identify, in which that party suggested two things that really floored me: First, the people at DHS who made the statements at the briefings might have really meant what they were trying to say (notwithstanding the fact that they confused control centers with control rooms): That the Russians had actually penetrated more than one utility control center, where they actually could control the flow of power on the grid itself. That meant to me that they had penetrated the Energy Management System (EMS) which forms the core of the mission of most utilities, and really does allow them to control power flows in a certain domain.
- The second thing this person said was that there may be two warring camps at DHS. One camp is the people putting out the story that the Russians penetrated utility control centers. The other camp is the people who are trying to walk this story back by saying that only one small generation asset was penetrated. This is why the two contradictory stories have both continued in a kind of state of quantum coherence, just like an electron can be in two positions at the same time. While I was quite surprised to hear this, I continued to believe in the scenario I outlined in paragraph 12.[iii]
- The second event last Friday – and this did cause me to question the position I held on Thursday – was when the same longtime industry observer emailed me to point out a contradiction he’d seen: As I pointed out in paragraph 4, the HMI screen shot shown in the webinar was of a combustion turbine (CT), and it could only have been obtained by the Russians penetrating the control systems that controlled that turbine – and by implication the plant itself.[iv] So the screen shot was taken when the Russians penetrated a gas CT plant.
This person had previously pointed this out to me. But now he said something I hadn’t thought of:
a) As described in paragraph 5, Leslie Fulop of DHS said that in fact only one small generating plant was compromised. Since this was right after the second DHS briefing where the HMI screen shot of the CT had been shown, it would be logical to conclude that the small plant she was referring to was a gas CT plant.
b) Yet a week later in New York, Christopher Krebs of DHS said only two wind turbines had been compromised. Since an HMI screen shot of those turbines would have looked completely different from what was shown in the briefing, there is no possibility that the single small plant referred to by Ms. Fulop was the wind farm referred to by Mr. Krebs (and if it had really just been two wind turbines that were compromised, Ms. Fulop would certainly have said that, not used the words she did).
So it appears that either Ms. Fulop was wrong when she said that one small plant was compromised, or Mr. Krebs was wrong when he said that just two wind turbines were compromised. At the minimum, it seems two plants were compromised – one wind farm and one gas CT plant – unless Ms. Fulop or Mr. Krebs is simply lying. Had these two statements been made the same day or one day apart, one could attribute this discrepancy to the likelihood that DHS people were rushing to walk back the initial story and two different people ended up with two different stories. This isn’t a good thing, but it wouldn’t be unprecedented in the annals of government. But the fact that there was a week (exactly) between the two statements and they still were contradictory, is disturbing. And it’s especially disturbing that, more than a month after all this started to happen, DHS hasn’t put out any statement or held any press conference to explain all of these discrepancies.
As I said, these contradictory statements – and the fact that they were separated by a whole week – made me reconsider the position I held last Thursday. While I still believe that the initial statements in the briefings were deliberate exaggerations made with good intentions (and that no utility control centers were penetrated), I am now suspicious of the two walk back attempts. I find it hard to believe that it’s just the normal fog of war that led to these contradictions. In any case, a single definitive statement of what happened – issued by someone presumably above the fray and the factions – could settle this and the other question. I hope that comes soon. This will be the fourth story that DHS has put out about the Russian attacks. I sincerely hope it will be the last!
Of course, as I just said, I continue to believe that no utility control centers were penetrated, even though the DHS briefings strongly implied that they had been. But if they were – even if only one small utility control center in West Texas was penetrated – this raises the seriousness of the situation to a higher level, and puts official DHS actions in a very different light from being mere bumbling. I will discuss this scenario in (hopefully) my next post, coming soon.
Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.
If you would like to comment on what you have read here, I would love to hear from you. Please email me at firstname.lastname@example.org. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. And if you’re a security vendor to the power industry, TALLC can help you by developing marketing materials, delivering webinars, etc. To discuss any of this, you can email me at the same address.
[i] For an in-depth attempt to clarify the language that DHS used in their briefings – which was repeated by the various news outlets – and determine what they really meant to say, see this post.
[ii] HMI stands for human-machine interface. HMIs are probably the most important component of any control system, since they gather all the data being put out by the various devices in the system and impose it on a real-time schematic drawing of the system; this is what enables operators to understand what is going on, as well as intervene to make changes.
[iii] To continue my quantum metaphor, I had made my observation that caused the wave function to collapse into one of the two contradictory possibilities, just as an observation of an electron causes its wave function to collapse to one position or the other – or at least that’s what the Copenhagen Interpretation of quantum mechanics says. But then, as Richard Feynman famously said, “Anyone who says they understand quantum mechanics doesn’t understand quantum mechanics.”
[iv] Power plants are often controlled by a Distributed Control System (DCS). This controls the combustion turbines, if they’re present. And the DCS usually resides in a single control room. If systems in the room control more than one plant, then the room is actually a control center, and subject to much more stringent NERC CIP controls, as well as other NERC standards. No generation operator in their right mind would ever describe a true control room as a control center.