Great. Another imaginary grid cybersecurity emergency for the press!

Yesterday, I posted about a New York Times article that pointed out some very interesting new information about the SolarWinds attacks. However, when I called the article “excellent” in the first sentence, I prefaced that with “(mostly)”; I didn’t elaborate on why I did that. No I’ll explain why I did that.

I thought the article was a great piece of reporting until I reached these paragraphs:

Publicly, officials have said they do not believe the hackers from Russia’s S.V.R. pierced classified systems containing sensitive communications and plans. But privately, officials say they still do not have a clear picture of what might have been stolen.

They said they worried about delicate but unclassified data the hackers might have taken from victims like the Federal Energy Regulatory Commission, including Black Start, the detailed technical blueprints for how the United States plans to restore power in the event of a cataclysmic blackout.

The plans would give Russia a hit list of systems to target to keep power from being restored in an attack like the one it pulled off in Ukraine in 2015, shutting off power for six hours in the dead of winter. Moscow long ago implanted malware in the American electric grid, and the United States has done the same to Russia as a deterrent.

Just about every word in the last two paragraphs is wrong, except for “the” and “for”. Here’s why:

Since not everyone who reads this may be familiar with the term blackstart (as the term is usually written in the power industry), I’ll summarize it in a way I can understand (so I know you will!). If you’re interested in learning more about this subject, you can go here. A lot of the information in the rest of this post comes from Kevin Perry, former Chief CIP auditor of SPP Regional Entity and before that, Supervisor-EMS and Director-IT Infrastructure for the Southwest Power Pool ISO/RTO.

1.      Electric power generators all require some power source in order to operate, since they depend on electromagnets, not the kind of magnets you used to pick up iron filings in grade school.

2.      Normally, that power source is the generator itself – i.e. a small amount of the power generated is used to keep the generator operating. In other cases, the plant needs to use power from the grid to start up. This is especially true for plants that use steam to operate turbines.

3.      Of course, as long as the grid is functioning in the area where the plant is located, there’s no problem – any plant can draw whatever amount of power it needs to restart.

4.      The problem arises when there has been a widespread outage, which almost always means a cascading outage. Almost all outages are local, meaning the grid is still functioning close enough to the outage that power is available to restart any plants that may be down. A cascading outage is what happened in the Northeast Blackout of August 2003: An initial problem in northeastern Ohio was inadvertently allowed to spread to much of the northeastern US and most of Ontario, killing six people.

5.      Once the cascade ended (it only took four minutes from when the problem first spread beyond northeastern Ohio, to when the cascade ended and 55 million people were in the dark), 508 generating units at 265 power plants had shut down. In large portions of the blackout areas, there was no functioning power grid at all.

6.      So how did all of these plants start back up? Where did they get their power from? Fortunately, there are some types of generators that require very little or no power to start up. These include a) hydro plants, since the rivers keep running even though the power is out, b) backup generators running on diesel, kerosene, etc., and c) specially-configured combustion turbine plants, usually between 50 and 100 megawatts. These are all called blackstart generators.

7.      These “self-starting” generators can ultimately restart all of the other plants, but there’s a catch: If these generators just throw their power randomly on the grid, it will never reach the proper voltage to start the other generators. There needs to be a plan in place showing which blackstart generators will energize which lines going through which substations (these lines and substations are known as the “cranking path”. I love that term), so that one-by-one the larger generators can restart. Once something close to full generation capacity is online, all remaining power users (known as “load”) can be reconnected, and the region is back to normal.

8.      You probably get the idea: There needs to be an elaborate plan to make sure all of this happens, since without coordination, the grid will never be restored. Developing such a plan is the responsibility of two types of organizations (some are also electric utilities and some are not): Transmission Operators (TOP) and Reliability Coordinators (RC).

9.      Both TOPs and RCs are required to follow NERC Reliability Standard EOP-005, which governs the process of developing and coordinating blackstart Restoration Plans.

10.   Each TOP is required by EOP-005 to develop their own Restoration Plan, to restore power in their service territory after a partial or complete blackout. Their RC is required to coordinate and approve the plans.

The appropriate NERC Regional Entity (RE – there are six of these) will normally have a copy of the plan for each TOP in their footprint, because they audit the TOPs (and RCs) for compliance with EOP-005. However, Kevin believes it is unlikely that the RE’s routinely share these plans with NERC. Because blackstart restoration is a regionally-managed process, NERC is not usually directly involved with it.

And because FERC is not involved in any way with managing the power grid – including restoring it – it is very unlikely they would have these plans sitting around on their network, just waiting for a hacker to grab them. In fact, FERC, NERC and the six Regions are all quite aware of the danger posed by hackers; in general, they aren’t interested in holding any information – even briefly – that might be of value to an entity that might use it to attack the grid. As illustration of this point, some utilities will not send any sensitive information to their Region at all when they’re being audited; they require the auditors to read it while they are onsite.

All of this is to say it’s highly unlikely there were any blackstart restoration plans on storage at FERC when they were penetrated by the Russians. Whoever told the NYT reporters they were “worried” about this should find something better to worry about.

But suppose the hackers had their hands on the restoration plans. How would they be able to use them to attack the US grid? The plans list power plants, lines and substations that play a role in restoring power after a blackout. Of course, knowing those would allow the Russians to target a physical attack – say, dropping paratroopers to take out these resources. But no one considers that a likely prospect.

How about a cyberattack? What information would the Russians need to launch one of those to prevent restoration of power? They would need information about the specific “Cyber Assets” (a NERC term meaning “programmable electronic devices”) that control the physical systems in the power plants and substations listed in the restoration plan – device names and types, physical location, IP addresses, etc. Of course, the plans contain none of that information, since it isn’t at all required for the purpose of the plan.

Finally, let’s assume this information were in the plans and that by some chance the plans were stored at FERC when the Russians attacked them. Would the hackers actually be able to attack the Cyber Assets in question?

Kevin and I discussed this question. The Cyber Assets that are most likely to be attacked would be electronic relays, located in substations, that control circuit breakers. The circuit breakers can open or close a line, but since they themselves aren’t microprocessor-controlled, the relay tells them what to do. But the relays are all controlled by the Control Center for the utility that operates the lines and substations. Thus, any attack would have to come through the Control Center, which is exactly what happened in the case of the Ukraine attacks in 2015 and 2016.

But the difference between the US in 2021 and the Ukraine in 2015 is that the Control Centers are very well protected, and even more importantly the Control Center’s network is well segregated from the IT network. Since the Russians attacked the Ukrainian utilities by first penetrating the IT network through phishing emails (at least this was true for the 2015 attacks. The exact vector for the 2016 attacks is still not clear), this segregation would make it very hard for them to get into the Control Center network in a US utility – and thus for them to command the substation relays to open circuit breakers and trip lines, as happened in the Ukraine.

The moral of this story is that whoever expressed their “worries” to the NYT reporters obviously wasn’t knowledgeable about the US power industry. It would have been nice if one of the reporters had then picked up the phone and talked to someone directly involved with the industry. They would have set their fears at rest.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.