Battling the Hydra

In January, I wrote a post discussing a press release just put out by a group called the Foundation for Resilient Societies. The essence of this release was that communications between control centers and substations were being run unencrypted over the public Internet, and therefore pose a huge vulnerability for the power grid. Meanwhile, as the group asserted, FERC isn’t ordering NERC to put controls on substation communications, specifically encryption. Therefore, FERC’s inaction means the US grid remains substantially at risk.

I said in the post that my main problem with this was that I don’t know of a single utility that is using the public Internet to communicate with substations, with or without encryption; so the entire argument in the press release is based on a false assumption. I’m sure others made this argument in other venues as well.

Of course, I never thought that post would stop the Foundation from pursuing their campaign, and sure enough on Feb. 22 they filed an administrative Request for Rehearing with FERC, which asked them to revise Order 822 to require controls on substation communications. But I did still hope that people in the industry would realize these were not serious arguments.

This is why I was surprised to read, in the April Transmission and Distribution World, a short article entitled “Deficient Cybersecurity Standards Leave U.S. Electric Grid at Risk”. This article states in the first paragraph that FERC has recently approved a “NERC cybersecurity standard” that exempts “significant points of vulnerability, including communications between control rooms[i] and grid substations.” In the second and third paragraphs, the article mentions the Foundation’s FERC filing, and states that “Industry standards require encryption of credit card information transmitted over the Internet, but the same is not true for communications between grid control centers and substations. When hackers attacked the Ukrainian power grid, they attacked control centers, service call centers and substations.”

Note: T&D World had reported on the Foundation's filing with FERC in their March issue. Of course, the statements are very similar between the two articles.

So it seems this myth is like the Hydra, the multi-headed monster of Greek mythology. When you would cut off one head, another two would grow back. In this post, I’m going to take a broader approach than I did the first time, in the hopes of either killing the Hydra (by perhaps poisoning the monster itself) or at least cutting off more heads than can grow back.

As I said, the assertion that any substation communications run unencrypted over the public Internet is almost assuredly completely false. But let’s look at how utilities typically do communicate with their substations, to figure out where a grain or two of truth (or at least plausibility) might be found in this argument.[ii]

First, I’m sure the majority of substation communications are still serial, not routable. I won’t say serial communications are hack-proof, but I will say I have never heard of a successful serial hack (other than one proof of concept by a researcher). So, as we look for vulnerable communications, we need to stick to the minority that is routable.

If not the Internet, what channel carries the routable communications with substations? I believe Frame Relay and SONET are the prevalent technologies here. Neither one of these, of course, touches the public Internet in any way, and I have never heard of a successful attack on communications using either of these technologies.

But let’s say one of these could be hacked. Would this be a threat to the Bulk Electric System? If the substation in question were a distribution one, the answer is probably no. There might be a localized outage (as there was in the case of the Ukraine attack – multiple ones, since multiple distribution substations were attacked), but there wouldn’t be a cascading BES outage (as I discussed in this post). 

So what if the substation were a transmission (BES) one? For good measure, an important substation that would be Medium impact under CIP v5? First, could a hack of one substation lead to a hack of lots of others? The answer to that is almost certainly no. Unlike the idea that some people seem to have, substations aren’t connected to some vast flat network, in which an attack on one can lead to easy penetration of many others.  Communications between control centers and substations are very much hub-and-spoke, not meshed. Were the control center to be compromised, that would be another story, and for that reason the most stringent controls in NERC CIP are applied there (and FERC has just ordered controls on communications between control centers).

Then could an attack on a single BES substation cause a cascading outage through direct electrical effects? Not by itself, I’ve heard repeatedly; there are too many other controls in place to prevent this from happening. This means that an attack on the communications between a control center and a single substation can’t cause a cascading BES outage, either through cyber or physical means. Of course, were a hacker to attack multiple BES substations simultaneously, that in itself could conceivably cause a cascading outage. But that brings us back to the question of how that could possibly be done, given that there isn’t any obvious way to hack into a single substation, let alone a number of them simultaneously.

So it has to be said that the possibility of a successful cyberattack on the communications between a control center and a substation (transmission or distribution) is quite low – especially an attack that could cause a cascading outage (the Ukraine attacks did cause a substantial loss of load, but that was all restored within four to six hours[iii]).

However, note that I’m not saying the probability of success is zero; sooner or later I’m sure even serial or Frame Relay communications could be compromised. So, if FERC were to order controls on substation communications, would that be worthwhile? After all, there would be a small increase in security.

In the case of substations, that small increase in security might well be offset (or more) by a marked decrease in reliability. This is because communications between a control center and a substation are extremely sensitive to latency. If a circuit breaker needs to be opened or closed, this needs to be done with no delay at all – and barring that, within as few cycles as possible. And encryption always imposes some small amount of latency.

Note I’m not saying that encryption would never be possible for substation communications, but it is certainly true it shouldn’t be ordered without making sure it won’t literally cause more harm than good (and note this argument doesn’t apply to control center to control center communications, since that is usually just exchange of information. Any decision on what needs to be done as a result of the information will probably be made by a human, for whom a few cycles won’t make much difference either way).

But let’s now pretend the latency problem doesn’t exist; would it then be a good idea to impose cybersecurity controls on substation communications? After all, they will certainly provide some small increase in security.

I have two answers to this question: one in the context of the current prescriptive NERC CIP standards, the other under the assumption that sooner or later they will be replaced by risk-based standards. In the case of the current NERC CIP, these controls should not be prescribed. Whatever small benefit they might incur would be far outweighed by a huge increase in compliance costs for NERC entities.

So how about under a risk-based approach? That is, suppose we had a set of CIP standards that consisted of 1) a requirement to get a comprehensive threat and vulnerability assessment for the entire enterprise (not just the OT systems) and 2) a requirement to develop and implement a cybersecurity improvement plan, based on the results of this assessment?[iv] The standards would come with some sort of guide to areas that need to be examined in the assessment; one of those might be the question whether encrypting routable substation communications would produce a net benefit, in the case of that entity.[v] If it did, the entity would probably need to implement that encryption, unless there were other controls whose net cybersecurity benefits outweighed this.

To summarize, I continue to see no real merit for the Foundation for Resilient Society’s argument that FERC should order encryption of substation communications. However, I strongly suspect I haven’t even given this Hydra a glancing blow, let alone killed it. I’m no Hercules.

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte Advisory.

---

[i] Both this article and the original Foundation press release mistakenly use the phrase “control rooms”. What communicates with substations is a control center, not a control room, which typically controls a particular plant or substation.

[ii] As you’ll see, I make about four or five fairly implausible assumptions below, in order to make the Foundation’s argument at least have some validity. I kind of wish I didn’t have to make their argument for them!

[iii] I found this out in the FBI/IS-CERT briefing on the Ukraine attack in Chicago this morning. Note that the Ukraine attack certainly wasn’t on substation communications. The communications themselves were already compromised because the attackers had complete freedom to move around the IT network, and they took control of the HMIs with remote access to the substation relays.

[iv] Of course, this is a big oversimplification.

[v] One of the big benefits of the risk-based approach is you no longer have to make decisions on which controls are worth imposing and which aren’t, where the controls (requirements) apply to every NERC entity subject to CIP – as is the case today. The controls required in the risk-based approach are those that produce the greatest cybersecurity benefit for that entity, in their role as actor on the grid. In other words, the cyber controls that will produce the greatest reliability impact.