The plot thickens

Note from Tom: If you’re only looking for today’s pandemic post, please go to my new blog. If you’re looking for my cyber/NERC CIP post, you’ve come to the right place.

On Wednesday evening, Rebecca Smith of the Wall Street Journal published a great article on what has to be one of the stanger events in the ongoing story of efforts to improve the cybersecurity of the US electric power grid: Last summer, federal officials took control of a very large transformer that had arrived at the Port of Houston. It was custom built by the Jiangsu Huapeng Transformer Company in China for the Western Area Power Authority (WAPA) and was intended for installation in a WAPA-owned substation outside of Denver. It was taken to Sandia National Laboratory (part of the Department of Energy) outside of Albuquerque, NM and closely examined there. It most likely is still there.

Why did the authorities do this? Of course, nobody at Sandia or DoE would comment. The article does say “Other people, with more limited knowledge of the situation, said federal officials probably commandeered the transformer because they suspected its electronics had been secretly given malicious capabilities, possibly allowing a distant adversary to monitor or even disable it on command. But these people said they didn’t know whether any such alterations were found.” But exactly how would “malicious capabilities” – presumably malicious cyber capabilities – be embedded in a transformer?

Three weeks ago, in a post written with Kevin Perry, I wrote

 ..transformers are extremely important to the grid, since the grid wouldn’t work without them – in fact, news articles I’ve seen consider these to be a big target of the order. Yet these don’t have microprocessors. They don’t need direction in order to do their job, either; the laws of physics give them almost all the instructions they need. My friend Kevin Perry wrote “There may be some new, smart transformers that have microprocessors, but as a general rule, I don’t think the high voltage transformer has electronic systems that can be hacked.  At best, there are sensors throughout the transformer that allow operating conditions to be monitored.  That is not much different than the transducers scattered around a generating plant.  To the extent the transducer voltage output can be recalibrated to produce false readings is about the only issue I am aware of.  But usually you need to be in close proximity to be able to manage such a device.”

Rebecca says something similar in her article:

Federal officials have long worried that foreign adversaries might hack into the utility computer networks that control power flows on transmission lines and cause blackouts.

However, transformers hadn’t typically been seen as products that could be easily isolated and hacked. That is because they don’t contain the software-based control systems that foreign actors could access. They are passive devices that increase or reduce voltages in switchyards, substations and on power poles according to the laws of physics.

So it seems the problem that the government was looking for wasn’t a cyber problem at all, but a physical one. Of course, there are certainly lots of ways a transformer might be rigged to suddenly start malfunctioning at some point, or even to blow up. If a transformer like this failed, it would certainly cause problems for the power distribution system in the area served by that transformer; there might even be an outage as a result of it.

But local outages happen all the time. If you’re looking to greatly reduce the threat of local outages, I recommend you focus on the number one cause of those outages: squirrels (although Kevin pointed out to me that in larger substations, snakes are a bigger problem). If a genetic modification were introduced into the squirrel population so they no longer see insulated wires as a possible food source, that would be a huge step forward in the fight against local outages. But local outages that are caused by substation events are rare, since the grid has tremendous redundancy built in. Even if all the transformers in a substation are brought down (as was the case with the Metcalf attack in California in 2013), that usually wouldn’t lead to an actual outage (for example, there was no outage due to the Metcalf attack).

However, when we talk about attacks on the power grid, we’re not talking about a local outage, but some sort of event – the worst being a cascading outage like the 2003 Northeast Blackout - on the Bulk Electric System (or Bulk Power System, the term used in the Executive Order) itself. This is the network of high voltage power lines and substations that moves power around the country and feeds it into local distribution substations. A true BES attack would have to affect multiple transformers in multiple substations at the same time.

In principle, a BES event could be accomplished by a cyber attack. If a number of these transformers were microprocessor-controlled and connected to a routable (IP) network, and if all of these had some sort of malicious logic embedded in software or firmware, a foreign attacker could in theory send a signal and cause a number of these to go down or malfunction at the same time, which might cause a large-scale grid event. However, transformers aren’t microprocessor-controlled, so this isn’t a realistic scenario.

The WSJ article pointed out that the transformer has sensors to monitor the level of insulating oil (and presumably some sort of microprocessor and communications link to relay that information to the control center), since transformers generate a lot of heat, which needs to be carried away to keep the unit from frying. In theory, the sensors could be recalibrated to report false readings, but as Kevin pointed out in the last three sentences of his quote above, that would almost certainly require someone being onsite to recalibrate the sensors.

Note from Tom 5/29: Kevin provided some good clarifying information to me this afternoon, which I'll pass on in a post tomorrow. It doesn't change any of the conclusions of this post, but it does fill in a few logical gaps.


There are also some smaller transformers that have tap changers, which are controlled by a microprocessor. It's unclear what harm could be caused by misusing a tap changer, but in any case, we don't believe one would be found in a large transformer like the one in question.

Since there’s nothing in the transformer normally that could be the vehicle for a successful cyberattack, it seems the people at Sandia must be looking for some logical device that was implanted in the system in China. What would this device do? It might trigger a bomb to blow the transformer up. It’s hard to see anything else that it could do, since the transformer operates according to the laws of physics – it doesn’t need any sort of commands to operate.

But remember, just having one transformer blow up isn’t going to cause a BES event, and it probably won’t cause even a local outage. There would have to be a coordinated attack on multiple transformers. The article says that Jiangsu Huapeng has installed about 100 of these units in the last decade in the US and Canada. If a significant number of these had some sort of microprocessor attached to a bomb inside of them, some of them might blow up at the same time – which would probably cause a big problem for the grid. But for that to happen, they would all have to receive some signal telling them it’s time to blow up. Since this implanted processor is unlikely to have an external communications port (which would be immediately noticed), then it would need some sort of satellite receiver embedded in it.

So yes, if Jiangsu Huapeng has been implanting these devices in their transformers for some time, and if nobody has ever noticed them before this, then a BES attack might be possible. But here’s the bigger question: What possible benefit would the Chinese reap from conducting such an attack? If it happened, it would be immediately traced back to China and would be rightly treated by the US as an act of war.  And given the relative sizes of the two countries’ militaries, I’d say China’s guaranteed to come out on the worse end of the deal.

Of course, the real question is whether the folks at Sandia have found anything wrong in the transformer. If they had, US utilities – especially the ones who have purchased these transformers – would presumably have been notified immediately. And that would explain why the Executive Order was released very precipitously about a month ago. The industry would have been placed on high alert.

Did that happen? E&E News published a very good article today that quotes a number of industry executives saying they were left completely in the dark about the EO until it was published. And the Department of Energy made clear in a couple calls with the industry recently that the EO doesn’t require utilities to do anything different now that what they were doing before the EO was issued - this despite what seems to be clear language in the EO saying that all procurements of Bulk Power System equipment need to be cleared with DoE as of the day of the order. So this is another good indication that no problems have been found so far with the transformer at Sandia.

Ironically, while the government is making a big effort to find a problem which seems unlikely to exist, there’s a serious foreign cybersecurity threat that the government itself has warned about multiple times (as described in this post, this one and this one, and in this WSJ article from 2018), which still has never even been investigated. Why don’t we investigate that one, too? Either we’ll find something, or the industry can sleep a lot more soundly, knowing that all of these reports were wrong.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Are you working on your CIP-013 plan and you would like some help on it? Or would you like me to review what you’ve written so far and let you know what could be improved? Just drop me an email!