Joe Weiss solves the mystery!

Note from Tom: If you’re only looking for today’s pandemic post, please go to my new blog. If you’re looking for my cyber/NERC CIP post, you’ve come to the right place.

The Wall Street Journal, E&E News and this blog have all written about the fact that a large transformer, custom-made in China for the Western Area Power Administration, was transferred from the port of Houston – where it arrived last summer – to Sandia National Laboratory (owned by the Department of Energy) in Albuquerque, New Mexico; it evidently remains there as of today, presumably being examined for…what? What could be planted in a transformer, that would pose a threat to the Bulk Power System? And was it something that was found during this examination that led to the May 1 Executive Order? None of us know the answer to that question.

However, it seems that longtime control system security guru Joe Weiss knew about the transformer being diverted more than two weeks before the WSJ article, judging by the date on this post by him. Not only that, he says he knows exactly what was found by the examiners – and he is quite definite that a serious problem was found. There’s only one problem with what he says: it doesn’t make sense.

I’ll let you read Joe’s post (and he certainly makes some good general points in it; I’m not disputing those). Here are the problems I’ve found with it:

1.      In the fourth paragraph, he says “Government and public utility procurement rules often push organizations into buying equipment due to price and without regard to origin or risk. In this case, it resulted in a utility having to procure a very large bulk transmission transformer from China.” I pointed out recently that utilities definitely don’t procure sensitive grid equipment based just on cost. But in this case, there are two additional problems with Joe’s statement.

2.      The first of these is that the utility, Western Area Power Authority (WAPA) isn’t strictly speaking a utility at all. It’s one of four Power Marketing Agencies owned and run by the Department of Energy; WAPA’s job is to distribute power from federal dams to cooperative and municipal utilities in the West. And I can assure you that WAPA wouldn’t think two seconds about what to do, if someone informed them that the transformer they were about to buy could be purchased for less somewhere else, but with perhaps a lesser degree of security. Of course, Joe didn’t seem to know, when he wrote the post, that the “utility” was WAPA, but the same can be said for any other utility. It’s too bad to see this old canard still alive.

3.      Joe continues to say “When the Chinese transformer was delivered to a US utility, the site acceptance testing identified electronics that should NOT have been part of the transformer – hardware backdoors.” First off, the WSJ article makes clear that the transformer was never even delivered to WAPA – it went right from the port of Houston to Sandia National Labs. But this in itself doesn’t invalidate Joe’s point that a “hardware backdoor” was discovered, since he may not have known this.

4.      But I’ve never heard of a “hardware backdoor”. I have only heard of software backdoors; these are a big supply chain risk, as various entities like Juniper and Delta Airlines have found out to their chagrin. Since I’m sure Joe doesn’t mean a literal back door in the housing of the transformer, he must mean firmware (i.e. software that is embedded in chips, not read from a storage device like a hard drive) that controls a microprocessor performing some function within the transformer. But as Kevin Perry and I have pointed out in this post and this one, there is no microprocessor[i] that controls the transformer in any way; at most, there’s usually one that reports operational data out to the control center. So Kevin’s and my question from yesterday remains: Where is the microprocessor that’s going to be affected by this “backdoor”?

5.      Joe goes on to say, in the same paragraph “It is unclear just how widespread the impact of compromised transformers and other grid equipment are (sic) though it is safe to say it is more than just one transformer. Could this be considered an act of war?” Sure it could, if this “hardware backdoor” were found in multiple transformers. But first I want to know what this miraculous hardware backdoor is, which seems to be able to cripple a transformer without having a microprocessor to run on.

6.      The next paragraph begins “The need for having spare transformers started almost 20 years ago because it was recognized these very expensive, long-term procurement items could have a major impact on grid availability. However, unless the devices that are inside or supporting the operation of the transformers (and generators, motors, valves, capacitor banks, etc.) are also addressed, the pool of spare transformers and other large equipment can be quickly exhausted by damaging the equipment from “within”.”

7.      Wow! This one is the mother of all FUD. Let’s try to unpack it. Joe talks about “devices that are inside or supporting the operation of the transformers”. Then he lists four “devices”; none of them are either found inside a transformer or support it. He’s correct that all of these devices have something to do with electricity, but that’s about all they have in common with a transformer. And his phrase “the pool of spare transformers and other large equipment can be quickly exhausted by damaging the equipment from ‘within’” seems to say that spare transformers – which of course won’t be connected to the grid at all – will be “exhausted” because of some unnamed attacks (perhaps the “hardware backdoor” attacks?). Or something like that. But who cares what this means? It sure sounds serious!

Finally, Joe brings up the Aurora vulnerability, which was used in a demonstration by Idaho National Labs in 2007 to cause a generator to literally blow itself to pieces. In fact, what could be considered the summation of his whole argument is printed in boldface type: "What the Chinese did was install hardware backdoors that can cause an Aurora or other type of damaging event at a time of their choosing." However, the Aurora vulnerability affects rotating equipment like the generator. It couldn’t affect a large transformer at all, since there are no moving parts in a transformer[ii], rotating or not.

Joe is obviously aware of this objection, since he goes on to say “Remotely accessing the protective relays can cause an Aurora event damaging the transformer and AC rotating equipment such as generators and motors connected to that substation. What the Chinese did was install hardware backdoors that can cause an Aurora or other type of damaging event at a time of their choosing.” So it seems the “hardware backdoor” – embedded in firmware that controls the non-existent processor that “controls” the transformer, even though the latter is controlled by nothing other than the laws of physics – is somehow able to damage not only the transformer, but generators and motors “connected” to the substation. Yet nobody has even suggested before that Aurora could damage anything more than the generator it directly attacks. I sure don’t understand this, but that obviously means this is a super-serious problem! Maybe we should call in the air force...

As Kevin pointed out in an email, “…the Aurora test is designed to destroy large rotating machines, such as generators, by connecting them to the grid out of phase.  120 degrees out of phase produces maximum damage.  No such vulnerability exists with breakers, transformers, and the like.  I have never seen a phase synchronization process for closing a breaker and energizing a transformer.”

But here’s another reason why it’s not believable that the people at Sandia found something really amiss with the transformer: There would surely have been some sort of notice to the industry, since presumably this whatever-it-is would be found in other Chinese transformers as well. If it’s such a big threat to the grid, you don’t want to hide the news. Of course, since they would undoubtedly be classified, the authorities wouldn’t publish the details in the newspaper; but they would set up classified briefings, etc. And the notifications of these briefings would go to the entire utility community. Neither Kevin nor I have heard anything about this.

And here’s yet another reason: DoE held a couple briefings for the industry after the EO came out. In those briefings, they bent over backwards to assure the listeners that nothing needs to be done now, other than what they’ve always been doing. This hardly sounds like the EO was issued in response to some grave danger.

So definitely take everything that Joe says in his post with a grain of salt. Unless, like me, you’re on a low-sodium diet. Then skip the salt.

Tom 5/31: Orlando Stevenson of NERC pointed out in a comment on Friday's post that tap changers have their own microprocessor-based controllers. If that were to be compromised and the tap changer itself malfunctioned, there could be a BES impact, although this would probably have to occur in multiple substations simultaneously. Kevin agrees with that, although he points out that the controller is always external to the transformer itself, and sometimes it resides in the substation control house (and it is sometimes made by a different manufacturer than the manfuracturer of the transformer. For example, GE makes a tap changer controller that works with multiple manufacturers' transformers, not just their own). And BTW, Kevin - ever the auditor! - adds that these tap changer controllers should be identified as BES Cyber Assets, since they could have a 15-minute BES impact.

So this means there might be a way for the Chinese to affect the BES through a transformer, by planting malware in the external tap changer controller (and remember, they'd have to do this in multiple transformers in multiple substations, in order to have a BES impact). But now I have to go back to the question I asked in yesterday's post: Why on earth would the Chinese want to do this, since it would likely be interpreted as an act of war?

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Are you working on your CIP-013 plan and you would like some help on it? Or would you like me to review what you’ve written so far and let you know what could be improved? Just drop me an email!

---

[i] Dick Brooks pointed out to me that Field Programmable Gate Array (FPGA) chips can execute commands, so a microprocessor isn’t necessarily required. But it still comes down to the fact that the transformer doesn’t operate based on controls; it operates according to the laws of physics. The only thing that Kevin and I can think of, that could impact the operation of the transformer, is if the microprocessor/FPGA activated a bomb to blow the transformer up.

[ii] Other than a tap changer. But these aren’t found in large transformers like the type in question.