Thursday, April 22, 2021

Will someone please drive a stake through the heart of this lie?

This morning, my good friend Mark Weatherford sent me this article about an interview he’d given in conjunction with notable ICS security consultant Joe Weiss.

I read through most of it, thinking that both Mark and Joe made good points and this was a very reasonable discussion. However, that changed when I came to the part where Joe suddenly said “I'd like to mention one thing. And this actually goes back to what Mark first brought up about supply chain. The Executive Order 13920 came out because there was a Chinese-made transformer that had hardware backdoors preinstalled coming from China.”

This immediately set off alarm bells for me. Let’s be clear: Joe has been pressing this fable about the Chinese-made transformer since early last May. While Joe’s story is based on a real incident, first reported by Rebecca Smith in the Wall Street Journal on May 27 (and written about by me in this post on May 29), it’s 100% false.

I had thought this story was dead on each of these occasions:

1.      When I wrote a post about this on May 31.

2.      When Robert M. Lee, Tim Conway and Jeff Shearer of SANS wrote a Defense Use Case that stated that Joe’s report had zero credibility and was based on zero evidence (although other than that, they thought it was great – for instance, they just loved the font it was written in).

3.      When I put up a post describing the SANS document and proposing what I thought was a quite reasonable alternative explanation.

4.      And finally, when I put up a post on February 1 in response to an interview in Forbes where Joe repeated this lie.

Let me be clear: At no point has Joe ever pointed to any evidence to verify his claims, other than the easily-debunked “evidence” he has provided in his posts and interviews. Yet he keeps bringing this lie up again and again, and he did so again in his interview with Mark. I’m going to go though what Joe said on this topic – and Mark’s responses, which never yielded an inch, bless his heart – blow by blow, in the hope that I can finally drive a stake through the heart of this zombie lie:

1.      When Joe first brings the WAPA transformer up and Mark immediately challenges him on it, Joe says “I was on a call with people who were physically at the substation where that transformer was, as it was being installed.” Joe’s point being that the people installing the transformer realized something was wrong with it.

2.      Mark then correctly notes that the transformer was never even delivered to WAPA (it was intended for WAPA’s Ault substation in Colorado). As Rebecca Smith’s great article points out, it was transported directly from the port of Houston to Sandia National Laboratories when it arrived in early 2020. Moreover, the Jiangsu Huapeng Transformer Company, which made the transformer, was ordered by WAPA in June of 2019, while the transformer was still being manufactured, to change the delivery point for the order from the Ault substation in Colorado to a warehouse at the port of Houston. From there, it was transported – presumably by the US government – to Sandia. So obviously Joe’s statement was wrong.

3.      But Joe wasn’t fazed by this. He then said “There were two transformers involved. The first transformer was installed in the WAPA [Western Area Power Administration] Ault substation, Mark, not far from you, outside of Denver. It was installed in August 2019. When WAPA was doing the site acceptance testing, the mechanical and electrical engineers found the extra electronics in that transformer.”

4.      Of course, it’s hard to believe that this squares with Joe’s first statement, and it certainly doesn’t square at all with his statement (in bold type) in his blog post of May 11, 2020, to the effect that “When the Chinese transformer was delivered to a US utility, the site acceptance testing identified electronics that should NOT have been part of the transformer – hardware backdoors.”

5.      I’ll leave behind the issue of a “hardware backdoor”, which seems to have no discernable meaning, as Robert, Jeff and Tim described in their DUC and I pointed out in my post a few days after the WSJ article came out.

6.      Of course, this didn’t faze Joe! He replies “I have pictures of both transformers—Ault and Houston. As a result of that, the next transformer that arrived at the Port of Houston in early 2020 was intercepted by DOE and taken to Sandia [National Laboratories]. There is a utility missing a transformer. It would have never, ever happened if DOE wasn't so concerned about what they found with the first. What’s missing is what DOE found at Sandia.”

7.      Mark asks for evidence, and Joe provides a masterpiece of obfuscation: “Mark, you were within the government. Go ask DOE.” Note he does what I’ve seen other people do when they have just told a lie and are challenged for evidence: They tell the challenger that they can easily find the evidence for themselves; in other words, it’s an insult to them to even be asked for evidence.

8.      But Joe “backs this up” by continuing “I can read you—I won't even mention the country—an email I got from one of our closest allies. From someone very senior. And it's saying, ‘I am hoping you can help me with something. Regarding the transformer issue you discuss, can you please tell me to what level that information is confirmed?’”

9.      In other words, Joe was challenged for evidence for his story by someone overseas. He doesn’t describe any evidence he gave that person – since he didn’t have any – but he seems to be saying that the fact that this person asked him for evidence somehow indicates the evidence exists in the first place. I don’t quite understand his reasoning in this, but of course the whole idea was to shut Mark up, not to answer his question. 

10.   However, Mark didn’t shut up. He said “Well, I think you just confirmed my point, Joe, and that is, if they don't know, we don't know. Maybe there's nothing to know.”

11.   Joe’s reply to Mark is very interesting. He says “We have a utility missing a transformer. Mark, that has never, ever, ever happened. You don't buy a transformer like it without an absolute need to have it installed.” In other words, he seems to be asking “Why would WAPA have ordered the transformer in the first place, if they just wanted to have it shipped to a warehouse and torn apart?” I’ll address that in a moment. Let’s continue to Joe’s next lie.

12.   Joe goes on to say “When you look at Executive Order 13920, they give a detailed list of all of the equipment that is in scope for Executive Order 13920. Every single item in that executive order is out of scope for NERC CIP. Every single thing in NERC CIP, and in the supply chain, is out of scope for the executive order. We have a problem here. This is a real, honest hardware implant. There are over 200 large Chinese electric transformers in our electric grids today. We have no idea how many of them have these hardware backdoors.” I do have an idea how many large Chinese transformers have hardware backdoors: zero, since there’s no such thing.

13.   But Joe is absolutely right that EO 13920 provides a detailed list of equipment in scope – in fact, there are around 25 items on the list. He’s also right that almost all of those items (at least 21, but not all of them) aren’t in scope for the NERC CIP standards. But there’s a good reason for that: The NERC CIP standards only apply to devices that are operated by – or at least contain – a microprocessor or some other logical hardware (e.g. in a PLC), since only a processor can be subject to a cyberattack. Almost all of the devices in the EO don’t have a processor at all – meaning they are no more subject to a cyberattack than a 1920 automobile, my $5 steam iron, or for that matter a brick. Kevin Perry and I documented that in this post.

14.   This includes transformers. They operate according to the laws of physics, period, not the commands from some processor. They run day and night and don’t need external power to operate their core function. The last time I checked, the laws of physics still apply in China. It’s true that a transformer can have ancillary devices with processors, including load tap changers and dissolved gas analyzers (the former are often external to the transformer itself, and are often made by a different manufacturer than the one that made the transformer). But it’s very hard to see how they could be attacked. Moreover, it’s just as hard to see how a successful attack on one of them could lead to anything more than a brief local outage – and if you’re concerned about local outages, I suggest you figure out a way to address the number one cause of those, which is squirrels. The big national security concern is a widespread, cascading outage, not a local one.

Now, let’s get down to the question Joe (implicitly) asked: “Why would WAPA have ordered the transformer if they had no intention of using it?” That was certainly a question I asked myself when I read the WSJ article last May. It didn’t take too long to figure it out, but I didn’t want to raise this point until now, since it wasn’t required by any discussion. However, since Joe has kindly asked me to provide it, I will now. Here’s what I think happened:

1.      It’s no secret that the Trump administration in 2019 was looking for ways to decrease imports from China. Someone pointed out that WAPA – part of DoE – had bought Chinese large transformers and had one on order at the time.

2.      Someone had the bright idea that they could take a look at the transformer when it arrived, to find out if it contained some sort of “hardware backdoor” that would allow the Chinese to compromise it through some sort of cyberattack (never mind that transformers don’t have a processor to attack) launched over some sort of internet connection (never mind that a device without a processor can’t be connected to any communications network, any more than your living room sofa can).

3.      This also ignored a fact that was pointed out in the WSJ article: Since WAPA isn’t staffed with dumb bunnies, they knew perfectly well they had to be very careful when ordering any grid equipment from China; they left nothing to chance. The article says “…the transformer had been built to WAPA’s exact specifications, down to the parts numbers for the electronics that were sourced from companies WAPA chose in the U.S. and U.K.”

4.      Of course, a privately owned utility would have raised big objections to diverting a huge piece of equipment that was – as Joe points out – needed to maintain grid reliability. But since WAPA is 100% controlled by DoE they had to comply, although – knowing a number of people who work for WAPA – I can assure you they must have been furious, both at losing the transformer and for the implicit judgment that they were too stupid to know they should be very careful when ordering any grid hardware from China (the WSJ article points out that there are a number of US utilities that buy transformers from the same supplier. And there are other transformer suppliers in China as well).

5.      So the effort to find a “hardware backdoor” failed, but of course the transformer was totally destroyed.

6.      As I described in this post, shortly after the EO came out, DoE held two briefings for utility executives and made a point of declaring that they didn’t have to do anything differently now (since many utility executives were under the naïve impression that, because the EO required all purchases of equipment for the Bulk Power System to be stopped pending review of the risk by the Secretary of Energy, this actually meant that they had to do that. How silly of them); if they did, they’d be given plenty of warning before they had to do anything.

7.      If DoE had just discovered a serious hardware backdoor in the WAPA transformer at Sandia (which of course is owned by DoE), don’t you think they would have phrased this a little differently? In fact, wouldn’t they have already held a series of briefings – both classified and unclassified - for the industry? That’s what DHS (I believe) did in 2016, in the wake of the first Ukraine attacks.

So I hope Joe stops peddling this lie. The irony is that he’s done a lifetime of good work and is really one of the founders of ICS security. To have all of that tainted this way is really a shame.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com

 

 

No comments:

Post a Comment