This morning, my good friend Mark Weatherford sent me this article
about an interview he’d given in conjunction with notable ICS security
consultant Joe Weiss.
I read through most of it, thinking that both Mark and Joe
made good points and this was a very reasonable discussion. However, that
changed when I came to the part where Joe suddenly said “I'd like to mention
one thing. And this actually goes back to what Mark first brought up about
supply chain. The Executive
Order 13920 came out because there was a Chinese-made transformer that
had hardware backdoors preinstalled coming from China.”
This immediately set off alarm bells for me. Let’s be clear:
Joe has been pressing this fable
about the Chinese-made transformer since early last May. While Joe’s story is
based on a real incident, first reported
by Rebecca Smith in the Wall Street Journal on May 27 (and written about
by me in this
post on May 29), it’s 100% false.
I had thought this story was dead on each of these
occasions:
1.
When I wrote a post
about this on May 31.
2.
When Robert M. Lee, Tim Conway and Jeff Shearer
of SANS wrote a Defense
Use Case that stated that Joe’s report had zero credibility and was
based on zero evidence (although other than that, they thought it was great –
for instance, they just loved the font it was written in).
3.
When I put up a post
describing the SANS document and proposing what I thought was a quite reasonable
alternative explanation.
4.
And finally, when I put up a post
on February 1 in response to an interview in Forbes where Joe repeated
this lie.
Let me be clear: At no point has Joe ever pointed to any
evidence to verify his claims, other than the easily-debunked “evidence” he has
provided in his posts and interviews. Yet he keeps bringing this lie up again
and again, and he did so again in his interview with Mark. I’m going to go
though what Joe said on this topic – and Mark’s responses, which never yielded
an inch, bless his heart – blow by blow, in the hope that I can finally drive a
stake through the heart of this zombie lie:
1.
When Joe first brings the WAPA transformer up
and Mark immediately challenges him on it, Joe says “I was on a call with
people who were physically at the substation where that transformer was, as it
was being installed.” Joe’s point being that the people installing the transformer
realized something was wrong with it.
2.
Mark then correctly notes that the transformer was
never even delivered to WAPA (it was intended for WAPA’s Ault substation
in Colorado). As Rebecca Smith’s great article points out, it was transported
directly from the port of Houston to Sandia National Laboratories when it
arrived in early 2020. Moreover, the Jiangsu Huapeng Transformer Company, which
made the transformer, was ordered by WAPA in June of 2019, while the
transformer was still being manufactured, to change the delivery point for the
order from the Ault substation in Colorado to a warehouse at the port of
Houston. From there, it was transported – presumably by the US government – to
Sandia. So obviously Joe’s statement was wrong.
3.
But Joe wasn’t fazed by this. He then said “There
were two transformers involved. The first transformer was installed in the WAPA
[Western Area Power Administration] Ault substation, Mark, not far from you,
outside of Denver. It was installed in August 2019. When WAPA was doing the
site acceptance testing, the mechanical and electrical engineers found the
extra electronics in that transformer.”
4.
Of course, it’s hard to believe that this
squares with Joe’s first statement, and it certainly doesn’t square at all with
his statement (in bold type) in his blog
post of May 11, 2020, to the effect that “When the Chinese
transformer was delivered to a US utility, the site acceptance testing
identified electronics that should NOT have been part of the transformer –
hardware backdoors.”
5.
I’ll leave behind the issue of a “hardware
backdoor”, which seems to have no discernable meaning, as Robert, Jeff and Tim described
in their DUC and I pointed out in my post
a few days after the WSJ article came out.
6.
Of course, this didn’t faze Joe! He replies “I
have pictures of both transformers—Ault and Houston. As a result of that, the
next transformer that arrived at the Port of Houston in early 2020 was
intercepted by DOE and taken to Sandia [National Laboratories]. There is a
utility missing a transformer. It would have never, ever happened if DOE wasn't
so concerned about what they found with the first. What’s missing is what DOE
found at Sandia.”
7.
Mark asks for evidence, and Joe provides a
masterpiece of obfuscation: “Mark, you were within the government. Go ask DOE.”
Note he does what I’ve seen other people do when they have just told a lie and
are challenged for evidence: They tell the challenger that they can easily find
the evidence for themselves; in other words, it’s an insult to them to even be
asked for evidence.
8.
But Joe “backs this up” by continuing “I can
read you—I won't even mention the country—an email I got from one of our
closest allies. From someone very senior. And it's saying, ‘I am hoping you can
help me with something. Regarding the transformer issue you discuss, can you
please tell me to what level that information is confirmed?’”
9.
In other words, Joe was challenged for evidence for
his story by someone overseas. He doesn’t describe any evidence he gave that
person – since he didn’t have any – but he seems to be saying that the fact
that this person asked him for evidence somehow indicates the evidence exists
in the first place. I don’t quite understand his reasoning in this, but of
course the whole idea was to shut Mark up, not to answer his question.
10.
However, Mark didn’t shut up. He said “Well, I
think you just confirmed my point, Joe, and that is, if they don't know, we
don't know. Maybe there's nothing to know.”
11.
Joe’s reply to Mark is very interesting. He says
“We have a utility missing a transformer. Mark, that has never, ever, ever
happened. You don't buy a transformer like it without an absolute need to have
it installed.” In other words, he seems to be asking “Why would WAPA have
ordered the transformer in the first place, if they just wanted to have it
shipped to a warehouse and torn apart?” I’ll address that in a moment. Let’s
continue to Joe’s next lie.
12.
Joe goes on to say “When you look at Executive
Order 13920, they give a detailed list of all of the equipment that is in scope
for Executive Order 13920. Every single item in that executive order is out of
scope for NERC CIP. Every single thing in NERC CIP, and in the supply chain, is
out of scope for the executive order. We have a problem here. This is a real,
honest hardware implant. There are over 200 large Chinese electric transformers
in our electric grids today. We have no idea how many of them have these
hardware backdoors.” I do have an idea how many large Chinese transformers have
hardware backdoors: zero, since there’s no such thing.
13.
But Joe is absolutely right that EO 13920
provides a detailed list of equipment in scope – in fact, there are around 25
items on the list. He’s also right that almost all of those items (at least 21,
but not all of them) aren’t in scope for the NERC CIP standards. But there’s a
good reason for that: The NERC CIP standards only apply to devices that are
operated by – or at least contain – a microprocessor or some other logical
hardware (e.g. in a PLC), since only a processor can be subject to a
cyberattack. Almost all of the devices in the EO don’t have a processor at all
– meaning they are no more subject to a cyberattack than a 1920 automobile, my
$5 steam iron, or for that matter a brick. Kevin Perry and I documented that in
this
post.
14.
This includes transformers. They operate according
to the laws of physics, period, not the commands from some processor. They run
day and night and don’t need external power to operate their core function. The
last time I checked, the laws of physics still apply in China. It’s true that a
transformer can have ancillary devices with processors, including load tap
changers and dissolved gas analyzers (the former are often external to the
transformer itself, and are often made by a different manufacturer than the one
that made the transformer). But it’s very hard to see how they could be
attacked. Moreover, it’s just as hard to see how a successful attack on one of
them could lead to anything more than a brief local outage – and if you’re
concerned about local outages, I suggest you figure out a way to address the
number one cause of those, which is squirrels. The big national security
concern is a widespread, cascading outage, not a local one.
Now, let’s get down to the question Joe (implicitly) asked: “Why
would WAPA have ordered the transformer if they had no intention of using it?” That
was certainly a question I asked myself when I read the WSJ article last May.
It didn’t take too long to figure it out, but I didn’t want to raise this point
until now, since it wasn’t required by any discussion. However, since Joe has
kindly asked me to provide it, I will now. Here’s what I think happened:
1.
It’s no secret that the Trump administration in
2019 was looking for ways to decrease imports from China. Someone pointed out
that WAPA – part of DoE – had bought Chinese large transformers and had one on
order at the time.
2.
Someone had the bright idea that they could take
a look at the transformer when it arrived, to find out if it contained some
sort of “hardware backdoor” that would allow the Chinese to compromise it
through some sort of cyberattack (never mind that transformers don’t have a
processor to attack) launched over some sort of internet connection (never mind
that a device without a processor can’t be connected to any communications
network, any more than your living room sofa can).
3.
This also ignored a fact that was pointed out in
the WSJ article: Since WAPA isn’t staffed with dumb bunnies, they knew
perfectly well they had to be very careful when ordering any grid equipment
from China; they left nothing to chance. The article says “…the transformer had
been built to WAPA’s exact specifications, down to the parts numbers for the
electronics that were sourced from companies WAPA chose in the U.S. and U.K.”
4.
Of course, a privately owned utility would have
raised big objections to diverting a huge piece of equipment that was – as Joe
points out – needed to maintain grid reliability. But since WAPA is 100%
controlled by DoE they had to comply, although – knowing a number of people who
work for WAPA – I can assure you they must have been furious, both at losing
the transformer and for the implicit judgment that they were too stupid to know
they should be very careful when ordering any grid hardware from China (the WSJ
article points out that there are a number of US utilities that buy
transformers from the same supplier. And there are other transformer suppliers
in China as well).
5.
So the effort to find a “hardware backdoor”
failed, but of course the transformer was totally destroyed.
6.
As I described in this post,
shortly after the EO came out, DoE held two briefings for utility executives
and made a point of declaring that they didn’t have to do anything differently
now (since many utility executives were under the naïve impression that,
because the EO required all purchases of equipment for the Bulk Power System to
be stopped pending review of the risk by the Secretary of Energy, this actually
meant that they had to do that. How silly of them); if they did, they’d be
given plenty of warning before they had to do anything.
7.
If DoE had just discovered a serious hardware
backdoor in the WAPA transformer at Sandia (which of course is owned by DoE),
don’t you think they would have phrased this a little differently? In fact,
wouldn’t they have already held a series of briefings – both classified and unclassified
- for the industry? That’s what DHS (I believe) did in 2016, in the wake of the
first Ukraine attacks.
So I hope Joe stops peddling this lie. The irony is that he’s
done a lifetime of good work and is really one of the founders of ICS security.
To have all of that tainted this way is really a shame.
Any opinions expressed in this
blog post are strictly mine and are not necessarily shared by any of the
clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would
love to hear from you. Please email me at tom@tomalrich.com.
