If you're
looking for my pandemic posts, go here.
Tim Conway, Patrick Miller and
Jason Christopher posted a draft response to the DoE Request for Information on Friday July 24, on the SANS ICS Community site. They
asked for comments on it, so here are mine. I have already written two posts (here and here) on the RFI, addressing four of the questions in
the RFI (and since a number of questions have a lot of sub-questions, there are
probably 20-30 questions in total in the document). Reading the draft response
made me want to discuss the RFI in general, since I haven’t done that yet.
Guys, I think you did an
excellent job; you’ve clearly put a huge amount of time into this. But I feel a
sense of regret that the three of you – all of whom have much better uses for
your time – should have even had to make this effort. Because here’s the
problem: If I were to have read the RFI with no knowledge of the Executive
Order that’s behind it, my initial response would be “You gotta be kidding!”
And I would have dumped it in the bit bucket. The same with the EO itself,
which is of course what drove the RFI.
But the problem is that neither
the EO nor the RFI can be ignored, simply because the EO itself presents a threat
to the industry – almost certainly greater than the close-to-infinitesimal threat that the
Chinese will cause a cascading outage on the US grid by implanting malware in a
bunch of devices like transformers that aren’t controlled by any sort of processor
anyway.
Yet the EO on its surface seems
to say that, as of the date it was released (May 1), all purchases of any of the
25-odd types of equipment listed in the EO need to be paused, pending DoE
giving guidance on “safe” vendors and “safe” products.
At this time I want to point
out, as I have before, that I don’t think for one second that DoE was behind
this EO. It came from the White House. I imagine that, when the people at DoE heard
they were going to be in charge of implementing this order, they didn’t exactly
jump for joy.
So here’s how my reply would
read:
Dear DoE:
I will have some strong words
below, but my objections are to the Executive Order, not to your attempts to
make it into something that might at least lead to some benefits to the US
electric grid. The basic problem is that the EO as written will lead to the electric
power industry wasting huge amounts of money and time chasing highly unlikely –
or even impossible – supply chain threats to grid control systems, while
completely ignoring supply chain cybersecurity threats to the grid that have
actually been realized and have been identified by multiple government agencies
as well as the media – yet they haven’t even been investigated yet.
First, let’s be clear about what
constitutes a cybersecurity threat to the grid and what doesn’t:
1.
Only devices that are controlled by some sort of built-in control logic
(usually a microprocessor, but also FPGAs and perhaps other capabilities) can
be subject to a cyberattack – these devices are all in some way “programmable”.
Of the 25-odd devices listed in the EO, Kevin Perry and I, in a post in early May, only identified
five that either always or sometimes meet this criterion. Since the NERC
definition of Cyber Asset is “programmable electronic device”, we can summarize
this criterion by saying only Cyber Assets are subject to cyberattack.
2.
It’s important to point out that a device like a transformer – whose
operation isn’t controlled by anything other than the laws of physics – might sometimes
have auxiliary devices included with it, that might themselves be programmable.
In the case of a transformer, it sometimes has a load tap changer (LTC) associated
with it (which might be installed within the housing of the transformer itself,
or in some cases outside of it. It might or might not be obtained from the supplier of the transformer iteself), which does have a microprocessor. The LTC is a
Cyber Asset, but the transformer itself isn’t one. So transformers shouldn’t
even be on the list of devices covered by the EO (although LTC’s might be added),
as well as any of the other 20-odd devices that aren’t Cyber Assets.
3.
And why do you (i.e. DoE) even ask about minerals? They’re obviously
not subject to a cyberattack. Why are they in the EO at all?
4.
But even being a Cyber Asset shouldn’t mean a device should be a concern. The device has to be capable of impacting the Bulk Electric System if compromised
or destroyed. Cyber Assets that meet this higher criterion are called BES Cyber
Assets - and the ones that can actually damage the BES if compromised are Medium and High impact BCAs. In our post referenced above, Kevin Perry and I couldn’t identify any
devices that meet the definition of BCA, that a) are actually found on the US grid
now and b) are sold by entities headquartered in one of the six “foreign
adversaries” identified in the RFI.
5.
In fact, the only BCAs that we could identify that might even be assembled in one of the six adversary countries are
servers and workstations from Dell or HP, which are sometimes assembled in
China. But if these included an embedded supply chain attack, it would
obviously affect every business in the US that buys HP or Dell computers –
there would be no way to target the attack at the power industry.
6.
In other words, there are simply no devices that are sold or assembled
by any of the six adversary nations, that could be subject to a grid cyberattack of
any kind. But there’s a further qualification: The attack would need to cause something
more than a local outage, to justify the power industry devoting any sort of
significant resources to preventing the attacks (probably the number one cause
of local outages is squirrels. If the goal of the EO is preventing local
outages, then we need to figure out what can be done about them. There might be a genetic engineering solution…). And that means it
would almost have to be a coordinated attack on multiple devices at multiple
locations on the grid. But there must be some means of triggering this attack
remotely. Since medium and high impact transmission substations and grid
Control Centers are already protected to a very high degree by the CIP
standards, it would be very difficult to launch such an attack in the first
place, except perhaps if there were a satellite transceiver embedded in the
device, by whoever implanted the vulnerability itself.
7.
But the EO and the RFI also address the problem of vulnerabilities that
might be embedded in components of these devices, like chips. Since it’s hard
to know what attacks might even come through components, it’s certainly
impossible to rule out those attacks now. But component vulnerabilities – as long
as they meet the criteria enumerated above – could conceivably be a worthwhile
subject of investigation.
8.
However, no utility in the US has the capability even to determine
provenance of components (since they mostly come through a bewildering array of
middlemen and brokers), let alone to subject them to the kinds of analysis
required to identify embedded vulnerabilities or backdoors – which requires
poring over schematic diagrams (and good luck getting those from the component
manufacturer, if you can even identify the manufacturer in the first place) and examining
traces with electron microscopes (of course, even if your average utility had a lab with electron microscopes, this analysis also requires
pulling apart devices the utility has paid for, which will at a minimum void the
warranty and may well make the device inoperable). DoD definitely has this
capability, and I suggest that DoE talk with them about setting up a program
like that – assuming this is really such a big problem. And if it is, then I
suggest you ask for a substantial increase in your budget for next year,
since you’ll need it.
9.
But the whole idea that vulnerabilities embedded in hardware – at the
device level or at the component level – are a serious threat is simply bogus. I
have never heard of a supply chain cyberattack on hardware that could meet all the criteria above - in fact, I've only heard of one successful supply chain attack on hardware, period, although there are probably more. And this was on consumer IoT devices.
10.
However, there is definitely one serious threat to BES Cyber Assets that
comes through the supply chain, that isn’t addressed at all in the EO: That’s
the idea of vulnerabilities and backdoors embedded in software. There have been
a number of successful supply chain attacks on software in other industries,
such as this
one against Delta Airlines. So far, there has been no such attack against the
power industry, although there was a famous – and successful – attack
on Juniper Networks in 2015 (which is suspected to have been carried out by the US
government, although I don’t believe the US is on the list of adversaries).
11.
Much more so than for hardware components, it would be a legitimate exercise
to go over all the software that controls the grid and look for vulnerabilities
or backdoors, especially on the level of third-party or open source components
of that software (usually third-party libraries used by and included with the software). There’s no question there are a lot of vulnerabilities in almost
any software package, but of course the EO isn’t at all concerned with software for some reason, perhaps because China isn't thought of as a software supplier. There hasn’t been any software sold by a foreign
adversary that is installed on BES Cyber Assets, since Kaspersky software was
removed a couple of years ago, but since just about any software you buy now has loads of third-party and open source components embedded in it, there's no telling what you might find if you got down to that level.
However, there is one foreign
adversary that – according to at least five agencies of our own government –
has not only launched supply chain cyber attacks against the US grid, but has
succeeded, through those attacks, in embedding malware at multiple locations in the grid, probably even
in Control Centers. I wrote this
post about that adversary in December, and followed it up with this
open email to Karen Evans of DoE a five days later. Ms. Evans left DoE shortly
after that, and of course I’ve never received a response from anybody on this.
My main point here is that the
Russians have implanted malware in the US grid entirely through supply chain attacks,
of two types:
1.
In their July 2018 briefings, DHS stated that at least two hundred
vendors to the power industry had been penetrated through their remote access
systems – which in very few cases were protected by two factor authentication. The briefings strongly implied that the Russians had succeeded in penetrating more than one utility through this vector, and planting malware in control systems. By the way, this
alone shows that requiring vendors to secure their own remote access systems should
be a big concern of NERC entities as they develop their CIP-013 supply chain
cybersecurity risk management plans. This isn’t explicitly stated in the
requirements (as the risks found in R1.2 are), but it should definitely be
identified as a risk to be mitigated in R1.1.
2.
In January 2019, the Wall Street Journal published a great article describing how the
Russians had conducted (and continued to conduct) an extensive and successful campaign
using phishing emails to penetrate vendors to the grid (my post on the article
is here.
I don’t have a free link to the article itself, but if you drop me an email I’ll
copy the text and send it to you); the article quotes Vikram Thakur of Symantec as saying that eight utility Control Centers were penetrated, and malware was planted. This shows that NERC entities should
identify phishing as another significant risk to vendors (as well as to the
entities themselves, of course) that needs to be mitigated in their CIP 13
program. Yet there has been no investigation of Vikram's statement, of DHS' statements, or of the statements by the FBI and CIA in the Worldwide Threat Assessment of 2019.
Of course, for the specific threat
of malware already implanted in the US grid, the only mitigation is to investigate
whether in fact the government reports are true, and if so describe the malware
and immediately get that information out to US utilities. The fact that this threat
has never even been investigated is an unending source of amazement (and disappointment, to be sure) to me.
So let’s be clear: (1) We have at least three government agencies and a major news outlet saying that supply chain attacks by a foreign adversary have
succeeded not only in penetrating vendors to the power industry but penetrating
grid assets like Control Centers. (2) Yet the White
House says that the biggest threat to the US grid is a mainly theoretical type
of supply chain attack for which it’s just about impossible to identify any
vector that would have any likelihood at all of succeeding. But which of these is the subject of an Executive Order that could well cost US
electric utilities tens of millions of dollars to comply with, and more importantly
could hold up needed improvements to the US grid for maybe years? If you
guessed Door No. 2, you’re right!
Any opinions expressed in this blog post are strictly mine
and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment
on what you have read here, I would love to hear from you. Please email me at
tom@tomalrich.com. Are you hot at work – or should be – on getting ready for
CIP-013-1 compliance on October 1? Here is my summary of what you need to do between now and then.
Or are you a vendor to the power industry that is wondering
what your obligations will be under CIP-013, and how you might meet those
obligations? Contact me as well.
No comments:
Post a Comment