Sunday, July 26, 2020

The DoE RFI: No, there’s no monster under your bed, but let’s investigate the one peeking in your window




If you're looking for my pandemic posts, go here.


Tim Conway, Patrick Miller and Jason Christopher posted a draft response to the DoE Request for Information on Friday July 24, on the SANS ICS Community site. They asked for comments on it, so here are mine. I have already written two posts (here and here) on the RFI, addressing four of the questions in the RFI (and since a number of questions have a lot of sub-questions, there are probably 20-30 questions in total in the document). Reading the draft response made me want to discuss the RFI in general, since I haven’t done that yet.

Guys, I think you did an excellent job; you’ve clearly put a huge amount of time into this. But I feel a sense of regret that the three of you – all of whom have much better uses for your time – should have even had to make this effort. Because here’s the problem: If I were to have read the RFI with no knowledge of the Executive Order that’s behind it, my initial response would be “You gotta be kidding!” And I would have dumped it in the bit bucket. The same with the EO itself, which is of course what drove the RFI.

But the problem is that neither the EO nor the RFI can be ignored, simply because the EO itself presents a threat to the industry – almost certainly greater than the close-to-infinitesimal threat that the Chinese will cause a cascading outage on the US grid by implanting malware in a bunch of devices like transformers that aren’t controlled by any sort of processor anyway.

Yet the EO on its surface seems to say that, as of the date it was released (May 1), all purchases of any of the 25-odd types of equipment listed in the EO need to be paused, pending DoE giving guidance on “safe” vendors and “safe” products.

At this time I want to point out, as I have before, that I don’t think for one second that DoE was behind this EO. It came from the White House. I imagine that, when the people at DoE heard they were going to be in charge of implementing this order, they didn’t exactly jump for joy.

So here’s how my reply would read:

Dear DoE:

I will have some strong words below, but my objections are to the Executive Order, not to your attempts to make it into something that might at least lead to some benefits to the US electric grid. The basic problem is that the EO as written will lead to the electric power industry wasting huge amounts of money and time chasing highly unlikely – or even impossible – supply chain threats to grid control systems, while completely ignoring supply chain cybersecurity threats to the grid that have actually been realized and have been identified by multiple government agencies as well as the media – yet they haven’t even been investigated yet.

First, let’s be clear about what constitutes a cybersecurity threat to the grid and what doesn’t:

1.      Only devices that are controlled by some sort of built-in control logic (usually a microprocessor, but also FPGAs and perhaps other capabilities) can be subject to a cyberattack – these devices are all in some way “programmable”. Of the 25-odd devices listed in the EO, Kevin Perry and I, in a post in early May, only identified five that either always or sometimes meet this criterion. Since the NERC definition of Cyber Asset is “programmable electronic device”, we can summarize this criterion by saying only Cyber Assets are subject to cyberattack.

2.      It’s important to point out that a device like a transformer – whose operation isn’t controlled by anything other than the laws of physics – might sometimes have auxiliary devices included with it, that might themselves be programmable. In the case of a transformer, it sometimes has a load tap changer (LTC) associated with it (which might be installed within the housing of the transformer itself, or in some cases outside of it. It might or might not be obtained from the supplier of the transformer iteself), which does have a microprocessor. The LTC is a Cyber Asset, but the transformer itself isn’t one. So transformers shouldn’t even be on the list of devices covered by the EO (although LTC’s might be added), as well as any of the other 20-odd devices that aren’t Cyber Assets.

3.      And why do you (i.e. DoE) even ask about minerals? They’re obviously not subject to a cyberattack. Why are they in the EO at all? 

4.      But even being a Cyber Asset shouldn’t mean a device should be a concern. The device has to be capable of impacting the Bulk Electric System if compromised or destroyed. Cyber Assets that meet this higher criterion are called BES Cyber Assets - and the ones that can actually damage the BES if compromised are Medium and High impact BCAs. In our post referenced above, Kevin Perry and I couldn’t identify any devices that meet the definition of BCA, that a) are actually found on the US grid now and b) are sold by entities headquartered in one of the six “foreign adversaries” identified in the RFI.

5.      In fact, the only BCAs that we could identify that might even be assembled in one of the six adversary countries are servers and workstations from Dell or HP, which are sometimes assembled in China. But if these included an embedded supply chain attack, it would obviously affect every business in the US that buys HP or Dell computers – there would be no way to target the attack at the power industry. 

6.      In other words, there are simply no devices that are sold or assembled by any of the six adversary nations, that could be subject to a grid cyberattack of any kind. But there’s a further qualification: The attack would need to cause something more than a local outage, to justify the power industry devoting any sort of significant resources to preventing the attacks (probably the number one cause of local outages is squirrels. If the goal of the EO is preventing local outages, then we need to figure out what can be done about them. There might be a genetic engineering solution…). And that means it would almost have to be a coordinated attack on multiple devices at multiple locations on the grid. But there must be some means of triggering this attack remotely. Since medium and high impact transmission substations and grid Control Centers are already protected to a very high degree by the CIP standards, it would be very difficult to launch such an attack in the first place, except perhaps if there were a satellite transceiver embedded in the device, by whoever implanted the vulnerability itself.

7.      But the EO and the RFI also address the problem of vulnerabilities that might be embedded in components of these devices, like chips. Since it’s hard to know what attacks might even come through components, it’s certainly impossible to rule out those attacks now. But component vulnerabilities – as long as they meet the criteria enumerated above – could conceivably be a worthwhile subject of investigation.

8.      However, no utility in the US has the capability even to determine provenance of components (since they mostly come through a bewildering array of middlemen and brokers), let alone to subject them to the kinds of analysis required to identify embedded vulnerabilities or backdoors – which requires poring over schematic diagrams (and good luck getting those from the component manufacturer, if you can even identify the manufacturer in the first place) and examining traces with electron microscopes (of course, even if your average utility had a lab with electron microscopes, this analysis also requires pulling apart devices the utility has paid for, which will at a minimum void the warranty and may well make the device inoperable). DoD definitely has this capability, and I suggest that DoE talk with them about setting up a program like that – assuming this is really such a big problem. And if it is, then I suggest you ask for a substantial increase in your budget for next year, since you’ll need it.

9.      But the whole idea that vulnerabilities embedded in hardware – at the device level or at the component level – are a serious threat is simply bogus. I have never heard of a supply chain cyberattack on hardware that could meet all the criteria above - in fact, I've only heard of one successful supply chain attack on hardware, period, although there are probably more. And this was on consumer IoT devices.

10.   However, there is definitely one serious threat to BES Cyber Assets that comes through the supply chain, that isn’t addressed at all in the EO: That’s the idea of vulnerabilities and backdoors embedded in software. There have been a number of successful supply chain attacks on software in other industries, such as this one against Delta Airlines. So far, there has been no such attack against the power industry, although there was a famous – and successful – attack on Juniper Networks in 2015 (which is suspected to have been carried out by the US government, although I don’t believe the US is on the list of adversaries).

11.   Much more so than for hardware components, it would be a legitimate exercise to go over all the software that controls the grid and look for vulnerabilities or backdoors, especially on the level of third-party or open source components of that software (usually third-party libraries used by and included with the software). There’s no question there are a lot of vulnerabilities in almost any software package, but of course the EO isn’t at all concerned with software for some reason, perhaps because China isn't thought of as a software supplier. There hasn’t been any software sold by a foreign adversary that is installed on BES Cyber Assets, since Kaspersky software was removed a couple of years ago, but since just about any software you buy now has loads of third-party and open source components embedded in it, there's no telling what you might find if you got down to that level.

However, there is one foreign adversary that – according to at least five agencies of our own government – has not only launched supply chain cyber attacks against the US grid, but has succeeded, through those attacks, in embedding malware at multiple locations in the grid, probably even in Control Centers. I wrote this post about that adversary in December, and followed it up with this open email to Karen Evans of DoE a five days later. Ms. Evans left DoE shortly after that, and of course I’ve never received a response from anybody on this.

My main point here is that the Russians have implanted malware in the US grid entirely through supply chain attacks, of two types:

1.      In their July 2018 briefings, DHS stated that at least two hundred vendors to the power industry had been penetrated through their remote access systems – which in very few cases were protected by two factor authentication. The briefings strongly implied that the Russians had succeeded in penetrating more than one utility through this vector, and planting malware in control systems. By the way, this alone shows that requiring vendors to secure their own remote access systems should be a big concern of NERC entities as they develop their CIP-013 supply chain cybersecurity risk management plans. This isn’t explicitly stated in the requirements (as the risks found in R1.2 are), but it should definitely be identified as a risk to be mitigated in R1.1.

2.      In January 2019, the Wall Street Journal published a great article describing how the Russians had conducted (and continued to conduct) an extensive and successful campaign using phishing emails to penetrate vendors to the grid (my post on the article is here. I don’t have a free link to the article itself, but if you drop me an email I’ll copy the text and send it to you); the article quotes Vikram Thakur of Symantec as saying that eight utility Control Centers were penetrated, and malware was planted. This shows that NERC entities should identify phishing as another significant risk to vendors (as well as to the entities themselves, of course) that needs to be mitigated in their CIP 13 program. Yet there has been no investigation of Vikram's statement, of DHS' statements, or of the statements by the FBI and CIA in the Worldwide Threat Assessment of 2019.

Of course, for the specific threat of malware already implanted in the US grid, the only mitigation is to investigate whether in fact the government reports are true, and if so describe the malware and immediately get that information out to US utilities. The fact that this threat has never even been investigated is an unending source of amazement (and disappointment, to be sure) to me.

So let’s be clear: (1) We have at least three government agencies and a major news outlet saying that supply chain attacks by a foreign adversary have succeeded not only in penetrating vendors to the power industry but penetrating grid assets like Control Centers. (2) Yet the White House says that the biggest threat to the US grid is a mainly theoretical type of supply chain attack for which it’s just about impossible to identify any vector that would have any likelihood at all of succeeding. But which of these is the subject of an Executive Order that could well cost US electric utilities tens of millions of dollars to comply with, and more importantly could hold up needed improvements to the US grid for maybe years? If you guessed Door No. 2, you’re right!


Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Are you hot at work – or should be – on getting ready for CIP-013-1 compliance on October 1? Here is my summary of what you need to do between now and then.

Or are you a vendor to the power industry that is wondering what your obligations will be under CIP-013, and how you might meet those obligations? Contact me as well. 




No comments:

Post a Comment