What exactly is the goal of this Executive Order, anyway?

Note from Tom: If you’re only looking for today’s pandemic post, please go to my new blog. If you’re looking for my cyber/NERC CIP posts, you’ve come to the right place.

The Executive Order doesn’t make any more sense to me today than it did when I wrote this post on Saturday. I simply can’t understand what it’s trying to achieve.

Of course, the stated purpose is very clear. The first paragraph reads

I, DONALD J. TRUMP, President of the United States of America, find that foreign adversaries are increasingly creating and exploiting vulnerabilities in the United States bulk-power system, which provides the electricity that supports our national defense, vital emergency services, critical infrastructure, economy, and way of life.  The bulk-power system is a target of those seeking to commit malicious acts against the United States and its people, including malicious cyber activities, because a successful attack on our bulk-power system would present significant risks to our economy, human health and safety, and would render the United States less capable of acting in defense of itself and its allies.

So it’s all about preventing cyber attacks, right? I totally agree that bad people are targeting the bulk power system with cyber attacks all the time. The number one culprit in that regard is Russia, and the 2019 Worldwide Threat Assessment stated that Russia has planted malware in the grid and could cause outages at any time; yet there’s been no investigation of this threat, let alone rooting out the malware, etc.

But of course, the EO isn’t targeting that kind of attack. It targets supply chain cyber attacks, in which malware is embedded in software or firmware on devices that run the grid. The general idea is that the malware will be activated at a later date, usually by some sort of signal sent through the internet; this will cause the device to misoperate or not operate at all, leading to some sort of effect on the grid. Yet I can see just about zero possibility that the EO could help mitigate this threat.

First, what kind of devices are the subject of cyber attacks? Ones with microprocessors in them, of course. I’m not worried about a cyber attack taking down my steam iron or toaster, but I could worry about an attack on my computer, my smart phone, my car, etc. – these all have microprocessors. 

So what devices on the power grid have microprocessors (or more generally, programmable logic circuits, since there wouldn’t necessarily have to be a microprocessor for it to be attacked by cyber means)? Fortunately, due to the NERC CIP standards, we have a collective term for these: Cyber Assets, defined as “programmable electronic devices”.

What does the EO apply to? That’s very clear (Sec. 4(b)): “..items used in bulk-power system substations, control rooms, or power generating stations, including reactors, capacitors, substation transformers, current coupling capacitors, large generators, backup generators, substation voltage regulators, shunt capacitor equipment, automatic circuit reclosers, instrument transformers, coupling capacity voltage transformers, protective relaying, metering equipment, high voltage circuit breakers, generation turbines, industrial control systems, distributed control systems, and safety instrumented systems.  Items not included in the preceding list and that have broader application of use beyond the bulk-power system are outside the scope of this order.”

So here’s the question: Which of these items typically have a microprocessor? This comes down to the question “Which of these items are Cyber Assets?” There are only three that are guaranteed to have a microprocessor – “industrial control systems, distributed control systems, and safety instrumented systems”. That’s because these are by definition programmable electronic devices. The only other two devices that usually have microprocessors are protective relays and usually backup generators.

As far as the other devices go, they are dumb and they’re deliberately dumb. They’re meant to take their orders from a control system. For example, circuit breakers are probably the devices that can have the greatest impact on the grid if misoperated. But they don’t make decisions on their own; they’re always controlled by a relay, which nowadays almost always has a microprocessor in it.

As another example, transformers are extremely important to the grid, since the grid wouldn’t work without them – in fact, news articles I’ve seen consider these to be a big target of the order. Yet these don’t have microprocessors. They don’t need direction in order to do their job, either; the laws of physics give them almost all the instructions they need. My friend Kevin Perry wrote “There may be some new, smart transformers that have microprocessors, but as a general rule, I don’t think the high voltage transformer has electronic systems that can be hacked.  At best, there are sensors throughout the transformer that allow operating conditions to be monitored.  That is not much different than the transducers scattered around a generating plant.  To the extent the transducer voltage output can be recalibrated to produce false readings is about the only issue I am aware of.  But usually you need to be in close proximity to be able to manage such a device.”

So transformers can’t be hacked, as well as most of the other devices on the list in the Order. This leads to the question: Why does the Order apply to essentially all devices, dumb or smart, on the grid, when only smart devices can be targeted in a cyber attack? Remember, for every one of these devices, utilities are going to have to get permission in order to buy them now. Of course, the Order does say that the Secretary of Energy can decide to pre-qualify vendors and products, to maybe speed up the process. But how long will it take for that to happen? The Order says the Secretary will publish rules in 150 days, yet it took effect immediately last Thursday. So all purchasing of grid equipment is going to be very uncertain for at least 150 days and the uncertainty will go on from there, unless the White House puts out an amended order right away that says the only devices it applies to are Cyber Assets. Of course, they’re not going to do that – and this means we’re placing a big burden on the grid right now (as if the current Covid-19 situation weren’t already a pretty significant burden!), in order to protect a small subset of the devices the order applies to.

Now let’s focus on that small subset: protective relays, backup generators, industrial control systems, distributed control systems, and safety instrumented systems. Which of these could have a significant impact on the grid, if compromised or destroyed? Here’s another way the NERC CIP terms can help us. Within the universe of Cyber Assets, there’s a subset called BES Cyber Assets. These are devices whose misoperation or loss could impact the Bulk Electric System (essentially, the same as the Bulk Power System). The Order should really just be focused on these, not even Cyber Assets in general.

Now the question becomes: What BES Cyber Assets are made with some sort of significant foreign involvement, by organizations headquartered in or possibly controlled by governments that pose a threat to the US? And face it: The governments we’re talking about here are Russia and China. On Saturday and today, Kevin Perry and I brainstormed about this question. What was our answer?

First, there’s an important semantic question to be resolved. The order says it applies to “items used in bulk-power system substations, control rooms, or power generating stations..” Substations and generating stations are clear, but “control rooms” isn’t. This is because that term usually applies to a room in a generating plant that contains the control systems that run the equipment in the plant. But since “generating stations” are already included in the list, Kevin and I guessed that what was really meant were Control Centers. These are really the heart of the grid, since their job is to balance electric power and supply in real time (i.e. microsecond-to-microsecond). You can’t be five minutes late in delivering something that’s needed right away!

I wish to point out at this point that the fact that whoever drafted this Order didn’t know the difference between a control room and a Control Center just reinforces my suspicion that the Order was put together with very little input from the electric power industry. After all, why ask them? What do they know, anyway?

So let’s assume Control Centers are in scope for the Order. There’s an important difference between Control Centers and the other two types of locations: substations and generating stations: The equipment in Control Centers is almost all the same kind of low-voltage computing equipment that you would find on any corporate desktop or data center in the country: mostly Dell or HP servers and workstations, Cisco firewalls and switches, etc. More specifically, there is no “bulk-power system equipment” in a control center, since that is invariably operated at a much higher voltage than is found in any office or data center.

So if Control Centers are covered by this order (and as I mentioned, they’re the heart of the grid. If you’re trying to disrupt the bulk power system, you would first try to attack a Control Center in some way, but lots of luck even getting in – they’re extremely well protected, both physically and electronically), then does that mean you’re going to say that generic Dell, HP and Cisco products – which are used by all industries and government agencies – are going to be covered as “bulk-power system equipment”?

To be honest, there are certainly real risks that apply to these products. Even though they’re sold by big US companies, they’re often assembled or even manufactured in China. A lot of HP workstations are manufactured in China; Dell assembles some of their servers there; and Cisco manufactures their “Fixed Switching products” and “Power Supplies” there, according to their web site.

So it might be worthwhile investigating HP, Dell and Cisco, to confirm they’re vetting their suppliers in China very carefully (I’ll guess they are, but it wouldn’t hurt to look at that). But let’s say the Secretary of Energy (who is in charge of this whole program, although my guess is there weren’t champagne corks  popping across DoE when they heard that this was getting dumped in their laps) decides that one of these vendors actually does a poor job of vetting their Chinese suppliers – so their products that are made or assembled in China should be banned.

Is Cisco going to be banned from selling any switches in the US? Will HP be banned from selling any workstations? And if not (and they certainly won’t be banned!), then how can DoE tell HP that they can sell workstations to any industry but power? Even more importantly, is it at all likely that the nefarious Chinese government would plant attacks designed just to damage the power industry – yet embed these in every server made or assembled in China, since there’s no way they could know in advance which ones were going to the power industry and which weren’t?

So outside of Control Centers, which Cyber Assets in substations and generating stations would qualify as BES Cyber Assets? Kevin could only identify a) the generic HP, Dell and Cisco systems just discussed; b) protective relays; c) backup generators; and d) programmable logic controllers (and Kevin has been in lots of substations and generating plants – as well as Control Centers – in his long power industry career, especially his almost ten-year gig as Chief CIP Auditor for SPP Regional Entity).

How many Chinese companies sell these devices to the power industry? Kevin doesn’t know of any. Then how many of these devices are often assembled or manufactured in China? Again, Kevin doesn’t know of any. Therefore, if you want to ask what computing devices are actually covered by this order (which means they qualify as BES Cyber Assets and are sometimes manufactured in China), the answer has to be: just switches, routers, firewalls, servers and workstations. None of these are actually sold by Chinese companies, but some of them are manufactured or assembled in China. But none of these are at all unique to the power industry, so they really don’t meeting the definition of “bulk-power system equipment” – which is supposedly what the order applies to. This means, that technically speaking - i.e. what a lawyer would argue in court - the EO doesn't apply to these devices.

What if we get below the device level? Is there risk there? There certainly is, because now we get into the chips (microprocessors and other kinds of chips) that form the smarts of the Cyber Assets that are found just in substations and generating plants, including relays and PLCs. There’s no question that a lot of these are made in China, so there’s certainly some risk from them – if their manufacturer were controlled by a “hostile power”.

However, asking each utility to ramp up a big program to a) determine what chips will be found in a product they’re purchasing – before they buy it!; b) determine the real provenance of those chips (not just the middle man that sold them to the device manufacturer, if you can even learn that); c) research each of the companies in the supply chain of the chip to determine what risk it posed; and d) document all of this for the Secretary of Energy so that he or she can determine the degree of risk posed to the BPS by that chip, would be nuts, period.

When it comes to chips and other small components like them, the only course for an organization smaller than say the Department of Defense is to make sure that the manufacturers whose products you’re buying take great care to vet their chip suppliers, and they don’t just buy the cheapest chips they can find, without any concern about who made them or handled them along the way.

As an illustration of this point, I want to bring up Schweitzer Engineering Labs. They are by far the largest maker of relays in the US (probably the world?); I would further add that they're almost certainly one of the few largest manufacturers of Cyber Assets that are essential to the operation of the US power grid. Dave Whitehead, their CEO, spoke on the panel on supply chain security that I moderated at NERC GridSecCon last year, and afterward put together a very good document called “How does SEL address supply chain vulnerabilities?” I included the document in this post last year.

The whole post is worth reading, but note this section:

Ensure Component Integrity

• SEL verifies the performance of ALL purchased components against our supplier product specs.
• We continuously test our products throughout the manufacturing process.
• We take additional steps to ensure the integrity of the components in our products. For example, we use x-rays, inspect packaging, and consult the manufacturer’s design drawings. (I invite anyone who wants to come to our HQ in Pullman to visit and check out our x-ray machines, manufacturing operations, and R&D facilities).  

I really think this is the only way to address the question of chip integrity: buy only from suppliers that you trust to have a good handle on the vetting and testing process. There’s simply no way that any utility could possibly make these judgments on its own. If the Federal government feels this is a huge issue, then they should stand up the mechanism to do this for the utilities (in fact, I’m sure it already exists in the Dept. of Defense).

In conclusion…

In order to address a very narrow problem – Intel-based servers and workstations, as well as networking devices, that are manufactured or assembled in China – the Executive Order requires any purchase of any type of “bulk-power system equipment” to be approved by the Secretary of Energy, even if it has no more smarts than my alarm clock. Moreover, since the devices just mentioned are sold to all industries, not just the power industry, they really can’t be banned. Plus it’s almost impossible to see how they could be used to carry out a supply chain attack on the power industry in the first place.

This would all be funny, were it not likely that most purchasing of products - with or without a microprocessor - for the Bulk Power System are likely to be significantly delayed, until the promised guidelines are released in 150 days. Let's hope that was just a high estimate, and that the guidelines will actually be clear and helpful, once released.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Are you working on your CIP-013 plan and you would like some help on it? Or would you like me to review what you’ve written so far and let you know what could be improved? Just drop me an email!