Tim Roxey on the “end of days”

Note from Tom: I’ve moved my email feed from FeedBurner (who’s getting out of this business in July) to Follow.It. If you aren’t getting my posts by email anymore, just hit the Subscribe button in the top right. And if you’d like to start receiving these posts in your email inbox, also hit the Subscribe button.

The day after I put up my post describing why it’s literally impossible for a single cyberattack (or a single set of coordinated cyberattacks) to shut down the entire US grid, I was pleased to receive an email from Tim Roxey, former NERC VP and CSO, on the subject. As usual, he brought a really interesting perspective to the topic.

To briefly summarize my post, I said that

a)      You can divide the assets in the Bulk Electric System into three types: generation, distribution substations and control centers, and transmission substations and control centers.

b)     Generation and distribution are fairly easily dismissed as attack vectors, leaving transmission substations and control centers as the likeliest vectors.

c)      However, I showed that penetrating the control systems in transmission substations and control centers would be extremely hard (and has never been accomplished in North America), even in the case of a single asset, due in part to the rigorous controls required by the NERC CIP standards.

d)     But attacking a single asset won’t get you very far if your goal is to bring down the entire grid. I estimated that you’d have to carry out a very well-coordinated attack on at least 40 transmission assets (10 in ERCOT, and 15 in both the Eastern and Western Interconnects. And if you want to include Quebec in your continent-wide blackout, then you have to add at least 10 assets there, since Quebec has its own grid – plus I know it’s stretching the truth a lot to say that Alberta is connected to the Western Interconnect. I believe it’s just in recent years that there’s been any connection at all, and even now I think it’s just one line. You’d probably have to attack at least 10 assets in Alberta as well, for a total of at least 60 all told), and even that is probably a woeful underestimate.

e)     I said this would be simply impossible. My reasoning – which I should have stated – was that OT networks are incredibly diverse in the power industry. The devices on the networks are quite variable, as are the configuration and technologies behind the networks.

Of course, other industries might consider it very inefficient to have so much diversity, since it means that suppliers can’t realize the huge economies of scale that for example Dell, HP and Cisco have realized on the IT side. There’s no doubt this is true, but at the same time it makes it literally impossible for the grid to be the subject of a massive, coordinated attack.

This diversity wasn’t planned, of course. It just happened because decision-making is so decentralized in the power industry. I’ve always said that planning is great, but in the end there’s no substitute for dumb luck! The industry - and North American power users - have benefited greatly from that dumb luck.

Tim wrote in to say he agreed with my general argument, but (and here I’m paraphrasing him) I’d overlooked another type of assets: IT assets. In fact, the only thing that generation, distribution and transmission operations have in common is that they all rely on IT assets, not just OT ones. A coordinated attack on IT assets throughout the industry could conceivably be the vector for a takedown of the entire North American power grid.

Tim’s point was that, since it would be normal to expect IT networks to be fairly homogeneous, that means those networks – and the devices attached to them - might well be the vector that would enable an attack to occur. However, once again the power industry has saved itself because of diversity. This time, it’s not diversity in the technologies involved in the IT networks – they literally all run IP, I’m sure, on Intel-standard devices. There’s no DECNet or Novell IPX anymore, although I can remember when these were present in lots of IT networks. And the machines on the networks almost all run Windows, with some Linux. No MS-DOS, MacOS, VMS, etc.

So where does the diversity come from? It’s in the network architecture. Electric utilities have realized the benefits of network segmentation, firewalling off different areas of the network, different WAN technologies, etc. None of this is great for pure efficiency, but it’s great for preventing a small number of hackers from carrying out a massive, simultaneous attack on lots of different grid assets in every Interconnect. And it would take such an attack to bring down the US (or North American) grid in its entirety.

Here is what Tim wrote. Lots of wisdom in here!

Tom Yes – Scalability is directly related to variability in the environment. Very little variation – broader span. Larger variability, then more effort for each unique piece of variability. 

 

If an environment is very homogeneous, then a successful exploit at one interface is likely useful at a second or third interface.

 

1.      Homogenous is bad.

a)      Network architecture using the same make and model for all switches, hubs, routers, servers, etc.

b)     Desktop environment consistent across the enterprise.

c)      Lack of principle of least privilege. 

d)     Lack of application White Listing.

 

If an environment is Heterogeneous, then a successful exploit at one interface does not necessarily mean it will work at a second or third interface.  

 

2.      Heterogeneity is good. 

a)      Network architecture mixed with different vendors supplying parts of the environment. 

b)     A desktop environment consisting of different Operating Systems. 

c)      Full implementation of the principle of least privilege 

d)     Full implementation of application whitelisting

 

In number 1, the Adversary only needs to understand one (or a few) different types of network technology. Perhaps the same firewalls are used everywhere for segmentation. In this case, the same exploit used for layer 1 is useful for layers two and layer 3.

 

If the victim changes firewalls at every boundary level, then the Adversary must deal with a different set of exploits for each of the different levels. 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the National Technology and Information Administration’s Software Component Transparency Initiative, for which I volunteer. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.