Do electric utilities sacrifice security for a few dollars in product costs?

Note from Tom: If you’re only looking for today’s pandemic post, please go to my new blog. If you’re looking for my cyber/NERC CIP posts, you’ve come to the right place.

Last Wednesday, I put up this post, my second on the recent Executive Order. Written with Kevin Perry, it made the following points (and I’ve just made a couple revisions in it to clarify ambiguities, so you might want to look at it again - even if you’ve already read it):

1. The EO is primarily aimed at deterring cyberattacks, given what the EO itself says and what literally all the news articles and comments have said. Yet it applies to all equipment for the Bulk Power System, whether or not it is controlled by a microprocessor. The vast majority of items on the list of equipment in the EO (including transformers) are devices that aren’t controlled by a microprocessor.
2. Of all the microprocessor-controlled systems, only a portion of those are “BES Cyber Assets” – i.e. they’re deployed in a role in which their failure or misoperation could have an adverse impact on the grid.
3. Among BES Cyber Assets, Kevin and I know of none that are sold by Chinese companies. The only ones that we can think of, that are sometimes manufactured or assembled in China, are off-the-shelf items that are used by just about every commercial or industrial company or government agency in the US – HP servers and workstations, Dell servers and workstations, Cisco networking and security devices, etc.
4. Yet it’s very hard to see how the Chinese could possibly launch a cyberattack against the US grid using those devices, since there’s no way they would know, while the device is being assembled in China, whether a particular server will end up in a Control Center that controls the New York City power grid, or a dry cleaners’ store in a town in the desert somewhere. And even if there were a real vector for a supply chain attack, the only effective way to prevent it would be to ban HP, Dell and Cisco from doing any manufacturing or assembling in China, for any of these products. That’s probably doable, but it will undoubtedly result in lost production and higher prices – for all companies and government agencies, not just utilities. There will be widespread unhappiness about this, especially when everybody learns that this drastic move by the government probably prevented exactly zero attacks on the power grid.

I was prompted to write this post when I saw a link in the weekly energy cybersecurity newsletter put out by Blake Sobczak of E&E News, which – as always – I turned to as soon as I saw it in my inbox this morning. The link was to an Energywire article by Christian Vasquez and David Iaconangelo that came out last Friday. While I saw the article last week, I hadn’t written about it because…well, I was busy.

The article was evidently prompted by a Moody’s report, which said the EO was “credit positive” for electric utilities because it “promotes ‘needed investments in cybersecurity preparedness’ among power providers, curbing their exposure to supply chain risks.” I wondered, how exactly will this happen? The article continues:

Moody's said that the order would create an incentive for critical equipment suppliers to invest in and develop stronger cybersecurity practices, citing the move as a positive change from the current preference for "lowest-cost bids."

Foreign adversaries may find it easier to "take advantage" of those less-expensive defenses, said Leroy Terrelonge, cyber risk analyst at Moody's.

"Security is effectively a cost center," said Moody's senior analyst Lesley Ritter. "So if you're bidding for a contract, you're going to try to put it in the lowest bid."

Utilities are particularly sensitive to costs, as they need regulatory approval to change prices for their customers, said Ritter — and leaving out some security features is an easy area to save money.

Let’s unpack what’s said in these four paragraphs:

1. Utilities almost always go for “lower-cost bids”. Because of that, they’re quite happy to have vendors leave out security features – anything to get their price lower.
2. They do this because they are constrained by the need to get rate increases when their acquisition costs exceed some presumed bare minimum.
3. The EO will do the utilities a huge favor, since it will force them to consider something besides price when they buy systems that power the grid. In turn, the PUC Commissioners will have no choice but to approve the higher-cost purchases.

I don’t know about you, but were I reading this with no knowledge of the power industry, I would conclude that people who run electric utilities, as well as the commissioners who regulate them, are a bunch of idiots: Given that it’s clearly much more costly in the long run to put in place equipment to run the grid that is inherently unsafe than it is to go for the cheapest option, they instead act against their own best interests, put blinders on, and mindlessly just focus on buying stuff at the lowest price.

Of course, this isn’t true at all, as the people from Moody’s would have found out if they had bothered to…you know…actually talk to one or two electric utility executives, or for that matter one or two state Public Utility Commissioners. However, that seems to have been too much work for them to do, so I’ll do their work for them now:

a)      I have no doubt that electric utilities go for the lowest price on true commodity items like paper clips. But when it comes to any operational equipment – that could potentially lead to some sort of outage or disturbance if it fails – the very last thing they concern themselves with is price. Their first and last concern is reliability; any equipment that might possibly lead to a loss in reliability of the grid is never even considered, even if the supplier is willing to pay them to take it off their hands. And one of the most important things they look at nowadays – and it has become much more important with the advent of NERC CIP-013 – is the cyber security capabilities of the product. If it doesn’t have the capabilities they need – or if they suspect the supplier has been cutting corners in say their software development environment, that might result in malware being planted in a product – they’ll never even consider it.

b)      Far from putting everything out to bid, most utilities stick with existing suppliers for decades, if not centuries – mainly because they trust the manufacturers not to cut corners that will result in decreased reliability (and I’m not kidding about centuries. General Electric is still a big supplier of a lot of hardware and software that runs the power grid, and I know there are at least a few utilities that have been buying from GE since close to its inception. When was that? GE was founded as the Edison Electric Light Company in 1878, by Thomas Edison). One large municipal utility told me that, when it comes to operational systems – those that directly run the grid – they do a Request for Proposal perhaps once every five years, if that. And I’ve yet to see a utility RFP that doesn’t say explicitly that cost is just one of the criteria – and often one of the smaller criteria – that they will use to choose the winning vendor.

c)       When utilities put together their rate requests – usually years in advance of needing the funds – they always include in it the cost of the products they know they will want to buy, not what might be the lowest cost available. This isn’t to say that sometimes the cost of a product won’t be higher than they anticipated, or that they will face some emergency they didn’t budget for – just that the idea that they would plan 2-3 years ahead to simply buy the lowest-cost product is ridiculous.

d)      And the Public Utility Commissioners who have to approve the rate requests aren’t some sort of clueless dolts who understand nothing but the fact that 3 is lower than 4. In fact, if the people from Moody’s want to attend one of the meetings of NARUC, the National Association of Regulatory Utility Commissioners, they’ll find that it seems every other presentation has something to do with cybersecurity. The Commissioners are keenly aware of cybersecurity threats, and they work very closely with the utilities they regulate to make sure they are addressing those threats.

I heard today that the Department of Energy is telling electric utilities that they’re not prohibited by the EO from buying any products at all at the current time – so there’s no reason for them not to carry on business as usual (or as much as usual as possible, given the Covid-19 situation). This is good news, but I’d have to say that the wording of the EO seems to say that all purchasing of equipment for the BPS needs to grind to a halt until there’s guidance on what’s “safe” to buy (and the whole idea of products or vendors being safe or unsafe is a real canard. But that’s for another post). I’m glad to hear DoE isn’t interpreting it that way.

This also confirms my suspicion that the EO will end up being something like the Huawei ban, which keeps getting postponed. The current start date is – yikes! – tomorrow, May 15. I’d say that date is no more likely to hold than the previous ones.

This also confirms my and Kevin’s suspicion that the EO wasn’t written by DoE at all, even though they're charged with administering it. DoE's job amounts to staying a few steps behind the elephant in the parade, shovel in hand.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Are you working on your CIP-013 plan and you would like some help on it? Or would you like me to review what you’ve written so far and let you know what could be improved? Just drop me an email!