The Great Texas Power Pricing Fiasco

I put up this post two days ago, which described the first of two primary causes that I see for the Texas power grid problems last week (I will address the second primary cause next week). This first cause was economic: to quote the author of that post, “Texas trusted the markets too much and went way too far in deregulation.”

The next morning, I got a long email from my friend Kevin Perry, which provided a lot of context on the markets problem. Kevin is in a position to know this, since he was for a number of years a member of the IT leadership team at Southwest Power Pool, the Regional Transmission Organization based in Little Rock which now, according to their web site, “oversees the bulk electric grid and wholesale power market in the central United States on behalf of a diverse group of utilities and transmission companies in 17 states.” Kevin was also the Chief CIP Auditor for ten years for SPP Regional Entity, the NERC Regional Entity that covered what was at the time the SPP footprint.

Of course, SPP is next door to Texas, where the grid is run by ERCOT. ERCOT just oversees the Texas market, since most of Texas is on a separate grid from the rest of the country (more about that when I discuss the second primary cause of the outage). So he knows a lot about power markets, both in ERCOT and SPP. Kevin described to me two important reasons why the power market broke down so badly in Texas last week. I’ll describe one of the problems in this post, and the other in (hopefully) my next post.

Kevin pointed out that Texas is a retail choice state, meaning consumers have a choice of who they can buy their power from. In other retail choice states (including Illinois, where I live), consumers can buy from their local utility, which also delivers the power (and charges a set fee for that, set by the regulators), or they can buy their power from one of the 220 or so retail choice providers (it's still delivered by their local utility). Those providers enter into contracts with power generators to meet their normal demand, but in cases of market turmoil those providers have to go on the spot market and pay the going rate. In some cases, they simply pass these prices on to their consumers (although some retail choice providers sell their power to consumers at fixed rates).

However, in Texas 60% of consumers didn't have a choice whether or not to buy power from their local utility; their only choice was which retail choice provider they wanted. According to this WSJ article, since 2004, customers of the retail choice providers have paid $28 billion more for power than have customers of traditional utilities. 

And there was plenty of turmoil last week. As everyone knows, power supply was severely constrained because so many natural gas plants couldn’t produce at all, and also because natural gas pipelines couldn’t transmit gas to the plants that burn it (about 40% of Texas generation) because of freeze-ups in the pipelines and at gas wellheads; both coal plants and wind/solar farms also had a lot of outages due to the cold. When you add to that the fact that most Texas homes are heated by electricity, the soaring demand combined with plummeting supply led to…high prices.

But that wasn’t the whole story. The normal settlement price on the spot market is under $50 per megawatt-hour (MWH); when Kevin emailed me yesterday morning it was $18. But for a couple of days last week, it was $9,000 per MWH. Was this price a result of pure supply and demand? Kevin had heard the settlement price was $1200 at the worst of the crisis, but then it suddenly jumped to $9,000. Why did that happen? Kevin wasn’t sure.

I found out why it happened this morning when I read this article in the Wall Street Journal. It provided a great picture of what happened, and points the finger at the Texas Public Utility Commission: 

Hours into widespread blackouts in Texas last week, the state’s power regulator took an unusual step: It stopped relying on the deregulated market to set electricity prices and did so itself.

The Texas Public Utility Commission said it raised prices to a market cap of $9,000 per megawatt hour during a six-minute emergency meeting Feb. 15, up from recent prices as low as $1,200 a megawatt hour, because the computer that was supposed to help match supply and demand on the power grid wasn’t working properly, and it needed to intervene to relieve a growing crisis.

But the higher prices didn’t result in additional power production, because many generators were dealing with frozen equipment or fuel shortages, and were unable to deliver more megawatts, no matter the price. Some electric-market participants now say the commission’s action turned an energy crisis into a financial catastrophe for many electricity buyers, who were left paying billions of dollars more for the same limited supply of electricity as before.

The role of the PUC, a three-member panel appointed by Texas Gov. Greg Abbott, in last week’s power fiasco is poised to garner more attention as state lawmakers review what went wrong. Up to now, most attention has focused on the Electric Reliability Council of Texas, or Ercot, the state’s nonprofit grid operator, but the PUC is the state’s chief electric regulator, and took key actions during the crisis.

State hearings examining the causes of the power collapse began Thursday.

PUC officials told The Wall Street Journal that, while Ercot had begun ordering blackouts as power supplies fell short last week, its computer that ran the market was apparently confused by what was happening. Ercot was trying to stabilize the grid by building up reserves of available generation. The computer was “misinterpreting those reserves as abundance and turning off the more expensive natural gas plants,” exacerbating power supply problems, said PUC spokesman Andrew Barlow.

Ercot spokeswoman Leslie Sopko disputed that the computer was turning off gas plants.

At the time, the situation left the PUC members dumbfounded. Chairman DeAnn Walker described herself during the Feb. 15 meeting as surprised by the market’s prices, which were hovering around $1,200 a megawatt hour at the time. Commissioner Arthur D’Andrea added: “We are not calculating prices correctly.”

The commission moved to set prices at the $9,000 cap, concluding that the prices at that time were “inconsistent with the fundamental design of the Ercot market. Energy prices should reflect scarcity of the supply.” That was intended to encourage power generation to come back online and allow Ercot to end the blackouts, which had plunged millions of homes into the dark in subfreezing temperatures, triggering a humanitarian crisis in the nation’s second-largest state.

But the Monday order didn’t immediately have the intended effect. At the time of the order, there was about 50,000 megawatts offline—out of 107,500 megawatts. This would remain the case through midday last Wednesday, according to a presentation by Ercot this week.

While the Ercot computer glitch may have turned off some plants, many more were shut down because of freezing conditions, fluctuations on the power grid and natural gas shortages.

Let’s be clear about this:

·        The settlement price was $1,200 per MWH at the worst of the crisis.

·        However, the PUC decided that this price didn’t reflect the real scarcity (which was possible), so they decided to assist the market a little in finding the right price. They pushed it up to the maximum allowed, or $9,000.

·        Why did they do this? Just to make sure the generators made a real killing - which is in fact what happened? No, they had a noble goal: They knew – because their Economics 101 class had taught them that – that high prices induce big supply increases.

·        Unfortunately, they never reached Economics 201, where they would have learned that supply never increases instantaneously, but takes some time to ramp up.

·        And if they’d taken Common Sense 101, they would have realized that, given that plants and pipelines were shutting down because of cold weather, only a return to warmer weather was going to allow them to increase output, no matter how high they raised the price.

·        And it seems the PUC learns slowly. After making sure that the price consumers had to pay was more than seven times what it would have been without their intervention, they kept prices high for the rest of the week, thinking…hmm, what were they thinking, if they were thinking at all?

All of this isn’t a story of a few people getting gouged. Billions of dollars were transferred from residential, commercial and industrial customers, municipal utilities and Retail Choice Providers (some of which, like NRG, sell their power at fixed prices, so they had to eat any overage) to generators and gas pipeline operators – more specifically, to the ones that were able to keep operating last week. There will be hell to pay for this, and we haven't seen the end of it.

The big problem in this case is that some people put way too much faith in their theories and way too little in common sense. This includes the PUC members and Prof. Hogan of Harvard (discussed in my last post) – who designed this wonderful system and reassured Texans last week that his handiwork had performed exactly as it was intended to perform. I’m sure that was very comforting to those Texans, especially the parents of the 11-year-old boy who froze to death in his bed.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

Cold Comfort in Texas

The situation in Texas last week was a shock to everybody, but especially those of us who are involved with the electric power industry in one way or another. The press and politicians have identified a number of reasons why things went so badly wrong; all of them are in one sense true. However, I believe there are two primary reasons why the outages happened; I’ll discuss one in this post and the second in (hopefully) my next post.

First, I’ll mention three “also ran” reasons. Each of these derives from one of the primary reasons.

1.      ERCOT failed to plan adequately. Of course, a statement like that is always 100% accurate after the fact, because it’s essentially a tautology: An outage happened. Planning happened beforehand. Therefore the planning failed.

2.      The real substance behind the failure to plan reason is that there wasn’t enough backup generation available. But in this case, even more backup wouldn’t necessarily have done any good, since so many plants were disabled anyway. Just having a few more plants that couldn’t generate power when needed obviously wouldn’t have saved the day in Texas.

3.      The primary power generation fuel in Texas, natural gas, is the least resilient in cold weather, because of the huge, exposed infrastructure required to get the gas from wellhead to power plant. So even if the gas-fired generation plants had all been working perfectly, they wouldn’t have been able to generate enough power because the fuel couldn’t get to them.

But the reason behind all three of the above reasons is that Texas trusted the markets too much and went way too far in deregulation. The theory of deregulation is that the possibility of making big gains in periods of stress will cause generators to invest what they need to be able to take advantage of the high prices that come with that stress.

Of course, competition works fine in normal circumstances, as long as you have a large enough number of independent suppliers in the market. Each supplier is incented to produce as much power as they can, since no single supplier can dictate the price they’ll receive. If they feel the current price is too low, they can stop producing altogether, or they can take a chance on pricing themselves above the market, on the hope that there will be certain market imperfections that allow them to get away with a higher price. But in the end, absent some long term imperfections, they won’t be able to sustain a higher price.

On the other hand, if enough suppliers stop producing and supply is constrained at least temporarily, the price will move back up until once again there is enough supply to meet demand.

But as we know, when the cold weather hit Texas last week, lots of suppliers stopped producing, for one of two reasons:

1.      Their own equipment had been frozen or otherwise negatively impacted; or

2.      In the case of many natural gas generators, they could have kept running if they’d had a good gas supply available. But they didn’t, because of wellhead and pipeline equipment that froze.

As a result of these outages, prices spiked to astronomical levels (as they were intended to do), but this didn’t bring forth a flood of new supply – it simply wasn’t there. So while some people suffered terribly because of lack of power, many of the “lucky” people whose power had remained on didn’t feel so lucky when they opened their power bills to find costs in the thousands of dollars.

You might think this situation would cause the people who designed the deregulated system to admit there was a problem. However, the main architect of this system, Dr. William Hogan of Harvard, said a few days ago “As you get closer and closer to the bare minimum (of production, meaning more and more generators are failing), these prices get higher and higher, which is what you want.”

I have to admit, I’m not sure I could find a better example of an academic so wrapped up in his ideas that he has lost all connection with reality. It’s crystal clear that the higher prices wouldn’t solve the immediate problem at all – the needed generation just wasn’t there, period. What he presumably had in mind (if anything) was that the high prices will make a lot of generators and would-be generators sit up and take notice, so that the next day they’ll visit their bankers and ask for money to expand their capacity. But this ignores a few bracing realities:

1.      These high prices, unlike the not-quite-as-high prices in previous cold weather outages like 2011, are unlikely to stick. There is so much outcry that they’ll be at least partially rolled back. The generators will be forced to swallow this, even though there’s probably no way that can be done legally. This will have the opposite effect on any ideas they might have of expanding.

2.      These cold weather events are very infrequent. Who knows? What Texas just went through might be the worst such event in the next 50 years. How can you plan on such a rare occurrence?

3.      And even if Texas gets more events like this one, what’s to say the same thing won’t happen – that lots of plants will be disabled, so they can’t take advantage of the high prices anyway? In fact, that’s close to certain, unless the new plants are winterized much more than the current ones were. That might get them through the spike, but then you face Problem 1: Who says the high prices will be allowed to stand?

Almost all states have deregulated generation to some degree, but Texas stands out in the degree to which it pushed the idea to its logical extreme. And it seems they have a Harvard professor to blame for that.

But there is an even more fundamental reason for the power grid failures. That story is coming soon to a blog near you. 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

Reminder: SBoM webinar on Thursday

Just a reminder that the second in a series of 3 or 4 webinars on software bills of materials for the electric power industry, sponsored by the National Technology and Information Administration of the US Department of Commerce, will be held on Thursday. If you’d like to view the first webinar, which was very well attended and received, go here.

The primary purpose of these webinars is to help members of the power industry (and any other interested parties, of course) understand how the distribution and use of SBoMs can improve the level of software security for the industry; and in case you haven’t noticed, the need for better software security has become painfully evident in recent months!

The webinars are leading up to a proof of concept, expected to start in the next 2-3 months. In the PoC, industry software suppliers and users will work together to develop and test formats and procedures for production and use of SBoMs in the industry. The group certainly won’t start from scratch. They will build on the great work of the healthcare community, which started their first PoC in 2018. They’re still doing PoCs, but each one is more comprehensive. The Energy PoC will be able to take advantage of their work. In fact, participants in the healthcare PoC will discuss their experiences in the third webinar, coming within a few weeks.

Attendance information is below. As with all of these introductory webinars, no pre-registration is required.

 

SBOM Technical Info Session for the Energy Community

Thursday, February 18

12:00pm – 1:00pm ET

Teams link: https://teams.microsoft.com/l/meetup-join/19%3ameeting_YmMzZmRhYjQtMTU4YS00MDYzLTk3ZmYtODcyMzg1OGExOWMw%40thread.v2/0?context=%7b%22Tid%22%3a%22d6cff1bd-67dd-4ce8-945d-d07dc775672f%22%2c%22Oid%22%3a%22a62b8f72-7ed2-4d55-9358-cfe7b3e4f3ed%22%7d

Dial-in: +1-202-886-0111,,315018361#

Other Numbers: https://dialin.teams.microsoft.com/2e8e819f-8605-44d3-a7b9-d176414fe81a?id=315018361

 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

In spite of itself, the new FERC NOPR will bring some improvement

This is my third (and last for the moment) post on FERC’s NOPR on cybersecurity incentives – the first was this one. The first two posts dealt with the financial side of the NOPR, pointing out that the financial “stimulus” that FERC is proposing is likely to end up being fairly modest. This post deals with the substance of what FERC is proposing – i.e. what those utilities lucky enough to be able to take advantage of FERC’s proposal will have to do in order to have their investments approved for reimbursement through transmission rate relief.

I must admit that my first emotion on reading FERC’s description of the substance of their proposal was embarrassment. Doesn’t anybody at FERC know anything about the NERC CIP standards? After all, they did approve them, as I recall. I also have to ask how much they know about cybersecurity in general, since a lot of what they say in the NOPR seems to reflect the view that cybersecurity controls are a kind of magic pixie dust. Since I know FERC has some very good cybersecurity people, I have to assume the cybersecurity pronouncements in the NOPR were the product of too much holiday partying.

The NOPR describes two different approaches to cybersecurity investments, that (some) electric utilities can make in order to receive rate-based financial incentives under FERC’s proposal.

I. NERC CIP Incentives Approach

The first approach is described starting on page 24 of the NOPR. FERC describes this approach as “voluntarily applying identified CIP Reliability Standards to facilities that are not currently subject to those requirements”. Essentially, FERC is asking utilities to “voluntarily” (as if anything the government asks you to do is done voluntarily. I remember Milton Friedman describing, during one of the economics classes I took with him a long time ago, President Ford’s recently-announced “voluntary” wage and price controls thusly: “This is an Orwellian use of the term ‘voluntary’: You do this voluntarily or your throat gets cut.”) extend the CIP standards beyond what they apply to now.

The idea behind that is FERC doesn’t want to pay utilities to do what they’re required to do anyway by the CIP standards – but they’re fine with having them go beyond what’s required and paying them for that (of course, I’m simplifying things a lot by speaking of FERC “paying” anybody). There are two ways in which a utility can follow this approach:

First, the utility can “voluntarily” (there’s that word again!) apply “the requirements for medium or high impact systems to low impact systems, and/or the requirements for high impact systems to medium impact systems”.

FERC offers two methods (or “incentives”) for accomplishing this goal. The first – the “Med/High incentive” - is applying the requirements for high impact facilities to medium impact ones (although strictly speaking the high and medium designations apply to systems, not facilities. But we’ll skip the religious arguments for the moment). This is a fairly straightforward task: where a requirement only applies to systems at a high impact facility like a Control Center, the utility applies it to systems at a medium impact facility like a transmission substation.

On the other hand, I find it hard to believe that any utility will seriously want to use this method, unless the description of it is completely revised. By suggesting utilities can apply medium/high requirements to low systems, or high requirements to mediums, FERC seems to be saying (perhaps inadvertently) that the utilities will “voluntarily” (again!) declare their lower-impact facilities to have a higher impact than they’re otherwise required to have, by CIP-002-5.1a R1 Attachment 1. Without going into detail on why this is a crazy idea, I’ll just say that anybody at an electric utility who suggested uprating the impact ratings on facilities subject to NERC CIP probably wouldn’t live to see tomorrow’s dawn.

If FERC wants this method to have any chance of finding some takers, they will have to make it clear that the utility will “voluntarily” apply particular controls that are only required of higher-impact BCS to lower-impact ones – and that they can apply those controls in any way they want, rather than strictly according to what CIP requires. If they did that, I think some utilities might be interested in this.

However, FERC really goes to pieces when they try to describe the second method, the “Hub-Spoke incentive. They say “Under the Hub-Spoke Incentive, a public utility is eligible for incentives if its investment applies CIP Reliability Standard security controls inherited from a high or medium impact BES Cyber System at locations containing low impact BES Cyber Systems by ensuring all external routable connectivity to and from the low impact system connect (sic) to a high or medium impact BES Cyber System.”

Let’s try to pull this apart. First, FERC is trying to make the case that it would be a good idea to connect low impact (and therefore lower security) systems to high or medium impact (and therefore higher security) ones – that somehow the higher security will magically flow down the wire connecting the two systems (which would be in two different facilities).

Let’s think about this: Staff members who have access to BCS at medium and high impact facilities have to have background checks, whereas they don’t at low impact facilities. This presumably makes the systems at the former more secure than the latter. Does connecting them with a wire give the people who work in the low impact facility the equivalent of background checks (they probably have them anyways, but that’s not the point here)?

Here’s another example: To comply with CIP, utilities have to spend a lot of effort on configuration and patch management at medium and high impact facilities, but not at lows. If they run a wire between the two, will the systems at the low facilities automatically “inherit” (FERC’s word, not mine!) the benefits of the configuration and patch management that’s done on the high or medium impact systems?

Of course not. The only cybersecurity benefit that could be conferred by connecting the two levels of facilities has to do with external routable connectivity, which FERC mentions. But BCS at medium and high impact facilities aren’t connected directly to ERC; instead, there needs to be some device like a firewall that makes the connection to the outside world and extends that connection to the systems within the facility.

But a firewall isn’t a BCS; it’s an EACMS (electronic access control and monitoring system). And the low BCS wouldn’t connect directly to the medium or high EACMS, but to an EACMS at the low facility, either a router or a firewall. That would in turn connect to the high or medium EACMS.

Moreover, it wouldn’t be good security practice for a low impact BCS to have any true external routable connectivity (i.e. a connection to the internet or to networks outside of the utility’s) in the first place, no matter what system it came through. Low impact systems, if they’re connected to anything at all, are connected to the SCADA or EMS system in a (usually) medium or high impact Control Center. Only the latter system would connect to any systems outside the utility, including the internet. I might be speaking two categorically here, but I can’t think of any reason why it would be a good idea to give ERC to individual BES Cyber Systems at low impact assets.

In short, the “Hub-Spoke Incentive” is described very poorly by FERC, and even describing it correctly would probably lead to a less secure arrangement than what is in place now at most electric utilities. But other than these quibbles, I think it’s a wonderful idea.

II. NIST Framework Approach

FERC’s second approach (found beginning on page 28 of the NOPR) is described by FERC this way: “public utility may receive incentive rate treatment for implementing certain security controls included in the NIST Framework”. Specifically, FERC suggests that five types of controls found in the NIST Framework would be eligible for incentives: “(1) automated and continuous monitoring; (2) access control; (3) data protection; (4) incident response; and (5) physical security of cyber systems”. However, the NOPR then says that initially, FERC will only consider the first type of controls for incentives: automated and continuous monitoring.

I don’t have any problem with FERC limiting the types of controls to five, and perhaps even limiting them to one (which makes one wonder why they listed the other four in the first place). There’s no doubt that one of the big deficiencies in the CIP standards is they don’t require utilities to implement some sort of monitoring, even though many are already doing it on their own (and that really is “voluntary”!). If more utilities start doing this because of FERC’s proposed incentives – or more likely expand on the monitoring they’re already doing – that in itself would lead to a higher level of grid security.

My main question about this approach is: What does this have to do with the NIST Framework? There are all sorts of cybersecurity frameworks and standards that mandate some or all of the five types of controls FERC suggests, and most of them predate the Framework. In other words, I think FERC wasted a lot of their time writing many pages about the Framework, and I think readers of the NOPR will waste a lot of their time reading about it, when this Approach could have been much better categorized as “It’s important to monitor your network, your external connections, and the systems on your network. Do it and you’ll get rate relief.”

But there is one thing that FERC might have added to these two sentences: “Utilities may want to look for a different vendor for their monitoring software than the (until recently, anyway) leading vendor: SolarWinds”. However, that’s a different issue.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

Just who benefits from FERC’s NOPR? Anyone?

In a recent post, I discussed FERC’s new Notice of Proposed Rulemaking (NOPR) on “cybersecurity incentives” for electric utilities. I pointed out that, while the benefits to utilities will be substantial, they will – as far as I could see at the time – go entirely to about 35 organizations of investor-owned utilities (IOUs). Although, to be fair, those 35 organizations account for at least 75% of total load served in the US. The thousands of cooperative and municipal utilities would be left out altogether.

However, I realized the next day, in the course of emailing with Kevin Perry, that I was wrong about both the number of organizations that will benefit (I was too low on that), and the total amount of benefits (which is likely to be much lower than I was thinking, although I didn’t try to estimate the amount when I wrote the first post, and I certainly won’t try to estimate it now).

Essentially, the NOPR states that FERC will provide utilities (BTW, IPPs are completely out of the picture on this, since except in very few cases their rates aren’t regulated by anything but the market) with rate relief – meaning they can charge their customers more - to allow them to increase the amount they spend on BES cybersecurity, as long as they spend it in particular ways.  I assumed FERC was talking about the rates the utilities charge electric power users – i.e. everyday schmoes like you and me (well me, anyway. I know you’re perfect).

I was quite aware that FERC has no direct jurisdiction over those rates. Those rates are set by the state Public Utility Commissions (PUCs, as if I weren’t already over my quota for acronyms in this post. I might get fined), but only for IOUs, which is why the coops and munis were left out in the cold. But I assumed that FERC’s guidance would be followed by the PUCs, so they would automatically let the utilities include in their rate base whatever FERC said they could include.

However, when Kevin and I had our email conversation, we realized that what FERC must be talking about is the rates they do have direct authority over: rates for power transmission, meaning the rates that Transmission Owners (the organizations that own transmission lines, substations and control centers) charge their customers; they are called TO’s in the industry.

FERC has jurisdiction over interstate transmission of power, but Kevin pointed out to me that there are lots of transmission lines that are entirely within one state. After all, there are few long lines that cross multiple state boundaries, so power that’s flowing from say Louisiana to Indiana will pass over a number of different TO’s lines. FERC is proposing to let TOs charge their customers a little more, to fund additional cybersecurity expenditures (by the TOs themselves, of course). 

So who are these TOs that are going to reap the benefits of FERC’s proposal? I haven’t studied any of this, but from what I know, they fall roughly into three categories:

1.      Transmission-only companies, of which there are…get ready…a total of two: ATC and ITC (I want to thank Lew Folkerth for reminding me of these. Lew, by the way, is Kevin’s evil twin. Or Kevin’s Lew’s evil twin, I’m never sure. It depends on who’s speaking). They’re both privately owned by utility groups.

2.      Transmission arms of large utilities, primarily IOUs, but including at least a few large coops and munis.

3.      Transmission arms of smaller utilities, consisting almost entirely of coops and munis.

Since this post is all about money (that makes two in a row), the two questions now are

1.      Is the total amount of monetary benefit from FERC’s proposal likely to be larger or smaller than what it would be if I’d been correct when I said in my last post that the benefit was based on sales of power to end users?

2.      How will it be distributed among the three groups above?

To answer 1, I’d say the aggregate monetary benefit must be much less than I originally thought, since undoubtedly the total amount spent on power transmission in the US must be much smaller than what’s spent by end users (households, commercial and industrial) on power. Of course, if someone knows better and wants to enlighten me on this, I’d love to hear from you.

Regarding question 2, it seems almost axiomatic that most of the benefits of FERC’s proposal will flow to the first two groups. But keep in mind that, when it comes to the second and third groups (i.e. utilities), their total transmission revenues must be a small fraction of their distribution revenues – since transmission is a sideline but distribution is their raison d’etre of almost any electric utility. Except for ATC and ITC, I don’t think there will be any kind of huge increase in cybersecurity expenditures by TO’s this year, or at least much more than they were already planning (which was undoubtedly a lot).

So what’s my verdict on FERC’s proposal, from the fiscal side (I still haven’t discussed it from the cybersecurity side. I’ll get to that soon)? I certainly think it’s better than nothing. But it’s not the answer to the long-term problem: The need for cybersecurity spending for the grid will continue to grow much faster than utility revenues. FERC is doing what it can, but ultimately the solution has to lie in Congress and the White House.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

FERC’s NOPR on Cybersecurity Incentives – part I

On Friday morning, just about the first email I received for the day was from a good friend who’s getting ready to take a cybersecurity job in the new administration and seems to read the Federal Register with his morning coffee (me, I read the Times and WaPo online, along with the WSJ - for balance, dontcha know - in hard copy).

His email included the FR appearance (starting on page 8309) of FERC’s new NOPR on Cybersecurity Incentives, which I hadn’t seen. He wanted to know what I thought of it, and I promised to write a post about it on Sunday. Here’s that post, but I must admit there’s a lot more meat in it than I thought. This post discusses one important aspect of the NOPR – i.e. the money side – but I also want to weigh in on the cybersecurity side as well; I’ll do that very soon.

The post builds on the white paper on Cybersecurity Incentives Policy, put out by FERC staff last June. To sum it up very briefly, the NOPR proposes that FERC will provide “transmission incentives” to electric utilities that go beyond the NERC CIP standards to implement certain designated cybersecurity measures that FERC deems valuable to the interstate transmission system (which is really what FERC is concerned with, as opposed to the BES that NERC concerns itself with. I can guess at the difference between the two, but if someone knowledgeable wants to explain it to me, I’d appreciate that).

Of course, when FERC talks about incentives, they’re not talking about money they can hand out – FERC doesn’t have a pot of gold sitting in its headquarters, ready to fork over to deserving utilities. They’re talking about rate relief, meaning the money will come from…guess who?...the ratepayers, of course.

But I was surprised that neither the NOPR nor the white paper mentioned one fact: there are only about 35 utility organizations in the US who will benefit from these incentives. As you probably know, there are three main types of utilities in the US: investor-owned (IOUs), municipals and cooperatives. There might be about 200 IOUs, but since the majority are included in multi-utility holding companies, there are only about 35 separate investor-owned utility organizations, vs. literally thousands of coops and municipals.

The IOUs are all regulated by the state public utility commissions, who set their rates. FERC’s new regulations – if they are implemented – will essentially tell the PUCs that they should allow additional rate relief to the IOUs for the cybersecurity improvements described in the NOPR. So the ratepayers that are served by IOUs will have to cough up a little more on their monthly bills, to fund these improvements.

But what about the coops and munis? Don’t they have ratepayers? No, they don’t. Coops have members, who essentially share the cost of running the utility – there are no profits strictly speaking. If the staff of the coop decides they need to spend more on cybersecurity, they’ll need to get permission to spend that from the board, which is elected by the members. FERC can say whatever they want, but a coop isn’t going to spend a dime extra on cybersecurity unless their members agree to it.

And who are the “ratepayers” of a municipal utility? They’re the citizens who live in the service area (which can be a city, town, state, or region of a state – or, in the case of TVA, multiple states). And who decides the rates? The utility is usually a department of a local government, so they’re subject to the same accountability that any other branch of that government is. If you don’t like the way the Roads Department is patching potholes, you might vote against the current mayor in the next election. And if you don’t like the electric rates charged by the Power Department (although it doesn’t usually have that exact name), you also vote against the mayor (of course, there are certainly other ways to pressure a municipal utility over rates, such as petitions, rallies, etc).

Here’s the bottom line: There are 35 utility organizations in the US that will be able to take advantage of FERC’s “generosity” (with their ratepayers’ money). But there are thousands of utilities that won’t make a dime off of this proposal. However, it does need to be said that those 35 utility organizations account for around ¾ (or more) of total electric load served in the US – so the proposal will certainly lead to an increase in overall power cybersecurity (and hopefully reliability) for the great majority of American electricity users.

At the same time, it would be nice if the coops and munis, along with their members and voters, also got a little piece of the action. However, as I said, FERC has no pot of gold to divvy up with them.

But maybe they should. For a while, I’ve believed that the cybersecurity of the power grid is really a national concern and should be addressed on a national basis. Rather than putting the squeeze on IOU ratepayers to pay for the greater part of grid security (which they undoubtedly are at the moment. Munis and coops are chronically underfunded for cybersecurity expenditures, although I’m very impressed by how well they spend what they have available), it would be much better if Congress would ask a few questions:

1.      What will it cost to secure the US bulk electric system to the degree necessary for all of us to feel that we’re safe from power outages due to cyberattacks? I feel pretty safe right now, so I don’t think we need to spend a lot more than we are now, although I do think that what we do spend on cybersecurity could go a lot further if the NERC CIP standards were rewritten to be completely risk based (plus a few other important changes). On the other hand, there are certainly a lot of people who believe that the country needs to be spending a lot more on grid cybersecurity than even FERC’s proposing.

2.      What would be the right distribution of that spending among the US population? The answer certainly wouldn’t be the simplest: divide the amount from item 1 by the number of households in the US. There are many considerations that would go into determining this optimal distribution of costs, and it’s certain that lots of people will be unhappy with any solution proposed. But Congress is there in order to allow the different voices to be heard, and reach…not the perfect solution, but a solution that’s one of the least hated.

3.      Once Congress knows what item 2 is, they then need to look at what customers/members/voters are paying to each utility for cybersecurity expenditures and determine who’s overpaying (if anybody is currently overpaying) and who’s underpaying, relative to what their share should be.

4.      But instead of raising everybody’s rates until we’ve reached the total in item 1, Congress now needs to consider: Since the security of the power grid benefits everybody, why should it be financed any differently than say the military? We don’t all pay a monthly bill based on how much we use military services, since most of the benefits from the military come about when they’re needed most – during war. When we’re not at war, the cost of having the military doesn’t fall to zero, even though its benefits aren’t anywhere near as visible. We need to fund the military continually, so we’re ready for the next war (although obviously the military performs lots of important functions between wars, one of the most important nowadays being cybersecurity).

5.      I’m not saying we should have the government fund 100% of cybersecurity expenditures by electric utilities, any more than the government should fund 100% of expenditures on say transmission line construction. But, given that the expectations for cybersecurity expenditures by electric utilities are growing all the time, I am saying that expecting their ratepayers/members/voters to keep paying increasing amounts ad infinitum isn’t realistic at all. A certain portion will need to be funded out of general revenues on the national level.

I also have something to say on the cybersecurity aspects of the NOPR. More on that soon.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

Don’t worry, Vlad. Your record’s safe.

There have been a number of news stories about the fact that the Chinese were able to exploit a vulnerability in SolarWinds Orion software to attack the US Dept. of Agriculture’s National Finance Center. Most of the articles have made it sound like the attack was déjà vu all over again: another supply chain attack on SolarWinds!

However, only a few articles (like this one) have gotten it right: This wasn’t a supply chain attack. This was just your garden-variety attack on software, in which some nefarious party exploits a vulnerability in software (and there are lots of vulnerabilities out there!) to penetrate an organization. The attack is on software that is already installed.

A supply chain attack on software usually starts long before the software is installed. In fact, in the case of the Russian attack on SolarWinds, it started about 15 months before the attack was discovered, while the software was being developed. The Russians planted the SUNBURST malware in updates to Orion, using the amazing SUNSPOT malware, which I described recently. This was the first stage of the attack.

The SUSNBURST malware then opened up a backdoor when the tainted update was installed on a customer’s network. It beaconed to the Russians that it was active, at which point they were able to exploit the backdoor to perform their dirty work. This was the second stage of the attack.

The big difference between the Russian and the Chinese attacks was that the latter was essentially equivalent to just the second stage. Both attacks exploited a vulnerability, but the vulnerability that the Chinese exploited existed in Orion before they attacked it. The vulnerability the Russians exploited was one they had placed there themselves. That’s why the vulnerability was called a backdoor.

If you read my post on SUNSPOT, you know that it was an exquisitely-designed piece of malware that rivaled Stuxnet. The Russians conducted a careful campaign that started with a proof of concept that placed a benign piece of code in a few Orion updates, just to make sure that could be done. Then they developed SUNSPOT and deployed it a few months later. After that, SUNSPOT had to run on its own for months inside the SolarWinds development environment, without any direct Russian intervention. Yet it was completely successful in placing SUNBURST in about seven or eight Orion updates.

Compared to the Russian campaign, the Chinese attack was a skirmish. The Russians penetrated maybe a few hundred targets (some very high value). They could presumably have penetrated another 17,750, since about 18,000 customers downloaded the tainted Orion updates, but they just didn’t have the time. Meanwhile, the Chinese penetrated one Orion customer: an agency inside USDA. Do you see the immense power of a supply chain attack, vs. a garden-variety software attack?

I’ve said at least twice (here and here) that Uncle Vlad is the king of supply chain attacks. The Chinese will never displace him!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

The recording of last week’s SBoM webinar is up!

Dr. Allan Friedman, the leader of the NTIA Software Transparency Initiative, sent around the email below today. It provides a link to the recording of last week’s webinar. It also provides the connection information for the next webinar in this series (there will be a third webinar around the end of February, date and time TBD).

I hope you’ll listen to the recording (if you weren’t there last week, or if we spoke too fast and you want to listen again) and also attend the next info session (which will only be given at one time, not twice as last week’s webinar was). If you’d like to get on Allan’s email list, please email him at afriedman@ntia.gov.

The next info session on SBOM for the Energy and Bulk Power community will be on February 18, from 12-1pm ET. This session will be a deeper technical dive into the specifications of the basic, "minimum viable" SBOM, and highlight the existing data formats that can convey this data and are being implemented today.  A third info session with more detailed lessons from other proof-of-concept exercises will be announced for later in the month.

·         For those interested, the slides and a video recording of last week's introduction info session are now available at https://www.ntia.gov/softwaretransparency and are linked to below. 

·         Video: SBOM Information Session for the Energy Community (1 hour)

·         Slides: An overview of SBOM

·         Slides: SBOM use cases for the energy sector

·         Slides: Experimenting with SBOM – lessons from the healthcare sector

·         Slides: Experimenting with SBOM – early steps in the Automotive sector

 

SBOM Technical Info Session for the Energy Community

Thursday, February 18

12:00pm – 1:00pm ET

Teams link: https://teams.microsoft.com/l/meetup-join/19%3ameeting_YmMzZmRhYjQtMTU4YS00MDYzLTk3ZmYtODcyMzg1OGExOWMw%40thread.v2/0?context=%7b%22Tid%22%3a%22d6cff1bd-67dd-4ce8-945d-d07dc775672f%22%2c%22Oid%22%3a%22a62b8f72-7ed2-4d55-9358-cfe7b3e4f3ed%22%7d

Dial-in: +1-202-886-0111,,315018361#

Other Numbers: https://dialin.teams.microsoft.com/2e8e819f-8605-44d3-a7b9-d176414fe81a?id=315018361

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

The lie that won’t die

In June, SANS put out a Defense Use Case that took issue with an assertion that the consultant Joe Weiss made in a May blog post. He asserted that a large transformer intended for use by the Western Area Power Administration (WAPA) had been discovered to contain a “hardware backdoor”, that could have allowed attackers to penetrate the transformer in some way, and thereby damage the US power grid in some way.

The SANS authors (Robert M. Lee, Tim Conway and Jeff Shearer) assigned the following scores to Joe’s assertion:

·        Credibility: 0

·        Amount of technical information available: 0

So other than the fact that the post can’t be believed and Joe provided no technical information to support his assertion, they liked the post…😊

While SANS didn’t go this far, I have a word I use for assertions that aren’t credible and aren’t supported by any technical information: a lie. I also wrote about Joe’s post, and I went a little further than SANS did (I wrote about five or six more posts on the WAPA transformer after that, mostly on the question of whether there’s any way a transformer even could be the subject of a cyberattack. The answer to that question, IMO, is that a transformer considered by itself can never be subject to a cyberattack, since it isn’t controlled by a microprocessor. There are a couple common add-on devices like load tap changers or dissolved gas analyzers that do have microprocessors, but it’s not at all clear how they could be attacked, or what effect that would have – if any – on the operation of the transformer itself. These are often sold and installed by a separate company, not the transformer manufacturer).

When the SANS document came out, there was a lot of discussion on LinkedIn. In the discussion, Joe said he would provide backup documentation for his claim, but I never saw any. However, I considered Joe’s assertion to be dead and buried.

I was quite surprised, then, to read in an online article by Joe a week or two ago that he was again claiming the Chinese had planted a backdoor in the WAPA transformer (he was also criticizing DoE for somehow covering up that “fact”). I posted a comment to the article in which I linked the DUC; in a day or two the article had been amended to remove the assertion.

So I was quite surprised yesterday to see Joe post on LinkedIn a link to a Forbes article that repeated the story! I commented on Joe’s post that I hadn’t seen any documentation from him. Somebody deleted that comment, so I just put it up again. I’ve also sent an email to the editor of Forbes about this.

This might all fall in the Low Comedy department, were it not for one thing: In stating that the Chinese launched a supply chain attack on the US power grid (and implying that the WAPA transformer might have been one of many that were attacked), Joe was implicitly stating that the Chinese had attacked US national security. If that’s the case, why didn’t we take some stern measures against the Chinese? Perhaps fly a B52 over Taiwan as a demonstration that we can retaliate massively (as we just did with Iran)?

Fortunately, I don’t think anyone in the national security establishment believes this ridiculous story either. I sure hope we don’t see any more of it.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.