Meanwhile, back at the (Texas) ranch…

I haven’t written about the Texas power crisis lately, but I’ve been following the story in the press – to the extent possible, since this isn’t exactly my day job. I’ve learned some interesting things lately, and I’d like to share them with you. I’ll do that in this post and a few subsequent ones (although not immediately following this one).

To summarize where I think things stand, I’ll say that it’s becoming clear what a huge event this was: from the power engineering standpoint, the human impact standpoint, and the financial standpoint. Are any of these three concerns closer to being at least comprehended – let alone resolved - than they were say a month ago? In the case of just the power engineering standpoint, I’d say it’s at least understood now, even if what’s not understandable is how the blackout was allowed to happen in the first place. And as far as resolution goes, none of the three is anywhere close to being resolved.

Let’s start with the power engineering standpoint. I’m very far from being a power engineer, but I was quite impressed with a few of the things I learned from reading this recent article in T&D World:

·        The “theme” of the article is stated early: “In the aftermath of the events taking place from Feb. 14 through Feb. 17, the ERCOT leadership made a series of statements. One quote caught some attention: The Texas power grid had come within four minutes and 37 seconds of total collapse. They went on to say that had the system collapsed, it would have required what is called a “black start” of the entire ERCOT system. Furthermore, the resulting blackout could have lasted weeks or even months: Rebuilding a grid from scratch takes time.”

·        The article goes on to ask and answer three questions: a) “What exactly does ERCOT’s statement mean?”; b) “Is it probably an exaggeration, or did the grid really come close to total collapse?”; and finally c) “Had ‘total collapse’ actually occurred, what would have been the consequences for Texas?”

·        The article answers a) and b) together. It describes (and graphs, using synchrophasor data) the minute-by-minute sequence of events that occurred on the ERCOT grid (monitored from the control center in Round Rock, Texas) between around 1:43 and 2:00 AM on February 15. At around 1:50 AM, the ERCOT system frequency fell below 59.4 Hz. This triggered a nine-minute window. Had the operators not been able to bring frequency above 59.4 Hz by nine minutes later, a protection scheme called “generator under frequency ride-through” (I hadn’t heard of that, either) would have been automatically activated.

·        This scheme is designed to keep generators from suffering permanent damage if the frequency stays too low for too long. It does this by shutting all generators down. As the article states, “The entire Texas power grid controlled by ERCOT would have collapsed and approximately 26 million customers would have been without power.”

·        Why is this scheme necessary? If this protection scheme weren’t in place and frequency didn’t rise sufficiently, the grid would have collapsed anyway, probably because the protection relays on most of the generators would have tripped them. But if those protection relays hadn’t worked, the ERCOT grid wouldn’t just have been shut down; it might have taken literally months (and huge expenditures) to restore it (the article doesn’t go as far as these last two sentences do, but it seems to me – Tom Alrich, Boy Engineer - that this in fact is what would happen in the worst case scenario).

·        Spoiler alert: The Texas grid didn’t collapse. The article describes how, after the frequency dropped below the critical level, the ERCOT operators (or really the under-frequency load shedding system, or UFLS) dropped 6,500 MW of load (a huge amount, needless to say – i.e. 6.5 gigawatts. 11 gigawatt-hours is roughly enough to power New York City for one day).

·        Unlike two earlier 1 GW load sheds, this one (it was really two, spaced a few minutes apart) did the trick and reversed the trend in frequency. Frequency passed out of the critical range with four minutes and 37 seconds remaining in the nine-minute window, which prevented the generator under frequency ride-through protection scheme from kicking in.

·        Had they not succeeded and the nine-minute clock had run out, “The entire Texas power grid controlled by ERCOT would have collapsed and approximately 26 million customers would have been without power.” The article continues, “What the men and women of ERCOT did was not an easy task. Their efforts kept the power flowing and saved their grid to operate another day.” Can you imagine the sigh of relief that was collectively let out in the control room when the frequency went back above 59.4 Hz?

·        Of course, this story doesn’t have a happy ending, since the financial meltdown kicked in in the early afternoon, when the Texas PUC made the disastrous decision that the current $1,200/megawatt-hour wholesale power price wasn’t high enough (despite the normal level being about $25/mwh) and bumped it up to $9,000. And ERCOT decided to keep that rate in place through Friday, even though the market price returned to about $25 on Thursday. So while the mean and women who worked in the ERCOT control center were genuine heroes, the men and women who worked in the rates department (or whatever it’s called) were…not heroes.

·        But this isn’t the whole story. If the 9-minute window had passed without frequency going back above the critical level, the protection system would have brought all generation in ERCOT to a “graceful” shutdown – meaning the fact that the protection was in place prevented permanent damage to a lot of generation, which would have taken months to repair. But once the graceful shutdown had happened, how would ERCOT have restarted the grid? Would they just hit another switch and all the generators would immediately kick in and start producing power?

·        Unfortunately, it’s not that simple. This is because of a dirty little secret: It takes power to produce power. Generators require electromagnets in order to work. But electromagnets require…electric power! So if there’s no electric power to be had for love or money – as would have been the case if the ERCOT grid had totally collapsed -  how does any power get produced?

·        This is where “blackstart”, referred to in the ERCOT citation at the beginning of this article, comes in. I described blackstart in more detail in this post, but suffice it to say that the grid can be restarted if you start small and build up. You start with smaller plants that can be restarted with a backup generator or a big battery (as well as hydro plants, whose power source never stops, so the electromagnets are always energized. Of course, Texas isn’t known for an abundance of hydro power, like for example the Pacific Northwest). They energize particular lines, which then energize larger plants, which energize other lines…until finally the grid is operating again.

·        Doing this requires that very detailed procedures be followed in the proper order, which is all laid out in a blackstart plan. Every grid operator is required by FERC and NERC to have a blackstart plan. Did ERCOT have one?

·        The writer of the article wondered about this and checked out what ERCOT has said. He found that “there is a reference saying ERCOT has a black start plan, but it has never been used since there has never been a system-wide blackout.” Fair enough, but the plan should be tested regularly through non-intrusive means. Was this done? The writer couldn’t find any reference to drills.

·        In fact, he found another reference that said “…there are 13 units capable of black start operations in ERCOT, but six of those units experienced outages because of the extreme weather.” In other words, even if the blackstart plan had been tested, it might not have worked if needed, probably because the plan was written in anticipation of a hot-weather outage, when the generation would still all have been available.

·        So the people in the ERCOT service area dodged one bullet, due to the quick-thinking of people in the ERCOT control center. But had they not dodged that bullet and there had been a total (but graceful) grid shutdown, they might have found they had to wait a few days without power (in a severe cold snap, of course), while the ERCOT staff tried to improvise to get the grid running again. So an outage of hopefully just a few hours would have turned into one of at least a few days, and maybe longer.

·        And multi-day outages, especially over most of a big state like Texas, aren’t pretty. 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.