Back at the ranch – part 3

This is the third installment of my post following up on the Texas power (and financial and human) disaster, which I call the Valentine’s Day Storm. In the second installment, I posted the comments of Andrew Gallo, a former ERCOT employee on the first post, primarily pointing out mistakes (nicely, of course) in my discussion of the ERCOT blackstart plan, and what FERC does and doesn’t regulate in Texas.

Now I’m posting the comments of Kevin Perry, former Chief CIP Auditor of the SPP Regional Entity (next door to Texas, of course). He provides tons of detail on blackstart plans, and provides a great explanation of how the restoration process works after blackstart (which of course wasn’t needed in Texas, but it almost was). He also addresses my assertion (based on news articles I read during the crisis) that the whole ERCOT grid might have been down for months, had a particular system (generation ride-through, which works in conjunction with UFLS) not worked as designed between about 1:50 and 2:00 AM on February 15. My comments on what he wrote are in red.

Here is some enlightenment to pass along to Tom Alrich, Boy Engineer, when you next see him…(oh, the snark! I self-deprecatingly referred to myself this way, and now he repeats it like it’s true! The ingratitude is astounding…After I taught him everything he ever knew about NERC CIP…😊).

Texas RE, the Regional Entity that provides oversight of the ERCOT entities, does enforce the NERC Reliability Standards.  ERCOT is registered as a Balancing Authority, Planning Authority/Planning Coordinator, Reliability Coordinator, Resource Planner, Transmission Operator, and Transmission Service Provider.  So, ERCOT is subject to a large number of NERC Reliability Standards, including those applicable to BAs, TOPs, and RCs.  There are 19 TOPs in the ERCOT Interconnection, and I will admit I am not sure what role ERCOT plays in this space.  I suspect they are contractually the TOP for some TOs that are not TOPs themselves.  That is more common than one might think.  They are the only BA and the only RC.  To save you some time attempting the near impossible task of finding anything on the NERC website (hear, hear! I’ve often said that I know a surefire way to protect critical infrastructure information from ISIS: post it on the NERC website. They’ll never find it there!), the registration files are found at Organization Registration and Organization Certification (nerc.com).  Scroll down to the Registration: Compliance Registry Files ‎(3) link, expand the link, and download the “NCR Active Entities List” spreadsheet.

 So, why is this important?  It is important because system restoration (black start) requirements and applicability are spelled out in EOP-005.  EOP-005 is applicable to Transmission Operators, Generator Operators, Transmission Owners identified in the Transmission Operator's restoration plan, and Distribution Providers identified in the Transmission Operator's restoration plan.  ERCOT is a TOP and therefore must have a system restoration plan.  ERCOT is also a Reliability Coordinator.  The Reliability Coordinator used to be required to develop a detailed Regional Restoration Plan, which was basically a merging of the individual TOP plans.  That was done away with some years ago and today the RC role is to have a high-level restoration plan that mostly addresses TOP restoration coordination, and to review and approve the TOP plans on an annual (or as otherwise agreed to) schedule (per EOP-006). 

So, getting back to the TOP where the heavy lifting is performed, the TOP plan must also be submitted to the RC when it substantively changes.  The TOP must verify that its plan will work through testing, steady state and dynamic simulations, or analysis of actual events at least every five years.  And the TOP must test each black start resource at least every three years to make sure it will start without BES support and can energize a bus to provide the expected cranking power. 

Each TOP has to perform system restoration training to its operators annually.  And applicable TOs and DPs (referenced in the TOP plan) have to have at least two hours of training for their field switching personnel every two calendar years.  SPP (the RC) conducts Regional training annually, which accomplishes the TOP training requirement.  If ERCOT (the RC) is smart, they will do the same.  GOPs with black start resources must provide a minimum of two hours training in starting the resource every two years.  TOPs and GOPs are required to participate in the RC restoration drills as requested by the RC.  O&P violations are published with detail by FERC, so if ERCOT was not complying with EOP-005 (and EOP-006), that information would be readily available within a couple of years of the violation. 

Now, system restoration plans are very detailed, with unit starting sequences and cranking path switching procedures a small portion of the overall plan.  They usually anticipate problems and thus have options.  One entity I am familiar with will try starting each one of its black start units in a defined sequence until they get one to come up, and if all else fails, will wait on their neighbor to provide sufficient energy on the grid to provide cranking power. Another entity I am familiar with relies entirely on its neighbors.  Relying on the neighbor requires defined cranking power switching procedures from defined inter-utility connection substations (tie line points). 

So, what if all of ERCOT fails and none of the black start units can fire up for some reason (and I suspect you have greater chances of getting a blood clot from the J&J COVID vaccine)?  The DC Ties between the Eastern/Western Interconnections and ERCOT come to the rescue (You may believe the ERCOT grid is totally isolated from the two big Interconnects, but there are actually DC ties that can import and export power. However, DC ties require manual activation – they don’t pass power solely according to the laws of physics, like AC ties do. At the height of the crisis on February 15, the DC ties wouldn’t have helped since they take some time to activate – AC ties would have. But DC ties can certainly help in restoring the grid after a disaster, as Kevin describes, since timing isn’t so critical then).

You need roughly 40-50 MW power to energize the station auxiliaries.  The DC ties typically pass several hundred MW of power through the tie station.  So, you build a cranking path to start up a fossil unit (and more importantly to restore external station power to the nukes) and once you get one or more fossil units up, you rebuild more and more of the grid from there.  Regardless of how you do the restoration, you have to carefully balance generation and load to keep the energized system stable, or the whole thing collapses and you have to start all over again.  But if all works to plan, you will bring up islands across the blacked out area and the RC then coordinates the knitting of the islands together.

 The generating unit stators, magnets, etc., (what you referred to as electro-magnets) do not need power themselves in order to generate power.  It is the pumps, blowers, pressure systems, even the control room gear, what are referred to as station auxiliaries, that need the power in order to fire up the fossil unit.  The turning of the turbine shaft connected to the generator itself creates power through the magic of coiled wires and magnetic fields.  Maybe I am wrong, but I am not aware that the magnets in the generator are themselves energized in order to create the necessary magnetic field.  And, oh by the way, even hydro units need station power to operate the wicket gates.  At a hydro unit, you will find a very small water turbine that always runs.  It provides enough station power to keep the lights on and bring the big turbines online. 

Moving on to under-frequency load shedding, I am not ready to give all the folks at ERCOT a congratulatory shout out just yet.  Under frequency load shedding (UFLS) requirements are defined in PRC-006, which is applicable to Planning Coordinators and all entities responsible for the ownership, operation, or control of UFLS equipment as required by the UFLS program established by the Planning Coordinators.  That includes TOs, DPs, and UFLS-only DPs (not sure what the distinction is there).  The Planning Coordinator designs and coordinates the UFLS program and load shed schedules, and has to assess the UFLS plan at least once every five years using dynamic simulation.  It is not that the ERCOT RC staff had nine minutes in which to scramble and shed load.  UFLS relaying is automatic.  So maybe ERCOT, as the PC, gets a shout out for designing a plan that worked.  The UFLS relay owners get a shout out for properly implementing the UFLS schedules in the relays they own.  SPP’s plan, as an example, is three tiered.  More and more load is automatically dumped as frequency declines.  You might even see deliberate separation (islanding) as part of the plan in order to avoid a complete collapse. 

What I have the hardest time with is the notion that the ERCOT grid would be down for possibly months if it totally collapsed.  When the grid collapses, the generating units are tripped offline and substations go on battery backup.  Fossil units disconnect from the grid and big valves open up with a tremendous bang and an ear splitting shriek (you really need to be there and experience it one time) as they dump steam pressure.  The fossil unit turbines come to a near stop (they need to be kept slowly turning because they are hot and will distort if completely stopped).  Now, maybe if all of ERCOT collapsed, that is the concern – no power to turn the turbines and resulting turbine damage will prevent restart (the news articles I based this on said that there would be damage to the equipment in the plants, which probably meant the turbines). 

Otherwise, a typical fossil unit can warm-start in roughly 24 hours, often less.  Cold starting takes a bit longer because you gotta boil water to turn the turbine.  Nukes have to be cleared by the NRC before they can restart, and that could be days once external station power is restored.  Wind turbines are simply parked and locked until ready to turn back on.  So, unless there were significant turbine damage, I cannot conceive how ERCOT would be down for months (but I think that was the point of the articles I read – there could have been turbine damage. Fortunately, this theory wasn’t put to the test).  And I suspect it takes longer than a few months to get a replacement turbine and install it, much less one for each and every unit in ERCOT-land (yes, that’s the idea. Once you have widespread turbine damage, you’re in big trouble. And of course, we’re not talking about wind turbines here. We’re talking about huge monsters costing sometimes tens of millions of dollars, that rotate at 1800-3600 rpm, i.e. 30-60 times a second. You don’t just order a new one on Amazon and have it show up in a week).  In the event there is some damage, you may see Texas turn into post-9/11 Baghdad with rotating load shed (rolling blackouts for the lay person), until enough generation is back online to fully serve load.  But I cannot see Texas being in the dark for months (I never thought I’d see planes fly into buildings in NYC, but it happened. Again, the UFLS system did what it was supposed to do at 1:50 AM on February 15, so this is now an academic question). 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.