NERC CIP: Is there a risk of over-concentration in the cloud?

Most of the discussion of cloud risks in the power industry has focused on cybersecurity: Is it safe to deploy critical infrastructure systems and data in the cloud? And if so, how can we fix the NERC CIP standards so that electric utilities and Independent Power Producers (IPPs) feel comfortable using the cloud for all types of workloads? Currently, NERC entities don’t feel comfortable using the cloud for systems subject to CIP compliance (even low impact ones). You could say quite truthfully that the electric power industry is seriously underutilizing the cloud because of fears of becoming non-compliant with CIP, even though those fears are often not justified.

But there has lately been discussion of the opposite risk: that too many power industry assets will be deployed in the cloud, so that a big outage at one of the major CSPs will cause a substantial adverse impact on the power grid – or something like that. 

Before we go any further, let’s have a reality check on whether this is a current problem. How many assets subject to NERC CIP are deployed in the cloud today? At NERC’s GridSecCon conference last month, I heard a person who is in a position to know this answer the question:…(drumroll, please….) There are currently two (count them!) small low-impact Control Centers deployed in the cloud (he didn’t say whether they’re deployed with the same platform CSP or not, let alone whether they’re in the same data center, etc).

I don’t know about you, but this hardly strikes me as an imminent danger. Perhaps in five years or more we will need to think harder about this problem. But since the subject comes up repeatedly (as it did during NERC’s Cloud Technical Conference three weeks ago), it seems we can’t avoid it. What is the nature of this threat?

This threat is usually described as due to the over-concentration of generation assets in the cloud – that is, systems that control generation. Let’s say the two Control Centers just mentioned control generating assets (although they’re much more likely to control renewables like wind and solar. Given the low latency requirements of synchronous – i.e., mostly fossil – generation, I find it doubtful that any such generation will ever be deployed in the cloud. Given that renewables generation is interrupted all the time when the sun doesn’t shine and the wind doesn’t blow, it’s hard to see how interruption of any quantity of renewable generation would lead to a grid emergency. But let’s pretend that it would…).

Furthermore, let’s pretend (isn’t this fun?) that those two Control Centers are now twenty, and they’re all deployed with the same platform CSP. Additionally, let's say these each control 1,000MW (I’m keeping that number under 1500MW, since that would make the Control Center medium impact. Currently, medium and high impact systems cannot be deployed in the cloud, without putting their owners into non-compliance with multiple CIP requirements).

Now, let's say the CSP experiences a catastrophic outage that brings it mostly down (which of course has hardly ever happened). Will this bring all that generation to an immediate halt? Only if none of these 1,000MW Control Centers has a backup somewhere else, perhaps in another CSP’s cloud or on premises at the IPP – which I hope is not the case, although again the risk with renewables is much lower. So now we must imagine a much larger outage that affects multiple platform CSPs. And since each CSP should (and does) have complete redundancy for all their workloads, we also must imagine that none of those redundant workloads is operational, either.

Of course, if someone with a vivid imagination (and I don’t deny those people exist) wants to assume that all the unfortunate events just described can easily come to pass, how will we prevent this from happening? The answer seems quite simple: We reduce the concentration of Control Centers in the cloud.

One way to do this is to simply ban cloud use by the power industry. But let’s say that’s too drastic. How do we reduce the concentration of grid control in the cloud without totally banning it? To do this, we need to bring the CSP into the compliance picture by developing special NERC CIP standards that apply to them. They will need to figure out some way to limit their number of NERC CIP customers (perhaps by limiting them to entities whose names begin with A through L?) and be subject to penalties if they don’t do that.

Today, no CSP is subject to compliance with NERC standards, and it’s hard to identify any Functional Model designation they would fall into. One that has been mentioned is Generation Operator (GOP)? After all, the CSP in the above example hosts assets that control at least 20,000MW. If you bring in four times as many 1,000MW Control Centers, that amount of generation would bring the CSP over the threshold (75MW) to be low impact under CIP.

But just because the CSP hosts a bunch of servers that are controlling generation, does that make the CSP a GOP – as opposed to the entity that actually controls those servers? If so, then a CSP that hosts servers involved in pharmaceutical manufacturing should be declared a drugmaker and become subject to FDA regulations. And a CSP that hosted software for manufacturing aircraft would be declared an aircraft manufacturer and would perhaps be subject to FAA regulations.

The fact is that the CSP is just a platform used by the actual GOP, in the same way that HP or Dell servers are used on premises to control generation. Just like HP and Dell, CSPs will never consent to become NERC entities, nor will it ever make sense to try to force them to do that. If over-concentration in the cloud ever becomes a problem, the way to combat it will be to take other measures to prevent at least some NERC entities from using the cloud. Come to think of it, we’re doing a great job of that today. We just need to keep up what we’re doing!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.