Most of the discussion of cloud risks in the power industry has focused on cybersecurity: Is it safe to deploy critical infrastructure systems and data in the cloud? And if so, how can we fix the NERC CIP standards so that electric utilities and Independent Power Producers (IPPs) feel comfortable using the cloud for all types of workloads? Currently, NERC entities don’t feel comfortable using the cloud for systems subject to CIP compliance (even low impact ones). You could say quite truthfully that the electric power industry is seriously underutilizing the cloud because of fears of becoming non-compliant with CIP, even though those fears are often not justified.
But there has lately been discussion of the opposite risk: that
too many power industry assets will be deployed in the cloud, so that a big
outage at one of the major CSPs will cause a substantial adverse impact on the
power grid – or something like that.
Before we go any further, let’s have a reality check on
whether this is a current problem. How many assets subject to NERC CIP are deployed
in the cloud today? At NERC’s GridSecCon conference last month, I heard a
person who is in a position to know this answer the question:…(drumroll, please….)
There are currently two (count them!) small low-impact Control Centers deployed
in the cloud (he didn’t say whether they’re deployed with the same platform CSP
or not, let alone whether they’re in the same data center, etc).
I don’t know about you, but this hardly strikes me as an
imminent danger. Perhaps in five years or more we will need to think harder
about this problem. But since the subject comes up repeatedly (as it did during
NERC’s Cloud Technical Conference three weeks ago), it seems we can’t avoid it.
What is the nature of this threat?
This threat is usually described as due to the
over-concentration of generation assets in the cloud – that is, systems that
control generation. Let’s say the two Control Centers just mentioned control
generating assets (although they’re much more likely to control renewables like
wind and solar. Given the low latency requirements of synchronous – i.e.,
mostly fossil – generation, I find it doubtful that any such generation will
ever be deployed in the cloud. Given that renewables generation is interrupted
all the time when the sun doesn’t shine and the wind doesn’t blow, it’s hard to
see how interruption of any quantity of renewable generation would lead to a
grid emergency. But let’s pretend that it would…).
Furthermore, let’s pretend (isn’t this fun?) that those two
Control Centers are now twenty, and they’re all deployed with the same platform
CSP. Additionally, let's say these each control 1,000MW (I’m keeping that
number under 1500MW, since that would make the Control Center medium impact. Currently,
medium and high impact systems cannot be deployed in the cloud, without putting
their owners into non-compliance with multiple CIP requirements).
Now, let's say the CSP experiences a catastrophic outage
that brings it mostly down (which of course has hardly ever happened). Will this
bring all that generation to an immediate halt? Only if none of these 1,000MW
Control Centers has a backup somewhere else, perhaps in another CSP’s cloud or
on premises at the IPP – which I hope is not the case, although again the risk with
renewables is much lower. So now we must imagine a much larger outage that
affects multiple platform CSPs. And since each CSP should (and does) have
complete redundancy for all their workloads, we also must imagine that none of
those redundant workloads is operational, either.
Of course, if someone with a vivid imagination (and I don’t
deny those people exist) wants to assume that all the unfortunate events just
described can easily come to pass, how will we prevent this from happening? The
answer seems quite simple: We reduce the concentration of Control Centers in
the cloud.
One way to do this is to simply ban cloud use by the power
industry. But let’s say that’s too drastic. How do we reduce the concentration
of grid control in the cloud without totally banning it? To do this, we need to
bring the CSP into the compliance picture by developing special NERC CIP standards
that apply to them. They will need to figure out some way to limit their number
of NERC CIP customers (perhaps by limiting them to entities whose names begin
with A through L?) and be subject to penalties if they don’t do that.
Today, no CSP is subject to compliance with NERC standards,
and it’s hard to identify any Functional Model designation they would fall
into. One that has been mentioned is Generation Operator (GOP)? After all, the CSP
in the above example hosts assets that control at least 20,000MW. If you bring
in four times as many 1,000MW Control Centers, that amount of generation would
bring the CSP over the threshold (75MW) to be low impact under CIP.
But just because the CSP hosts a bunch of servers that are
controlling generation, does that make the CSP a GOP – as opposed to the entity
that actually controls those servers? If so, then a CSP that hosts servers
involved in pharmaceutical manufacturing should be declared a drugmaker and
become subject to FDA regulations. And a CSP that hosted software for manufacturing
aircraft would be declared an aircraft manufacturer and would perhaps be
subject to FAA regulations.
The fact is that the CSP is just a platform used by the
actual GOP, in the same way that HP or Dell servers are used on premises to
control generation. Just like HP and Dell, CSPs will never consent to become
NERC entities, nor will it ever make sense to try to force them to do that. If
over-concentration in the cloud ever becomes a problem, the way to combat it
will be to take other measures to prevent at least some NERC entities from
using the cloud. Come to think of it, we’re doing a great job of that today. We
just need to keep up what we’re doing!
Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.