Last week’s
NERC CIPC meeting in Atlanta was the first I attended this year (the last was a
year ago in December, at the same hotel), and it was certainly one of the best
I’ve attended. The highlight of the meeting, for me and many others, was the
presentation from Dave Revill, vice-chair of the CIPC and (I believe) co-chair
of the CIP Modifications Standards Drafting Team.
Dave’s topic
was everything that the team is working on now, but his big focus – as I
expected – was the modifications the team is proposing in order to allow the
standards to finally take account of virtualization. The team recently posted
these modifications, which include new definitions, retired definitions, and
some significant changes to various requirements. The modifications were posted
for comment (not ballot), but the team intends to post modifications for ballot
early next year. The comment period expires tomorrow (Dec. 18), so I realize
that what I’ll say in this post is too late for anybody to use in their
comments – but that’s OK, since as you’ll see my main concern isn’t tweaking
what the SDT posted. I think there needs to be a very different approach.
If you want
to get the full picture of what the team is proposing, you should go to the
team’s
web
site and go to the set of documents posted under “CIP Virtualization Updates”.
I will very briefly summarize what they are proposing to do (this isn’t hugely
different from what they first outlined in a white paper and webinar this June.
I wrote three posts on what they were proposing, starting
here):
- The team wants to move CIP from being based on individual
Cyber Assets to being based on systems (specifically, BES Cyber Systems).
Of course, a systems-based approach is needed if you’re going to include
virtualization, since a Cyber Asset is a “device”. There is no
NERC-approved definition of this term, but my unofficial definition is “Something
is a device if it will hurt you if dropped on your foot”. Even if you can
figure out how to drop a virtual system on your foot, it won’t hurt it. So
it isn’t a device. Q.E.D.
- They are eliminating the definitions of both Cyber Asset
and BES Cyber Asset, both of which were very problematic. Instead, they’ve
transferred everything important from the BCA definition to the BES Cyber
System definition (most notably, the provision about 15-minute BES
impact). Entities will now start their BCS identification process by
looking at all the functions performed at an asset (especially the BES
Reliability Operating Services, although those still have no official
status in the requirements) and identifying those that can have a
15-minute impact on the BES if misused, etc. Those systems are BCS,
whether they’re entirely physical, entirely virtual or a mixture of the
two.
- Of course, since the BCS definition is now changed, the
question becomes what changes will need to be made in the requirements in
CIP-003 through CIP-011. The good news is that a lot of the current
requirements (including all requirements in the current CIP-003, -004,
-008, and -009) require exactly zero changes, since they will work fine
with any system, physical or virtual. The somewhat-good news is that many
of the other requirements don’t require any substantial changes.
- But the bad news is that a few of the most prescriptive
requirements, like CIP-005 R1, CIP-007 R2 and CIP-010 R1, require very
substantial changes due to the new systems-based approach. Trying to make
all these changes to those requirements would mire the SDT in an endless
fight and probably ten years of NERC ballots - as I estimated
a couple years ago, when the SDT was discussing a different approach which
would also require a huge number of specific changes to individual
requirements. Instead, the SDT is doing the smart thing and rewriting
those requirements – or incorporating them into others – in a series of
steps that I don’t want to go into here, since that isn’t my purpose in
this post.
I agree with
the SDT’s whole approach, which I think is absolutely the right one if they are
to fulfill their mandate to incorporate virtualization into CIP – although there
would need to be a lot of tweaks to what they wrote, which just about everybody
that commented at the CIPC meeting agreed on (just about everyone started out by
saying “I applaud the SDT since it seems they’re on the right course. But I see
some big problems that need to be addressed…). But there’s one fundamental
problem with what the SDT is proposing (and I’ll admit I didn’t realize this
until after the CIPC session): I don’t think there’s any way the NERC ballot
body will approve the virtualization changes, no matter how many tweaks the SDT
makes to wording.
I will write
a post soon on why I make this assertion, but I’ll say for the moment that
there are two basic reasons:
- There is no pressing need to officially incorporate
virtualization technologies into CIP, because lots of entities have
already done that. Many (even most?) are already using virtual machines in
Control Centers, and the NERC Regions are all supporting them in that
effort (in fact, NERC itself has acknowledged that virtualization is
allowed, as long as no VM hosts mix ESP and non-ESP virtual machines). I
know one entity that virtualized their Control Center in the CIP v1 or V2
days, and passed an audit with flying colors in 2010. Of course, I’m sure
everyone will agree that it would be better to make it official that
virtualization is allowed. And I’ll also grant that use of switch and storage
virtualization is probably being held back some by the lack of official
status for those two technologies. But I am also fairly sure that these
considerations alone won’t be enough for NERC entities to accept the big
changes the SDT is proposing, which brings me to the second reason.
- Why won’t NERC entities accept the big changes the SDT is
proposing? Is it just because utilities are very conservative and don’t
like to change something that is working, however flawed it may be? I don’t
think so. In fact, in a minute I’m going to discuss an even-more-radical
change that I think the industry would accept in a heartbeat. The reason
why I don’t see the SDT’s proposed changes succeeding, no matter how much
they’re tweaked, is because the smell of the bad experience with the CIP
version 5 rollout still hangs in the air in CIP circles.
As I will
discuss in a post soon, the CIP v5 rollout was quite flawed (despite the words
of a NERC official who spoke at the CIPC meeting and seemed to think it was a
huge success), and has left a huge set of unanswered questions about what the requirements
and definitions mean – that are only papered over by a tacit mutual agreement
between the Regions and the entities that I call Don’t Ask, Don’t Tell. There
is every expectation that, given the current NERC auditing program (and again I’ll
have to wait until a new post to explain what I mean by this), exactly the same
thing will happen with the new changes as happened with CIP v5 (one questioner
raised exactly this point, suggesting that the many lingering uncertainties in
the “v5” requirements and definitions should be fixed before we introduce a
bunch more due to the virtualization changes). Given this situation and the
fact that there isn’t a pressing need to change the CIP standards to allow
virtualization, I believe NERC entities will choose the status quo over what
the SDT is proposing.
However,
rather than being a big defeat for CIP reform, I think this is actually a big
opportunity. That’s because there is
one very pressing concern that almost all – if not all – NERC entities share,
and that could be addressed by changes to the CIP standards and definitions
along the lines of what the SDT is proposing, except going further.
Of course,
what I’m talking about here is use of the cloud in CIP environments. Currently,
there are two big issues: a) BCS Information (BCSI) in the cloud, and b) actual
BCS in the cloud. BCSI in the cloud is currently allowed by the standards,
simply because there is no prohibition against having a third party hold BCSI,
whether they’re a vendor or a cloud provider (in fact, I know that a number of
entities with High and/or Medium impact BCS are using cloud-based configuration
management software and have passed audits. There are probably also some entities
using cloud-based security monitoring services. Both services store BCSI in the
cloud). But the problem is that the third party will have to comply with a set
of four or five requirement parts from CIP-004, which I discussed in
this
post.
Let’s consider
just one of these requirement parts, CIP-004-6 R5.3; this requires the entity
(whether the Registered Entity or a third party holding their BCSI) to remove
physical access to storage locations for BCSI within a day of an employee being
terminated. This shouldn’t be a problem for a regular vendor. All of your BCSI
is probably stored on servers in their single data center. It’s no problem to
just remove that terminated employee’s access to that data center in a day or
even a few hours – in fact, they probably do that now.
But what
about a cloud vendor? By the very nature of their business, different parts of your
BCSI could be flitting among servers by the minute, and jumping from one data
center to another across the country any moment. And just one of their data
centers might have in the thousands of employees with physical access to
systems in that center. How could they prove, for every individual employee
that is terminated across all of their data centers, that the person’s physical
access to all of their other data centers was removed within 24 hours?
I’m not
saying it would be completely impossible for a cloud vendor to prove this, or to
mitigate the risk by say keeping all of your BCSI on servers in one particular
data center. But doing so would break their whole business model, and they
simply aren’t going to do that for one customer, or even for all utilities
acting together (and the utilities can’t band together to force these
concessions for antitrust reasons anyway – even if they had the collective
clout for it).
Of course,
there’s little question that any big cloud provider probably has security practices
that go far beyond what any utility could think of implementing on their own –
and they have certifications like FedRAMP to prove this. But where in the
Measures section of any of the CIP requirements will you find a phrase like “…or
submit evidence of a current FedRAMP certification”? Obviously, the answer to
that question is “Nowhere”. I will point
out that the CEWG, a subcommittee of the CIPC whose acronym – I believe - means
“Compliance and Enforcement Working Group”, is working on the issue of BCSI in
the cloud – in fact they held a meeting after the CIPC itself concluded, which
I attended. I’ll hope to report on that in a post in the somewhat near future,
although I’ll summarize by saying this group is doing great work, but don’t
look for any official change to the Measures for quite some time.
So the
problem with BCSI in the cloud isn’t that it’s not allowed by the CIP
standards, but that there is currently no workable way for a cloud vendor to
prove compliance with the requirement parts that apply to them. On the other
hand, like server virtualization, there are definitely a number of entities
with High and/or Medium impact BCS that are in fact storing BCSI in the cloud,
with the tacit approval (remember: Don’t Ask, Don’t Tell!) of their Regions. In
other words, if the SDT were to decide they should extend the modifications
they’re now proposing for virtualization so that BCSI could be realistically
stored in the cloud (which could be done, and would perhaps be easier than
trying to get the Measures changed, as the CEWG is currently thinking), NERC
entities would be pleased to hear this. However, as with the virtualization
changes by themselves, I doubt including both virtualization and
BCSI-in-the-cloud would be enough to overcome the industry’s reluctance to go
through a wrenching change in the CIP standards like CIP v5 was.
But what
about storing actual BES Cyber Systems in the cloud? There, the situation is
very different, for two reasons. For one, while I can’t guarantee that no
entity with Medium or High impact BCS is storing any of them in the cloud (and
of course, outsourced SCADA would be the most tempting way to do so, for many
smaller entities), I am pretty sure they’re not doing it with the tacit or
explicit approval of their Region – meaning they could have a big problem come
audit time, if they haven’t let their Region know they’re doing this.
The other
reason is that having actual BCS in the cloud would require huge changes to a number
of the CIP requirements, even after all of the changes that the SDT is
proposing for virtualization (and BTW, all of the virtualization changes are also
necessary for BCS-in-the-cloud to be allowed by CIP in the future. The problem
is those changes alone aren’t anywhere near enough). Even if an entity were to
get their Region’s agreement that they could store one or more BCS in the
cloud, they would have to sit down with the Region and essentially rewrite most
of the requirements – either the requirements themselves or at least the
Measures.
This leads
me to the point I’ve been hinting at: Were the SDT to decide to extend the
changes they’re proposing so that BCS could be stored in the cloud (as well as
to incorporate virtualization and BCSI in the cloud), I think this would be
something that could actually pass the NERC ballot body (and by the way, at
last December’s CIPC meeting, both NERC and even a senior FERC staff member –
as always, speaking for himself, not the Commissioners – indicated they don’t
have fundamental objections to BCS in the cloud).
Yes, this
would require much more thorough changes to the CIP standards and definitions
than even what the SDT is now proposing for virtualization. But I can already
see a lot of utility financial types, hearing that BCS-in-the-cloud might
actually be happening, jumping for joy and telling their NERC compliance people
to do everything they can to help NERC bring this to fruition.
So I really
think that we shouldn’t mess around with changing CIP just in order to
officially incorporate virtualization. As I said, it’s not really solving any
big problems, but will nevertheless consume at least a couple years of SDT
time. Why put off even talking about addressing BCS-in-the-cloud until the
hoped-for day 2-3 years (at least) from now when virtualization is finally written
into the CIP standards? Why not do both of them at once, even though this will
be a much bigger effort?
I’ll tell
you right now the main reason the SDT won’t consider this now – because their
Standards Authorization Request (SAR) only lists virtualization as something
they need to address. It will be a big deal to go back to NERC for a new SAR,
then have that drafted and balloted, since SARs also need to be approved by the
NERC ballot body. In fact, I think just drafting the SAR as I’m proposing might
require a lot of debate and meetings. Why go through all of that, when the SDT
can just get virtualization passed (and a couple other things), then ride off
into the sunset with the endless thanks of the NERC community?
The answer
to that is: It’s not at all guaranteed that the SDT will ever be able to get the virtualization changes passed, even if they
go back now and make a bunch of changes before they even go to the first
ballot. It’s very possible – and I think likely – that they will spend 1-2
years on this effort and finally decide it’s time to “declare victory and go
home” (as was suggested by Sen. George Aiken as a solution to the Vietnam War
quagmire).
I can see that
some of the current SDT members wouldn’t feel like gearing up now for a massive
change in CIP (much more far-reaching than CIP v5 was), so it’s very likely
that NERC would need to empanel a new SDT to deal with this new CIP version.
And it’s very likely that, even if the work to draft the new SAR started today,
whatever changes finally resulted wouldn’t be in place for at least five years
and even longer than that.
But guess
what? I think that there are a couple even further changes that could literally
make this the last CIP version that would need to be drafted. I’ve actually
outlined these – at least mentally – already, since I’ve been working on a book
(now suspended because I’ve been too busy, but I still hope to get back to it
in the future) on the problems of CIP and how they can be addressed. You might
think it’s impossible for me to say (at least without fingers crossed) that
there could be a set of CIP changes that would ensure no further changes would
ever be needed. But this is what I believe, although I’ll admit this would
require changes in how the CIP standards are enforced, not just in the
standards themselves. That is beyond the capacity of any SDT acting on their
own, but isn’t beyond the capacity of NERC, working in consensus with FERC – i.e.
it wouldn’t require going back to Congress, which could literally make it
impossible.
However,
even if we don’t go even further and implement the changes that would make this
the last time CIP has to be redrafted, I definitely think it’s worth going back
and drawing up a new SAR so that both virtualization and the cloud can be
officially included in CIP. In fact, since I don’t see the SDT’s virtualization
changes passing without this, I’d say it’s the only way to proceed. Sometimes
you have to aim high if you want to accomplish anything at all.
Any opinions expressed in this blog post are strictly mine
and are not necessarily shared by any of the clients of Tom Alrich LLC.
If you would like to comment on what you have read here, I
would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that
if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or
challenges like what is discussed in this post – especially on compliance with
CIP-013; we also work with security product or service vendors that need help
articulating their message to the power industry. To discuss this, you can
email me at the same address.