When I talk
about the costs of NERC CIP version 5, I have focused so far on the costs in
dollars, and on my impression
from talking with a number of entities that a large portion (perhaps the greater
part) of the amount being spent on CIP v5 is going to compliance paperwork and
other tasks, not to enhancing cybersecurity.
But there is
another hidden cost to the current CIP approach, which is the fact that
confusion over what the CIP requirements mean can inhibit implementation of
technologies that will greatly enhance productivity at NERC entities; this is
as much of a “compliance cost” as any dollars actually spent on CIP. Exhibit A
for this proposition is virtualization.
Here is my
impressionistic history of CIP and virtualization. In this discussion, I’m primarily
referring to server virtualization, although virtual LANs and virtualized
storage are probably in the same situation.
- CIP versions 1 through 3 didn’t mention virtualization at
all. However, I don’t believe there was a lot of complaining about this
fact. From what I’ve heard, most entities just decided they weren’t going
to virtualize any NERC CIP cyber assets – not wishing to spend a lot of
effort on something that might end up getting shot down by the auditors
anyway. This isn’t too surprising, considering that when CIP v1 was
developed in 2006 and 2007, virtualization was nowhere near as widespread
in the IT environment as it is today.
- As v5 was developed starting in 2011, I was expecting that
virtualization would have been addressed, since it was already prevalent
in IT environments at the time and the potential productivity gains and
hardware cost savings clearly could be substantial.
- Unfortunately, CIP v5 (and v6) didn’t address virtualization,
either. However, by now many NERC entities have gotten a taste of what
server virtualization can do for them, based on their experience with IT
data centers. They are anxious to use their experience to virtualize their
control centers as well.[i]
- NERC and the Regions agreed with this sentiment. After v5
was approved in late 2013, NERC started talking about developing guidance
for virtualization, and even had a conference on the topic in 2014.
However, nothing came of this, and in the SAR
for CIP v7, NERC included virtualization as one topic the new Standards
Drafting Team needed to address – thus acknowledging that there was no way
to address this topic other than revising the standards.
I agree with
this: There is no way that virtualization could be treated as a Lesson Learned
or something like that.[ii] Since
it’s not mentioned in the standards at all now, there’s no way you can examine
the current wording to see how virtualization is to be treated; you’ll have to
wait for v7 to get certainty. However, CIP v7 is unlikely to be in effect until
three or four years from now.
At the same
time, NERC and the Regions are taking pains not to discourage entities that
want to implement virtualization within their ESPs. But from some informal
discussions I’ve had, it seems very few entities are taking them up on doing
this. This is because, even though NERC has provided a few pieces of “guidance”
on handling virtualization, there is a lot more that is now simply
undetermined.
To be
specific, NERC has provided the following “guidance”:
- VMs that are within an ESP need to be protected with the
same requirements as they would if they were physical Cyber Assets.
- It is forbidden to mix ESP and non-ESP VMs within a single
host server. For example, if any VMs are BCA or BCS, the entire server
(i.e. the physical box) needs to be treated as a BCA, and there can’t be
any VMs running on the server that aren’t inside the ESP.
- NERC has said (although I can’t find where they said it;
it must have been a draft Lesson Learned that was withdrawn) that VMs of
different impact levels in a single host server must be classified
according to the high water mark; that is, if there are both High and Low
impact BCAs or BCS on VMs hosted within the box, all of the VMs will be at
least PCAs, and will be High impact.
- In NERC’s CIP v5 Evidence Request User Guide, issued in
December, NERC pointed
out that, when VMs can be shared among multiple physical servers, all
of the servers need to be protected at the highest impact rating of any of
the VMs being shared. So if there is one High VM in a server farm, all of
the servers will need to be treated as High impact BCAs, and all the VMs
will have to be High impact BCAs or PCAs.
However, the
gaps in v5 that NERC hasn’t addressed outweigh the three items that it has
addressed. The basic problem for virtualization in v5 is that a Cyber Asset is
defined as a “Programmable electronic device
(my emphasis)”; and only Cyber Assets are in scope for v5. We all know that
“programmable” isn’t defined in v5, but how about “device”? That isn’t defined
either, but I think it has a fairly common-sense meaning. Among its other attributes,
a device is something that, if you drop it on your foot, it will hurt. If you
drop a computer on your foot, it will hurt; so it is a device. However, a VM is
software; you can’t even pick it up, let alone drop it. Thus a VM isn’t a Cyber
Asset, and therefore it isn’t a BES Cyber Asset. Q.E.D.
What this
means is that VMs are officially out of scope for CIP v5 and v6, just as are
any computers that aren’t within the ESP (or say the coffee maker in the
control center – although some have said the coffee maker should be protected
anyway, since it’s probably essential to the stability of the BES, especially
late at night!). So technically, you are free to do what you want with a VM,
whether or not it’s within an ESP. You can protect it according to the
standards, or not. You might get a PV for – say – not patching the OS on a VM,
but it is highly unlikely that PV will become an actual violation and even more
unlikely that, were you to appeal a violation to the civil courts, it would be
upheld.
Of course, I
don’t recommend that you not protect any VMs that are within an ESP! Especially
since this contradicts one of the four pieces of “guidance” that NERC has issued.
But there are two problems with that “guidance”:
- It has no official status. As I just said, this means it’s
not enforceable in the strict sense of being upheld by the court system.
- It doesn’t address the many other issues that naturally
arise when an entity tries to utilize virtualization in a CIP v5/v6 ESP.
What are
some of the other issues that arise when virtualization is used in a v5/v6
environment? Here are a few[iii]:
- How should remote management of VMs that are within an ESP
be handled? Since the VMs are ESP devices, does the remote management need
to come through an Intermediate System?
- Can ESP VMs ever be hosted outside of a PSP?
- Do purely virtual ESPs need to be monitored in the same
way as non-virtual ones?
- If VMs are replicated between primary and backup control centers, does the communication medium between them need to be within a PSP?
I’m sure there are many other questions that
arise. And that’s the problem: Even though entities may be satisfied with the “guidance”
that NERC has provided on virtualization, there are many questions that aren’t
addressed in that guidance. And if an auditor disagrees with an entity’s own
answer to one of these questions, the entity will be on the losing end of that
transaction! Frankly, I think this is why NERC entities are so reluctant to
implement virtualization in control centers[iv].
So entities need a comprehensive guide to
implementing virtualization within an ESP, but none has been provided yet. Moreover,
since CIP v7 won’t be approved for probably at least three years, anything that
the entity does in the process of complying in a virtualized environment will
be subject to second-guessing by an auditor, when there is no guidance on the
issue one way or the other.
What could be done to fix this? Well, there
certainly isn’t anything that can be done officially, except for all entities to
wait for v7 to implement any virtualization within an ESP. But how about
unofficially? Could say one of the regions develop their own informal guide,
then let the other regions adopt it as well?
I honestly think this would be the best
course. I also honestly don’t believe it will happen. However, I also know that
use of virtualization within ESPs (and especially control centers) would make
utilities much more efficient and arguably more secure as well. It’s a shame
this will have to wait for v7.
The views and opinions expressed here are my own and don’t
necessarily represent the views or opinions of Deloitte Advisory.
[i]
One huge advantage of virtualization is in backup/DR. Instead of having to
maintain a secondary control center filled with exactly the same pieces of
hardware as the primary, and to meticulously replicate every configuration
change in the primary to the secondary, the entity can simply replicate VMs
from the primary to secondary whenever there’s a change.
[ii]
In fact, there was never any way that almost any of the big questions about v5
could be addressed, other than through rewriting the requirements.
Unfortunately, some of those big questions (and a lot of small ones) won’t even
be addressed
in v7. But virtualization will be addressed, at least.
[iii]
All of these questions were discussed in a TRE webinar last year. The speaker
mentioned that there were almost certainly many more questions that will come
up as an entity gets serious about implementing virtualization within an ESP.
[iv]
Of course, the benefits of virtualization in generating stations and
substations are far smaller than they are for control centers.
This is a really well written blog. I’ll be sure to bookmark it and return to read more of your useful information. Thanks for the post.virtualization technology services
ReplyDelete