The Virtualization Conundrum

When I talk about the costs of NERC CIP version 5, I have focused so far on the costs in dollars, and on my impression from talking with a number of entities that a large portion (perhaps the greater part) of the amount being spent on CIP v5 is going to compliance paperwork and other tasks, not to enhancing cybersecurity.

But there is another hidden cost to the current CIP approach, which is the fact that confusion over what the CIP requirements mean can inhibit implementation of technologies that will greatly enhance productivity at NERC entities; this is as much of a “compliance cost” as any dollars actually spent on CIP. Exhibit A for this proposition is virtualization.

Here is my impressionistic history of CIP and virtualization. In this discussion, I’m primarily referring to server virtualization, although virtual LANs and virtualized storage are probably in the same situation.

1. CIP versions 1 through 3 didn’t mention virtualization at all. However, I don’t believe there was a lot of complaining about this fact. From what I’ve heard, most entities just decided they weren’t going to virtualize any NERC CIP cyber assets – not wishing to spend a lot of effort on something that might end up getting shot down by the auditors anyway. This isn’t too surprising, considering that when CIP v1 was developed in 2006 and 2007, virtualization was nowhere near as widespread in the IT environment as it is today.
2. As v5 was developed starting in 2011, I was expecting that virtualization would have been addressed, since it was already prevalent in IT environments at the time and the potential productivity gains and hardware cost savings clearly could be substantial.
3. Unfortunately, CIP v5 (and v6) didn’t address virtualization, either. However, by now many NERC entities have gotten a taste of what server virtualization can do for them, based on their experience with IT data centers. They are anxious to use their experience to virtualize their control centers as well.[i]
4. NERC and the Regions agreed with this sentiment. After v5 was approved in late 2013, NERC started talking about developing guidance for virtualization, and even had a conference on the topic in 2014. However, nothing came of this, and in the SAR for CIP v7, NERC included virtualization as one topic the new Standards Drafting Team needed to address – thus acknowledging that there was no way to address this topic other than revising the standards.

I agree with this: There is no way that virtualization could be treated as a Lesson Learned or something like that.[ii] Since it’s not mentioned in the standards at all now, there’s no way you can examine the current wording to see how virtualization is to be treated; you’ll have to wait for v7 to get certainty. However, CIP v7 is unlikely to be in effect until three or four years from now.

At the same time, NERC and the Regions are taking pains not to discourage entities that want to implement virtualization within their ESPs. But from some informal discussions I’ve had, it seems very few entities are taking them up on doing this. This is because, even though NERC has provided a few pieces of “guidance” on handling virtualization, there is a lot more that is now simply undetermined.

To be specific, NERC has provided the following “guidance”:

• VMs that are within an ESP need to be protected with the same requirements as they would if they were physical Cyber Assets.
• It is forbidden to mix ESP and non-ESP VMs within a single host server. For example, if any VMs are BCA or BCS, the entire server (i.e. the physical box) needs to be treated as a BCA, and there can’t be any VMs running on the server that aren’t inside the ESP.
• NERC has said (although I can’t find where they said it; it must have been a draft Lesson Learned that was withdrawn) that VMs of different impact levels in a single host server must be classified according to the high water mark; that is, if there are both High and Low impact BCAs or BCS on VMs hosted within the box, all of the VMs will be at least PCAs, and will be High impact.
• In NERC’s CIP v5 Evidence Request User Guide, issued in December, NERC pointed out that, when VMs can be shared among multiple physical servers, all of the servers need to be protected at the highest impact rating of any of the VMs being shared. So if there is one High VM in a server farm, all of the servers will need to be treated as High impact BCAs, and all the VMs will have to be High impact BCAs or PCAs.

However, the gaps in v5 that NERC hasn’t addressed outweigh the three items that it has addressed. The basic problem for virtualization in v5 is that a Cyber Asset is defined as a “Programmable electronic device (my emphasis)”; and only Cyber Assets are in scope for v5. We all know that “programmable” isn’t defined in v5, but how about “device”? That isn’t defined either, but I think it has a fairly common-sense meaning. Among its other attributes, a device is something that, if you drop it on your foot, it will hurt. If you drop a computer on your foot, it will hurt; so it is a device. However, a VM is software; you can’t even pick it up, let alone drop it. Thus a VM isn’t a Cyber Asset, and therefore it isn’t a BES Cyber Asset. Q.E.D.

What this means is that VMs are officially out of scope for CIP v5 and v6, just as are any computers that aren’t within the ESP (or say the coffee maker in the control center – although some have said the coffee maker should be protected anyway, since it’s probably essential to the stability of the BES, especially late at night!). So technically, you are free to do what you want with a VM, whether or not it’s within an ESP. You can protect it according to the standards, or not. You might get a PV for – say – not patching the OS on a VM, but it is highly unlikely that PV will become an actual violation and even more unlikely that, were you to appeal a violation to the civil courts, it would be upheld.

Of course, I don’t recommend that you not protect any VMs that are within an ESP! Especially since this contradicts one of the four pieces of “guidance” that NERC has issued. But there are two problems with that “guidance”:

1. It has no official status. As I just said, this means it’s not enforceable in the strict sense of being upheld by the court system.
2. It doesn’t address the many other issues that naturally arise when an entity tries to utilize virtualization in a CIP v5/v6 ESP.

What are some of the other issues that arise when virtualization is used in a v5/v6 environment? Here are a few[iii]:

• How should remote management of VMs that are within an ESP be handled? Since the VMs are ESP devices, does the remote management need to come through an Intermediate System?
• Can ESP VMs ever be hosted outside of a PSP?
• Do purely virtual ESPs need to be monitored in the same way as non-virtual ones?
• If VMs are replicated between primary and backup control centers, does the communication medium between them need to be within a PSP?

I’m sure there are many other questions that arise. And that’s the problem: Even though entities may be satisfied with the “guidance” that NERC has provided on virtualization, there are many questions that aren’t addressed in that guidance. And if an auditor disagrees with an entity’s own answer to one of these questions, the entity will be on the losing end of that transaction! Frankly, I think this is why NERC entities are so reluctant to implement virtualization in control centers[iv].

So entities need a comprehensive guide to implementing virtualization within an ESP, but none has been provided yet. Moreover, since CIP v7 won’t be approved for probably at least three years, anything that the entity does in the process of complying in a virtualized environment will be subject to second-guessing by an auditor, when there is no guidance on the issue one way or the other.

What could be done to fix this? Well, there certainly isn’t anything that can be done officially, except for all entities to wait for v7 to implement any virtualization within an ESP. But how about unofficially? Could say one of the regions develop their own informal guide, then let the other regions adopt it as well?

I honestly think this would be the best course. I also honestly don’t believe it will happen. However, I also know that use of virtualization within ESPs (and especially control centers) would make utilities much more efficient and arguably more secure as well. It’s a shame this will have to wait for v7.

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte Advisory.

---

[i] One huge advantage of virtualization is in backup/DR. Instead of having to maintain a secondary control center filled with exactly the same pieces of hardware as the primary, and to meticulously replicate every configuration change in the primary to the secondary, the entity can simply replicate VMs from the primary to secondary whenever there’s a change.

[ii] In fact, there was never any way that almost any of the big questions about v5 could be addressed, other than through rewriting the requirements. Unfortunately, some of those big questions (and a lot of small ones) won’t even be addressed in v7. But virtualization will be addressed, at least.

[iii] All of these questions were discussed in a TRE webinar last year. The speaker mentioned that there were almost certainly many more questions that will come up as an entity gets serious about implementing virtualization within an ESP.

[iv] Of course, the benefits of virtualization in generating stations and substations are far smaller than they are for control centers.