The Virtualization Conundrum, Part II

I wrote a post in May in which I pointed out that many NERC entities – and perhaps even the majority of them – are holding off implementing virtualization (server, switch and storage) within their ESPs. They are doing this in spite of encouragement from both NERC and the regions. Their reasoning is that, since the CIP standards through version 6 have been entirely silent on virtualization, there is a good chance they will fall afoul of one or multiple CIP requirements if they do this. Of course, this is a shame, since there are huge benefits to be realized through virtualization – as almost any IT manager can attest!

In the post, I pointed out that help is coming on two fronts. First, NERC is quite concerned about the fact (which they don’t dispute) that CIP is inhibiting technological innovation in several important areas, including virtualization. So it is likely that your region will bend over backwards to make sure you don’t create any egregious faux pas (like mixing ESP and non-ESP VMs in a single server). I and several others speakers will be making this point in a webinar on September 15.

Second, there is a permanent “fix” to the problem coming, in that the Standards Drafting Team currently at work on CIP version 7 (although I’m not sure I’m allowed to use that phrase!) has virtualization on its plate. In other words, v7 will finally a) recognize that virtualization is an important technology that can help NERC entities become much more efficient in their OT operations (especially in their control centers), and b) provide definitions and modified requirements that let entities know what the rules are for virtualized environments – so that the current huge level of uncertainty will finally be removed. This is the good news; the bad news is that it will be 2-3 years before the revised CIP standards are in effect.

The SDT has held a number of discussions of this virtualization question, both in weekly calls and in their monthly face-to-face meetings. I confess that I hadn’t had time to attend most of these, so I didn’t follow the discussion much until I attended the SDT’s three-day meeting in southern California in August. At that meeting, I was able to listen to (and participate in) the virtualization discussion, and found it very interesting.

I will point out that a good technical understanding of virtualization technologies is definitely above my pay grade, so frankly some of the discussion was over my head. But I understood enough to realize that my fairly simplistic idea of what needs to be done to incorporate virtualization into CIP is wrong.

I had previously thought that the biggest problem with server virtualization in CIP was that the definition of Cyber Asset includes the word “device”. In my mind a device is something that, if you drop it on your foot, it will hurt. Even if you can figure out how to drop a VM on your foot, it won’t hurt if you do so; ergo, a VM can’t be a Cyber Asset. Strictly speaking, this means that VMs are completely outside of CIP, even if implemented within an ESP. You should be able to virtualize all you want, while applying no protections at all to the VMs – and never face any consequences due to CIP non-compliance. Of course we all know that, even if an entity actually tried to do this, they would face overwhelming pressure from their NERC Regional auditors to either treat VMs as Cyber Assets (and BES Cyber Systems, EAPs, PACS or PCAs if applicable), or simply remove them from the ESP altogether.

This is why I previously thought that the fundamental step the SDT needed to take toward incorporating virtualization into CIP was to revise the Cyber Asset definition to include virtual cyber assets. Once they did that, I reasoned, a lot of the other requirements wouldn’t need tweaking at all, and the few that did would be fairly easy to modify. However, the discussion at the August SDT meeting quickly convinced me that incorporating virtualization into CIP will be a much bigger job than I had thought.

The clue for me came during a discussion of separating ESP VLANs from non-ESP VLANs in a virtualized switch environment. It was pointed out that simply setting up the VLANs so that they are separated isn’t good enough; that separation needs to be maintained over time. If these were physical networks implemented on separate switches, maintaining the separation would be much easier – since that separation could only be changed if there were some sort of hardware change in the switches (e.g. a new cable is plugged in, connecting an ESP switch with a non-ESP switch).

But since VLANs are virtual networks, not physical ones, maintaining their separation requires maintaining the switch software configurations so that an errant command doesn’t cause the two networks to become one. Therefore, there probably needs to be an ongoing software configuration management requirement for VLANs, that does not apply to physical networks. Of course, the SDT can certainly write such a requirement, but it will certainly not be an easy task.[i]

But there was yet another turn of the screw[ii] in this discussion. If software configuration management is required to maintain the integrity of an ESP, then the ESP definition itself may have to be changed, to call attention to the fact that an ESP can change or even be eliminated, due to a single errant switch configuration change. Again, this definition change is certainly doable, but it will not be a piece of cake as I’d originally anticipated.

The moral of this story is that dealing with virtualization alone – let alone the other items on the SDT’s plate – may turn out to require a good deal more time than the SDT is currently anticipating. I believe they are hoping to have the first draft of the new standards finished by the end of this year; that may be a stretch, just because of the issues that need to be addressed for virtualization.

But there is another moral to this story. It’s one that I have been harping on a lot lately, and you can anticipate I will continue to harp on it in the future. The big problem here isn’t just that the CIP standards don’t address virtualization now; there are a number of other technologies (like the cloud) that they don’t address either. Rather, the problem is that the prescriptive nature of the NERC CIP standards makes extending them to address new areas very hard.

I believe that the right approach would be to make the NERC CIP standards what I and others call “threat-based” (or perhaps “outcomes-based”. Even though at first glance it might seem these two terms are very different, I think in this case they might turn out to be effectively synonyms). Entities would be required to address certain cyber security practices like patch management and network separation[iii]; and they would have to do this across certain domains like OT systems.

Including a new technology like virtualization in threat-based standards would be easy. There would be an Applicability section, which described what the requirements applied to.  Adding virtualization to the CIP scope would only require creating a definition and stating that virtualized systems and networks were now also in scope.[iv]

For a deeper discussion of virtualization and how it interfaces with NERC CIP, you should consider attending the upcoming Deloitte/Cisco/UTC webinar on this topic.

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte Advisory.

---

[i] I would think there will be need for a similar requirement applying to virtualized servers. Since VMs can easily be migrated from one network to another, there probably needs to be a requirement to ensure that a VM isn’t migrated from an ESP into a non-ESP network, or vice versa. As with VLANs, this requirement can be written, but it won’t be particularly easy.

[ii] Of course, this is a deliberate reference to Henry James’ great ghost story (actually a novella), The Turn of the Screw.

[iii] And while the entity would be required to address each practice, there wouldn’t be prescriptive requirements saying, for example, that entities have to assess new patches for applicability within 35 days, and woe betide you if you take 36 days for one particular patch! There would be guidelines for how to structure – again as an example - an effective patch management program; the auditor would need to decide whether or not the entity’s program was effective, based on the evidence.

[iv] Hopefully, it wouldn’t even take an SDT to make this sort of change. I envision an industry body that would manage the CIP program and be responsible for decisions like this. It would be constituted by the CIP standards, but it wouldn’t need to amend those standards for changes like this one.