Going for the whole thing

Last week’s NERC CIPC meeting in Atlanta was the first I attended this year (the last was a year ago in December, at the same hotel), and it was certainly one of the best I’ve attended. The highlight of the meeting, for me and many others, was the presentation from Dave Revill, vice-chair of the CIPC and (I believe) co-chair of the CIP Modifications Standards Drafting Team.

Dave’s topic was everything that the team is working on now, but his big focus – as I expected – was the modifications the team is proposing in order to allow the standards to finally take account of virtualization. The team recently posted these modifications, which include new definitions, retired definitions, and some significant changes to various requirements. The modifications were posted for comment (not ballot), but the team intends to post modifications for ballot early next year. The comment period expires tomorrow (Dec. 18), so I realize that what I’ll say in this post is too late for anybody to use in their comments – but that’s OK, since as you’ll see my main concern isn’t tweaking what the SDT posted. I think there needs to be a very different approach.

If you want to get the full picture of what the team is proposing, you should go to the team’s web site and go to the set of documents posted under “CIP Virtualization Updates”. I will very briefly summarize what they are proposing to do (this isn’t hugely different from what they first outlined in a white paper and webinar this June. I wrote three posts on what they were proposing, starting here):

1. The team wants to move CIP from being based on individual Cyber Assets to being based on systems (specifically, BES Cyber Systems). Of course, a systems-based approach is needed if you’re going to include virtualization, since a Cyber Asset is a “device”. There is no NERC-approved definition of this term, but my unofficial definition is “Something is a device if it will hurt you if dropped on your foot”. Even if you can figure out how to drop a virtual system on your foot, it won’t hurt it. So it isn’t a device. Q.E.D.
2. They are eliminating the definitions of both Cyber Asset and BES Cyber Asset, both of which were very problematic. Instead, they’ve transferred everything important from the BCA definition to the BES Cyber System definition (most notably, the provision about 15-minute BES impact). Entities will now start their BCS identification process by looking at all the functions performed at an asset (especially the BES Reliability Operating Services, although those still have no official status in the requirements) and identifying those that can have a 15-minute impact on the BES if misused, etc. Those systems are BCS, whether they’re entirely physical, entirely virtual or a mixture of the two.
3. Of course, since the BCS definition is now changed, the question becomes what changes will need to be made in the requirements in CIP-003 through CIP-011. The good news is that a lot of the current requirements (including all requirements in the current CIP-003, -004, -008, and -009) require exactly zero changes, since they will work fine with any system, physical or virtual. The somewhat-good news is that many of the other requirements don’t require any substantial changes.
4. But the bad news is that a few of the most prescriptive requirements, like CIP-005 R1, CIP-007 R2 and CIP-010 R1, require very substantial changes due to the new systems-based approach. Trying to make all these changes to those requirements would mire the SDT in an endless fight and probably ten years of NERC ballots - as I estimated a couple years ago, when the SDT was discussing a different approach which would also require a huge number of specific changes to individual requirements. Instead, the SDT is doing the smart thing and rewriting those requirements – or incorporating them into others – in a series of steps that I don’t want to go into here, since that isn’t my purpose in this post.

I agree with the SDT’s whole approach, which I think is absolutely the right one if they are to fulfill their mandate to incorporate virtualization into CIP – although there would need to be a lot of tweaks to what they wrote, which just about everybody that commented at the CIPC meeting agreed on (just about everyone started out by saying “I applaud the SDT since it seems they’re on the right course. But I see some big problems that need to be addressed…). But there’s one fundamental problem with what the SDT is proposing (and I’ll admit I didn’t realize this until after the CIPC session): I don’t think there’s any way the NERC ballot body will approve the virtualization changes, no matter how many tweaks the SDT makes to wording.

I will write a post soon on why I make this assertion, but I’ll say for the moment that there are two basic reasons:

1. There is no pressing need to officially incorporate virtualization technologies into CIP, because lots of entities have already done that. Many (even most?) are already using virtual machines in Control Centers, and the NERC Regions are all supporting them in that effort (in fact, NERC itself has acknowledged that virtualization is allowed, as long as no VM hosts mix ESP and non-ESP virtual machines). I know one entity that virtualized their Control Center in the CIP v1 or V2 days, and passed an audit with flying colors in 2010. Of course, I’m sure everyone will agree that it would be better to make it official that virtualization is allowed. And I’ll also grant that use of switch and storage virtualization is probably being held back some by the lack of official status for those two technologies. But I am also fairly sure that these considerations alone won’t be enough for NERC entities to accept the big changes the SDT is proposing, which brings me to the second reason.
2. Why won’t NERC entities accept the big changes the SDT is proposing? Is it just because utilities are very conservative and don’t like to change something that is working, however flawed it may be? I don’t think so. In fact, in a minute I’m going to discuss an even-more-radical change that I think the industry would accept in a heartbeat. The reason why I don’t see the SDT’s proposed changes succeeding, no matter how much they’re tweaked, is because the smell of the bad experience with the CIP version 5 rollout still hangs in the air in CIP circles.

As I will discuss in a post soon, the CIP v5 rollout was quite flawed (despite the words of a NERC official who spoke at the CIPC meeting and seemed to think it was a huge success), and has left a huge set of unanswered questions about what the requirements and definitions mean – that are only papered over by a tacit mutual agreement between the Regions and the entities that I call Don’t Ask, Don’t Tell. There is every expectation that, given the current NERC auditing program (and again I’ll have to wait until a new post to explain what I mean by this), exactly the same thing will happen with the new changes as happened with CIP v5 (one questioner raised exactly this point, suggesting that the many lingering uncertainties in the “v5” requirements and definitions should be fixed before we introduce a bunch more due to the virtualization changes). Given this situation and the fact that there isn’t a pressing need to change the CIP standards to allow virtualization, I believe NERC entities will choose the status quo over what the SDT is proposing.

However, rather than being a big defeat for CIP reform, I think this is actually a big opportunity. That’s because there is one very pressing concern that almost all – if not all – NERC entities share, and that could be addressed by changes to the CIP standards and definitions along the lines of what the SDT is proposing, except going further.

Of course, what I’m talking about here is use of the cloud in CIP environments. Currently, there are two big issues: a) BCS Information (BCSI) in the cloud, and b) actual BCS in the cloud. BCSI in the cloud is currently allowed by the standards, simply because there is no prohibition against having a third party hold BCSI, whether they’re a vendor or a cloud provider (in fact, I know that a number of entities with High and/or Medium impact BCS are using cloud-based configuration management software and have passed audits. There are probably also some entities using cloud-based security monitoring services. Both services store BCSI in the cloud). But the problem is that the third party will have to comply with a set of four or five requirement parts from CIP-004, which I discussed in this post.

Let’s consider just one of these requirement parts, CIP-004-6 R5.3; this requires the entity (whether the Registered Entity or a third party holding their BCSI) to remove physical access to storage locations for BCSI within a day of an employee being terminated. This shouldn’t be a problem for a regular vendor. All of your BCSI is probably stored on servers in their single data center. It’s no problem to just remove that terminated employee’s access to that data center in a day or even a few hours – in fact, they probably do that now.

But what about a cloud vendor? By the very nature of their business, different parts of your BCSI could be flitting among servers by the minute, and jumping from one data center to another across the country any moment. And just one of their data centers might have in the thousands of employees with physical access to systems in that center. How could they prove, for every individual employee that is terminated across all of their data centers, that the person’s physical access to all of their other data centers was removed within 24 hours?

I’m not saying it would be completely impossible for a cloud vendor to prove this, or to mitigate the risk by say keeping all of your BCSI on servers in one particular data center. But doing so would break their whole business model, and they simply aren’t going to do that for one customer, or even for all utilities acting together (and the utilities can’t band together to force these concessions for antitrust reasons anyway – even if they had the collective clout for it).

Of course, there’s little question that any big cloud provider probably has security practices that go far beyond what any utility could think of implementing on their own – and they have certifications like FedRAMP to prove this. But where in the Measures section of any of the CIP requirements will you find a phrase like “…or submit evidence of a current FedRAMP certification”? Obviously, the answer to that question is “Nowhere”.  I will point out that the CEWG, a subcommittee of the CIPC whose acronym – I believe - means “Compliance and Enforcement Working Group”, is working on the issue of BCSI in the cloud – in fact they held a meeting after the CIPC itself concluded, which I attended. I’ll hope to report on that in a post in the somewhat near future, although I’ll summarize by saying this group is doing great work, but don’t look for any official change to the Measures for quite some time.

So the problem with BCSI in the cloud isn’t that it’s not allowed by the CIP standards, but that there is currently no workable way for a cloud vendor to prove compliance with the requirement parts that apply to them. On the other hand, like server virtualization, there are definitely a number of entities with High and/or Medium impact BCS that are in fact storing BCSI in the cloud, with the tacit approval (remember: Don’t Ask, Don’t Tell!) of their Regions. In other words, if the SDT were to decide they should extend the modifications they’re now proposing for virtualization so that BCSI could be realistically stored in the cloud (which could be done, and would perhaps be easier than trying to get the Measures changed, as the CEWG is currently thinking), NERC entities would be pleased to hear this. However, as with the virtualization changes by themselves, I doubt including both virtualization and BCSI-in-the-cloud would be enough to overcome the industry’s reluctance to go through a wrenching change in the CIP standards like CIP v5 was.

But what about storing actual BES Cyber Systems in the cloud? There, the situation is very different, for two reasons. For one, while I can’t guarantee that no entity with Medium or High impact BCS is storing any of them in the cloud (and of course, outsourced SCADA would be the most tempting way to do so, for many smaller entities), I am pretty sure they’re not doing it with the tacit or explicit approval of their Region – meaning they could have a big problem come audit time, if they haven’t let their Region know they’re doing this.

The other reason is that having actual BCS in the cloud would require huge changes to a number of the CIP requirements, even after all of the changes that the SDT is proposing for virtualization (and BTW, all of the virtualization changes are also necessary for BCS-in-the-cloud to be allowed by CIP in the future. The problem is those changes alone aren’t anywhere near enough). Even if an entity were to get their Region’s agreement that they could store one or more BCS in the cloud, they would have to sit down with the Region and essentially rewrite most of the requirements – either the requirements themselves or at least the Measures.

This leads me to the point I’ve been hinting at: Were the SDT to decide to extend the changes they’re proposing so that BCS could be stored in the cloud (as well as to incorporate virtualization and BCSI in the cloud), I think this would be something that could actually pass the NERC ballot body (and by the way, at last December’s CIPC meeting, both NERC and even a senior FERC staff member – as always, speaking for himself, not the Commissioners – indicated they don’t have fundamental objections to BCS in the cloud).

Yes, this would require much more thorough changes to the CIP standards and definitions than even what the SDT is now proposing for virtualization. But I can already see a lot of utility financial types, hearing that BCS-in-the-cloud might actually be happening, jumping for joy and telling their NERC compliance people to do everything they can to help NERC bring this to fruition.

So I really think that we shouldn’t mess around with changing CIP just in order to officially incorporate virtualization. As I said, it’s not really solving any big problems, but will nevertheless consume at least a couple years of SDT time. Why put off even talking about addressing BCS-in-the-cloud until the hoped-for day 2-3 years (at least) from now when virtualization is finally written into the CIP standards? Why not do both of them at once, even though this will be a much bigger effort?

I’ll tell you right now the main reason the SDT won’t consider this now – because their Standards Authorization Request (SAR) only lists virtualization as something they need to address. It will be a big deal to go back to NERC for a new SAR, then have that drafted and balloted, since SARs also need to be approved by the NERC ballot body. In fact, I think just drafting the SAR as I’m proposing might require a lot of debate and meetings. Why go through all of that, when the SDT can just get virtualization passed (and a couple other things), then ride off into the sunset with the endless thanks of the NERC community?

The answer to that is: It’s not at all guaranteed that the SDT will ever be able to get the virtualization changes passed, even if they go back now and make a bunch of changes before they even go to the first ballot. It’s very possible – and I think likely – that they will spend 1-2 years on this effort and finally decide it’s time to “declare victory and go home” (as was suggested by Sen. George Aiken as a solution to the Vietnam War quagmire).

I can see that some of the current SDT members wouldn’t feel like gearing up now for a massive change in CIP (much more far-reaching than CIP v5 was), so it’s very likely that NERC would need to empanel a new SDT to deal with this new CIP version. And it’s very likely that, even if the work to draft the new SAR started today, whatever changes finally resulted wouldn’t be in place for at least five years and even longer than that.

But guess what? I think that there are a couple even further changes that could literally make this the last CIP version that would need to be drafted. I’ve actually outlined these – at least mentally – already, since I’ve been working on a book (now suspended because I’ve been too busy, but I still hope to get back to it in the future) on the problems of CIP and how they can be addressed. You might think it’s impossible for me to say (at least without fingers crossed) that there could be a set of CIP changes that would ensure no further changes would ever be needed. But this is what I believe, although I’ll admit this would require changes in how the CIP standards are enforced, not just in the standards themselves. That is beyond the capacity of any SDT acting on their own, but isn’t beyond the capacity of NERC, working in consensus with FERC – i.e. it wouldn’t require going back to Congress, which could literally make it impossible.

However, even if we don’t go even further and implement the changes that would make this the last time CIP has to be redrafted, I definitely think it’s worth going back and drawing up a new SAR so that both virtualization and the cloud can be officially included in CIP. In fact, since I don’t see the SDT’s virtualization changes passing without this, I’d say it’s the only way to proceed. Sometimes you have to aim high if you want to accomplish anything at all.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013; we also work with security product or service vendors that need help articulating their message to the power industry. To discuss this, you can email me at the same address.