What about Virtualization?

In my last post, I listed six items in the Standards Authorization Request (SAR) for the current CIP Modifications drafting team that I believe will never be accomplished. I discussed my reasons for thinking the first five won’t be accomplished, but I deliberately left the last one – virtualization – for another day. Another day has come, and I’ll explain why I don’t think the drafting team will accomplish this task, either.

First off, what is this task? NERC included virtualization in this drafting team’s SAR because CIP v5 and v6, like all of the other CIP versions before them, say nothing about virtualized cyber assets. But many NERC entities with High and Medium impact assets have been using virtualization – of systems, of networks, and of storage – for years. This is especially true in control centers, where the benefits of virtualization are especially pronounced.

So what does it mean that CIP doesn’t mention virtualization? Does that mean it’s forbidden? No, not at all. It means it’s completely off the radar for CIP. For instance, if you use virtualized servers in your control centers, all you have to do is treat the single piece of hardware that they all run on as a BES Cyber System; this is because CIP currently only applies to Cyber Assets, defined as “Programmable electronic devices.” And a “device”, in my naïve way of looking at things, is something that will hurt if you drop it on your foot. Even if you can figure out how to drop a virtual machine on your foot, it’s pretty clear that it won’t hurt you.

This means that you don’t have to declare your virtual servers themselves as BES Cyber Assets/Systems, and therefore that the requirements in CIP-003 through -011 don’t apply to them. In practice, of course, I’m sure that any NERC entity with High or Medium assets is declaring all of their virtual servers within the ESP to be either BCAs or PCAs; I’m also sure every Regional Entity is keeping an eagle eye out for VMs that haven’t been declared. But if a NERC entity were to be fined for not declaring virtual servers to be BCAs and they were to challenge the fine in court (which they are always allowed to do, since all NERC standards are administrative law, arbitrated by administrative law courts), I believe it is highly likely the fine would be thrown out, simply because they aren’t violating any current CIP requirements.[i]

But even leaving the question of virtual servers aside, there are lots of thorny questions that come up when you look at virtual networks and virtual storage – and these can’t be fixed with just a tweak or reinterpretation of the word “devices” in the Cyber Asset definition. The CIP Modifications drafting team didn’t try to sweep any of these questions under the rug; on the contrary, they actively sought them out.

Early in the team’s existence (they started operation in the spring of 2016), they formed a committee to work on virtualization. That committee quickly realized that there are some very fundamental concepts in CIP – besides the concept of Cyber Asset – that need to be changed to allow virtualization to be encompassed. For example, the idea of an Electronic Security Perimeter has limited use in a virtualized environment, since it focuses entirely on devices – the physical device is either in or out of the ESP, along with all of the virtual devices that may be housed in it. But since switches, storage and servers can all be “divided” in some way into different virtual components, and since these components won’t always be all part of the ESP, there needs to be another way of talking about a security perimeter when these are present; so the committee came up with the concept of an “Electronic Security Zone”.

I have good news and bad news. The good news is that this committee has really done some pioneering work, and they have developed a new way of thinking about cyber security in virtualized environments (not just on OT networks, either); to see some of their work, you can go to the drafting team’s web site and look at the slides and listen to the recordings from the webinars they’ve done. The team could document what they’ve done in a really nifty white paper (or even a book), which should have applicability far beyond the CIP world.

But here’s the bad news: Implementing the new concepts the committee is talking about will require a huge number of new or changed CIP requirements and definitions. Each of these will need to be drafted, posted for 2-4 ballots, commented on, revised multiple times, and finally approved. In this post from early January, I did a little math. I pointed out that, in the case of the changes to the LERC definition and the underlying requirement, this whole process had taken more than six months of this drafting team’s almost exclusive time in the second half of 2016 and the first 2-3 months of 2017.

I then combined this with an estimate that one of the virtualization committee members had given me, that there were probably 15-20 (or even more) definitions or requirements that would need to be changed or developed from scratch. Assuming this is true, and that each of these will take up 6 months of the SDT’s time to get approved by the ballot body, I came up with the (perhaps conservative) estimate that just virtualization will take ten years for the drafting team to accomplish! Needless to say, this is never going to happen.

I know that, at one point at least, there were some on the drafting team who were advocating they just draft all of the changes required for virtualization and submit these as one huge ballot (or maybe 2 or 3 big ballots), that NERC entities could either approve or disapprove in toto. This strikes me as a wonderful prescription for spending an incredible amount of time (just this effort would take well over a year, maybe two years), and likely coming up with exactly zero to show for your efforts, since – unlike the case with LERC – there is no mandate from FERC that this be done. And that seems to be what it takes to get new CIP requirements, or substantial changes to existing ones, approved nowadays, as was also demonstrated by CIP-013.

At one point, I heard a member of the drafting team compare the work they’re doing to the CIP v5 standards. Those standards were of course submitted as a whole to the NERC ballot body, and – while their path to passage certainly wasn’t easy, taking close to two years and at least four ballots – they were ultimately approved and are now in effect. And it’s certainly true that CIP v5 required many more individual changes than virtualization will. So why shouldn’t we expect the same thing to happen here - that it might be a rough year or two, but ultimately the changes required for virtualization will pass?

I’ll tell you why the same thing won’t happen with virtualization as happened with CIP v5. First off, remember that CIP v5 was very much ordered by FERC, in this case Order 706 from January 2008, although it took five years for NERC to “respond” to the order[ii] (there were of course a lot of diversions along the way, including CIP versions 2-4. Of these, v3 was due to a new FERC directive, while v2 and v4 were deliberate choices on NERC’s part). So the NERC ballot body was very much aware that FERC wanted them to approve v5. And FERC had added another incentive when they unexpectedly approved[iii] CIP v4 in April 2012. In doing that, they in effect said “You’d better approve v5 soon. If you don’t, you’ll end up having to comply with v4 and then with v5. Does that sound like much fun?”

The other reason why the same thing won’t happen has to do with the CIP v5 experience itself. While there was a lot of optimism around v5 initially (including on my part), as NERC entities started their v5 implementation work in early 2014 all sorts of questions about what the requirements and definitions really meant started popping up. NERC started trying various expedients to address these questions without doing the one thing that is allowed by the Rules of Procedure: develop a drafting team and task them with fixing many if not most of the worst problems; most of the “fixes” NERC tried ended up being withdrawn, resulting in the fact that today, August 31, 2017, NERC entities are no closer to having answers to their CIP v5 interpretation questions than they were the day FERC approved v5 in 2013 (this sorry process is discussed in a little bit of detail in this post. If you want to read the whole story, I suggest you read every post I wrote between this one and say January 1, 2016. Then maybe you can summarize them for me, since I have no intention of reading them all any time soon. A human being can only take so much punishment).

I’m not a psychologist, but I think the NERC community has been in a kind of shell-shock ever since the CIP v5 experience. This has manifested itself in two ways. The first is a suspicion of anything that a NERC drafting team puts out for a ballot, no matter how good it is. I saw this most vividly in the debate over the “LERC” changes last year, where there was a firestorm of opposition to the first ballot, and the second ballot probably only passed because there was a deadline of early February for NERC to get this change to FERC. This in spite of my opinions that a) the second ballot, which passed, wasn’t substantially different from the first one, which didn’t; and b) both ballots (actually, drafts) should have been no brainers to approve – they allowed the entity to do everything allowed in the CIP v6 version of this requirement, plus much more.

The second manifestation of shell-shock was vividly brought home in a regional CIP workshop I attended last year. At that meeting, a member of the CIP Modifications drafting team gave a presentation on their progress, including some of the changes that virtualization would require. Then a member of the CIP v5 drafting team (in fact, the team that drafted CIP versions 2-5) made an eloquent appeal that the team not subject the NERC membership to debate and vote on any more far-reaching requirements or definitions (and I believe he had the LERC experience in mind).

Think about this: Here was a very well-respected member of the CIP v5 drafting team essentially saying there shouldn’t be any more improvement or expansion of NERC CIP! And simply because the NERC membership was “too tired” (I think he used some words to that effect) to be able to debate these issues anymore.

Of course, what this person was talking about wasn’t just virtualization, but the whole agenda (SAR) for that team. If the team actually came out with a new requirement and/or definition that addressed any one of items 1-5 in the list of SAR directives in my last post (e.g. clarification of what “Programmable” means in the Cyber Asset definition), it’s almost inevitable there would be another big, divisive debate, just as there would be if the team starts to try to codify far-reaching changes like the Electronic Security Zone. So this is another reason why I think the drafting team will need to get their SAR reduced to the items they’ve addressed already and the two items they’re now actively working on: CIP-012 and the TOCC issue. If they try to slog through with the rest of their agenda, they should all literally ask for a reduction in their day job responsibilities for the next 15-20 years; I’m sure that won’t be any problem at all.

And given that, assuming the SDT does get their SAR reduced, the chance there will ever be a future SAR aimed at addressing the fundamental problems in CIP v5/v6 is exactly zero, this is another reason (besides the one discussed in my previous post) why I see no likelihood there will ever be any further changes or additions to the current CIP standards, except where explicitly allowed by FERC.

There is also a corollary to this statement, which I will bring up in one of my next posts.

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte.

---

[i] I have heard a few people around NERC make the argument that, because the definition of Cyber Asset reads “Programmable electronic devices, and the hardware, software and data in those devices” (my emphasis), this means that virtual servers all do have to be treated just the same as physical servers. This is a pretty tenuous argument in my opinion, since this wording clearly was intended to cover the normal situation where application software is stored on a device and utilized in that device’s normal operation, not where virtual devices themselves – each with their own application software – are housed in a physical server.

[ii] And the fact that it did take five years was very much on FERC’s mind from then on. Since then, every significant new or changed standard that they have ordered has come with a deadline – this applies to CIP-014, LERC and CIP-013.

[iii] I didn’t start this blog until January 2013, but I did write about that event, and what it meant, on a Honeywell blog that no longer exists. I can send anyone who’s interested the post I wrote discussing this idea, which was later obliquely confirmed to me by a source who knows something about what goes on internally at FERC. The fact that FERC took this maneuver turned out to have some serious unfortunate consequences, as discussed in a series of posts starting with this one.

The Horror, Part 2: The End is Near

It isn’t my intention from now on to only use titles for my posts that sound like they come from bad movies. But this post is a follow up to a post that described what I believe is a genuine horror: the fact that there isn’t now and never will be any guidance published on NERC CIP that is in any meaningful way binding on the auditors. NERC entities are truly on their own when it comes to deciding how to interpret CIP v5/v6, given the ambiguities, missing definitions and inconsistencies that are found in those standards as written. I’m sure most NERC entities understand this already, even though they may not have articulated it so far (however, I’m not at all sure most people at NERC understand this).

But I regret to say that the horror I described in the previous post is only the lesser of two horrors, and the second horror is a lot scarier than the first. In other words, this post is one sequel that may actually be scarier than the original. Parental discretion advised.

This summer, I was in Montreal for the meeting of the CIP Modifications Standards Drafting Team (the team that’s working on all CIP changes except the new Supply Chain Security Management standard, CIP-013. That one has its own SDT). This was the first meeting of this team that I’d been able to attend in six months, and I’m very glad I did attend.

I wanted to come to this meeting primarily to get updated on what this team is doing. But I also came because I wanted to see if they felt the same sense of alarm about their mission as I do. As I pointed out in this post in January, I have come to doubt that there is any way for this team ever to accomplish everything that is in their Standards Authorization Request (SAR).

So I have good news and bad news. The good news is that I no longer have any doubts about whether this team can accomplish everything in their SAR. The bad news is I’m now certain they can’t ever do this. This is due to no fault of their own; it just turns out their marching orders came from Mission: Impossible.

So – since I’m just a helpful guy by nature - I came to Montreal prepared to bring up to the team the fact that they really need to think now about what they can and can’t accomplish; after all, I don’t think any of them joined the drafting team with the expectation that they’d serve on it until retirement! One conversation I’d had with a team member at the NERC CIPC meeting two weeks previous had led me to believe it would be difficult for the team members to even have this conversation.

However, I was glad to see that one of the first items discussed at the meeting was the question whether they need to revisit their SAR now. Of course, doing this will be no small deal. A NERC drafting team doesn’t get to decide for themselves which parts of their SAR they’ll be able to fulfill; if they want to change it, they need to somehow get NERC to modify it, which may require ballots (and I don’t know whether this has ever happened in NERC’s history, and certainly not with any of the CIP drafting teams).

So what is in this team’s SAR? I described the SAR in this post in early 2016. I’ll now list each of the items in the SAR and discuss its status (although I’ll group them differently than I did in the previous post):

1. The team has finished with the FERC directive to clarify the meaning of LERC. This was approved by the NERC ballot body – although it proved much more contentious among the NERC membership than I ever imagined it would – and sent to FERC in early March (FERC, of course, hasn’t had a quorum since then, but now they do again. Given that there are a number of urgent projects like pipelines that need to be approved, my guess is they won’t get to this for six months or more).
2. Similarly, the team has finished with the FERC directive to develop requirements for Transient Electronic Devices for Low impact BES Cyber Systems. This was also sent to FERC in March, thus saving NERC the cost of another postage stamp.
3. The FERC directive to protect “communications network components” between Control Centers is being addressed in the new CIP-012. This has been drafted and is now out for a first ballot.

These were the three items that were ordered by FERC, meaning neither NERC nor the drafting team has any choice but to fulfill them. Even though the last of these isn’t yet put to bed, it’s well on its way, and there’s no question that the SDT will fulfill it. But what about the items in the SAR that weren’t ordered by FERC?

First, the Transmission Operator Control Centers (TOCC) issue – which I admit I understand only on a fairly superficial level – has been the subject of a lot of debate and comment (I referred to it as the “TO/TOP Issue” in my 2016 post). I’m sure it will be resolved, since it is a very contentious issue for a large number of (relatively small) NERC entities. This one doesn’t pose any philosophical issues: Everyone seems to agree on what needs to happen, but they haven’t yet agreed on the wording that will make it happen. I’m sure they will agree in the near future, though.

What about the other items in the SDT’s SAR? As discussed in depth in the previous post, these include

a)       Clarifying the definition of Cyber Asset (especially the meaning of “Programmable”);

b)      Preventing the BES Cyber Asset definition from “subsuming” all other asset types (and here we have another potential horror movie title: “The Definition that ate all the Assets!”);

c)       Determining how to set a “lower bound” for “impact on the BES” in the BCA definition;

d)      The question of “double impact” and preventing this from leading to the N-1 criterion raising its ugly head to exempt lots of assets from the scope of CIP v5 (as discussed in the previous post, I thought the way this question was phrased was wrong, but I was also surprised that the N-1 criterion was still the bugaboo it had been in the CIP v1 days. I thought it had been laid to rest with a stake through its heart then);

e)       The “Network and externally accessible devices” question; and

f)        Virtualization (i.e. trying to fix the problem that CIP is now completely silent on virtualization, meaning that – strictly technically speaking – any virtualized cyber assets are completely out of scope).

These might seem like very different issues, but items a) through e) all have to do with asset identification – that is, CIP-002 R1. And in the Montreal meeting, one of the SDT members pointed out one very important reason - which I hadn’t thought of - why it would be better to leave these items alone (in other words, to ask NERC to remove them from the SAR): If one or more of these were changed, NERC entities with Medium or High impact assets would probably have to re-run their entire asset identification process. Whether or not this resulted in a lot of new cyber assets coming into scope for CIP v5/v6 or maybe a lot of existing ones being removed from scope, just having to go through this exercise again would require a huge investment of staff time and money, especially on the part of large entities that have hundreds or thousands of BES Cyber Assets.

Of course (and here I’m going beyond what was discussed in Montreal; the discussion of this issue ended after the above point was raised, when one of the team leaders noted that at the moment there was still one item on the SDT’s plate that had to be dealt with due to a FERC order, so the question of their SAR didn’t need to be settled right away), the fact that a huge investment would be required doesn’t itself mean the effort shouldn’t happen; after all, we’re talking about protecting the US electric grid! But what would this effort accomplish? Even with all the ambiguities and missing definitions in CIP-002 R1, Attachment 1, and the Cyber Asset and BCA definitions – and there are a lot more that never made it into the SAR - there has been a remarkable consensus among entities and the NERC Regions as to how cyber assets should be evaluated to identify BCAs.

As proof of this consensus, I haven’t heard of a single NERC entity that has been told they were way off the mark on how they identified BCAs, and they’re now in deep doo-doo because they have a bunch of – say – relays that now need to be brought into compliance with all of the requirements in CIP-003 through -011. In other words, if there were even one sizable entity that had woefully under-identified its BCAs and had therefore put the security of the BES at risk, my guess is I would have heard something about that by now. In still other words, if there’s no problem to fix, why require entities to go through this huge effort?

Those of you (both of you, I think) that read this blog in 2014 and 2015 may perhaps remember that I was very worried about the fact that there was so much ambiguity, contradiction, and missing definitions in the CIP v5 asset identification process. In fact, I’m sure I wrote at least 50 posts on various aspects of this problem. As I recounted in my previous Horror post, I had at first hoped the CIP v6 drafting team would be charged with fixing these problems, then I hoped that some sort of guidance could be put out to do this.

When it became clear in 2016 that neither of these would happen, I went through all the stages of grief, and I finally came to realize there would be no true fix for these problems. NERC entities would simply have to in effect write their own CIP v5 standards, following whatever unofficial (and almost always unwritten) guidance their regions will give them. And I began to realize that NERC could ask Mahatma Gandhi, Mother Theresa, Alexander Hamilton, James Madison, Solon, and the Buddha to sit down at a table and hammer out a final solution to these problems, and these august personages still wouldn’t be able to do so, any more than the members of the original CIP v5 SDT (who I know were aware there were a number of serious issues in v5, but were under enormous pressure to finally deliver FERC a response to the directives in Order 706, issued in 2008) were able to do so. Indeed, I came to realize that only a wholesale rewriting of the CIP standards (and, as I’ve also come to realize more recently, NERC’s CMEP and maybe even the Rules of Procedure) will “solve” these problems for good.

So I totally agree with the sentiment among at least some of the CIP Modifications SDT that all of the “asset identification” issues in their SAR should be removed from it (and I also believe item f above -virtualization - should be removed, although for a very different reason. I will discuss that in a separate post soon). What does this mean? It means I believe we’ve come to the end of the line for further developments in NERC CIP, unless FERC orders a new standard.

Specifically, there will be no further efforts to rewrite CIP v5 or v6 requirements and definitions to clarify the many wording problems, including the efforts in the current SDT’s SAR. How could it be otherwise? If the current SDT succeeds in getting these items removed from their SAR, who will ever even suggest in the future that a new SDT be constituted to try to fix problems with CIP v5? We are truly at the end of the line for any further refinement of the current CIP standards.

The lesson of the first Horror post was that there is not now and will never be any guidance on NERC CIP – published by NERC, the Regions, or any other organization or individual (which includes, I hate to say, my blog posts) – that can be considered in any way binding on CIP auditors. Entities need to decide for themselves how to address each interpretation issue, and document why they think that way.

And the lesson of this second Horror post is that the only possible means of resolving these issues – writing a new SAR and revising the existing CIP requirements and definitions (or developing new requirements or definitions to fix issues like the meaning of Programmable) is now closed. Thus, the CIP v5 and v6 standards will always be ambiguous and inconsistent, until they are replaced with new standards that take a very different approach.

From now on, I plan to mostly write about what kind of standards (or maybe just one standard) could replace the current CIP standards and eliminate these ambiguities (more specifically, the new standards would make ambiguities, like the ones in CIP-002 R1 and Attachment 1, irrelevant). And I will also write a lot about CIP-013, which – despite its many flaws – moves a long way toward what I think all of the CIP standards should be (and this creates a lot of interesting problems, since of course CIP-013 will be enforced using the same prescriptive CMEP as all the other NERC standards).

However, I don’t intend to ignore CIP v5 and v6 from now on; after all, they will remain the law of the land for at least 2-3 more years. But at every possible occasion I’ll let my inner snark appear and point out that whatever problem I’m discussing at the moment can only be finally resolved when the CIP standards are rewritten. During the turbulent 60’s, I and my radical friends had a running joke that every problem – no matter how pedestrian – would be resolved “come the Revolution”. But now I’m not joking: I hope I’ve demonstrated to your satisfaction in these two Horror posts that there will be no resolution to the problems of CIP v5 and v6 until the standards are replaced with ones that take a very different approach.

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte.

Another great news article, this time on EMP

It's almost embarrassing that, two days after I put up a post about an article in Energy and Environment News, I'm putting up a new one (I swear, they're not paying me!). This is an article that just came out today, and it's about EMP. It's really good. Like most E&E News articles, it's written without a concern to fit in a particular space - unlike another good (but too short) article on this topic from the Wall Street Journal. Of course, people who write long articles are near and dear to my heart!

One small quibble I have with today's article has to do with the last paragraph, where the writer, John Fialka, mentions that a number of other countries have started to take steps to harden against EMP. He adds "Whether the United States will join them remains a work in progress." NERC has just passed a standard to do just that, TPL-007-2, although it still requires FERC approval. Of course, opinions differ as to whether the standard does enough, but understanding those arguments is far above my pay grade.

Another good thing about this article is that it's part 1 of 2. I'll hopefully be able to give you a link to the second one after it's published next week.

The best press article on NERC CIP ever written

After yesterday's post, someone reminded me about a press article that appeared in Energy and Environment News in January 2016, which I had quoted in full in this post. It is very relevant to what I was writing about yesterday, and goes beyond that.

It is without a doubt the best news article I've ever seen on NERC CIP. Almost every other article I've read makes me want to cringe, although I'll admit CIP is a very difficult subject to get your arms around. You need to really be willing to sit down and devote a lot of time to learning about CIP and its whole context. Pete Behr is one of the few reporters that has done that. In fact, I'd say he's the only one who has done that.

The Horror! The Horror!

There has been a lot of talk in NERC circles lately about guidance for the CIP standards. This is largely driven by NERC’s recent efforts to “clarify” the status of the many types of guidance they have put out about CIP v5 and v6, and now CIP-013. I would like to give your my own account of those guidance efforts.

In the long-ago days of CIP v3, concern began to grow about inconsistency between auditors and between regions in the interpretation of the CIP requirements. This led to NERC’s creating two series of documents: Compliance Application Notices (CANs) and Compliance Application Reports (CARs).  NERC thought – along with most of the industry, to be sure – that simply having NERC state its opinion on certain controversial topics would lead the regional auditors to put aside their differences and all start singing from the same page in the hymnal.

Unfortunately, things didn’t work out too well for the CANs and CARs. They were attacked roundly from many sides, and most importantly the auditors saw no reason to feel bound by what these documents said. After all, where was the basis for them in the NERC Rules of Procedure? The answer is “Nowhere”. This led to most, although not all, of the CANs and CARs being withdrawn (a few of the less controversial ones remain technically in force).

This was considered to be a good learning experience for NERC. People said, “Well, at least NERC will make sure that the next CIP version (which was expected to be numbered v4 at the time) doesn’t have these ambiguities, so there will be no need for these extraordinary measures in the future.” However, I would say that many people in the NERC community today would gladly exchange the huge level of uncertainty in CIP v5 and v6 with the much more modest level of uncertainty in CIP v3 (and coming soon to a NERC Regional Entity near you: CIP-013!). Yes, those were the days…

When FERC issued their Notice of Proposed Rulemaking (NOPR) in April 2013, which said they intended to approve CIP v5 (and would send CIP v4, which had been approved for implementation in April 2014, to sleep with the fishes), I decided to write a series of posts on v5.

What I found was disturbing. I started out with CIP-002, since that is the first standard. I tried to figure out exactly what CIP-002 R1 (with Attachment 1) required the entity to do. And I literally came to a dead end: The logic broke down so completely that there was no way to go forward without taking a big leap of faith. I went on to write probably 100-150 more posts on problems with CIP v5 over the next 2-3 years and cataloged a wide range of problems, especially having to do with CIP-002 and its associated definitions.

At this point I started wondering how these problems could be fixed. My first hope was for FERC – when they actually approved v5 – to simultaneously order NERC to fix the problems, or at least the really fundamental ones in CIP-002-5.1 R1 (since that is the foundation of the rest of the current CIP standards).

However, when FERC approved CIP v5 in Order 791 in November 2013, they broke my heart by not telling NERC to address any of these problems. And the next month, at a NERC CIPC meeting in Atlanta, I asked a highly-placed NERC staff member whether NERC would of its own accord include this problem in the Standards Authorization Request (SAR) that would guide the drafting team for CIP v6[i]. His answer was as concise as possible: “No”.

This was a very disappointing answer, since I believed it meant there was now no way to truly fix the problems with CIP v5. I believed this (and still do!) because the NERC Rules of Procedure allow no other mechanism to address problems with a standard than to write a SAR and convene a Standards Drafting Team to revise the standards. Yes, this is a very time consuming process – especially given the magnitude of the problems in CIP v5 – but it is the only way to fix problems, rather than simply attempt to paper them over.

However, life goes on. The fact that there were a lot of problems with CIP v5 didn’t mean that NERC entities didn’t have to comply on April 1, 2016 (the original compliance date) – they still had to do that. My attention then turned to the next question: What would NERC do to at least mitigate these interpretation problems? I first asked this question in this post, and you could say that each of the next 100 posts asked the same question.

I won’t reiterate for you all the many twists and turns of NERC’s admittedly well-intended efforts to provide guidance on complying with CIP v5. At first the Guidelines and Technical Basis were going to do the trick, then the RSAWs, then the CIP v5 Implementation Study, then the FAQs, then the Lessons Learned, and finally the Memoranda (I’m probably missing three or four things in this list and I know they overlapped, so the order isn’t at all hard and fast).

Each of these different efforts was touted by NERC at one point as being the final answer to the ambiguities of CIP v5, yet each of them was ultimately abandoned. What finally brought this process to an end was the Memoranda, which caused huge contention and were withdrawn in spectacular fashion at a meeting on July 1, 2015.

At that point, NERC seemed to me to have raised the white flag and admitted that there was no definitive way – other than by writing a SAR and convening a new SDT – to address problems with standards; they said they would do exactly this (and that team is still working today). They also seemed to be pointing toward a more ecumenical guidance process where other groups could also provide guidance and NERC would publish those documents that it believed had merit. And here’s the kicker: It seemed they were finally admitting that all credible guidance, from whatever source, should be given consideration by both entities and auditors.

But there was another implication to what NERC said: that in the case of ambiguity, it is ultimately up to the entity to decide what the CIP v5 requirements and definitions mean. Because if a) the standards are ambiguous (which NERC admitted) and b) NERC can’t provide definitive guidance (by which I mean guidance that the auditors are bound to follow in their audits), then there really is no 100% right or wrong way to comply with a CIP requirement.

And here’s where “Roll your own” comes in. In September 2014, I wrote the first in what turned out to be a series of posts on how NERC entities were dealing with ambiguity in CIP v5. That post described how one entity had decided they couldn’t wait for NERC to come out with definitive guidance on v5 – specifically, on what “programmable” means in the Cyber Asset definition – and had simply developed their own guidance. Just as importantly, they had documented what they had done. The person I talked with argued that, if an auditor three years from now disagrees with the definition they came up with, they will simply show him or her the documentation of how they arrived at this definition, including the fact that they reviewed all available guidance before doing this. There is simply no way the auditor can assess a potential violation (or at least make it hold up after they have assessed it), given that the requirement is ambiguous.

This was a turning point for me, because in the almost three years since I wrote that post it has now become completely clear to me, as well as almost all of the rest of the NERC community (including entities and auditors), that this is the only way to comply with CIP v5 and v6: You simply have to get out your plywood and nails and patch over whatever logical chasms you come across, so that you can cross them and get on with compliance. But the key is documenting what you did; I hope you all did that (at least if you have High or Medium impact assets), but even if you didn’t, it’s not too late to do so.

Since July 2015, NERC has more or less adhered to what they said that month. They have convened an SDT to address at least some of the problems with CIP v5[ii], and they have moved to a guidance framework that allows a number of organizations to develop guidance and have it “approved” by NERC. However, there is one way in which NERC seems to be relapsing into its old mindset: It once again seems to believe that it can develop guidance (or approve particular guidance developed by others) that is better than anybody else’s guidance, and therefore will be given some sort of “priority” by the auditors when they audit. I believe the current idea is that “implementation guidance” written by the SDT that developed a standard should and will be given extra attention, both by entities and auditors.

But don’t believe it. Let me repeat, in case you weren’t paying attention earlier:

1. No CIP guidance of any kind, whether written by a NERC SDT, the NERC Board of Trustees, Thomas Jefferson, Baha'u'llah, Saint Paul, or the Dalai Lama, has any greater validity than any other guidance. In particular, the auditors aren’t bound to follow any particular guidance.
2. However, you should consider all available guidance as you do the only thing you can do when faced with an ambiguous requirement or missing definition: decide for yourself the best approach, and document how you came to that conclusion (for an alternative and more far-reaching approach than “Roll your own”, see this 2014 post about an article by Lew Folkerth of RF).

Of course, now we have CIP-013 coming up, and that presents a whole different set of guidance issues…

The Horror!

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte.

---

[i] The v6 SAR only included the four things FERC had mandated in Order 791. None of them were fixes to the numerous wording problems I and others had found in v5 thus far.

[ii] Although, as I will say in an upcoming post, I don’t believe that SDT will ever address everything that is on its plate. And I also don’t think, absent new FERC orders, there will be any further changes or updates in NERC CIP – unless the standards are completely rewritten from scratch.

A Good Idea

My most recent post lamented that every NERC entity subject to CIP-013 (i.e. those with High or Medium impact assets) has to, every 15 months, identify new supply chain security threats and mitigation measures, and incorporate the relevant ones into their Supply Chain Security Management Plan; this per CIP-013 R3. I pointed out (very astutely, I might add) that it didn’t make sense for each entity to have to review the same information and draw the same conclusions. Why couldn’t there just be one body that did this for all NERC entities (although the entities would be free to add to the list of new threats and mitigations provided by this body)?

My answer to that question asserted that there is no provision in the NERC Compliance Monitoring and Enforcement Program (CMEP) for this; therefore, NERC entities are doomed to comply with R3 completely on their own. I then went on to point out that this is a general problem for CIP: there is simply no way to incorporate new threats into CIP and require entities to comply with them, other than writing a Standards Authorization Request (SAR), convening a Standards Drafting Team, going through 3 or 4 NERC ballots, submitting the new or revised standard to FERC, waiting for them to approve it, etc. At the most optimistic, that’s a 3-year process, but I will soon write a post that asserts that the window has closed for any future modifications to CIP, except modifications ordered by FERC (there are various reasons for this, but it is primarily because the industry has been exhausted by all the interpretation issues with CIP v5 – which will never be resolved – and isn’t exactly looking forward to having a bunch of new ambiguous standards dumped on their table).

However, an auditor emailed me to say that he thought there were at least 4 existing organizations that could fulfill this role. When I replied that the problem wasn’t that no organization could do this but that there was no provision in CMEP allowing them to do so, he pointed out to me that “The Standard guidance suggests that entities need to review actionable information to identify needed changes to their plans.  No one said where that actionable information has to come from.”

And this makes sense to me. For the purposes of CIP-013 R3 compliance, I believe it would be fine if some third party organization, like one of the trade associations but not limited to them, committed to doing this for all NERC entities. That is, they would continually look for new supply chain security threats and mitigation measures and publish these for the whole NERC community (and if an organization just wanted to do this for their members, then hopefully other organizations would do the same for their members). Any takers for this? I won’t name names, but I can think of at least a couple organizations who would be ideal for this.

However, my larger point in the post was that a procedure like this is really needed for all cyber security threats to BES Cyber Systems, not just supply chain threats in CIP-013. So it would be nice if there were a body that would regularly (or even continually) review all new cyber threats as well as all new mitigations to cyber threats, identify those that are relevant to the electric power industry, and publish these for the industry (perhaps on a need-to-know basis). If CIP were rewritten along the lines of the six principles in the post I referenced above, then NERC entities (probably above a certain size threshold) would have to get an assessment based on the threats on the current list, then mitigate those threats.

But that can’t happen now, given the current CIP standards and CMEP. When a new threat arises now, like phishing or ransomware, the only “legal” way to address it, according to the NERC Rules of Procedure, is to go through the SAR process I described above. Obviously, in the case of phishing and ransomware (both of which have been threats for years), nobody has even suggested a SAR, and as I already said, I don’t think there will be any more changes to CIP that aren’t FERC-ordered. So phishing and ransomware will never be addressed within the current CIP framework (despite the fact that the Ukraine attacks all started through phishing). This also applies to as cloud threats, and many more current and future ones. I believe that none of these will ever be addressed, given the current CIP-002 through -011 wording, which doesn’t have any provision for addressing new threats, as in CIP-013 R3.

But in my ideal world, which I described in this post (the last few paragraphs, although to understand them well you need to read the whole thing), CIP would be totally rewritten and CMEP would be revised[i], allowing threats and mitigations to be continually updated without having to revise the standard itself. If you look at number 3 of the six principles I list in that post, you’ll see it calls for the entity to continually update its list of threats; all the threats on the list have to be addressed in some way, although on a risk-adjusted basis (so threats that pose less risk would require less mitigation work and might in some circumstances be completely ignored if they really don’t apply to the particular entity). This list could – and should – be maintained by a central industry body, although the entity would be free to add other threats to the list if they felt this was warranted.

So this is where I see an industry body – which could well be one of the trade associations, etc. – being able to finally solve the problem caused by the fact that CIP doesn’t have a good way to respond to new threats (other than CIP-013, of course). But this can’t happen until CIP is completely rewritten and CMEP is revised. As I said in the last post, I don’t think NERC is likely to do either of these any time soon, which is why I’m pessimistic that FERC and NERC will be allowed to keep responsibility for cyber regulation of the power grid much longer. But I’ve been wrong before!

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte.

---

[i] Although maybe there would need to be two NERC CMEPs. The existing one would be for the 693 standards, and the new one would be for CIP. If you look at the list of six principles for the “new CIP” listed in the post just referenced and you try to figure out how the new CIP standard would be audited, I think you’ll agree that it would be almost impossible to do that with the existing CMEP. In fact, I think auditing CIP-013 and CIP-012 will be very challenging with the existing CMEP, since these are much more like what I would like to see in the new CIP. But even the existing CIP-002 through CIP-011 don’t fit in very well with CMEP, which is one of the reasons there are so many continuing problems.

CIP-013 R3

I intend to start writing a lot about CIP-013 in the coming few months. There are two reasons for this:

First, the standard has recently made great strides toward coming into effect. With the third and final ballot passing, the chances are just about 100% that the NERC Board of Trustees will approve CIP-013 (and CIP-005-6 and CIP-010-3) this month. This will then go to FERC. A day ago, I would have pointed out sarcastically that FERC doesn’t have a quorum and may not for a while, so it’s questionable whether the standard will be returned to NERC with the notice “Nobody at this address”.

However, just a couple hours ago the US Senate confirmed two new Commissioners, meaning that soon FERC will have a quorum. Given the huge backlog that FERC has (including CIP-003-7, which includes the revised “LERC” requirement), I’m betting it may take up to a year for them to approve CIP-013. And there’s still a small chance they will simply remand it to NERC (or even kill Order 829 altogether). But my guess is they will approve it.

Second, CIP-013 is a very interesting standard. Complying with it is completely different from complying with any of the previous CIP standards; in fact, it’s completely different from any previous NERC standard (although CIP-014 comes closest to it). The good news is that it is (almost) entirely an objectives-based standard, which is what I (and many others) have been saying for some time is how all of the CIP standards should be written (there are some objectives-based requirements in the existing CIP standards, like CIP-007 R3 and CIP-010 R4. But none of the standards themselves are objectives-based. This fact is actually quite significant, as I’ll explain in later posts). This means that compliance with the standard is much more like what your organization would do if you were mandated by your organization to address cyber risks in your supply chain, and given a healthy pot of resources to do it with: You would rank the threats by the degree of risk they pose, and allocate your funds for remediation on that basis.

That was the good news. The bad news is that auditing CIP-013 is also very different, and the auditors are still going to have to follow all the provisions in NERC’s Compliance Monitoring Enforcement Program (CMEP) and Rules of Procedure; these two documents are completely focused on prescriptive, non-risk based standards.[i] So it’s very much an open question whether the auditors will be able to completely change their style of auditing, while still paying obeisance to those two documents.

In any case, I will be writing a lot about CIP-013, and here’s the first in this series. What I want to discuss now is – as those of you who are super-alert may have noticed from the title – CIP-013 Requirement 3. If all you do is read the requirement as it stands in the final draft of CIP-013, you might be excused for thinking it’s fairly innocuous. It reads “Each Responsible Entity shall review and obtain CIP Senior Manager or delegate approval of its supply chain cyber security risk management plan(s) specified in Requirement R1 at least once every 15 calendar months.”

You might think, “Hey, this one’s a piece of cake. All I have to do is stand outside some suit’s door for five minutes once every 15 months to get them to sign a piece of paper. I wish everything else in CIP were this easy.” But you’re wrong about that. This requirement just reveals the tip of the iceberg. To understand why, I’ll quote the first draft of CIP-013 (you remember that? The one that got about nine percent approval?), where this is requirement 2, not 3:

“R2. Each Responsible Entity shall review and update, as necessary, its supply chain cyber security risk management plan(s) specified in Requirement R1 at least once every 15 calendar months, which shall include:

2.1. Evaluation of revisions, if any, to address applicable new supply chain security risks and mitigation measures; and

2.2. Obtaining CIP Senior Manager or delegate approval.”

As you can see, the only real difference between the two drafts is part 2.1 in the first draft. Let’s unpack what it says:

1. There may be new supply chain security risks and mitigation measures that have appeared since your plan was last approved.
2. You need to consider these – and my guess is auditors will want some sort of assurance that you at least considered all new risks and mitigation measures, in some way or another. They won’t let you get away with saying you considered just every other threat, or just all those threats that begin with letters A through G. There isn’t really a good way to limit what you have to consider.
3. If you find any new threats or mitigation measures (and guess what? In the world of cyber security, there are new threats all the time. Fortunately, there are new mitigation measures all the time as well), you would be well advised to modify your plan to address them.
4. And since you’re not treating all the vendors and systems the same (remember, this is a risk based standard?), you really need to look – at least in principle - at how these new threats and mitigation measures apply to each BCS you purchase - or each piece of software that goes on a BCS – as well as to each vendor you purchase these from.

Does this strike any of you as a lot of work? Some people who voted no on the first ballot commented that this is really an open-ended commitment. How many news articles, blog posts, vendor notices, threat intel feeds, Tweets, etc. do you need to peruse in order to be able to say that you have at least considered all threats and mitigation measures?

You may ask me, “Why are you bringing this up? After all, the first draft of CIP-013 was voted down; what finally passed (the third draft) says nothing about addressing new risks or mitigation measures.” And I will agree with you that this directive is no longer in CIP-013 R3. However, it’s not really gone. And to find that out, you need to look no further than the Implementation Guidance written by the SDT (and here I’m quoting from the revised version that was recently circulated by the SDT – page 9, in the discussion of R3):

“A team of subject matter experts from across the organization representing appropriate business operations, security architecture, information communications and technology, supply chain, compliance, legal, etc. reviews the supply chain cyber security risk management plan at least once every 15 calendar months to reassess for any changes needed. Sources of information for changes include, but are not limited to:

Requirements or guidelines from regulatory agencies

Industry best practices and guidance that improve supply chain cyber security risk management controls (e.g. NERC, DOE, DHS, ICS-CERT, Canadian Cyber Incident Response Center (CCIRC), and NIST).

Mitigating controls to address new and emerging supply chain-related cyber security concerns and vulnerabilities

Internal organizational continuous improvement feedback regarding identified deficiencies, opportunities for improvement, and lessons learned.”

Doesn’t this look a lot like the SDT is saying you still need to do what was in Section 2.1 of R2 in the first draft? I’ll answer that question for you (since I didn’t hear anybody say anything): Yes, it does. You might then ask, “Well, how can they remove part of a requirement from one draft to the next, but still say in their guidance that you need to effectively comply with the first draft?” I really don’t know what to say to that, except “They did it.”

And I wouldn’t suggest that you tell your auditor that you're ignoring what is in the Implementation Guidance, since in this case it goes beyond what the requirement says. Yes, I know that no NERC guidance is supposed to go beyond what the strict wording of the requirement says – that’s perfectly true. But it also may be perfectly true that your auditor’s baby is ugly, and I don’t recommend you tell him that either.

To be honest (every now and then I try to be honest. It’s a good exercise), I think this requirement is a step toward addressing one of the biggest problems with the current CIP standards: the only way they can be made to address new threats is by someone writing a SAR (which gets approved by NERC and FERC), a drafting team being constituted, a new standard or requirement being drafted, several NERC ballots, NERC BoT approval, and finally FERC approval. This process always takes years, and I’ll be writing in a new post shortly that I’m now convinced there will never be any new additions to the CIP standards unless FERC explicitly orders them (Note from Tom 12/2/2019: I never wrote this post, but I continue to believe this is the case). In other words, the threats currently not addressed by CIP, including phishing, ransomware, cloud-based threats and more, will never be addressed unless FERC issues a new order (and given the political makeup of the new FERC, it’s not likely these Commissioners will be eagerly looking for new ways to extend any regulations).

So the fact that the SDT decided (with prodding from FERC) to provide some mechanism for addressing new threats to supply chain security on an ongoing basis is a good thing. What’s not a good thing is this: Why should each NERC entity have to go through the same set of blog posts, news articles, etc. to find out if there are any new supply chain security threats or mitigation measures? Why not have an industry body which does that regularly, and provides the information to all NERC entities?[ii]

The answer is this: There is no mechanism in the NERC Rules of Procedure or CMEP to have such a body. Think about it: This body would essentially be writing new CIP requirements, since entities would have to address these new threats in some way. And the current NERC operating environment allows no way to officially incorporate new threats into CIP, within any period much less than 3 years.[iii]

And this leads me to my final question: What would need to change in order for CIP to be able to quickly address new threats? Obviously, the existing standards would have to be changed, but more importantly the whole NERC standards environment would need to be changed as well. That is, there would need to be a new CMEP (and perhaps a new RoP), or at least a new CMEP that only applies to cybersecurity standards. That way, there could be a body that could continually examine new threats, and add important ones to a list of threats that must be addressed by NERC entities subject to CIP.

This would require NERC to make a huge change. Could they do this? Of course they could. Are they likely to do this? I’d say probably not. So does this mean things will stay as they are forever? Yes it does, until either a) NERC decides to change or b) NERC (and probably FERC) no longer have responsibility for the cyber security of the electric power industry. Currently, I’d say these are about equally probable. What isn’t probable at all is that this situation – the fact that the CIP standards have no effective way to be updated to incorporate new threats – will be allowed to stay in place forever. I believe Congress will ultimately step in if NERC doesn’t do anything about this, and they'll find another organization - perhaps DoE itself (and I know FERC is part of DoE. But there's a lot more to DoE than FERC!) - that will do something about it.

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte.

---

[i] And before you point out to me that the Reliability Assurance Initiative – now called Risk-Based CMEP – takes risk into account, I want to point out that the risk that RAI deals with is compliance risk – i.e. the risk that your organization won’t comply with a particular requirement. A risk-based standard is one in which the entity is allowed to align their controls with the risk posed by a particular threat – e.g. the risk that Vendor A’s patches will introduce malware into your system, vs the risk that Vendor B’s patches will do so. The requirements in CIP-013 allow that, whereas most of the other CIP requirements don’t.

[ii] Of course, the E-ISAC provides information on new technical threats, but there are a lot more threats that aren’t technical ones, plus new mitigation measures just aren’t in their bailiwick.

[iii] I know there are NERC Alerts, which try to do at least something when a new threat appears – as in the case of the Ukraine attacks. But these don’t mandate anything but a report on what the entity is doing about the new threat. While I’m sure most entities will take action on the alert, they don’t mandate the entity do something, as a new CIP requirement would.