Tuesday, September 19, 2017

Why is CIP-013 a Good Standard?


As I've hinted previously, I am now pivoting toward discussing CIP-013 (not that I'll completely ignore the rest of CIP, of course!). I'm doing this for two reasons. One is that CIP-013 compliance is going to require a huge effort, by utilities as well as vendors. The second is that CIP-013 is completely different from previous CIP standards, and compliance is completely different. Not only is it different, it's also very close to how I would like to see all of CIP be rewritten. In this post I discuss why I say that.

In July 2016, FERC issued Order 829, which ordered NERC to develop a standard for “supply chain risk management for industrial control system hardware, software, and computing and networking services associated with bulk electric system operations.” NERC developed and approved the standard and submitted it to FERC for their approval in September 2017. While FERC approval isn’t completely guaranteed, the new standard, CIP-013-1, will most likely come into effect in late 2019 or early 2020.

Experienced NERC CIP practitioners often repeat the mantra “Compliance doesn’t equal security.” This means that, while the current CIP standards CIP-002 through CIP-011 prescribe particular security practices to address particular threats, there are many threats (phishing, ransomware, cloud-based threats) that are simply not addressed at all. A comprehensive cyber security program needs to address all threats, not just those that happen to be included in NERC CIP.

But CIP-013 is different. CIP-013 essentially requires a NERC entity with High or Medium impact BES Cyber Systems to do two things: develop and implement a supply chain cyber security risk management plan (there are six particular items that must be addressed in that plan per CIP-013 R1.2, but the plan itself is in no way limited to those six items). Since a supply chain cyber risk management plan worthy of the name must by definition address all supply chain cyber risks, this in fact means that, as far as CIP-013 goes, there is no difference between compliance and security. Everything the NERC entity needs to do to secure their supply chain, they also need to do for CIP-013 compliance, and vice versa.

At first glance, NERC CIP practitioners may worry that CIP-013 will overwhelm them. It now seems that everything required for good supply chain cyber security is not only recommended but mandated by FERC and NERC – and therefore potentially subject to million-dollar-a-day penalties for non-compliance. How could this possibly be better than the existing CIP standards, which at least omit many important cyber threats like phishing and ransomware? How will NERC entities possibly comply with a standard that requires them to do “everything”?

Before answering this question, let’s consider why there need to be mandatory cyber security standards in the first place. Of course, there are few if any North American power market participants who don’t believe that cyber security is important. But they all face many competing demands for resources, and since security spending doesn’t usually produce immediately visible results, it often gets pushed to the bottom of the priority list, except when there are mandatory requirements to comply with.

Yet it is also true that mandatory standards like the current CIP-002 through CIP-011 distort cyber security spending priorities. This is because these standards (and similar standards in other industries) only address particular threats and security practices – for example, patch management – while completely omitting other threats and practices, like anti-phishing processes and technologies. But since patch management is required by CIP while anti-phishing is not, NERC entities will inevitably invest much more in patch management, even though phishing is one of the most important security threats today.

The ideal cyber regulation would be one that requires NERC entities to do what they would do anyway, given a hefty cyber security budget: They would identify the threats they face, determine the impact of each one, and remediate them based on their relative impact. Moreover, remediation of each particular threat is based on risk (so that, for example, a risky system gets much more attention than one that poses little risk). This ensures the most efficient allocation of security spending – i.e. the greatest “bang for the buck”. Yet this is almost exactly what CIP-013 does. Here’s why:

  • 1.      CIP-013 is objectives-based, meaning it states the objective and requires the entity to attain it. Too many of the other CIP requirements are prescriptive: They tell the entity how to achieve the objective, often in great detail. This creates a lot of compliance risk, and causes a big expenditure of resources on a lot of individual actions that in themselves do very little to enhance security. In CIP-013, the entity is required to develop and implement a good supply chain cyber security risk management plan, period. The content of the plan and the steps to implement it are completely up to the entity.
  • 2     CIP-013 is risk-based. While it is never officially stated in the requirements themselves, the excellent Implementation Guidance makes clear that risk is to be taken into account at every step of remediation. Vendors and systems should be categorized by risk, and remediation steps (including what is required of vendors in contract language and other commitments) should always be based on risk. This is what makes compliance with CIP-013 manageable: If it weren’t risk-based (and FERC specifically required this in Order 829), NERC entities would have to take exactly the same steps for their most strategic vendors as they do for their least strategic. And they would have to treat very important systems the same as fairly inconsequential ones.
  • 3.    The fact that CIP-013 requires a comprehensive plan means that the entity has to consider all supply chain security threats at one time, and prioritize their remediation based on impact. This is not possible under the other CIP standards, which simply require NERC entities to address particular threats without in any way allowing them to rank those threats against each other based on impact, and allocate remediation resources accordingly.





Friday, September 8, 2017

Here’s the Real Problem….


In two recent posts (here and here), I laid out the case for saying there will be no further development of the CIP standards in their present form, except for compliance with future FERC orders. But why is this a problem? Maybe NERC CIP is just fine as it is….No, I’m afraid that’s not the case.

In those two posts, I discussed one serious consequence of this situation: that NERC entities will never have definitive answers to any interpretation questions about CIP v5 and v6. And there are many of these!

This is of course a serious problem for NERC entities subject to the CIP standards, but it isn’t for the general public. The degree of protection against the consequences of a large-scale cyber attack that the CIP standards afford them (and I have said multiple times that CIP compliance has clearly made the industry much more secure than it would be otherwise; the problem is the huge and rising cost that goes along with the current prescriptive CIP compliance framework) is in no way diminished by the fact that there has been so much confusion over what the v5 and v6 standards mean. The lights will stay on, regardless of how many CIP compliance professionals spend long nights trying to figure out how to comply with ambiguously-worded requirements.

But there actually is another very serious problem that ultimately can and will affect the general public, due to the fact that CIP won’t ever be expanded: CIP will never be amended or expanded to address threats that the standards don’t currently address. CIP already doesn’t address some of the most important cyber threats today, and that problem will keep growing, since new threats are always appearing.

I’ll admit this is a problem that’s been around for a long time. Exhibit A is phishing – a threat that was probably recognized about 7 or 8 years ago (I don’t believe I’d ever heard this word until then, perhaps later than that). The current standards do nothing to address phishing (and don’t tell me that quarterly security awareness efforts are in any way adequate to mitigate any of the risk posed by phishing!); this isn’t surprising, given that the framework for CIP v5 and v6 was laid in 2008. That was when FERC issued Order 706, which approved CIP v1 but also ordered NERC to make a number of changes; CIP v5 was the culmination of the effort to address the directives in Order 706 (indeed, the drafting team that was put in place after 706, which drafted CIP versions 2 through 5, was called the CSO 706 drafting team, where CSO stands for “Cyber Security Order”).

A standard practice for NERC, in drafting a SAR for a new drafting team to address a FERC order, is to keep that SAR as narrow as possible – if possible, only including what is in the order itself. And with Order 706 – which weighed in at over 600 pages – NERC had all sorts of motivation not to go beyond the order. Even addressing what was in the order – or as much of it as NERC felt they could address – took five years! So, even though phishing quickly grew in importance as the years went on, NERC never even considered adding it to the threats addressed in CIP, since it wasn’t mentioned in Order 706. This means that now we have the situation in which at least one study found that 91% of cyberattacks start with a user clicking on a link or attachment in a phishing email (and both of the Ukraine attacks started that way!), yet CIP has absolutely nothing to say about phishing.

Of course, I’m not at all saying that utilities aren’t hyper-vigilant about phishing; in fact, the same study found that utilities had the third-lowest phishing success rate of the ten industries studied. I have heard of one large utility that has a “four strikes and you’re out” policy. Pursuant to that policy, regular test emails are sent to all employees. An employee who clicks for the first, second or third times is sent to increasingly serious training and given increasingly explicit warnings. On the fourth click, they’re fired, with no appeal possible (and in theory, the CEO isn’t exempt from this!). This may seem pretty harsh, but given the serious threat that phishing poses to any organization, in my opinion this is appropriate.

And phishing certainly isn’t the only threat not addressed by CIP. How about ransomware? How about purely destructive non-ransomware, exemplified by Not-Petya? How about the many possible threats that come with virtualization? How about the many threats attendant on storing BCSI in the cloud (which, as I discussed in a series of posts this year starting with this one, is allowed by CIP v5)? I’m sure there are a lot more that don’t come to mind at the moment.

None of these are new threats, yet not only are they not addressed by CIP now, none of them except virtualization have even been included in a NERC SAR (and as my recent post on virtualization asserted, the drafting team whose SAR this is included in will almost certainly never even submit the required new requirements and definitions to the NERC ballot body for approval). I think the last new threats that were addressed in CIP are the threats posed by Transient Electronic Devices and Removable Media; and these were only addressed because FERC ordered this (in Order 791 for Medium and High assets, and Order 822 for Lows).

And why aren’t these threats addressed in CIP v5 or v6, or at least set to be addressed by the CIP Modifications drafting team? Because addressing a new threat requires a tremendous multi-year effort: coming up with 3-4 major drafts of the new requirements and definitions, submitting these to the NERC ballot body, responding to the voluminous comments on each draft, submitting the final approved draft to the NERC Board, and finally to FERC for their approval. And should FERC decide they want some changes in the new requirements or definitions, a new drafting team (or perhaps the original one) will have to go through the same process again. It isn’t at all surprising that NERC practically has to have a gun at their head before they’ll propose that CIP be expanded to address a new threat.

Of course, NERC does have other ways of dealing with new threats, most notably the NERC Alerts, which have been used in the cases of the Aurora vulnerability and the Ukraine attacks, as well as in other cases. These aren’t mandatory standards, but they require NERC entities to report on what they are doing to address these threats – which is pretty close to mandatory, in my opinion. And as I said, probably all of the larger NERC entities are very attuned to new threats, and consider any new warning from the E-ISAC or other industry bodies as something that needs to be acted on.

But this just means that, in today’s utility environment, cyber threats are addressed using a hodge-podge of mandatory requirements (the CIP requirements), non-mandatory “requirements” (the NERC Alerts), and all sorts of different best practices. Except for CIP compliance, there is very little uniformity in how NERC entities are dealing with these threats, and even with CIP there is lots of variation among entities in how they comply.

Just as importantly, the fact that there isn’t any uniformity in how NERC entities deal with threats – both old and new ones – means that entities will almost always spend most of their time and resources on the threats included in the CIP standards (malware, remote access, etc), and less time and resources on threats not included in CIP (phishing, ransomware, etc). This is frankly a mis-allocation of resources, since in an ideal world each entity would look at all the threats it faces, rank them based on their potential impact, and prioritize their spending accordingly. Prescriptive CIP requirements inevitably result in too many resources being spent on certain activities, compared to others that might be of equal benefit but aren’t included in CIP at all. My poster children for this are CIP-007 R2 Patch Management and CIP-010 R1 Configuration Management. Both of these are important activities, but both consume disproportionate amounts of resources (especially in large organizations) vs. addressing threats like ransomware and phishing (if you have some time on your hands, you can find a rather voluminous discussion of this problem in this post).

You might now ask how I would fix this misallocation problem, given that there have to be mandatory standards (which I totally agree with). As some of you may know, this spring I started advocating a new critical infrastructure compliance regime that includes six parts:

1.                   The process being protected needs to be clearly defined (the Bulk Electric System, the interstate gas transmission system, a safe water supply for City X, etc).
2.                   The compliance regime must be threat-based, meaning there needs to be a list of threats that each entity should address (or else demonstrate why a particular threat doesn’t apply to it).
3.                   The list of threats needs to be regularly updated, as new threats emerge and perhaps some old ones become less important. Ideally, there will be an industry body that meets regularly to assess new threats, as well as document new mitigations. They will add important new threats to the list, and potentially remove threats that are now of less importance.
4.                   While no particular steps will be prescribed to mitigate any threat, the entity will need to be able to show that what they have done to mitigate each threat has been effective.
5.                   Mitigations need to apply to all assets and cyber assets in the entity’s control, although the degree of mitigation required will depend on the risk that misuse or loss of the particular asset or cyber asset poses to the process being protected.
6.                   It should be up to the entity to determine how it will prioritize its expenditures (expenditures of money and of time) on these threats, although it will need to document how it determined its prioritization.

Let’s focus on parts 2 and 3, since these are the ones that are important for this post. Part 2 essentially means that all of the CIP requirements should be replaced with just one, which reads something like “On a risk-adjusted basis, take steps to mitigate all threats on the current threat list, or document why a particular threat doesn’t apply to your organization.” And part 3 calls for an industry body to make regular updates to the threat list.

In my opinion, this is the right approach because it requires the entity to do what they would normally do in the absence of CIP: identify all current threats, rank them by impact, and allocate resources to mitigate them according to that ranking (with the stipulation that some threats may pose so little risk, or they may not be relevant to the entity for some reason or another, that they may be left out altogether – although the reasons for this will need to be documented!). But this process will be mandatory, not just one of those nice-to-haves that we all never quite get around to.

I hope you see how the subject of this post – the fact that it’s very unlikely any further cyber threats will be incorporated into the current NERC CIP compliance regime – is addressed by my third point. New threats will be fairly easily incorporated into my proposed new CIP compliance regime because there will no longer be any need to develop new requirements, or modify existing ones. The single requirement that I sketched out two paragraphs ago doesn’t ever need to be changed, regardless of the new threats that come along. The industry body I’m suggesting will be able to update the threat list simply through a majority vote (or perhaps a super-majority vote) of its members.

So could all of the problems with CIP be fixed by throwing out all of the current standards and replacing them with my single requirement, as well as the six points? The problem is that what I’m suggesting requires fundamental changes to NERC’s Compliance Monitoring and Enforcement Plan (CMEP) and to its Rules of Procedure (RoP), not just to the CIP standards.

Why do these documents need to be changed? For one example, just consider my point 3. The industry body I’m calling for would essentially have authority to modify what my single CIP requirement covers, by updating the threat list. There is certainly no provision in the two documents that would allow such a body to be constituted, or to have this authority.

Even more fundamentally, the way auditing would occur under my new compliance regime would be radically different from the way it is described in CMEP and the RoP now. These two documents describe auditing of requirements that mandate very specific procedures (e.g. almost all the requirements in the NERC 693 or “O&P” standards, as well as prescriptive CIP requirements like CIP-007 R2). These requirements can be audited very simply: Did the entity do what was required? If so, they are compliant; if not, they aren’t.

Think about how an auditor would have to determine whether the entity had complied with my single CIP requirement. Some of the components of that audit would include:

  1. Has the entity truly taken steps to address all of the threats on the current list, eliminating only those that aren’t applicable to it? And do I agree with the criteria the entity used to determine applicability of the threats?
  2. Since the entity is required to address threats based on the risk that each poses, have they developed a reasonable methodology for determining risk? And is the “risk” addressed in that methodology the risk to the Bulk Electric System, not to something else like the entity’s finances (of course, the whole point of CIP is to protect the BES, nothing more, nothing less)?
  3. Looking at each threat addressed by the entity, have they deployed effective means to address it? In other words, they may believe that a certain chant, repeated daily, will prevent malware infection of their BES Cyber Systems, but this isn’t an effective means of preventing malware.

I’m sure there’s more to auditing my single CIP requirement, but I think you can see that none of the three audit steps I’ve just listed would fit into the current NERC CMEP and RoP, although in fact I contend that CIP auditors are already partly moving in this direction. This is because some of the CIP requirements, like CIP-007 R3 Anti-Malware and CIP-010 R4 Transient Electronic Devices, are already non-prescriptive, objectives-based requirements. The only way to effectively audit these requirements is to take an approach something like this. But this situation – in which a few non-prescriptive CIP requirements are audited using a very prescriptive auditing framework – simply can’t last. And it would definitely not last if the current CIP regime were replaced with my single requirement and my six points.

Of course, we all know that the likelihood of CMEP and the RoP being so radically revised by NERC is pretty small. Does this mean I’m wasting my time by proposing a new compliance regime that will never be enacted? I don’t think so at all, and the reason I don’t is that I think there will be steadily growing pressure – mostly from Congress – on NERC and FERC to either a) make the necessary changes to the NERC compliance regime so that new cyber threats can be easily incorporated into CIP; or b) turn over cyber regulation of the electric power industry to another government body that isn’t constrained by CMEP and RoP. Consider this Congressional hearing sometime in the not-so-distant future.

Senate Committee Chairwoman: Thank you for coming before us, Mr. High NERC Official. We’ve asked you to come here today because we were disturbed to learn recently that the NERC CIP cyber security standards still haven’t been updated to include such important threats as phishing and ransomware. Can you explain to us why this is the case?

High NERC Official: I am pleased to explain that to you, Madame Chairwoman. This isn’t due to our not considering these threats to be of the highest importance. It is due to the fact that NERC’s Rules of Procedure require a deliberate, democratic process for drafting and approving proposed changes to the NERC CIP standards. We do intend to address all of these threats in the future, but we are currently working on some other changes to the CIP standards, such as new standards (ordered by FERC) addressing supply chain security and protecting communications between control centers. Once we have been able to draft and approve these and other new standards and requirements, we will be pleased to turn our attention to other threats like phishing and ransomware.

Senate Committee Chairwoman: Thank you for your honest answer. How long do you think it will be before we have new CIP requirements addressing these two threats, as well as other emerging threats that we don’t even know about today?

High NERC Official: We have already drafted the supply chain security standard and sent it to FERC for approval. We still have a number of other new standards and requirements – including some very important requirements regulating virtualization – that will take maybe three years to draft and approve. But at that point, I think we might very well be able to take up new threats like phishing.

Senate Committee Chairwoman: Excuse me, but I don’t consider phishing a “new” threat. But let’s go on. You say in three years you’ll be able to start addressing phishing. How long will it take, after that point, for a new NERC CIP anti-phishing requirement to be in effect?

High NERC Official: I would think we would have one in place in 2-3 years after we started drafting it.

Senate Committee Chairwoman: So it will be 5-6 years before there will be a requirement addressing phishing, is that correct? (HNO assents). And let’s suppose you could address ransomware on the same timetable, so an anti-ransomware requirement would also be in place 5-6 years from now? (HNO assents) Of course, I also have to assume that a new type of threat that appears say in 2018 will take much longer to address. Is that the case? (HNO assents) Mr. High NERC Official, I have to ask you: Do you agree that the North American Bulk Electric System is a critical national asset, whose substantial impairment would have a huge negative effect on the entire nation? (HNO assents) And do you agree that, while other methods are important, there needs to be a mandatory compliance regime, backed by substantial penalties, that will address all major threats to the security of the BES? And that the BES isn’t well protected until CIP addresses all major threats? (HNO assents, although he starts to look around nervously) Then how can you say that NERC is protecting the BES, when there are such gaping holes in the CIP standards – and there always will be such holes?

High NERC Official: (rising from his seat) I’m sorry, Madame Chairwoman. I need a short break to confer with some of my colleagues. Can we resume in half an hour?

Senate Committee Chairwoman: I’m afraid we don’t have time to do that. I want to thank you for your honest testimony today. I’m sure we will want to have further conversations with you in the future. This session is adjourned.

Senate Committee Chairwoman (aside to aide): Please contact the Secretaries of Homeland Security and Energy and ask them to come before this committee to let us know if they might possibly be able to address phishing and ransomware in something less than six years, if they were given responsibility for cyber regulation of the BES.


The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte.

Tuesday, September 5, 2017

EMP Part 2


Two weeks ago, I did a post calling your attention to an excellent article in Energy and Environment News about the dangers of an EMP attack. That article was just part 1 of 2, and here is part 2. Just as riveting – and perhaps more scary – than part 1!



The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte. 

Thursday, August 31, 2017

What about Virtualization?


In my last post, I listed six items in the Standards Authorization Request (SAR) for the current CIP Modifications drafting team that I believe will never be accomplished. I discussed my reasons for thinking the first five won’t be accomplished, but I deliberately left the last one – virtualization – for another day. Another day has come, and I’ll explain why I don’t think the drafting team will accomplish this task, either.

First off, what is this task? NERC included virtualization in this drafting team’s SAR because CIP v5 and v6, like all of the other CIP versions before them, say nothing about virtualized cyber assets. But many NERC entities with High and Medium impact assets have been using virtualization – of systems, of networks, and of storage – for years. This is especially true in control centers, where the benefits of virtualization are especially pronounced.

So what does it mean that CIP doesn’t mention virtualization? Does that mean it’s forbidden? No, not at all. It means it’s completely off the radar for CIP. For instance, if you use virtualized servers in your control centers, all you have to do is treat the single piece of hardware that they all run on as a BES Cyber System; this is because CIP currently only applies to Cyber Assets, defined as “Programmable electronic devices.” And a “device”, in my na├»ve way of looking at things, is something that will hurt if you drop it on your foot. Even if you can figure out how to drop a virtual machine on your foot, it’s pretty clear that it won’t hurt you.

This means that you don’t have to declare your virtual servers themselves as BES Cyber Assets/Systems, and therefore that the requirements in CIP-003 through -011 don’t apply to them. In practice, of course, I’m sure that any NERC entity with High or Medium assets is declaring all of their virtual servers within the ESP to be either BCAs or PCAs; I’m also sure every Regional Entity is keeping an eagle eye out for VMs that haven’t been declared. But if a NERC entity were to be fined for not declaring virtual servers to be BCAs and they were to challenge the fine in court (which they are always allowed to do, since all NERC standards are administrative law, arbitrated by administrative law courts), I believe it is highly likely the fine would be thrown out, simply because they aren’t violating any current CIP requirements.[i]

But even leaving the question of virtual servers aside, there are lots of thorny questions that come up when you look at virtual networks and virtual storage – and these can’t be fixed with just a tweak or reinterpretation of the word “devices” in the Cyber Asset definition. The CIP Modifications drafting team didn’t try to sweep any of these questions under the rug; on the contrary, they actively sought them out.

Early in the team’s existence (they started operation in the spring of 2016), they formed a committee to work on virtualization. That committee quickly realized that there are some very fundamental concepts in CIP – besides the concept of Cyber Asset – that need to be changed to allow virtualization to be encompassed. For example, the idea of an Electronic Security Perimeter has limited use in a virtualized environment, since it focuses entirely on devices – the physical device is either in or out of the ESP, along with all of the virtual devices that may be housed in it. But since switches, storage and servers can all be “divided” in some way into different virtual components, and since these components won’t always be all part of the ESP, there needs to be another way of talking about a security perimeter when these are present; so the committee came up with the concept of an “Electronic Security Zone”.

I have good news and bad news. The good news is that this committee has really done some pioneering work, and they have developed a new way of thinking about cyber security in virtualized environments (not just on OT networks, either); to see some of their work, you can go to the drafting team’s web site and look at the slides and listen to the recordings from the webinars they’ve done. The team could document what they’ve done in a really nifty white paper (or even a book), which should have applicability far beyond the CIP world.

But here’s the bad news: Implementing the new concepts the committee is talking about will require a huge number of new or changed CIP requirements and definitions. Each of these will need to be drafted, posted for 2-4 ballots, commented on, revised multiple times, and finally approved. In this post from early January, I did a little math. I pointed out that, in the case of the changes to the LERC definition and the underlying requirement, this whole process had taken more than six months of this drafting team’s almost exclusive time in the second half of 2016 and the first 2-3 months of 2017.

I then combined this with an estimate that one of the virtualization committee members had given me, that there were probably 15-20 (or even more) definitions or requirements that would need to be changed or developed from scratch. Assuming this is true, and that each of these will take up 6 months of the SDT’s time to get approved by the ballot body, I came up with the (perhaps conservative) estimate that just virtualization will take ten years for the drafting team to accomplish! Needless to say, this is never going to happen.

I know that, at one point at least, there were some on the drafting team who were advocating they just draft all of the changes required for virtualization and submit these as one huge ballot (or maybe 2 or 3 big ballots), that NERC entities could either approve or disapprove in toto. This strikes me as a wonderful prescription for spending an incredible amount of time (just this effort would take well over a year, maybe two years), and likely coming up with exactly zero to show for your efforts, since – unlike the case with LERC – there is no mandate from FERC that this be done. And that seems to be what it takes to get new CIP requirements, or substantial changes to existing ones, approved nowadays, as was also demonstrated by CIP-013.

At one point, I heard a member of the drafting team compare the work they’re doing to the CIP v5 standards. Those standards were of course submitted as a whole to the NERC ballot body, and – while their path to passage certainly wasn’t easy, taking close to two years and at least four ballots – they were ultimately approved and are now in effect. And it’s certainly true that CIP v5 required many more individual changes than virtualization will. So why shouldn’t we expect the same thing to happen here - that it might be a rough year or two, but ultimately the changes required for virtualization will pass?

I’ll tell you why the same thing won’t happen with virtualization as happened with CIP v5. First off, remember that CIP v5 was very much ordered by FERC, in this case Order 706 from January 2008, although it took five years for NERC to “respond” to the order[ii] (there were of course a lot of diversions along the way, including CIP versions 2-4. Of these, v3 was due to a new FERC directive, while v2 and v4 were deliberate choices on NERC’s part). So the NERC ballot body was very much aware that FERC wanted them to approve v5. And FERC had added another incentive when they unexpectedly approved[iii] CIP v4 in April 2012. In doing that, they in effect said “You’d better approve v5 soon. If you don’t, you’ll end up having to comply with v4 and then with v5. Does that sound like much fun?”

The other reason why the same thing won’t happen has to do with the CIP v5 experience itself. While there was a lot of optimism around v5 initially (including on my part), as NERC entities started their v5 implementation work in early 2014 all sorts of questions about what the requirements and definitions really meant started popping up. NERC started trying various expedients to address these questions without doing the one thing that is allowed by the Rules of Procedure: develop a drafting team and task them with fixing many if not most of the worst problems; most of the “fixes” NERC tried ended up being withdrawn, resulting in the fact that today, August 31, 2017, NERC entities are no closer to having answers to their CIP v5 interpretation questions than they were the day FERC approved v5 in 2013 (this sorry process is discussed in a little bit of detail in this post. If you want to read the whole story, I suggest you read every post I wrote between this one and say January 1, 2016. Then maybe you can summarize them for me, since I have no intention of reading them all any time soon. A human being can only take so much punishment).

I’m not a psychologist, but I think the NERC community has been in a kind of shell-shock ever since the CIP v5 experience. This has manifested itself in two ways. The first is a suspicion of anything that a NERC drafting team puts out for a ballot, no matter how good it is. I saw this most vividly in the debate over the “LERC” changes last year, where there was a firestorm of opposition to the first ballot, and the second ballot probably only passed because there was a deadline of early February for NERC to get this change to FERC. This in spite of my opinions that a) the second ballot, which passed, wasn’t substantially different from the first one, which didn’t; and b) both ballots (actually, drafts) should have been no brainers to approve – they allowed the entity to do everything allowed in the CIP v6 version of this requirement, plus much more.

The second manifestation of shell-shock was vividly brought home in a regional CIP workshop I attended last year. At that meeting, a member of the CIP Modifications drafting team gave a presentation on their progress, including some of the changes that virtualization would require. Then a member of the CIP v5 drafting team (in fact, the team that drafted CIP versions 2-5) made an eloquent appeal that the team not subject the NERC membership to debate and vote on any more far-reaching requirements or definitions (and I believe he had the LERC experience in mind).

Think about this: Here was a very well-respected member of the CIP v5 drafting team essentially saying there shouldn’t be any more improvement or expansion of NERC CIP! And simply because the NERC membership was “too tired” (I think he used some words to that effect) to be able to debate these issues anymore.

Of course, what this person was talking about wasn’t just virtualization, but the whole agenda (SAR) for that team. If the team actually came out with a new requirement and/or definition that addressed any one of items 1-5 in the list of SAR directives in my last post (e.g. clarification of what “Programmable” means in the Cyber Asset definition), it’s almost inevitable there would be another big, divisive debate, just as there would be if the team starts to try to codify far-reaching changes like the Electronic Security Zone. So this is another reason why I think the drafting team will need to get their SAR reduced to the items they’ve addressed already and the two items they’re now actively working on: CIP-012 and the TOCC issue. If they try to slog through with the rest of their agenda, they should all literally ask for a reduction in their day job responsibilities for the next 15-20 years; I’m sure that won’t be any problem at all.

And given that, assuming the SDT does get their SAR reduced, the chance there will ever be a future SAR aimed at addressing the fundamental problems in CIP v5/v6 is exactly zero, this is another reason (besides the one discussed in my previous post) why I see no likelihood there will ever be any further changes or additions to the current CIP standards, except where explicitly allowed by FERC.

There is also a corollary to this statement, which I will bring up in one of my next posts.


The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte.


[i] I have heard a few people around NERC make the argument that, because the definition of Cyber Asset reads “Programmable electronic devices, and the hardware, software and data in those devices” (my emphasis), this means that virtual servers all do have to be treated just the same as physical servers. This is a pretty tenuous argument in my opinion, since this wording clearly was intended to cover the normal situation where application software is stored on a device and utilized in that device’s normal operation, not where virtual devices themselves – each with their own application software – are housed in a physical server.

[ii] And the fact that it did take five years was very much on FERC’s mind from then on. Since then, every significant new or changed standard that they have ordered has come with a deadline – this applies to CIP-014, LERC and CIP-013.

[iii] I didn’t start this blog until January 2013, but I did write about that event, and what it meant, on a Honeywell blog that no longer exists. I can send anyone who’s interested the post I wrote discussing this idea, which was later obliquely confirmed to me by a source who knows something about what goes on internally at FERC. The fact that FERC took this maneuver turned out to have some serious unfortunate consequences, as discussed in a series of posts starting with this one.

Sunday, August 27, 2017

The Horror, Part 2: The End is Near


It isn’t my intention from now on to only use titles for my posts that sound like they come from bad movies. But this post is a follow up to a post that described what I believe is a genuine horror: the fact that there isn’t now and never will be any guidance published on NERC CIP that is in any meaningful way binding on the auditors. NERC entities are truly on their own when it comes to deciding how to interpret CIP v5/v6, given the ambiguities, missing definitions and inconsistencies that are found in those standards as written. I’m sure most NERC entities understand this already, even though they may not have articulated it so far (however, I’m not at all sure most people at NERC understand this).

But I regret to say that the horror I described in the previous post is only the lesser of two horrors, and the second horror is a lot scarier than the first. In other words, this post is one sequel that may actually be scarier than the original. Parental discretion advised.

This summer, I was in Montreal for the meeting of the CIP Modifications Standards Drafting Team (the team that’s working on all CIP changes except the new Supply Chain Security Management standard, CIP-013. That one has its own SDT). This was the first meeting of this team that I’d been able to attend in six months, and I’m very glad I did attend.

I wanted to come to this meeting primarily to get updated on what this team is doing. But I also came because I wanted to see if they felt the same sense of alarm about their mission as I do. As I pointed out in this post in January, I have come to doubt that there is any way for this team ever to accomplish everything that is in their Standards Authorization Request (SAR).

So I have good news and bad news. The good news is that I no longer have any doubts about whether this team can accomplish everything in their SAR. The bad news is I’m now certain they can’t ever do this. This is due to no fault of their own; it just turns out their marching orders came from Mission: Impossible.

So – since I’m just a helpful guy by nature - I came to Montreal prepared to bring up to the team the fact that they really need to think now about what they can and can’t accomplish; after all, I don’t think any of them joined the drafting team with the expectation that they’d serve on it until retirement! One conversation I’d had with a team member at the NERC CIPC meeting two weeks previous had led me to believe it would be difficult for the team members to even have this conversation.

However, I was glad to see that one of the first items discussed at the meeting was the question whether they need to revisit their SAR now. Of course, doing this will be no small deal. A NERC drafting team doesn’t get to decide for themselves which parts of their SAR they’ll be able to fulfill; if they want to change it, they need to somehow get NERC to modify it, which may require ballots (and I don’t know whether this has ever happened in NERC’s history, and certainly not with any of the CIP drafting teams).

So what is in this team’s SAR? I described the SAR in this post in early 2016. I’ll now list each of the items in the SAR and discuss its status (although I’ll group them differently than I did in the previous post):

  1. The team has finished with the FERC directive to clarify the meaning of LERC. This was approved by the NERC ballot body – although it proved much more contentious among the NERC membership than I ever imagined it would – and sent to FERC in early March (FERC, of course, hasn’t had a quorum since then, but now they do again. Given that there are a number of urgent projects like pipelines that need to be approved, my guess is they won’t get to this for six months or more).
  2. Similarly, the team has finished with the FERC directive to develop requirements for Transient Electronic Devices for Low impact BES Cyber Systems. This was also sent to FERC in March, thus saving NERC the cost of another postage stamp.
  3. The FERC directive to protect “communications network components” between Control Centers is being addressed in the new CIP-012. This has been drafted and is now out for a first ballot.

These were the three items that were ordered by FERC, meaning neither NERC nor the drafting team has any choice but to fulfill them. Even though the last of these isn’t yet put to bed, it’s well on its way, and there’s no question that the SDT will fulfill it. But what about the items in the SAR that weren’t ordered by FERC?

First, the Transmission Operator Control Centers (TOCC) issue – which I admit I understand only on a fairly superficial level – has been the subject of a lot of debate and comment (I referred to it as the “TO/TOP Issue” in my 2016 post). I’m sure it will be resolved, since it is a very contentious issue for a large number of (relatively small) NERC entities. This one doesn’t pose any philosophical issues: Everyone seems to agree on what needs to happen, but they haven’t yet agreed on the wording that will make it happen. I’m sure they will agree in the near future, though.

What about the other items in the SDT’s SAR? As discussed in depth in the previous post, these include

a)       Clarifying the definition of Cyber Asset (especially the meaning of “Programmable”);
b)      Preventing the BES Cyber Asset definition from “subsuming” all other asset types (and here we have another potential horror movie title: “The Definition that ate all the Assets!”);
c)       Determining how to set a “lower bound” for “impact on the BES” in the BCA definition;
d)      The question of “double impact” and preventing this from leading to the N-1 criterion raising its ugly head to exempt lots of assets from the scope of CIP v5 (as discussed in the previous post, I thought the way this question was phrased was wrong, but I was also surprised that the N-1 criterion was still the bugaboo it had been in the CIP v1 days. I thought it had been laid to rest with a stake through its heart then);
e)       The “Network and externally accessible devices” question; and
f)        Virtualization (i.e. trying to fix the problem that CIP is now completely silent on virtualization, meaning that – strictly technically speaking – any virtualized cyber assets are completely out of scope).

These might seem like very different issues, but items a) through e) all have to do with asset identification – that is, CIP-002 R1. And in the Montreal meeting, one of the SDT members pointed out one very important reason - which I hadn’t thought of - why it would be better to leave these items alone (in other words, to ask NERC to remove them from the SAR): If one or more of these were changed, NERC entities with Medium or High impact assets would probably have to re-run their entire asset identification process. Whether or not this resulted in a lot of new cyber assets coming into scope for CIP v5/v6 or maybe a lot of existing ones being removed from scope, just having to go through this exercise again would require a huge investment of staff time and money, especially on the part of large entities that have hundreds or thousands of BES Cyber Assets.

Of course (and here I’m going beyond what was discussed in Montreal; the discussion of this issue ended after the above point was raised, when one of the team leaders noted that at the moment there was still one item on the SDT’s plate that had to be dealt with due to a FERC order, so the question of their SAR didn’t need to be settled right away), the fact that a huge investment would be required doesn’t itself mean the effort shouldn’t happen; after all, we’re talking about protecting the US electric grid! But what would this effort accomplish? Even with all the ambiguities and missing definitions in CIP-002 R1, Attachment 1, and the Cyber Asset and BCA definitions – and there are a lot more that never made it into the SAR - there has been a remarkable consensus among entities and the NERC Regions as to how cyber assets should be evaluated to identify BCAs.

As proof of this consensus, I haven’t heard of a single NERC entity that has been told they were way off the mark on how they identified BCAs, and they’re now in deep doo-doo because they have a bunch of – say – relays that now need to be brought into compliance with all of the requirements in CIP-003 through -011. In other words, if there were even one sizable entity that had woefully under-identified its BCAs and had therefore put the security of the BES at risk, my guess is I would have heard something about that by now. In still other words, if there’s no problem to fix, why require entities to go through this huge effort?

Those of you (both of you, I think) that read this blog in 2014 and 2015 may perhaps remember that I was very worried about the fact that there was so much ambiguity, contradiction, and missing definitions in the CIP v5 asset identification process. In fact, I’m sure I wrote at least 50 posts on various aspects of this problem. As I recounted in my previous Horror post, I had at first hoped the CIP v6 drafting team would be charged with fixing these problems, then I hoped that some sort of guidance could be put out to do this.

When it became clear in 2016 that neither of these would happen, I went through all the stages of grief, and I finally came to realize there would be no true fix for these problems. NERC entities would simply have to in effect write their own CIP v5 standards, following whatever unofficial (and almost always unwritten) guidance their regions will give them. And I began to realize that NERC could ask Mahatma Gandhi, Mother Theresa, Alexander Hamilton, James Madison, Solon, and the Buddha to sit down at a table and hammer out a final solution to these problems, and these august personages still wouldn’t be able to do so, any more than the members of the original CIP v5 SDT (who I know were aware there were a number of serious issues in v5, but were under enormous pressure to finally deliver FERC a response to the directives in Order 706, issued in 2008) were able to do so. Indeed, I came to realize that only a wholesale rewriting of the CIP standards (and, as I’ve also come to realize more recently, NERC’s CMEP and maybe even the Rules of Procedure) will “solve” these problems for good.

So I totally agree with the sentiment among at least some of the CIP Modifications SDT that all of the “asset identification” issues in their SAR should be removed from it (and I also believe item f above -virtualization - should be removed, although for a very different reason. I will discuss that in a separate post soon). What does this mean? It means I believe we’ve come to the end of the line for further developments in NERC CIP, unless FERC orders a new standard.

Specifically, there will be no further efforts to rewrite CIP v5 or v6 requirements and definitions to clarify the many wording problems, including the efforts in the current SDT’s SAR. How could it be otherwise? If the current SDT succeeds in getting these items removed from their SAR, who will ever even suggest in the future that a new SDT be constituted to try to fix problems with CIP v5? We are truly at the end of the line for any further refinement of the current CIP standards.

The lesson of the first Horror post was that there is not now and will never be any guidance on NERC CIP – published by NERC, the Regions, or any other organization or individual (which includes, I hate to say, my blog posts) – that can be considered in any way binding on CIP auditors. Entities need to decide for themselves how to address each interpretation issue, and document why they think that way.

And the lesson of this second Horror post is that the only possible means of resolving these issues – writing a new SAR and revising the existing CIP requirements and definitions (or developing new requirements or definitions to fix issues like the meaning of Programmable) is now closed. Thus, the CIP v5 and v6 standards will always be ambiguous and inconsistent, until they are replaced with new standards that take a very different approach.

From now on, I plan to mostly write about what kind of standards (or maybe just one standard) could replace the current CIP standards and eliminate these ambiguities (more specifically, the new standards would make ambiguities, like the ones in CIP-002 R1 and Attachment 1, irrelevant). And I will also write a lot about CIP-013, which – despite its many flaws – moves a long way toward what I think all of the CIP standards should be (and this creates a lot of interesting problems, since of course CIP-013 will be enforced using the same prescriptive CMEP as all the other NERC standards).

However, I don’t intend to ignore CIP v5 and v6 from now on; after all, they will remain the law of the land for at least 2-3 more years. But at every possible occasion I’ll let my inner snark appear and point out that whatever problem I’m discussing at the moment can only be finally resolved when the CIP standards are rewritten. During the turbulent 60’s, I and my radical friends had a running joke that every problem – no matter how pedestrian – would be resolved “come the Revolution”. But now I’m not joking: I hope I’ve demonstrated to your satisfaction in these two Horror posts that there will be no resolution to the problems of CIP v5 and v6 until the standards are replaced with ones that take a very different approach.


The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte.

Friday, August 18, 2017

Another great news article, this time on EMP

It's almost embarrassing that, two days after I put up a post about an article in Energy and Environment News, I'm putting up a new one (I swear, they're not paying me!). This is an article that just came out today, and it's about EMP. It's really good. Like most E&E News articles, it's written without a concern to fit in a particular space - unlike another good (but too short) article on this topic from the Wall Street Journal. Of course, people who write long articles are near and dear to my heart!

One small quibble I have with today's article has to do with the last paragraph, where the writer, John Fialka, mentions that a number of other countries have started to take steps to harden against EMP. He adds "Whether the United States will join them remains a work in progress." NERC has just passed a standard to do just that, TPL-007-2, although it still requires FERC approval. Of course, opinions differ as to whether the standard does enough, but understanding those arguments is far above my pay grade.

Another good thing about this article is that it's part 1 of 2. I'll hopefully be able to give you a link to the second one after it's published next week.

Wednesday, August 16, 2017

The best press article on NERC CIP ever written

After yesterday's post, someone reminded me about a press article that appeared in Energy and Environment News in January 2016, which I had quoted in full in this post. It is very relevant to what I was writing about yesterday, and goes beyond that.

It is without a doubt the best news article I've ever seen on NERC CIP. Almost every other article I've read makes me want to cringe, although I'll admit CIP is a very difficult subject to get your arms around. You need to really be willing to sit down and devote a lot of time to learning about CIP and its whole context. Pete Behr is one of the few reporters that has done that. In fact, I'd say he's the only one who has done that.