The News from SPP

I spent most of the last week travelling to or attending two regional meetings: SPP’s CIP Workshop in Little Rock and WECC’s Low Impact Workshop in Salt Lake City. Both meetings were very well run and quite informative. Below are some high points I took away from the SPP meeting. My next post will do the same for the WECC meeting.

Before I start that, I want to point out that you can find all of the presentations from the SPP meeting here. They are all in one big file, but it downloads pretty quickly. There will also be videos of the presentations; you will be able to find them here. The WECC presentations are here.

While all of the presentations at SPP were good, I got the most out of Scott Mix’s presentation on how NERC will audit for compliance with the Low impact requirements. I had seen him do this presentation at RF’s CIP workshop in April, but there were a lot of points he made at SPP that I didn’t remember from then (of course, it’s possible he had made some changes. I highly recommend watching the video when it’s available).

Scott first addressed the question whether a list of Low impact BES Cyber Systems is actually required, even though the requirements practically do back flips to say it isn’t. This is a huge issue (it was at the WECC workshop as well). I think Scott did a very good job of addressing this, and rather than try to summarize what he said, I’ll just refer you to his discussion starting on slide 4.

One of the main points of Scott’s presentation was that, when it comes to Low impact assets, the idea of randomly sampling them to decide what to audit goes out the window. Since Low impact assets vary widely in terms of their impact on the BES, the focus of Low impact audits will always be on the most important assets. One example of this has to do with 1500+MW plants that are called out in criterion 2.1. If the entity is claiming a plant is segmented so that there are no Medium impact BES Cyber Systems, that plant will definitely be visited during the audit. In a similar vein, Scott pointed out that a substation that has multiple lines but doesn’t meet criterion 2.5 will “get more attention” than one with just a single line.

He discussed three areas where both Lows and Mediums have requirements (starting on slide 39). For incident response plans and awareness programs, Scott suggested it’s probably a lot easier for entities that have both Medium and Low impact assets to just use the Medium procedures. That way, people who aren’t working with CIP day-to-day can just refer to a single procedure, rather than have to figure out whether an asset is Medium or Low impact. He also mentioned that “configuration and management” procedures may be similar for LEAPS (Low impact Electronic Access Points) as for EACMS containing EAPs (of course, EACMS and EAP only come into play for Mediums. An EAP is an interface, which is part of an EACMS).

While the remaining presentations were all good[i] (although I had to miss the half-day session on Wednesday to get to Salt Lake City for the WECC meeting), I want to call your attention to the last presentation of the day by Robert Vaughn and Shon Austin, who are both SPP auditors (starting on slide 242). It is titled “Observations from our CIP V5 Outreach Visits”, and it’s based on a set of visits they evidently did to SPP entities to review their preparedness for v5 compliance.

I’ll let you read their slides (and see the video when it’s available), but I’ll say now that their presentation seems to raise some serious questions about how prepared NERC entities will really be for CIP v5 on July 1 (although, since I wasn’t there for the presentation, I’ll admit it’s possible that their spoken words may mitigate what seems to be the purport of their slides – I’m looking forward to the video!). It looks like I may have to revise my April Fool’s Day post.

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte Advisory.

---

[i] Kevin Perry of SPP gave a presentation (starting on slide 225) entitled “Could CIP Standards have Prevented the Ukraine Attack?” I missed that, but I will definitely watch the video when it’s available.

What's Going On?

Last Sunday, the number of daily page views for my blog suddenly spiked, and I had about four or five times my normal traffic. I thought this was just some one-day anomaly, yet the page views have continued at close to that level since then.

I first checked to see if I was suddenly getting a bunch of hits from a different country, but – as has always been the case – the vast majority of my readers still are in the US. I then wondered if it was a DDOS attack, but if so it is a very inept one. I also considered the idea that RF had offered the entity that got the $1.7MM fine a mitigation plan involving having lots of their employees read my old posts. But that would violate the Constitutional ban on cruel and unusual punishment.

I’m not complaining about this, of course, but I’d really like to know why this is happening. If anyone reading this knows why, I’d appreciate hearing from you. You can email me at talrich@deloitte.com or talrich@hotmail.com.

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte Advisory.

What I Mean when I Say “Enforceable”

A few discussions I’ve had recently with people who commented on my posts have revealed to me that I’m not being completely clear or consistent when I assert that some of the CIP v5/v6 requirements will probably not be “enforceable in the strict sense”.  I’d like to clear that up now, since I expect this theme to continue in future posts (and I’ll refer back to this post so I don’t have to explain this every time).

One of the main reasons there is confusion is that I haven’t been consistent in my wording. I have in some cases omitted the words “in the strict sense”. And I have in some cases asserted that all the requirements of v5 and v6 aren’t enforceable in the strict sense. I believe that could actually prove to be the case, but I am far less certain of the truth of that statement than I am of the narrower statement that some of the important requirements – especially CIP-002-5.1 R1 and CIP-005-5 R1 as it has to do with External Routable Connectivity – are not enforceable in the strict sense. I’m going to stick with the latter assertion for now.

What do I mean by “enforceable” in the strict sense? I mean that, if an entity receives a violation and fine due to alleged lack of compliance with a particular v5/v6 requirement and decides to appeal that violation to the US civil court system (which is allowed since CIP, like all FERC-approved NERC standards, is regulatory law), there is a substantial likelihood that the violation and fine will be thrown out.

For example, suppose you receive a violation and fine due to not having identified all the Cyber Assets that your auditor thinks you should have identified (presumably this led to your also not identifying all the BES Cyber Assets they thought you should have identified). The reasoning given is that you didn’t use an appropriate definition of “programmable” – of course, a Cyber Asset is defined by NERC as a “Programmable electronic device”. When you appear before the judge, you simply point out that there is no FERC-approved definition of “programmable”; therefore this violation is due to a mere difference of opinion between you and the auditor. I find it very hard to believe that any such case wouldn’t be dismissed.

The result of this lack of strict enforceability is that some PVs will probably never be written in the first place – one example being a PV for not using the “right” definition of programmable. No Regional Entity is interested in wasting their time writing up PVs, if there is no chance at all they will stand up if challenged in court. Thus, certain requirements are unenforceable in the strict sense.[i]

Now, what do I mean by enforceability in the general sense? This sense has nothing to do with the court system. In this sense, there is something like general agreement between NERC entities and the NERC regions about how to comply with a particular requirement. This will be evidenced by the fact that an auditor will feel comfortable writing a Potential Violation for a particular requirement, and the entity receiving it is likely to agree that the PV is legitimate – or at least to conclude that they should fight it through the normal NERC channels, not wait for it to become final and then appeal to the courts. So the question whether a requirement is enforceable in the general sense isn’t a legal one, it’s a sociological one. It’s simply a question whether there is broad agreement – perhaps in one region, perhaps across the NERC community – about the meaning of a particular requirement, regardless of whether parts of that requirement may be lacking needed definitions or may be ambiguous or contradictory in their wording.

And I further stipulate that certain violations will be unenforceable in the strict sense, but enforceable in the general sense. For an example of this, consider the fact that CIP-002-5.1 R1 never requires the entity to document a methodology for identifying BES Cyber Systems. In fact, it never requires the entity to identify BCS in the first place, although that requirement is certainly implied by the fact that all the other requirements apply to BCS. You obviously couldn’t comply with them unless you had already identified your BCS.

However, if you don’t develop and document such a methodology, you are going to have a pretty tough time at audit even though it is never explicitly required, if the auditor believes that you haven’t identified all of the BCS that you should have identified. You will have to be able to demonstrate that a) you did develop this methodology and b) you applied it consistently in your BCS identification process. The methodology you develop may differ from what the auditor would personally prefer, but as long as you can show that you put a lot of thought into developing and documenting it and that it is a reasonable interpretation of CIP-002-5.1 R1, I find it very hard to believe you will be issued a PV.

On the other hand, if you haven’t documented a methodology, or if you documented one but didn’t require that all groups within your organization use that methodology when identifying their BCS, you probably will be issued a PV. Since I believe that, if that PV became a violation and you appealed to the courts, it would be thrown out (since you haven’t actually violated any written requirement), you might be tempted to simply let the violation become final and then appeal to the courts.

However, I believe that 99.9% of NERC entities wouldn’t allow a violation to happen even if the “penalty” were a two-week vacation in France. The last thing they want is to become known as a serial NERC violator, regardless of what the actual penalties may be. Plus entities spend an enormous amount of time and money getting their legal teams involved whenever a PV appears headed to being a violation; they will do almost anything to avoid a long drawn-out fight with NERC and FERC, no matter what their ultimate chance of success will be in the courts.

This is why some requirements are unenforceable in the strict sense, but enforceable in the general sense. An entity would be ill-advised if it decided not to comply with an ambiguous requirement or an aspect of an ambiguous requirement, even though there is general agreement among NERC, the regions and the entities as to its meaning. The entity might be right in assuming it will be thrown out in court, but is it worth going through the big hassle and cost of getting that far, vs. just settling for a small fine and a mitigation plan? In some cases, the answer to this might be yes, in others no. But since no challenge to a NERC CIP fine has ever been adjudicated in the courts, I believe this indicates most entities will not go that far. This constitutes what I call enforceability in the general sense.

Another example of a class of violations that are very likely unenforceable in the strict sense, but enforceable in the general one, is PVs having to do with virtualization. It is my opinion (not universally shared, I’ll admit) that the fact that the definition of Cyber Asset is “programmable electronic device (my emphasis)” means that VMs are not Cyber Assets and thus not BES Cyber Assets (of course, in CIP version 7 I’m fairly sure this omission will be fixed, since virtualization is one of the biggest items on the menu for the new Standards Drafting Team).

So if an entity doesn’t install AV on all of its VMs that are within an ESP, will they be cited for violating CIP-007-6 R3? I believe they will be cited, and moreover they should be - although at the same time I believe that, if the entity appealed a violation to the courts, it would be thrown out. I believe most, if not all, of the regions have made it clear they will consider VMs to be Cyber Assets[ii], and NERC itself has at least once made that clear as well. No entity can argue that they properly identified their Cyber Assets if they ignored what NERC and their region were saying on this issue.

I’ve said what I wanted to about how I’m using the word “enforceability”, but I do want to point out one implication of this for the new SDT (although even if you’re not on the SDT, I’m fine if you listen in while I’m talking to them): As discussed above, there are three types of requirements[iii] in CIP v5:

1. Requirements that are enforceable in both the strict and general senses. For example, I think CIP-005-5 R2 is such a requirement, even though there are certainly cases that could be identified where a violation would be unenforceable in the strict sense. But I doubt there’s any CIP v5 or v6 requirement that wouldn’t be in that same boat.
2. Requirements that are unenforceable in both senses, like the “programmable” definition discussed above.
3. Requirements that are enforceable in the general sense, but unenforceable in the strict sense. In my opinion, any violation having to do with virtualization falls into this category.

Folks on the SDT, I believe your job is to address the second and third categories above. But I’ll also admit that doing this might require five or ten years of debate, so you will have to triage your efforts. What you need to focus on most of all is the second category. If you’re able to clear up some of these items, you will be doing the NERC community a lot of good, since these items are now the Wild West of CIP requirements; every entity is more or less on its own to determine how to comply with them.

I’ve previously provided a list of things that aren’t in the SAR but need to be addressed. I divided this list into things that are easy to address and things that are hard. The “easy” ones were mostly the third category; however, precisely because I think they are easy I still wish you would address them.

So, besides what is in the SAR, I'm recommending the SDT focus on the second category items, as well as the "easy" items in the third category. What are these second category items? The "programmable" example above is one; of course, this is already in the SAR so I know it will be addressed. I would also say items 2-6 under the section "What the V5TAG Requested" are second-category items - and they are also on the SDT's plate since they're in the SAR.

And, SDT members, if you would like to go for extra credit, I recommend you address items 25-29 in this post. These are all "hard" items that aren't included in your SAR. Unless these are addressed, a lot of CIP v5 won't be enforceable in either the strict or general sense. But I'll admit that addressing these might well add six months to a year to your efforts. Essentially, properly resolving these issues requires rewriting CIP-002 R1 and Attachment 1 - the Mt. Everest of drafting tasks.

Why should the SDT consider doing this? Because the fundamental problem in CIP v5 is the fact that the wording in CIP-002 that describes the process for identifying cyber assets in scope for the standards leaves huge gaps - and the wording that is there is confusing and contradictory (if you would like to learn more about this issue, just look at the 100 or more posts I've written on it, starting with this one). The question is whether this fundamental uncertainty about asset identification in CIP v5 and v6 will continue into v7 or not. I recommend it not be continued.

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte Advisory.

---

[i] I do need to point out, as I always do when discussing this subject, that even for requirements that are unenforceable in the strict sense, this unenforceability won’t apply in the case of an entity that has simply blown off compliance with the requirement – in this example CIP-002-5.1 R1 – in the first place. You have to have made a good faith effort to comply, which in this case includes developing and documenting your own definition of “programmable”, and documenting that this definition was applied consistently as your organization identified Cyber Assets.

[ii] It would be very hard for an entity to argue that, even though a VM might be a “device”, it still wouldn’t be a Cyber Asset since it isn’t programmable. How could a piece of software not be programmable?

[iii] When I say “requirements” here, I really mean “aspects of requirements” or something like that. CIP-002-5.1 is enforceable in some of its aspects, like the clearly implied requirement to have a BCS identification methodology. But it is unenforceable in other aspects, like its implied requirement to identify electronic devices that are programmable (i.e. Cyber Assets).

An Auditor Comments on a Recent Post

After my recent post on my dialog with Steve Parker – where the subject was whether NERC rules changes would be needed to implement the new audit regime I described in this post – an auditor emailed me the comments below:

“When we audit compliance with the CIP (or any NERC) Standards, we audit to the language of the requirement, tempered by any official guidance that may be available.  For a CIP Standard, that guidance includes the Guidelines and Technical Basis section of the Standard, the NERC CIP Lessons Learned, and the NERC CIP Frequently Asked Questions.  When we audit, we can make one of four determinations for each Requirement; No Finding, Possible Violation, Area of Concern, or Recommendation.

“A determination of No Finding does not mean the Registered Entity is compliant, just that the audit team did not find any issues of non-compliance with the evidence that was examined.  As the audit team does not examine every possible bit of evidence, we cannot make a determination of full compliance.  We strive to examine sufficient evidence to make a reasonable determination, nothing more.  The opposite of No Finding is a finding, and that is where the other three options come into play.

“A determination of Possible Violation is made when the audit team finds the Registered Entity is not compliant with the Requirement.  That could include the failure to patch a single member component of a BES Cyber System as referenced in your blog.  If a Registered Entity is expected to do something by the Requirement and fails to do so, no matter how insignificant, the Entity is non-compliant.  The determination is initially termed a Possible Violation until Enforcement reviews the facts and circumstances of the finding and either confirms or dismisses the finding.  A confirmed violation is then processed by Enforcement and can result in a number of possible outcomes, ranging from a Compliance Exception to a significant penalty, such as the $1.7 million fine that was recently reported.

“A determination of an Area of Concern is, and should be considered by the Registered Entity as a ‘fair warning.’ The audit team may write an Area of Concern for a number of reasons.  If the Registered Entity did not have an issue of non-compliance this time but the audit team is concerned that the Entity’s processes and procedures are such that the Entity may fall into non-compliance, the audit team will find an Area of Concern.  If the Registered Entity was non-compliant at some point during the audit period, but had remedied the failure and is currently OK, the audit team may find an Area of Concern as opposed to calling a Possible Violation.  Technically, the Registered Entity is non-compliant for the audit period and a Possible Violation is fully warranted.  The audit team, using its discretion, may determine to issue the Area of Concern, in effect warning the Entity to continue to manage its processes to stay in compliance. 

“The audit team may also choose to find an Area of Concern in lieu of a Possible Violation for a current issue of non-compliance, when the risk to the Bulk Electric System is so slight and the Registered Entity is reasonably expected to act promptly to remedy the concern.  Again, the audit team would be proper in finding a Possible Violation for technical non-compliance, but uses its discretion to determine a lesser finding.  This discretion is solely that of the audit team, which may consult with Enforcement before settling on the determination; the Registered Entity is well advised to not try to argue or persuade the audit team to downgrade a Possible Violation.  And, the Registered Entity should take an Area of Concern seriously.  While an Area of Concern is not processed through Enforcement, it is recorded in the audit report and does become part of the Registered Entity’s compliance history.  If the audit team finds a substantively similar issue at the next audit, a Possible Violation will likely be found if there is any issue of non-compliance during the new audit period.

“A determination of a Recommendation is exactly that, a recommendation for consideration by the Registered Entity.  Unlike an Area of Concern, where the audit team expects the Registered Entity to take some action to improve performance, a Recommendation is a suggestion for improvement.  The audit team may make a suggestion for improving the efficiency of a process, or for the implementation of a best practice that will increase the Registered Entity’s overall cybersecurity posture.  There are no down-road compliance implications if the Registered Entity evaluates the Recommendation and chooses to not implement it.  Issuing Recommendations is the audit team’s way of encouraging Registered Entities to focus on protecting their Cyber Assets and not just on compliance.

“With the onset of CIP Version 5, there is another use for both the Area of Concern and the Recommendation.  As you have pointed out on numerous occasions, there are some issues of ambiguity with the V5 Standards as written.  A number of these issues have been identified by NERC and the Regions, such as through the V5TAG Transfer Document and submitted Requests for Interpretation.  While NERC and the Regions may have a consensus on the expectations of an ambiguous Requirement, the audit team will likely find an Area of Concern in lieu of a Possible Violation if the Registered Entity’s actions differ from the expectations of the audit team.  This discretion will depend, in part, on how well the audit team believes the Registered Entity put forth a good faith effort and can justify its position.  Like most other Areas of Concern, the Registered Entity is put on notice that it may be non-compliant once the Requirement has been clarified.  The Registered Entity is well advised to closely monitor the movement to resolution and to take all appropriate action to achieve compliance as necessary by the time the issue is fully resolved.  Additionally, there are gaps in the Standards.  When an audit team finds the Registered Entity is not in Possible Violation of the Requirement as written, but there is a gap that poses a risk to the reliability of the Bulk Electric System, the audit team is expected to find a Recommendation and to document the specifics of the gap in the audit report.  As the audit reports are reviewed by NERC and FERC, these identified gaps will hopefully be scheduled for future standards development action.”

(back to Tom)

Besides providing some good information on audits, this auditor is also making a couple good points:

1. In the post referenced at the top of this one, I referred to the auditors issuing Areas of Concern to point out a way for the entity to improve its cybersecurity. I should have really called these findings Recommendations.
2. Note that the auditor says “There are no down-road compliance implications if the Registered Entity evaluates the Recommendation and chooses not to implement it.” This reinforces what I said in the previous post: that I think Steve Parker’s fears that entities will be expected to comply with Recommendations are exaggerated. But there’s no way I (or even the auditor) can say with 100% certainty that this will never happen.

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte Advisory.

The Virtualization Conundrum

When I talk about the costs of NERC CIP version 5, I have focused so far on the costs in dollars, and on my impression from talking with a number of entities that a large portion (perhaps the greater part) of the amount being spent on CIP v5 is going to compliance paperwork and other tasks, not to enhancing cybersecurity.

But there is another hidden cost to the current CIP approach, which is the fact that confusion over what the CIP requirements mean can inhibit implementation of technologies that will greatly enhance productivity at NERC entities; this is as much of a “compliance cost” as any dollars actually spent on CIP. Exhibit A for this proposition is virtualization.

Here is my impressionistic history of CIP and virtualization. In this discussion, I’m primarily referring to server virtualization, although virtual LANs and virtualized storage are probably in the same situation.

1. CIP versions 1 through 3 didn’t mention virtualization at all. However, I don’t believe there was a lot of complaining about this fact. From what I’ve heard, most entities just decided they weren’t going to virtualize any NERC CIP cyber assets – not wishing to spend a lot of effort on something that might end up getting shot down by the auditors anyway. This isn’t too surprising, considering that when CIP v1 was developed in 2006 and 2007, virtualization was nowhere near as widespread in the IT environment as it is today.
2. As v5 was developed starting in 2011, I was expecting that virtualization would have been addressed, since it was already prevalent in IT environments at the time and the potential productivity gains and hardware cost savings clearly could be substantial.
3. Unfortunately, CIP v5 (and v6) didn’t address virtualization, either. However, by now many NERC entities have gotten a taste of what server virtualization can do for them, based on their experience with IT data centers. They are anxious to use their experience to virtualize their control centers as well.[i]
4. NERC and the Regions agreed with this sentiment. After v5 was approved in late 2013, NERC started talking about developing guidance for virtualization, and even had a conference on the topic in 2014. However, nothing came of this, and in the SAR for CIP v7, NERC included virtualization as one topic the new Standards Drafting Team needed to address – thus acknowledging that there was no way to address this topic other than revising the standards.

I agree with this: There is no way that virtualization could be treated as a Lesson Learned or something like that.[ii] Since it’s not mentioned in the standards at all now, there’s no way you can examine the current wording to see how virtualization is to be treated; you’ll have to wait for v7 to get certainty. However, CIP v7 is unlikely to be in effect until three or four years from now.

At the same time, NERC and the Regions are taking pains not to discourage entities that want to implement virtualization within their ESPs. But from some informal discussions I’ve had, it seems very few entities are taking them up on doing this. This is because, even though NERC has provided a few pieces of “guidance” on handling virtualization, there is a lot more that is now simply undetermined.

To be specific, NERC has provided the following “guidance”:

• VMs that are within an ESP need to be protected with the same requirements as they would if they were physical Cyber Assets.
• It is forbidden to mix ESP and non-ESP VMs within a single host server. For example, if any VMs are BCA or BCS, the entire server (i.e. the physical box) needs to be treated as a BCA, and there can’t be any VMs running on the server that aren’t inside the ESP.
• NERC has said (although I can’t find where they said it; it must have been a draft Lesson Learned that was withdrawn) that VMs of different impact levels in a single host server must be classified according to the high water mark; that is, if there are both High and Low impact BCAs or BCS on VMs hosted within the box, all of the VMs will be at least PCAs, and will be High impact.
• In NERC’s CIP v5 Evidence Request User Guide, issued in December, NERC pointed out that, when VMs can be shared among multiple physical servers, all of the servers need to be protected at the highest impact rating of any of the VMs being shared. So if there is one High VM in a server farm, all of the servers will need to be treated as High impact BCAs, and all the VMs will have to be High impact BCAs or PCAs.

However, the gaps in v5 that NERC hasn’t addressed outweigh the three items that it has addressed. The basic problem for virtualization in v5 is that a Cyber Asset is defined as a “Programmable electronic device (my emphasis)”; and only Cyber Assets are in scope for v5. We all know that “programmable” isn’t defined in v5, but how about “device”? That isn’t defined either, but I think it has a fairly common-sense meaning. Among its other attributes, a device is something that, if you drop it on your foot, it will hurt. If you drop a computer on your foot, it will hurt; so it is a device. However, a VM is software; you can’t even pick it up, let alone drop it. Thus a VM isn’t a Cyber Asset, and therefore it isn’t a BES Cyber Asset. Q.E.D.

What this means is that VMs are officially out of scope for CIP v5 and v6, just as are any computers that aren’t within the ESP (or say the coffee maker in the control center – although some have said the coffee maker should be protected anyway, since it’s probably essential to the stability of the BES, especially late at night!). So technically, you are free to do what you want with a VM, whether or not it’s within an ESP. You can protect it according to the standards, or not. You might get a PV for – say – not patching the OS on a VM, but it is highly unlikely that PV will become an actual violation and even more unlikely that, were you to appeal a violation to the civil courts, it would be upheld.

Of course, I don’t recommend that you not protect any VMs that are within an ESP! Especially since this contradicts one of the four pieces of “guidance” that NERC has issued. But there are two problems with that “guidance”:

1. It has no official status. As I just said, this means it’s not enforceable in the strict sense of being upheld by the court system.
2. It doesn’t address the many other issues that naturally arise when an entity tries to utilize virtualization in a CIP v5/v6 ESP.

What are some of the other issues that arise when virtualization is used in a v5/v6 environment? Here are a few[iii]:

• How should remote management of VMs that are within an ESP be handled? Since the VMs are ESP devices, does the remote management need to come through an Intermediate System?
• Can ESP VMs ever be hosted outside of a PSP?
• Do purely virtual ESPs need to be monitored in the same way as non-virtual ones?
• If VMs are replicated between primary and backup control centers, does the communication medium between them need to be within a PSP?

I’m sure there are many other questions that arise. And that’s the problem: Even though entities may be satisfied with the “guidance” that NERC has provided on virtualization, there are many questions that aren’t addressed in that guidance. And if an auditor disagrees with an entity’s own answer to one of these questions, the entity will be on the losing end of that transaction! Frankly, I think this is why NERC entities are so reluctant to implement virtualization in control centers[iv].

So entities need a comprehensive guide to implementing virtualization within an ESP, but none has been provided yet. Moreover, since CIP v7 won’t be approved for probably at least three years, anything that the entity does in the process of complying in a virtualized environment will be subject to second-guessing by an auditor, when there is no guidance on the issue one way or the other.

What could be done to fix this? Well, there certainly isn’t anything that can be done officially, except for all entities to wait for v7 to implement any virtualization within an ESP. But how about unofficially? Could say one of the regions develop their own informal guide, then let the other regions adopt it as well?

I honestly think this would be the best course. I also honestly don’t believe it will happen. However, I also know that use of virtualization within ESPs (and especially control centers) would make utilities much more efficient and arguably more secure as well. It’s a shame this will have to wait for v7.

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte Advisory.

---

[i] One huge advantage of virtualization is in backup/DR. Instead of having to maintain a secondary control center filled with exactly the same pieces of hardware as the primary, and to meticulously replicate every configuration change in the primary to the secondary, the entity can simply replicate VMs from the primary to secondary whenever there’s a change.

[ii] In fact, there was never any way that almost any of the big questions about v5 could be addressed, other than through rewriting the requirements. Unfortunately, some of those big questions (and a lot of small ones) won’t even be addressed in v7. But virtualization will be addressed, at least.

[iii] All of these questions were discussed in a TRE webinar last year. The speaker mentioned that there were almost certainly many more questions that will come up as an entity gets serious about implementing virtualization within an ESP.

[iv] Of course, the benefits of virtualization in generating stations and substations are far smaller than they are for control centers.

Dialog with Steve Parker

Steve Parker, the President of EnergySec, left this comment on my recent post “On Results-Based Requirements”:

“Tom: First, thanks for the kind words about our newsletter. Second, I somewhat share your optimism, although mine is tempered by the reality of the current CMEP and NERC legal authority. The problem is that CIP audits are pass/fail performance audits. Although I like the idea of moving to assessments, the CMEP needs to be modified to account for that. I think you hinted at that in your post.

“I worry that trying to take an assessment based approach without modifying the underlying Rules of Procedure, legal authority, and CMEP could lead to unexpected and undesirable results. What if an entity ignores an Area of Concern because it was not a violation? Could it eventually become one? Would there be liability for not doing something? Could this ultimately give de facto authority to auditors that they do not currently have?

“As I said, I like the idea of assessment based approaches to entity oversight. In fact, I have advocated for that with state regulators. I just want to see it done properly.” (Steve made similar comments in the EnergySec Weekly Update newsletter that came out the same day).

My initial response to Steve (which I posted as a comment under his, although I’m expanding on it here) was quite simple: I don’t envision any of this happening because of, or in spite of, appropriate changes being made in the RoP, CMEP, etc. I see it as inevitably happening because much of CIP v5 is simply unenforceable in the strict sense – meaning that an entity that challenges a fine they receive will, in my personal opinion, almost inevitably be victorious in the civil courts if they challenge it. For that reason, and also because there are so many questions about what the requirements mean in v5 (almost all of which will never be finally resolved unless or until they are addressed in a new version of CIP, which is three years away at the minimum), I believe auditors won’t have much stomach for trying to issue a bunch of PVs over purely technical violations. Instead, they will focus on what really is important, as well as much more pleasant for them: namely, working with the entity to give them suggestions for improving their overall level of cyber security.

The caveat in this scenario is that it will only happen in the case of entities that are clearly doing their best to comply with CIP v5; so there’s nothing to be gained by fighting with them over compliance details that simply have no correct answer. For instance, let’s say the entity hasn’t patched one device that’s part of a BCS, but the rest have been properly patched. Since CIP-007-6 R2 applies to BCS, not components of BCS, have they violated anything or not? I think it’s clear that entities should interpret that requirement – and a lot of others – as applying to the components of BCS. But I personally believe there’s no way that, were the entity to be fined for this and appeal to the courts, the fine would be upheld. There are hundreds of similar examples I could drag up.

However, I had a conversation with Steve at the UTC conference in Denver this afternoon. He pointed out to me that NERC audits are supposed to be devoted to auditing; anything outside of that mandate would require changes to CMEP or the RoP. While you certainly don’t want to prohibit auditors from going beyond what CIP requires and issuing Areas of Concern for security issues that should be addressed, if this becomes a substantial part of most audits, it really needs to be recognized in those governing documents.

As Steve said in his comment, one danger is that an Area of Concern issued during one audit – for a security measure that isn’t mandated by the CIP requirements – would lead in a subsequent audit to a PV.  The way to guard against this would be to make changes to CMEP or RoP so that this or other abuses are guaranteed not to happen. So I stand corrected on this point, at least to the point that these changes should be considered.

But I don’t want the regions to suddenly pull back from their new approach and go back to the bad old days of just looking for violations (no matter how unimportant), pending revisions to those documents. The new auditing approach will help a lot toward the goal of increasing grid cybersecurity, even if at the moment the full legal authority for it isn’t in place. My personal opinion is that the auditors will do the right thing (and I know that Steve, being a former auditor himself, wouldn’t disagree with this!) and not pervert the process before CMEP and/or RoP can be revised.

But the new audit approach has to be seen as an imperfect solution to a much deeper problem: namely, that prescriptive standards just don’t work for cybersecurity regulation. I believe the solution is what I call a risk-based approach, something like CIP-014: the entity is required to get an assessment of its security threats and vulnerabilities and to put in place a plan to address those threats and remediate the vulnerabilities. The plan will most likely need to be approved by NERC, who could order revisions. The entity is then audited based on how well they have implemented the plan.

The hallmark of this new approach to CIP is that a threat and vulnerability assessment will be at its heart, just as it is in the case of CIP-014. While I don’t think that the Regional Entities have the staff to actually conduct all of the assessments, I do believe they should be required to review all of the reports, as well as the mitigation plans that result from them; they could then order changes to a plan or order an assessment be re-done if needed.

This is currently in the “I have a dream” stage[i], but – as I’ve tried to point out in the post in question – it may be hastened on precisely by the fact that (IMHO as always) CIP v5 is unenforceable in the strict sense. Since this “ultimate solution” has a security assessment at its heart, in a sense the new audit approach – discussed in the post - could be seen as a “halfway house” on the way to the ultimate solution to the problems of CIP. 

  

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte Advisory.

---

[i] Note that I’m not advocating that the new CIP v7 drafting team stop working on what’s in the SAR and just focus on this complete rewrite of CIP. On the other hand, I certainly hope v7 will be the last prescriptive CIP version. I want CIP v8 to be the new risk-based CIP!