An Auditor Responds to my Firmware Post

Judging by the large number of page views, my recent post on firmware – and the fact that firmware patches are subject to CIP-007-6 R2 – has clearly struck a chord. The day after the post appeared, I received an email from a well-known CIP auditor. As usual, this person had some very good points to make. I reproduce his email in full, with my comments interspersed in italics.

Auditor: First, firmware is nothing more than software burned onto a chip.  It might be updatable (through a process of “flashing” the PROM), or it might require replacement of the chip to implement the update.  However, to emphasize what you referenced in your blog, it is software and it is subject to CIP-007-6 until such time as an official, FERC-approved definition of Programmable Electronic Device determines otherwise.

Tom: It hadn’t occurred to me that the definition of “Programmable” – or rather the lack thereof – would have anything to do with this. In any case, I wouldn’t count on a definition coming out any time soon, and especially one that ruled out firmware updates being subject to CIP-007-6 R2.

A: Probably the biggest question is whether an update that provides for configuring and entering a complex password constitutes a security patch.  While I would dearly like to assert that it is, I cannot.  A security patch is an update that corrects a vulnerability in the affected code.  Modifying the code to allow a better password is a feature update and does not address a vulnerability.  Therefore, the patch that allows for stronger passwords is not subject to the CIP standards.  Having said that, I am not convinced that all Regional auditors share the same view, so the entity would be well advised to have this discussion with their Region sooner rather than later.

T: This point was also made by Brandon Workentin of EnergySec, in my long footnote quoting him. The bottom line is that, if you want to be safe, you need to consider a firmware patch adding a feature like stronger passwords to be a security update, even though it isn’t one from a purely technical point of view. Of course, like all CIP questions, you can always discuss this with your region. And like all questions since a few weeks ago, you probably can’t count on your region’s answer (if you get one) guiding your auditor, since your auditor may now be from FERC, not your region. In general, it’s better to take the more “hawkish” interpretation, until FERC comes out with guidance on these and a lot of other issues (and there is no assurance now that they’ll come out with guidance on any issue regarding CIP v5 interpretation).

A: And, to confirm what others have stated, you have to assess security patches for applicability.  You do not have to implement them as long as you have a mitigation plan that addresses the vulnerabilities covered by the patch.  I disagree that you have to test and determine with certainty that implementation of the patch “breaks” the system before you can choose to not implement the patch.  It is well established that outside of the traditional servers and workstations found in the Control Center, (1) updates are often cumulative, meaning you cannot pick and choose what you are going to implement, (2) vendors to this point have very frequently not identified whether a patch or update includes security fixes, (3) vendors have been reluctant to go back and try to provide information necessary to identify security patches for updates released over the past many years, and (4) even if the next released update does explicitly include a security patch, the fact that updates are generally cumulative and the vast majority of the previously released, unimplemented updates are feature releases still causes an entity to appropriately be wary of installing the update.  Extensive testing would be required and even then there is no guarantee that some feature or function change won’t be overlooked or misunderstood, introducing reliability risks that exceed whatever the security patch may be addressing.

T: The second part of this paragraph brings up something I’ve heard elsewhere: namely, that the patching process is very different once you move away from the standard IT OS platforms and go to devices like relays. As the auditor points out, the fact that the firmware updates are often cumulative, and that security updates are often not explicitly identified as such, means the entity is probably justified in being very cautious about applying even updates that are identified as being for security purposes. So having a way of mitigating the risk addressed by a security update – without actually applying the update – is especially important for non-standard OS’s.

A: The bottom line is that the entities need to make sure they are on the top of their game, that they have implemented as many defensive controls as they can regardless of whether there is a pending security patch, and that they are focused on cyber security rather than the absolute minimum necessary to demonstrate compliance.

T: This paragraph makes yet another good point: That it is important to have defensive controls (like IDS or firewall rules) that go beyond immediate compliance needs. You never know when a patch will come out that can’t be installed without impacting reliability (or for another reason). This will require you to have an alternative means of mitigating the risk, and those controls may turn out to be absolutely necessary for compliance.

T: I do have another point to make. I got an email from a compliance professional that referred to the part near the end of the post where I mentioned that firmware for video cards, etc. needs to be updated as well.  This person said:

“In line with this question I feel like there’s the potential for this to bring device drivers into scope…It could be argued that on top of checking my source for firmware updates for my network card, I also need to check my source for driver updates for the same NIC.”

I really don’t see why this wouldn’t be the case. Device drivers would seem to have the same status as firmware updates – they’re “software” updates that affect the basic operation of the system.[i] I can hear 1,000 hearts stop beating as I’ve just added yet another item to your list of “Things I Need to Address in my CIP v5 Compliance Program.”

Believe me, I’m not writing this post (and the previous one) because I’m trying to make your job harder. But that is the way CIP v5 has been ever since its approval in 2013 (two years and two days before the day I’m writing this): as entities (and auditors) get further into the implementation effort, they discover more and more “implicit requirements”. These were in theory implied by CIP v5 all along, but just haven’t been discovered until now. And guess what: These will continue, probably long after April 1, 2016.

Here’s a tip: I’m soon going to start writing about how I think CIP v5 needs to be rewritten, so we can finally get off this hamster wheel of constantly-increasing requirements, explicit or implicit. And I’ll be talking about it (although “preaching” might be a better word) at Digital Bond’s S4X16 conference in Miami Beach in January.

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte Advisory.

---

[i] I showed this comment to the auditor. He agreed with it, and had the following additional comments: “The CIP standard is explicit in that the registered entity is not expected to separately baseline drivers and other components that come with the operating system (see the discussion found in the Guidelines and Technical Basis section of CIP-010-1/2).  But, video drivers and network drivers are often provided by different vendors from the operating system vendor and are separately installed from the operating system.  In other words, these drivers are not part of the installed operating system.  So, yes, you must baseline, track, and patch the nvidia display driver and the Intel NIC driver software packages that were installed on the PC (and show up as installed, versioned software under Programs and Features on a Windows system).  This auditor, at least, is applying the same concepts to installed third-party software as apply to the operating system.  You must baseline and manage the package.  You do not have to separately baseline all of the drivers and other components comprising the package.  Again, registered entities should check with their Regions to make sure their auditors share the same view.”

Tom's Lessons Learned No. 4: “Firmware is Software”

Did you know that, under CIP-007 R2, you must evaluate patches (or updates) to firmware in the same way you track “pure” software patches? I must admit I didn’t realize this until recently, and it seems many others didn’t either. Here is how I was led to this conclusion, as well as a few other issues that came up along the way.

In my post on RF’s CIP v5 workshop in early October, I included this paragraph, discussing a presentation by Todd Thompson, one of the RF auditors:

“Todd Thompson addressed the monster requirement, CIP-007. After Todd’s presentation, a questioner asked if an entity must upgrade their firmware if the upgrade will assist with compliance – for example, it will enable longer passwords – given that firmware upgrades can sometimes lead to unanticipated issues. Todd answered that the entity needs to test to make sure the upgrade won’t cause a problem. If it does, they don’t need to apply the upgrade, but they do need to document why they couldn’t apply it.”

I must admit I didn’t think too much about the full meaning of this question when I wrote the post. I knew that a number of NERC entities were making a concerted effort to upgrade firmware on lots of devices in substations in advance of the v5 compliance date, April 1, 2016. I knew there was no requirement to have the latest firmware as of the compliance date, but I assumed the entities were doing this as a good practice, perhaps with a nudge from their NERC Region. I assumed the questioner was referring to this process.

However, one reader with whom I’ve exchanged other emails about interpretation issues in the past, Brandon Workentin of EnergySec, saw something different in there. He thought this was really a CIP-007-6 R2 question; in other words, the real meaning of the question was “I know that R2 covers firmware patches as well as pure ‘software’ ones. But am I still obligated to apply one if I believe it might lead to a reliability problem?” Firmware patches are nowhere explicitly mentioned in R2, nor are they explicitly mentioned in the Guidance and Technical Basis for R2.

He pointed out that the auditor’s answer just addressed compliance with R2.2; he was saying that, if you evaluate a firmware patch and determine it will cause a reliability problem, you need to document that finding but you don’t have to apply the patch. However, Brandon pointed out that you still need to comply with R2.3, which requires the entity to develop a mitigation plan that addresses the vulnerability the patch was meant to fix.

In other words, the questioner was probably pretty happy with the auditor’s response, since he may have thought that he had no further obligation for a particular firmware patch, as long as he could document that applying it would impact reliability. But the questioner would probably not be so happy if he had received the full answer, which is that he would still have to go through the full mitigation plan process for firmware patches, just like pure “software” patches.

This struck me as potentially a lot of work, and before I advised people that they need to treat firmware patches the same as pure “software” ones, I decided to check with some people I know on whether they agreed with this. Two of those people were auditors (different regions); they both agreed that firmware patches (although they’re usually called “updates”) are covered by CIP-007-6 R2, even though the requirement does not explicitly say that (as one auditor stated, “Firmware is software”).[i]

I also asked several entities (large and small), as well as one consultant, whether they knew that “Firmware is software”. Only one of them (the small entity) said he knew about this. He said he’d heard about it at a WECC (Western Electric Coordinating Council) meeting a year or two ago, but couldn’t remember which one. I have attended a lot of regional meetings, but I couldn’t remember hearing this at any of them. However, I did go back and read the presentation on CIP-007-6 that was given at WECC’s Advanced CIP Training in early September. While I can’t remember this being stated in the actual meeting, slides 66, 67 and 72 do say that firmware patches need to be treated like pure “software” ones.[ii]

Of course, just the fact that WECC has stated this in one or two meetings doesn’t mean that all NERC entities in the US and Canada have thereby received notification that firmware is software. In case you want to counter that by saying “Since firmware is software, the burden isn’t on NERC or the regions to notify people of this; they should have known that”, I wish to counter you by pointing out that Brandon did an excellent analysis (see the end note[iii]) of the wording in the Guidance and Technical Basis. His analysis indicates that even the Standards Drafting Team members who wrote the Guidance didn’t seem to always understand this. I think it can quite legitimately be argued that NERC entities have not been sufficiently made aware that they should treat firmware as software under CIP-007-6 R2, for them to be held to strict compliance with that particular “implied requirement” on April 1, 2016.  Even if NERC (or FERC) comes out with guidance on this issue tomorrow, I think it's too late for entities to be expected to be in compliance on that date.

Before I go, I’d like to point out three other items that came up in these discussions:

First, for both pure “software” and firmware patches (or “updates”), CIP-007-6 R2 requires the entity to distinguish security patches from non-security ones; you only need to deal with the former. However, as one of the auditors pointed out to me, these security patches are usually identified as such by the vendor releasing them, but not always. So for a vendor that doesn’t distinguish between the two patch types, it is important for the entity to read the description of every patch released to see if it is a security patch or not.

Second, at GridSecCon I met Monta Elkins of FoxGuard Solutions (who gave an excellent presentation on patch management). He pointed out to me that servers and workstations will usually have different pieces of hardware – video card, I/O controller, etc. – that are made by different vendors. Does this mean you have to inventory each component of each server or workstation and monitor each component vendor for firmware patches? This would lead to a huge effort.

I think the answer to that issue, and Monta agreed with me, is you should confirm with the server or workstation vendor that they will provide firmware patches for any of the third-party devices in your system. This way, you still just have to monitor one patch source for all of those patches.

Third, in Brandon Workentin’s analysis (in end note iii below) of the Guidance and Technical Basis for CIP-007-6 R2, he pointed out another issue on which the Standards Drafting Team left contradictory guidance. The first paragraph of the R2 guidance reads:

“The SDT’s intent of Requirement R2 is to require entities to know, track, and mitigate the known software vulnerabilities associated with their BES Cyber Assets. It is not strictly an ‘install every security patch’ requirement; the main intention is to ‘be aware of in a timely manner and manage all known vulnerabilities’ requirement.”

I, like probably most people, didn’t notice on first reading this that it raises a big problem, but after Brandon’s email it stands out to me like a sore thumb: My understanding (and I think most others’) was that R2 requires entities to evaluate security patches and make sure the vulnerabilities they address are mitigated, either through applying the patch or (if the patch can’t be applied) through alternative measures (described in the mitigation plan).

But here the SDT is saying something quite different. They’re saying that the entity is responsible for knowing, tracking and mitigating all of the “known” software vulnerabilities associated with BES Cyber Assets! How do they do this? Clearly, just considering all patches that are released isn’t sufficient, since there are many “known” vulnerabilities – in probably most software products – that are never patched, either because the software vendor doesn’t consider them serious vulnerabilities, or because they have determined that the cost of patching them is too high.

This wording seems to imply that entities must continually scan all of the many sources of vulnerability information for any mention of a vulnerability in any software or firmware that’s installed anywhere in their ESPs. When they find mention of a vulnerability they didn’t know about, they need to first see if there’s been a patch released that addresses it. Presumably, if a patch has already been released, the entity has either been patched or mitigated the vulnerability, through complying with R2. However, this can’t automatically be assumed.

But what if no patch has been released? According to this wording, the entity is still responsible for mitigating any “known” vulnerability. And how would your auditor determine whether or not you had mitigated all of those? Will NERC scan all of the literature and publish a continuously-updated list? Clearly, that’s not going to happen. Also quite clearly, R2 is for patching, not for vulnerability management.

I think it’s safe to say you don’t need to think that your CIP v5 compliance burden has just taken another significant leap up, and that you are now responsible for assessing all vulnerabilities, as well as patches. As everywhere in CIP v5, only the language of the Requirement is auditable, and that clearly states that this is a requirement for patch management, not vulnerability management. But it’s quite striking that the Guidance could be so very off base in this case.

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte Advisory.

---

[i] One of the auditors pointed out that CIP-010-2 R1.1.1 says configurations need to be tracked for “Operating system(s) (including version) or firmware where no independent operating system exists”. Of course, this doesn’t necessarily mean that firmware needs to be covered in CIP-007 R2. It also indicates that firmware configurations do not need to be tracked when the device runs an O/S.

[ii] Slide 91 also mentions that a device’s lack of External Routable Connectivity, while not getting the device in question off the hook for compliance with R2, will make the compliance burden for that device easier. This is because some patches may simply not be applicable to it, due to the device not having ERC.

[iii]This is the analysis Brandon provided, to show that the Guidance and Technical Basis for CIP-007-6 R2 seems to be confused as to whether or not firmware patches are in scope for the requirement:

·       R2 exclusively uses the term "security patches." There are various places this term is attempted to be defined, but using the definition from those various places can lead to differing conclusions.

·       Guidance and Technical Basis for Requirement R2 ¶ 1: "The SDT’s intent of Requirement R2 is to require entities to know, track, and mitigate the known software vulnerabilities associated with their BES Cyber Assets...the main intention is to 'be aware of in a timely manner and manage all known vulnerabilities'"

o   (Let’s take the example of a firmware patch that addresses security, such as increasing the password length.)  Password length functionality isn't generally considered a "vulnerability." This would imply that a firmware update which adds functionality, even if that functionality is security-related, wouldn't be applicable. A firmware update which removed/addressed a vulnerability, however, would. (An example of this is if an undocumented admin account were hard-coded into the firmware. When that happens and is discovered, those are generally addressed as vulnerabilities in the sense that they get an ICS-CERT alert or something similar.)

·       G&TB §2.1 ¶ 1: "The requirement applies to patches only, which are fixes released to handle a specific vulnerability in a hardware or software product."

o   Again, is the inability to have long passwords a vulnerability? I would say no. It would qualify as an "insecure-by-design" feature, but I don't think it's a "vulnerability."

o   Tom’s comment: Here, the SDT seems to move away from what they said about the focus of R2 being vulnerabilities. I would take this as the more authoritative statement, not the quote from the first paragraph of the Guidance that I cited in the last section of the post.

·       (A major software vendor defines “vulnerability” as) “a security exposure that results from a product weakness that the product developer did not intend to introduce and should fix once it is discovered." They formally define it as, "a weakness in a product that could allow an attacker to compromise the integrity, availability, or confidentiality of that product," and state, "Security vulnerabilities involve inadvertent weaknesses; by-design weaknesses may sometimes occur in a product, but these aren't security vulnerabilities."

o   Tom’s comment: I can’t believe that the vendor is saying that the same piece of code would be a vulnerability if not intended, but wouldn’t be one if intended!

·        G&TB §2.1 ¶ 1: "The requirement covers only patches that involve cyber security fixes and does not cover patches that are purely functionality related with no cyber security impact."

o   A firmware update which allows for longer passwords or more complex passwords pretty clearly has a "cyber security impact" (even though it would technically be a functionality update). If you ignore that this uses the term "patches" again, this sentence would imply that such a firmware update would be "in-scope" for the requirement.

·       G&TB §2.1 ¶ 1: "The National Vulnerability Database, Operating System vendors, or Control System vendors could all be sources to monitor for release of security related patches, hotfixes, and/or updates."

o   This one pretty clearly states that "updates" would need to go through the R2 process.

·       G&TB §2.1 ¶ 1: "A patch source is not required for Cyber Assets that have no updateable software or firmware (there is no user accessible way to update the internal software or firmware executing on the Cyber Asset)"

o   This part pretty clearly says that updatable firmware needs to be tracked, if possible.

·       G&TB § 2.2 ¶ 1: the first several sentences all use the term "patch," which as discussed above seems to have only a limited definition in this document.

·       G&TB § 2.2 ¶ 1: "Considerable care must be taken in applying security related patches, hotfixes, and/or updates or applying compensating measures to BES Cyber Systems or BES Cyber Assets that are no longer supported by vendors."

o   Again, uses the term "updates" and assumes they are included in the R2 process.

·       G&TB § 2.3: This section again uses "patch" and "vulnerability," so it implicitly includes the limitation those terms imply.

Whistling Past the Graveyard

On November 10-12, 2015, I attended the Northeast Power Coordinating Council’s (NPCC) Fall Compliance Workshop near White Plains, NY. It was a very good workshop, with good presentations on both CIP and more general NERC compliance issues such as the Reliability Assurance Initiative (or Risk-Based Compliance Monitoring Enforcement Plan, as it’s now called). I plan to have one or two more posts on that conference. This post discusses one interesting thing I noticed at the conference.

First, I’ll say there was a lot of discussion – both by the speakers and in informal conversations among participants – of the fact that FERC has announced that it will start auditing compliance with CIP v5 and with CIP-014 next year. It appeared that almost everyone was in agreement that the full implications of this announcement may not be known for a while; I have no argument with that idea.

However, there also seemed to be general agreement that this probably will not be such a big deal. I didn’t hear a single entity say they were going to start doing things differently because of the announcement; I also didn’t hear any of the speakers say that things were likely to be very different.

I’ll be blunt: There was a lot of skepticism that FERC really has the manpower and the industry knowledge to pull this off.[i] While I know they have some really top-notch cyber security professionals on their team, I also don’t believe they have the staff today to start doing a number of audits at once - although this partly depends on what you mean by “audit”. NERC’s model includes about 90 days of offsite document discovery and say 1-3 weeks of onsite audit. I’m told that FERC’s audits can – and often do – take years, and the entity being audited can go for months without ever hearing from their auditors. If FERC decides to use their traditional model, they actually could conduct a number of simultaneous audits without a huge staff increase. But I also don’t doubt they could get a lot more audit staff if needed. Remember, they likely won’t be doing any v5 audits until next fall; they certainly aren’t going to come knocking on your door on April 2, 2016. 

But the general feeling that things aren’t really going to be too different under FERC went beyond issues of staffing. I feel it was due to the very human reaction to any potentially big news that isn’t immediately accompanied by a change in circumstances. When something big happens far away – say, the stock market crash of 1929 – we cling to the idea that its full impact isn’t known, and we gravitate to the best possible interpretation of what might happen. Of course, just the news that the stock market had crashed didn’t cause any immediate change to the majority of US citizens in 1929, unless they owned significant stock holdings. The reaction of some was glee: “Those guys had it coming.” It was a couple years later, as the banks started failing and unemployment climbed, that there was no denying things had drastically changed.

FERC has confirmed that they will be doing auditing, but they’ve said nothing more. It’s natural to simply assume the best outcome will occur: That they will do a few audits, but just of the “really big guys” (of course, if you work for a “really big guy”, this isn’t much comfort). The regions will continue to do most of the audits, and their approach won’t really change from what it is now.

Folks, I beg to differ. I think FERC is under a lot of pressure today – mainly from their bosses, the US Congress – to crack down on what is perceived to be a lax attitude toward cyber security on the part of the electric power industry. Why do I think that? Just the fact that FERC is going to be doing this auditing, and that it is a big change from the past, makes me believe this isn’t some idle whim of the Commissioners. I’m sure they thought all of this out before making their move. And I’m sure they know how they are going to get the staff they need to handle the audits.

If you want to experience firsthand the pressure FERC is under, I recommend you pick up Ted Koppel’s new book, Lights Out. Whatever your opinion of the book may be, it is going to have a big influence on the public. I doubt the average man on the street has heard of cyber attacks on the grid, except possibly from Hollywood.  But that will be different now. In fact, Congressmen and women will read the book and try to jump ahead of their constituents by demanding changes.

I don’t want to exaggerate the influence of Ted Koppel’s book by itself. The point is that pressure on FERC is coming from a lot of directions and is growing, not diminishing. To expect them to back away from the audit idea due to not having the expertise or the manpower is very dangerous. If this is what we’re thinking, we’re just whistling past the graveyard.

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte Advisory.

---

[i] I’ve also heard a couple people question whether FERC even has the authority to conduct these audits. Rest assured, FERC has that authority, although they haven’t been exercising it too much in the past. They are the regulator, not NERC.

“Low Impact 1500+MW Plants” and other Fairy Stories

As almost everyone knows, the most important category of plants that can potentially be Medium impact under CIP v5 is those plants that meet Criterion 2.1 of Attachment 1 of CIP-002-5.1. These are, without exception, huge plants with lots of potential BES Cyber Assets, and to fully comply as Medium impact will be very expensive.

Fortunately, there is a provision in 2.1 that allows plants to “segment” themselves so that no BES Cyber System can impact more than 1500MW of capacity. This provision was also in CIP v4. I have heard some refer to it as a “loophole”; but it’s really not, as I’ll show now.

Any plant this size will have multiple units of say 3-600MW each. Let’s say that each of those units were set up as its own plant with a fence around it. Clearly, none of these smaller plants would be Medium impact (unless they met another Medium criterion like 2.3). Let’s say you take out all the fences around the individual “plants” and instead run one big fence around all of them. Voila! All of a sudden they turn into one plant of greater than 1500MW, with completely segmented units (of course, this is a bit of an exaggeration, but not by much). Without the special provision in 2.1, all of their BCS would become Mediums, just because the fence was changed. This is clearly not fair, and it is why this provision is in there. Having a multi-unit plant with complete segmentation is not—in principle—any different than having each unit be its own plant. The risk to the BES of one unit (or small plant) being lost is the same in either case. The same generation capacity is lost.

But I’m not writing this post to justify Criterion 2.1. I’m writing it because almost every compliance person from a generation entity I have ever heard talk about this says that, by segmenting a 1500 plant, they change it from being Medium to Low impact. In fact, I have heard NERC staffers say the same thing. The problem with this is it is wrong. The plant doesn’t cease to be Medium impact[i] when you segment it. Rather, it ceases to have Medium BCS, or as 2.1 says, BCS that “could, within 15 minutes, adversely impact the reliable operation of any combination of units that in aggregate equal or exceed 1500 MW in a single Interconnection.”[ii]

Of course, the result is substantially the same, no matter which way you say it: You will have a large plant with no Medium BCS, only Low BCS. Since a Low asset is “defined” as one that contains Low BCS, the plant will need to be listed on your list of Low assets. But most people say the plant will itself become Low impact when it no longer has Medium BCS. This is wrong. It is still Medium impact, and will remain so until its total output capacity drops below 1500MW. Is this just an academic question that doesn’t make a difference for your compliance program? Hardly. I can think of two areas where it does make a difference, one not-so-profound and the other profound.

To address the not-so-profound area first, I call your attention to Criterion 1.4 of Attachment 1 of CIP-002-5.1, which says “Each Control Center or backup Control Center used to perform the functional obligations of the Generator Operator for one or more of the assets that meet criterion 2.1, 2.3, 2.6, or 2.9.” In other words, if you have a plant that meets criterion 2.1, a control center that controls that plant will be High impact. Does it make a difference if you’ve segmented the plant and you don’t have Medium BCS there? No, it doesn’t. The Control Center is still High impact.[iii]

Now the profound reason: One appeal of having an asset be Low impact rather than Medium is that the work required for coming into compliance, as well as what is required to maintain compliance, will be much less. This is primarily because CIP-002-5.1 says in two places that there is no obligation to inventory Low impact BES Cyber Systems.

But this statement doesn’t apply in the case of a 2.1 plant that has all Low BCS because it has been segmented. Again, since the plant is Medium impact, it is up to the entity to demonstrate that the BCS have all been relegated to Low status; moreover, the entity has to do that every 15 months to comply with CIP-002 R2.

How do you show there are no Medium BCS? It will take network diagrams that show no single network, if completely brought down, would affect more than 1500MW. It also will take engineering diagrams for the physical systems, to show they don’t affect more than 1500MW. Both of these types of documents will have to be updated and made available each year, to demonstrate that no changes have been made that would create any Medium BCS.

But you also can’t rule out having to identify your Low BCS at the plant, which means going through the same process as for Medium and High impact assets: identify Cyber Assets (including documenting your definition of “programmable”), identify BES Cyber Assets (including documenting how you are interpreting “adverse impact” on the BES), group these into BCS, etc. (a light version of my methodology is available here).

If an auditor doesn’t think your network diagrams are comprehensive enough – and in large coal plants there are sometimes thousands of devices that aren’t linked by IP and may not be on network diagrams in the first place – they might require your BCS list. And remember, you can’t point to the statement about an inventory of Low BCS not being required. Your BCS will be presumed Medium until you prove them to be Low.[iv] And you will need to do that every year.

Of course, I still feel it is worthwhile for most entities with 1500+MW plants to make the effort to segment their systems. It’s a good security and reliability practice as well.

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte Advisory.

---

[i] Of course, strictly speaking there is no such thing as a Medium plant, Medium substation, High Control Center, etc. While CIP-002-5.1 R1 and Attachment 1 are contradictory, the wording favors the idea that the criteria apply to BES Cyber Systems, not to assets. There are Medium and High impact BCS, but not High or Medium assets (on the other hand, there are Low assets as well as Low BCS, but the latter are deliberately not dealt with by the requirements, at least not at the moment). As I’ve stated repeatedly, I am unaware of anybody complying in this way or of the regions teaching compliance in this way. Everyone I’ve talked to is following the approach of first identifying High, Medium and Low impact assets, then classifying the BCS according to the asset’s classification, despite the fact that the words of the requirement don’t read that way. I discussed this situation in a recent post, and others before that.

[ii] You may think the words “in a single Interconnection” are fairly fanciful. After all, how many single plants serve more than one Interconnection? Well, I know of at least one, in Wyoming. Some of its units serve the Western Interconnection; others serve the Eastern Interconnection.

[iii] This makes sense, of course. The Control Center presumably controls the whole plant, not just one unit. If the Control Center is compromised by somebody or some system with ill intent, the plant could presumably be brought down in its entirety, whether or not it is segmented.

[iv] I may be making the auditors sound tougher than they really will be; they may not require all of this. But they could require it, and since it’s very possible that FERC may be your CIP v5 auditor, you should not rely on your previous experience with your region’s auditors. You should be prepared for FERC – although it’s possible that, when FERC’s official announcement comes out, they will give an idea of who might be audit targets and who wouldn’t be. On the other hand, they may decide it’s more effective to leave every NERC entity wondering who they’ll be audited by. This may be great for sales of antacids, but not so great for the sanity of CIP compliance professionals.

FERC Confirms It

On November 4, 2015, EnergyWire ran a story[i] quoting a FERC spokesperson as confirming what I reported in a recent post – that FERC will be conducting CIP v5 audits for some NERC entities, in place of the NERC region. To quote[ii] from the story:

FERC's decision, which has not been officially announced, was confirmed yesterday by Mary O'Driscoll, commission director of media relations, in response to a query. FERC and NERC have concurrent authority to audit and enforce cybersecurity standards. "However, historically, FERC has exercised its audit authority in only a few rare cases," O'Driscoll said.

"FERC is conducting these audits because CIP v5 is a major change in applicable requirements in a very important area of the standards," she said. FERC's action was not motivated by concern over the quality of audits by the regional entities, she added.

The same article states:

Several other regional organizations have also reported the change. Kim Israelsson, compliance program coordinator for the Western Electricity Coordinating Council (WECC), sent an email this week noting that FERC would conduct "limited monitoring activities" on the CIP5 rules. WECC is the regional organization for 14 Western states and parts of Canada and Mexico.

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte Advisory.

---

[i] Since EnergyWire is a subscription-based service, non-subscribers will need to sign up for the free trial to read the story. In the interest of full disclosure, I want to point out that the article does quote liberally from my original post on this story.

[ii] Quotation is with permission of EnergyWire.

What I think FERC Needs to Do – Near Term

Last week, I posted that FERC will start auditing some entities for compliance with the NERC CIP version 5 standards and also CIP-014 – and since then, I’ve been thinking about what this means. It didn’t take much thinking to realize this will have a profound impact on all NERC entities that need to comply with the CIP standards. This post sets out steps I believe FERC needs to take in the near term to address the primary problems with CIP v5.

First, it’s important to know that FERC’s move will impact all NERC entities with CIP v5 obligations, even though it is clear they won’t be auditing all, or even the majority of, entities. This is because I believe the mere fact that FERC may be their auditor will reset the bar for compliance for most entities - since it is a fair assumption that FERC’s auditors will be at least as strict as the Regional auditors, and often more strict. All entities will now need to rethink their compliance programs based on the assumption that FERC will be their auditor, not their Region (or perhaps both, since FERC could conceivably just do one audit for an entity, while leaving other audits to the Region. Hopefully, this is one issue that will be addressed when FERC makes their announcement of this program).

FERC and the entire NERC community are aware that compliance with the CIP version 5 standards is due starting April 1, 2016 – i.e., less than five months from now. FERC is also undoubtedly aware that there are a number of serious interpretation issues with CIP v5 that haven’t yet been addressed by NERC, and are very unlikely to be addressed in any definitive way before April 1.

Moreover, even if these issues are addressed before April 1, it may be too late. This is because the big CIP v5 interpretation problems are all found in what I call the fundamental requirements of v5 – the requirements that determine what cyber assets are in scope for the standards, as well as whether or not they have External Routable Connectivity (the concept of ERC doesn’t determine whether or not a cyber asset is in scope for CIP, but it does determine how many requirements are applicable to it). Since everything else that needs to be done for v5 is based on how the entity complies with these fundamental requirements, it is very likely not possible that an entity of any size can re-engineer its compliance program now and still meet the 4/1/16 date, even if definitive guidance comes out tomorrow on all of these problems.

This means FERC will be auditing NERC entities for CIP v5 compliance, knowing they haven’t had clear guidance on the meaning of the fundamental requirements in question. These include CIP-002-5.1 R1 and Attachment 1 (and the definitions that are integral to R1, including those of Cyber Asset, BES Cyber Asset and BES Cyber System), as well as the different requirements that apply to BES Cyber Systems with ERC.[i] Needless to say, this is a problem.

How will FERC deal with this problem? I can think of several ways:

1.       FERC could, in effect, say to the entities “Too bad you didn’t have clear guidance on these issues when you needed it. But you’re still in violation of the requirements as we interpret them.” I don’t think this will win FERC too many friends; nor do I think they want to take this approach.

2.       Since the NERC Regional Entities have provided some guidance to their members – either in public meetings or in private written correspondence – FERC can say they will audit to that guidance, as long as it is documented.[ii] The problem here is that both NERC and the regions have also provided guidance that isn’t documented in the Small Group Advisory Sessions (SGAS), as well as in phone conversations and at meetings.[iii] What will FERC do when an entity swears up and down that they were told during their SGAS that procedure X is a good way to comply with requirement Y, yet FERC (meaning the audit team) doesn’t agree with this interpretation? Does FERC accept the entity’s word, or do they fall back on the approach described in the previous bullet point? Frankly, I don’t believe either option would be acceptable to FERC.

3.       FERC can put out their own “interpretations” of these issues. For example, they can come out with their own “definitions” of “Programmable” and “adverse impact on the BES”; they can provide their own methodology for complying with CIP-002-5.1 R1;[iv] they can provide their own interpretation of External Routable Connectivity;[v] etc.

I don’t think it will shock you to hear that I think the third method is the only workable one. But there is a problem with that as well: Given that FERC will be changing the fundamental rules of the CIP v5 game less than five months before the compliance date, they simply can’t expect entities to be compliant, based on the guidance FERC does put out, by 4/1/16. I think FERC will need to postpone the date on which v5 is enforced in some way.

I have suggested one way this could be done, which is to have an “enforcement date” for v5 one year after the “compliance date”. Since I was fairly sure that wouldn’t happen, I had recently pointed out that the “effective enforcement date” for v5 would be postponed regardless of any official action; this refers to the date that the auditors will actually feel comfortable issuing PVs for non-compliance. I think the effective enforcement date will be after 4/1/16 because I believe auditors are unlikely to assess violations for requirements where there is fundamental ambiguity – assuming the entity has done all it could to come up with its own interpretation of the ambiguous areas, while considering all available guidance from NERC and its region. 

However, I’m certainly not ready to make this assumption about FERC auditors; they may feel they have to issue violations whether or not there is ambiguity in the requirement. As I said earlier, even if FERC were to release tomorrow a complete set of interpretations of all the ambiguous requirements and definitions in CIP v5, it may be too late for entities to revise their v5 programs to take advantage of those interpretations. This means FERC needs to give them more time for compliance, whether through some formal Order or more likely through an informal understanding. I don’t know the exact amount of additional time that will be required for entities to come into full compliance after 4/1/16 due to the ambiguity in the requirements; I’d say it’s at a minimum six months, and probably closer to 12.

Note that I’m not saying the compliance date needs to be pushed back for all of the CIP v5 requirements; just those that are ambiguous enough that they require guidance – including the requirements discussed above. To give an example of how this might work for the “unambiguous” requirements (say for example the requirements of CIP-009-6, Recovery Plans for BES Cyber Systems), I would say the 4/1/16 compliance date can stand, provided FERC stipulates during an audit that the entity has correctly complied with CIP-002-5.1 R1 and identified and classified its BCS, including those with and without ERC. Once the “Enforceable” date has been reached for CIP-002-5.1 R1 and other “ambiguous” requirements, this stipulation can be removed and FERC can issue PVs for not identifying BCS properly.

As FERC probably realizes, what I’ve described above are short-term measures, designed to allow CIP v5 to be rolled out without holding back enforcement until it can be rewritten (since that will take years). In the second post, I will discuss the longer-term measures that are required to have a sustainable NERC CIP program. You may find what I say in that post to be surprising.

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte Advisory.

---

[i] There are certainly other areas of ambiguity in CIP v5. These will also have to be dealt with, but they don’t affect all of the other requirements like these “fundamental” requirements do.

[ii] Of course, NERC has provided a number of guidance documents, including the Lessons Learned, FAQs and the Memoranda. Since most of these are still in draft form or have been withdrawn, and the few that have been finalized don’t address the fundamental issues I’m referring to, I really can’t say that NERC has provided documented guidance on these issues.

[iii] The other problem is that even the documented guidance is almost always prefaced with some statement that this is just the opinion of the individual providing it, not of the Regional Entity. This of course is necessary since according to the NERC Rules of Procedure, the only definitive “guidance” on the meaning of a requirement is what is provided through the Request for Interpretation (RFI) process. An RFI accepted today will likely take at least a year (and I’d guess more like two years) to turn into an official, NERC and FERC-approved Interpretation. But guidance on CIP v5 obviously can’t wait until a couple years after the enforcement date.

[iv] As I have been saying regularly, but most recently in this post, the problem with interpreting this requirement is that there are so many ambiguities and contradictions that no finite methodology could ever be written down, that both could be followed and would be consistent with the words of the requirement (and Attachment 1). However, there is a methodology that currently guides how virtually all NERC entities are complying with the requirement. I summarized that methodology in five steps near the beginning of the post just referenced. I admit that was an over-simplification and there are probably more like ten steps. I will do a post in the (reasonably) near future that will outline what I consider a complete description of this methodology, which I’ll call the “effective” CIP-002-5.1 R1 compliance methodology. I highly recommend that FERC’s interpretation of R1 follow this methodology. After all, it’s the one that the entities are using and the regions are teaching, even though the methodology strays pretty far from the actual wording of R1 and Attachment 1. There might be a methodology that could actually be followed that would be closer to the wording. But to try to introduce that at this point – and essentially require all entities with High and Medium assets to go back to square one with their entire CIP v5 program – would be disastrous, in my opinion.

[v] Of course, I and many others think the discussion of Low-impact External Routable Connectivity (LERC) in FERC’s NOPR on CIP v6 actually constitutes an interpretation of the meaning of ERC as well. However, FERC needs to put out a document explicitly addressing ERC.