An (Impressionistic) History of NERC CIP

A friend – who probably knows a lot more than I do about NERC CIP, but came into the game later than I did - asked me recently to write a post on the history of CIP. I’m finally doing that, although I’ll say at the outset that this isn’t meant to be your standard Wikipedia article. It’s focused on what I know personally, and you have to be very careful on what you take away from this post as facts vs. opinions. That being said, I hope you’ll find it interesting. I was originally planning on calling this a “brief” history of CIP, but as you can see the brief part fell by the wayside (I've never been very good at brevity, anyway). You hereby have my permission to read this post in two or three sessions. I probably won’t be able to write another post this week, anyway.

The origins of CIP were long before I became involved with it (2008). I’m told there were some sort of market access standards (from FERC) that required cyber security measures around 1999 or 2000, but I’ve been unable to verify that in my extensive five-minute search of the internet. I have to assume this is actually some sort of creation myth dreamed up by pre-(cyber-)literate peoples as they chewed on mastodon bones while sitting around campfires.

NERC cyber security standards first enter the historical record around 2003 with NERC Urgent Action 1200, aka NERC Cyber Security Standards (CSS); like all NERC standards before passage of the Electric Power Act of 2005, compliance with them was voluntary. The standards applied only to control centers. They were later replaced by UA 1300, which now applied to generation as well as control centers.

After passage of EPAct, the UA 1300 standards were submitted to FERC as CIP-002 through CIP-009 in August 2006. After issuing a NOPR (Notice of Proposed Rulemaking) in July 2007, FERC approved these standards as CIP version 1 in January of 2008 in their landmark Order 706.

You’re free to read all 221 pages of Order 706, but the highlights of the order are:

1. FERC approved the CIP v1 standards.
2. At the same time, FERC ordered NERC to make “revisions” to these standards. Since NERC’s Rules of Procedure don’t allow any standard to be revised once approved by the NERC Board of Trustees, and since all standards are approved by the BoT before even being submitted to FERC for their approval, these changes would have to be in a new version of the standards, version 2. Accordingly, NERC constituted a new drafting team, dubbed CSO 706 for Cyber Security Order 706; the team first met in the fall of 2008.
3. IMHO, the primary changes FERC requested in Order 706 were:
1. The CIP v1 drafting team had used the phrase “reasonable business judgment” in a number of places in the standards, to indicate that certain questions were left up to the entity to decide. FERC didn’t like this language, and ordered it removed.
2. The team had also used the phrase “acceptance of risk” in the standards. This is a common phrase in cyber security standards and guidelines in general. It means that the entity, in cases where mitigating a particular cyber risk may be too difficult or costly, agrees to instead accept the risk and move on. FERC pointed out – correctly, in my opinion – that the risk involved in the CIP standards, as well as all NERC standards, is risk to the Bulk Electric System, not to the organization itself. No organization can decide to accept risk on behalf of the whole BES.
3. The drafting team had also used the words “where technically feasible” in multiple places, meaning that, if a particular requirement couldn’t be complied with for a particular type of device in scope (for example, a switch that couldn’t comply with the antivirus software requirement because there was no way for the end user to load such software on the switch), the entity would only need to point out that fact at audit. However, FERC ordered NERC to “develop specific conditions that a responsible entity must satisfy to invoke the ‘technical feasibility’ exception.” This led to NERC’s later development of the TFE process, which probably shortened the lives of a number of CIP compliance professionals by 5-10 years.[i]
4. The Commission directed NERC to “provide additional guidance regarding the development of a risk-based assessment methodology for the identification of critical assets pursuant to CIP-002-1.” They were already skeptical that NERC entities’ use of the RBAM, as specified in CIP-002-1 R1, would end up identifying many Critical Assets, and they were right. Other than Control Centers, very few generating stations were identified as Critical Assets under CIP v1-3. There were a number of substations that were so designated, but that designation was usually obviated by the fact that, in CIP v1-v3, if there was no external routable connectivity at an asset, it wouldn’t have any Critical Cyber Assets – meaning no systems would be in scope at the substation. Because the great majority of substations at the time were not routably connected to the outside world, there were very few substations with CCA’s. This meant that there were very few substations that were effectively in scope for CIP, even though they might have been designated Critical Assets.
5. FERC also ordered “external review of critical asset lists…” FERC didn’t directly tell NERC what they were asking for in this regard, but NERC did develop the “bright line criteria”, which first appeared in CIP v4. The criteria did away with the RBAM altogether and simply specified what types of assets would become Critical Assets, based on various technical conditions. When the CSO 706 drafting team started work on CIP v5, they modified the criteria so that they dropped the reference to Critical Assets, replacing it with High, Medium and Low impact levels.[ii]

When the CSO 706 SDT started meeting in 2008, they decided to triage the directives from Order 706. They would first address the “low-hanging fruit” in CIP v2, then address the rest of the directives in v3. So they immeidately developed CIP v2, which eliminated both the “reasonable business judgment” and “acceptance of risk” language from the v1 requirements. V2 was quickly balloted and passed, approved by the NERC Board, submitted to FERC and finally approved by FERC in September 2009.

When FERC approved CIP v2, they brought up a new requirement – for procedures for escorted physical access into Critical Assets by persons not authorized for direct access to Critical Cyber Assets - that they wanted to see added quickly; in fact they required that the new CIP version be approved and submitted to them in 90 days. NERC accomplished this feat, and FERC approved CIP v3 in March of 2010.[iii] It came into effect that October. 

Once the CSO 706 SDT had drafted CIP v3, they turned their attention to CIP v4. They had a radically different idea for v4, though: They would just have two standards. CIP-010 would replace CIP-002, and it would provide a new way of identifying assets in scope for CIP: through bright-line criteria. Also, those assets would no longer be classified as Critical Assets or not in scope; they would be classified High, Medium and Low impact. Furthermore, the systems in scope would be called by a new term, BES Cyber System. All of the requirements in standards CIP-003 through -009 would be combined into CIP-011, and there were many changes proposed for these requirements. The first drafts of these two proposed standards were developed in early 2010; the SDT expected to be able to get these two new standards approved by NERC by the summer of 2010.

This radical proposal led to a firestorm of suspicion among NERC entities, and 600 people signed up to attend as observers – in person or by phone – for the SDT’s next meeting, in Dallas in April 2010. At that meeting these observers weren’t exactly shy about registering their profound concern.[iv] It was clear that the SDT's proposal would never be approved in anything like its current form.

After this meeting, the SDT retreated to their phone meetings to decide what to do; they clearly couldn’t go ahead with the CIP-010/-011 proposal now. One of the big objections to the proposal might seem pretty mundane, but really wasn’t: By changing the numbering of the standards so drastically, NERC entities were going to have to make some huge changes in computer systems as well as other things like training programs, which were all based on the CIP-002 through CIP-009 scheme.

In the aftermath of the Dallas meeting, some CEOs of large utilities became convinced that, if NERC didn’t develop some new CIP version by the end of 2010, FERC would completely take away from NERC the task of developing new CIP standards. So somebody developed the idea of just replacing CIP-002-3 with a new CIP-002 that would implement bright-line criteria for identifying Critical Assets, but leave everything else in v3 (including identification of Critical Cyber Assets in CIP-002 as well as CIP-003 through -009 in their entirety) unchanged. This would address FERC’s primary concern, which was that too few assets had been designated Critical Assets, due to the looseness of the criteria for developing a risk-based assessment methodology in CIP-002-3 R1.

And this is what happened. The SDT got to work[v] adopting the bright-line criteria to CIP-002 (and making some changes to them from the original CIP-010 draft), but didn’t touch any of the other standards (except to change their version numbers, so all of the new draft standards were version 4). The final ballot approving CIP v4 occurred at the end of 2010, and v4 was submitted to FERC immediately.

With v4 behind them, the drafting team started work in January 2011 (at a meeting located at the headquarters of a large utility, during which meeting the power went out in most of the building for a couple hours - Oops!) on CIP v5, which was now going to be the version that finally provided most of the changes that FERC had asked for in Order 706 (plus some more changes that NERC proposed, like the concept of BES Cyber System. By the way, that term originated in a “concept paper” in June 2009. This paper introduced the BCS idea, as well as the idea of “BES Reliability Functions”, which later became the BES Reliability Operating Services).[vi]

And a funny thing happened: Even though what the drafting team was proposing was quite different from CIP v3, as the year 2011 wore on, popular support seemed to be growing for the new version; in fact, some SDT members began to regret that they’d been rushed into developing v4, when clearly v5 would do everything v4 does and a lot more. The hope was that FERC would never approve v4, and v5 would become the next version.

That’s why it was a shock when, in September 2011, FERC issued a NOPR saying they intended to approve CIP v4. At the time, I thought[vii] FERC wasn’t all that serious about this – that they were trying to get all of NERC to support v5. Why would it be in anyone’s interest to make a big transition to v4, then a year or two later make an even bigger transition to v5?

If FERC was trying to pressure the NERC ballot body to get behind v5, the strategy didn’t work. The first draft of v5 was posted in November 2011, and was roundly voted down in the ballot in December (the highest percentage of the vote that any of the standards achieved was around 25%). But the SDT wasn’t fazed, and at their next meeting, at ERCOT’s headquarters in January 2012[viii], they made a number of changes to the standards. Probably the most important change was eliminating any requirement for Low assets that might require identifying individual Low BCS; in fact, the SDT went further and inserted wording in two or three places in the standards to the effect that an inventory of Low BCS wasn’t required. Those assurances remain in place today, of course.

But FERC had another surprise up their sleeve. In April 2012, they approved CIP v4. My opinion remained[ix] that they weren’t serious about wanting v4 to come into effect (which was now scheduled for April 1, 2014). Instead, they were sending NERC entities another message: “You’d better get behind CIP v5 now. If you don’t, you’ll definitely have to comply with v4, plus you’ll still have to comply with v5 after that, because it will ultimately pass. Plus, the NERC Board has the power to write the standard and send it to us for approval without balloting, in case you’d prefer to do it that way.”

Whether the NERC membership was taking FERC seriously, or whether the further changes made to v5 over the course of 2012 convinced them that it was now a workable CIP version (I suspect the latter), the NERC membership did get behind CIP v5. It finally passed the fourth ballot late in the year, and v5 was submitted to FERC at the end of January 2013.

However, even though some people at NERC were expressing confidence that FERC would quickly approve v5 and at the same time say that v4 would never come into effect, I had become convinced that, since v4 was the law of the land (which it was after FERC approved it), NERC entities had no choice but to go full speed ahead in getting ready for v4. After all, as of January 2013 (when I started writing this blog) there were only 14 months left before the v4 compliance date. I even wrote a post in February 2013, saying that it was highly unlikely FERC would approve v5 anytime soon (in that post, I also reiterated a call I’d previously made for the compliance date for v4 to be moved back). I now call this my “Dewey Beats Truman” post, since two months later FERC issued a NOPR saying they intended to approve CIP v5, and that v4 would never come into effect.

FERC approved CIP v5 in Order 791 on November 22, 2013[x]. As with all their previous CIP orders, they also ordered some important changes, which of course became “CIP v6”. NERC put in place a new drafting team to implement these changes, having decided that, after four or five years of continual meetings, the CSO 706 team deserved a break (although two or three of the members, being gluttons for punishment, continued on the new team). Significantly, this new team was called the CIP Version 5 Revisions SDT. This wasn’t an accident; with all of the back-and-forth and “start-no, stop-no, start again!” of the CIP v4/v5 debacle, the industry was hardly in the mood for a nice new version to sink their teeth into. So the words CIP version 6 were evidently verboten in NERC circles; I didn’t hear or see the term used until Scott Mix of NERC used it at a meeting on the new version in Atlanta in early 2015.

Meanwhile, FERC had another surprise up their sleeve. In March 2014, FERC issued an emergency order in response to the Metcalf substation attack in 2013 (which had recently caused a lot of alarm in Congress), mandating that NERC develop a new standard for physical security of critical substations – and that they approve it and submit it to FERC in 90 days. Needless to say, this was a big task, but it was accomplished, and CIP-014 was approved by FERC that fall.

The “CIP v6” SDT worked throughout 2014 on drafting and then balloting their revised standards. There were some twists and turns along that way, but the "v6" standards were finally approved by the NERC ballot body at the end of 2014 and submitted to FERC in February 2015. In July of 2015, FERC issued a NOPR saying they intended to approve CIP v6. But FERC went beyond that to discuss new or modified standards they were considering ordering NERC to develop, probably the most important of which was a standard for supply chain security.

However, this wasn’t all that was going on in CIP World. In fact, the major story was that throughout 2014 and 2015 there were many discussions and battles in the industry about the meaning of the CIP v5 standards[xi]. NERC issued various documents – FAQs, Lessons Learned and Memoranda – many of which ended up being rescinded. NERC finally decided that only a new drafting team, developing even newer versions of some of the CIP standards, could clear up these issues.

As it turns out, the new SDT would have more on its plate than just trying to fix problems from CIP v5. In January 2016, FERC issued Order 822, approving CIP v6 and – as usual – also mandating some more changes in CIP. These changes were added to the new drafting team’s SAR (and the new drafting team was called the “Modifications to CIP Standards” SDT – note there’s no mention of any version numbers at all in that name). This team started meeting in the spring of 2016, and immediately focused on two items mandated by FERC, one of which had a one-year deadline: the clarification of the meaning of LERC and a requirement for Transient Cyber Assets used at Low impact assets. These changes were balloted and approved in 2016, and submitted to FERC in early 2017. In October, FERC issued a NOPR saying they intend to approve these changes and – of course – asking for comments on some further changes they may order regarding Lows. Assuming FERC orders those changes when they approve the two changes that were submitted to them early this year, these will most likely be added to the current SDT’s agenda.

Meanwhile, the other big development of 2016 (regarding CIP, of course. I think there was also some election in November 2016 that some people thought was significant. I, of course, was too caught up in NERC CIP issues to pay much attention to that) was that FERC ordered that NERC develop a supply chain security standard in July. This standard was indeed developed, approved and submitted to FERC at the end of September. It now awaits FERC’s approval.

The only other significant CIP development I haven’t already mentioned was the startling announcement that I relayed in my post dated April 1, 2017, that the US had decided to deploy NERC CIP to Mexico….And on that note, I’ll sign off. It’s been fun writing this summary of what’s gone on in CIP since 2008. I’ll plan on doing this every ten years from now on. Mark your 2028 calendar now!

Happy New Year!

The views and opinions expressed here are my own, and do not reflect those of any organization I work with. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] At the September NERC CIP meeting, Tobias Whitney of NERC presented a slide that showed the number of TFE’s submitted annually had fallen from in the thousands under CIP v3 to under 200 in 2016; of course, this was because CIP v5 was designed to be “TFE-friendly”. Instead of requiring particular technologies in all cases, CIP v5 deliberately makes use of terms like “per device capability”, so that the entity can choose different ways of mitigating a particular cyber risk, rather than have to implement just one particular technology. The best example of this is CIP-007-6 R3, where antivirus software is no longer specifically required. Instead, the objective of the requirement is malicious code prevention, and the entity can choose whatever method is best for any particular device in scope.

[ii] Of course, the drafting team made a further modification to the criteria, in which they tweaked the wording to indicate that the criteria applied to BES Cyber Systems, not actually to the assets themselves; however, the criteria themselves still apply to the asset, not the systems. As anyone who was following this blog in 2013-2015 knows, I thought this was a very bad decision that was going to cause endless confusion going forward. But guess what? I was totally wrong about this. Even though there is officially no such thing as a High impact, Medium impact or Low impact asset, I doubt that ten percent of NERC compliance professionals or auditors even know that this is the case. But it really doesn’t matter, since exactly 0% of compliance professionals and auditors are actually acting as if it isn’t the assets that are classified, but instead the BCS. And I’m fine with that; as long as there is complete agreement that this is what CIP-002 R1 and Attachment 1 say, there’s no point in standing on a rooftop and yelling that this isn’t the case.

I used to think that the chickens would come home to roost on this issue the first time a NERC entity was fined for not properly classifying their BCS at a particular asset – say, they classified BCS at a particular asset as Low impact when NERC thought they should have classified them Medium impact.  The entity would then take this dispute to an Administrative Law Judge (which NERC entities are always allowed to do, since all NERC standards, like all FERC-approved standards, are regulatory law). He or she would spend 15 minutes trying to make sense of CIP-002 R1 and Attachment 1 – as I did in my first-ever post on CIP v5 in 2003, although I spent a lot more than 15 minutes writing it! - and proclaim “What is this (stuff)?” He/she would throw out the penalty, but might even rule that any dispute over classification of BCS was undecidable because of the ambiguity and outright contradiction in CIP-002. I no longer think this is likely to happen, though, mainly because the ultimate enforceability of CIP v5/v6 has been undermined in other ways as well. In any case, no case against NERC for a CIP penalty has ever been fully adjudicated, and I doubt one ever will be. Too much depends on trust between NERC, the regions, and the entities, for an entity to want to break that trust by filing a lawsuit.

[iii] The single change that FERC ordered only applied to CIP-006, so NERC might technically have just revised that one standard to version 2, leaving the other standards at the v1 level. However, the drafting team decided to change the numbering embedded in all of the CIP standards to v3, even though that meant having to submit all of the standards for ballot. When the next drafting team – charged with implementing the changes that FERC ordered in Order 791, which approved CIP v5 in 2003 – faced a similar decision, they decided only to revise the standards where there had been a substantive change and leave the others at the v5 level, which is why NERC entities now have to comply with both CIP v5 and v6 standards. In the next couple of years, there will be CIP-003-7 (due to the LERC changes) and CIP-005-7 (to implement changes that accompany CIP-013); plus within 3-4 years CIP-003-8 will come along. And of course, CIP-010 and CIP-011 have their own version numbers; they’re both on v2, although CIP-010 will go to v3 once CIP-013 is approved by FERC. So, as I’ve pointed out recently, we all need to get rid of the idea that there are “CIP versions”. Instead, the CIP standards should all be considered as having their own version numbers; it is up to the entity to make sure they are always dealing with the current effective version of each standard.

[iv] I didn’t attend that meeting in person, but I did by phone. The tension in the room was readily apparent even 1,000 miles away.

[v] I attended the August 2010 SDT meeting in Chicago – the first one I’d ever attended – and wrote about it in an “open letter” published under the Matrikon name (Matrikon was the company I joined one month before they were acquired by Honeywell in June 2010, but for a year we continued to operate as Matrikon). I later published some posts in a Honeywell cyber security blog, but started my own blog in January 2013. If anyone wants to see this open letter, email me at tom@tomalrich.com. It also tries to explain the big change from the original CIP v4 proposal (i.e. the one rejected in Dallas) to what ultimately became CIP v4 – although of course that was never implemented.

[vi] I also attended this meeting and wrote about it in an open letter. You can email me if you’d like to see it.

[vii] And wrote about it in a post on a now-nonexistent Honeywell blog. I hopefully have the text of that post on a USB stick at home, but I won’t be there for a week. If you would like to see that, email me and if I’ve found it I’ll send it to you.

[viii] I also wrote about this meeting on the Honeywell blog. Again, I’ll know next week if I still have a copy of that post. Email me if you’d like to see it.

[ix] Again, I hope I still have the text of the post I wrote about this. Email me if you’d like to see it.

[x] 50 years almost to the hour after the assassination of John F. Kennedy. Coincidence, you say? Well, I don’t know…

[xi] I must have written 50-80 posts on various controversies and developments regarding CIP v5 in those two years. You’re welcome to go back and read all of them. And if you do that, please tell me what they all said! I myself have no stomach for doing that.