Tuesday, July 30, 2013

The Travails of a CIP Compliance Manager

This video was obviously made by someone who's been there in the CIP trenches!  It is wildly funny and deadly accurate, I'll warn you.  Not recommended for those with heart problems.


By the way, the film this is taken from, Downfall (2005), is excellent.  Just seeing this clip makes me want to watch it again.

Monday, July 29, 2013

Open Letter to NERC

All opinions expressed herein are mine, not necessarily those of Honeywell International, Inc.

July 31: I listened to Scott Mix's excellent presentation to TRE on CIP Version 5 today (7/31). I submitted the question of when NERC would tell entities it was OK not to worry about V4 anymore, but he didn't explicitly answer it. However, from other things he said I can infer his answer: NERC expects FERC to approve CIP Version 5 this year, but until then, NERC entities can't rest assured that Version 4 won't come into effect on 4/1/2014. He spent a lot of time discussing timeline for V4 compliance, etc.

This isn't great news, since I know it means some entities will continue to spend money on Version 4 compliance, even though that will in all likelihood be wasted. All I can say is I tried, and I hope FERC approves Version 5 in September, as NERC seems to think they will (although Scott said he thought it would be later this year).

Aug. 2:  I knew there was a reason why I am the only person in North America working on an August Friday afternoon.  NERC just officially released the proposed transition plan - the same one I saw a week ago.  However, they say this will be finalized on Aug. 14.  This is good because there will now be an official plan for transitioning to Version 5.  But it's bad because they clearly aren't going to address the issue of whether Version 4 will come into effect - and whether they'll audit against it if by some chance it does.  So entities who want a final word on this will have to wait until FERC approves Version 5.

Sept. 7: To nobody's surprise, the final version of the CIP Version 5 Transition Plan ignores my sage advice below.  So we'll have to wait for FERC to approve V5 directly and put an end once and for all to the idea that V4 might still happen.  
Dear NERC:

I was pleased with what was in the proposed CIP Version 5 Transition Plan released last week.  However, I have a big concern about something that wasn’t in it. 

My concern is about guidance on CIP Version 4.  As you know, FERC made it quite clear in their NOPR that they don’t intend to let Version 4 come into effect.   However, it was exactly one year before that NOPR, in Order 761, that they had made it very clear that Version 4 would come into effect. 

A number of NERC entities (and I talked to two of them just this morning) believe they can’t take a chance that FERC will change their mind again and V4 will come into effect.  Some of them are still going forward with Version 4 preparation, including things like documentation and training that will not be applicable to CIP Version 5 – i.e. these are probably stranded costs that might not be allowed by the PUC’s. 

I had reason to believe the Version 5 plan would indeed address this question.  I thought Scott Mix’s comments at the SPP CIP Workshop in Dallas in May (which I reported in this post, see the paragraph numbered 6) indicated the plan would do that.  However, there is no word at all about it.  I certainly hope the final plan – which I also hope will be issued soon – will address the issue.  Let me suggest some rough language that would, I believe, allow a lot of NERC compliance professionals (as well as utility and IPP CEO’s!) to sleep at night:

Should CIP Version 4 come into effect as currently scheduled, and absent some other FERC directive on this issue, NERC will encourage the Regional Entities not to audit for strict compliance with CIP Version 4.  Instead, NERC will encourage the Regional Entities to recommend to their members that any assets, not currently critical under CIP Version 3, be instead prepared for CIP Version 5 compliance.[i]

Oh, and one more thing before I let you go, NERC.  It seemed in the discussion of the Transition Implementation Study (included with the proposed transition plan) that the final plan might not come out until the study was completed – i.e. in Q2 2014.  I hope I’m wrong about this interpretation.  Needless to say, since Version 4 will come into effect on the first day of that quarter (if it comes into effect at all), it will obviously not help any NERC entity if the V5 plan – even with the statement above – comes out after that!  The final plan really needs to come out very soon (tomorrow would be fine with me), since some NERC entities are incurring stranded V4 compliance costs as I write this sentence.

Please let me know as soon as possible (you can comment below or send me an email at tom.alrich@honeywell.com) when and how you will address this issue.

Respectfully yours,

Tom Alrich
Overall Nuisance and NERC/FERC Scold

[i] This recommendation is valid because, as far as I – speaking as Tom Alrich – know, there are few if any assets that would be Critical Assets under the Version 4 bright-line criteria that wouldn’t also be High or Medium impact under the Version 5 criteria.  And also because any assets that are currently Critical Assets under Version 3, that would remain critical under the Version 4 criteria, wouldn’t have to have anything done to them to remain in compliance under V4 – since CIP-003 through CIP-009 remain the same in V4 as in V3. 

The only exception to this statement – and this once again proves my ironclad rule that no exception-less statement can be made about anything having to do with NERC – would be >1500MW plants, where the provision in CIP-002-4 R2 about Critical Cyber Assets would make V4 compliance different from V3.  But that’s not worth worrying about now, since again the chances of V4 actually coming into effect are very remote indeed.

Thursday, July 25, 2013

NERC’s CIP Version 5 Transition Plan

Nov. 26: For my analysis of what FERC Order 791 means, including timeline for CIP V5/V6 and the transition to them, please see this exceedingly long post.

Nov. 8: It is very likely FERC will approve CIP Version 5 before Thanksgiving, most likely at their meeting on Nov. 21.  Of course, what will be important is the Order they issue with V5.  When that is issued, your reporter will sequester himself until he has figured out what it means, and will post that as soon as possible thereafter.

September 22: (the add-on notes to this post are now almost as long as the post itself)  I wrote yesterday a new post that updates some of the timing information in this post).

September 5: NERC finally released the approved version of this plan today.  I can’t see any substantial change from the preliminary one, so this whole post is still good (this post has gathered a very large following – I guess a lot of people are concerned about the topic, for some odd reason).  Of note:

  1. NERC thinks RBAM means “Risk Based Asset Methodology”.  Oh well, I sympathize with whoever did that.  It's very hard to keep up with all the stuff that NERC throws at you!
  2. They assume FERC will approve V5 in the third quarter.  That seems quite soon to me (also to Scott Mix in his talk to TRE recently), although I think Q4 is realistic.
  3. They will publish their V5 RSAWs as soon as FERC approves V5.  I will be quite interested to see how someone could write an RSAW for CIP-002-5 that actually followed the wording of the standard.
  4. They still include discussion of the upcoming "Transition Implementation Study".  In this study, they will give a few chosen entities about six months to try to implement V5 and then report on their experiences - and NERC will publish lessons learned.  As long as they don't think anyone could seriously implement V5 in six months, I supposed this isn't bad - they'll learn something, anyway.  But I believe NERC will have to provide a lot of guidance on Version 5 - and hopefully they won't wait a couple years to publish those documents, as they did with the Version 1 guidance.
June 25: I have just updated two important (and short) posts I did a few months ago on Version 5: one on the sequence of events for a new CIP version to come into place, the other on the likely timeline for compliance.  You may want to check these out first, since they are background for this as well as a number of other posts.

June 29: I found myself with writer's remorse this morning, thinking I had been too easy on NERC in this post.  This happened because I communicated with two NERC entities today who both asked the same question: How can we be sure CIP Version 4 won't come into effect?  I had hoped/expected the V5 transition plan would address that question, but at least this draft does not.  I said in footnote vi below that I hoped NERC would address the question in their final draft, but I realize I should express this hope more prominently.  So I have just made this post of an open letter to NERC. 

August 2: I knew there was a reason why I am the only person in North America working on an August Friday afternoon.  NERC just officially released the proposed transition plan.  It's exactly the same as what I wrote this post about.  What's important is they say this will be finalized on August 14.  So the good news is entities will now be able to use either the Version 4 or 5 bright-line criteria instead of their RBAM going forward.

The bad news is they clearly aren't going to address the question of Version 4 - whether they won't audit against V4 in the very unlikely event that it comes into effect next year.  Again, hardly anyone believes it will actually come into effect, but I know some entities probably consider that still too much of a chance to take.  They will have to wait until FERC approves Version 5 - perhaps later this year.

NERC has been surprising me lately.  They said in May they would have their CIP Version 5 transition plan out around mid-July, and they actually did that (arguably, it’s a week late).[i]  This is in sharp contrast to their Version 4 transition plan; they promised that was coming “soon” after FERC approved V4 in April 2012.  It didn’t show up until April 11 of this year, after a few false starts.  Then exactly one week later, on April 18, FERC made the plan largely invalid by announcing they intend to approve CIP Version 5, and that V4 won’t come into effect.  What’s a poor ERO to do?

What they did was get back to the grindstone and work on a V5 transition plan.  And now a Boring Alert: I don’t see anything seriously wrong with this plan.  Either it’s a decent plan or I’m getting old.  Since I know the latter isn’t the case, it must be the former.  Below is my summary of the plan.  I have consulted with an Interested Party on this, who provided some valuable comments and insight.  As is my usual practice when providing this Party’s comments, I will mix them in with my own comments, and let you try to figure out which is which.  I will of course take credit for everything good, no matter the source.

Here is my summary of this document:

1.     NERC bases their plan on the assumption that FERC will approve Version 5 before April 1, 2014, thus “stopping the clock” on Version 4.  Since FERC said this very explicitly in their NOPR, this isn’t going very far out on a limb.

2.    They make it clear that, up until the date that Version 5 becomes enforceable (and see this post for a discussion of a possible timeline), Version 3 will remain in effect.  Again, nothing surprising here.

3.    As in the Version 4 plan, the interesting part is the discussion of options a NERC entity has for identifying their Critical Assets under Version 3, starting now.  The Version 4 plan gave just[ii] two options: a) keep using your existing V3 risk-based assessment methodology (RBAM), or b) adopt the Version 4 bright-line criteria in their entirety – so anything that would be critical under V4 would be critical today under V3.  There was an exception to the second option: NERC allowed entities to remove blackstart generating units and substations in the blackstart cranking path from their Critical Asset lists, and said these wouldn’t be treated as CAs once V4 came into effect, either.

4.     In the new V5 plan, there are now three options for identifying Critical Assets under V3: a) stick with your current RBAM, b) utilize the V4 bright-line criteria, minus blackstart resources as in the V4 plan[iii], or c) use the V5 criteria[iv] and identify all High and Medium Impact assets[v] from Attachment 1 of CIP-002-5 as Critical Assets under V3.  I contend that the Critical Asset lists will be fairly similar whether you use approach b) or c) (and for more on that, see this post, in the section numbered I). 

5.    Because some Critical Assets may be removed when an entity switches from their RBAM to either the V4 or V5 bright-line criteria, NERC inserts a requirement that the entity should, for criteria that involve third-party designations, provide 90 days notice to that third party (RC, PC, TP, etc) before doing so.  You can read about this in the NERC document (see footnote 1 for how to get it), starting at the bottom of page 4.

And that is the plan right there.  However, the document also contains an added bonus (at absolutely no additional charge!): announcement of a CIP Version 5 Transition Implementation Study.  This study will “help identify successful implementation methods and challenges that the industry may face in transitioning to CIP Version 5, including identifying circumstances where entities will not be able to maintain compliance with CIP Version 3 while implementing CIP Version 5.” 

NERC plans to choose six to eight entities (from among the thousands who will no doubt volunteer) to start implementing Version 5 compliance in October 2013 (that date was chosen because NERC is now developing Version 5 RSAWs, which won’t be ready until then).  They are expected to finish work in the first quarter of 2014, and report on their problems and experiences to NERC.  NERC will then prepare a report that will “synthesize the Responsible Entities’ experiences in applying CIP Version 5, focusing on the effectiveness of meeting the CIP Version 5 Requirements and the methods employed during implementation.”  This will in turn lead to the final “Cyber Security Standards Transition Guidance” document[vi] in Q2 2014.

I (and the Interested Party) see a big problem with this: The idea that an entity of any size could complete (or even make a big dent in) their implementation of CIP V5 in six months is very far-fetched (unless they have almost no assets in scope, which then makes their participation in the study meaningless).  I can see doing a gap assessment and then perhaps starting on implementation (assuming there’s no delay waiting for funding, which strikes me as pretty unlikely for most NERC entities) in six months, but that’s about it.  So I’m afraid the report won’t be the definitive guide NERC wants it to be.  In other words, look for the V5 situation to be even more confused a year from now (if such a thing is possible), despite the report.

The question then becomes, how else will NERC provide guidance on V5?  This is a huge change, and there will have to be a lot of education for NERC entities: on applying the bright-line criteria, on trying to muddle through the morass of non-sequiturs known as CIP-002-5[vii], on grouping BES Cyber Assets into BES Cyber Systems, on properly dividing substations into transmission elements (subject to Version 5) and distribution elements (not subject), etc.  I’m sure NERC’s honest answer would be, “Darned if we know[viii].”

The Interested Party brought up something else I hadn’t thought of: What about cases where implementing V5 compliance will actually cause an entity to fall out of compliance with V3 (e.g. replacing the annual requirements in V3 with the 15-month requirements in V5)?  This person thinks that auditors will have to be lenient with the entities in these cases, as long as it is clear they are really implementing V5 and not just violating V3.

If you haven't signed up for the joint Honeywell / EnergySec webinar on CIP Version 5 on August 21 - "Covering your Assets in CIP Version 5" - I recommend you do it today!  Seats are going fast, and you might end up sitting behind a pole if you wait too long.  Remember, even if you can't make that date, you should still sign up, so you'll receive the link to the recording when it's available a couple days after the webinar.  You can sign up here.

[i] The title of the plan says “Proposed”, so it doesn’t have the same status as the V4 plan released in April.  I hope NERC finalizes this soon; it doesn’t seem to need much more work, IMHO.  I would like to provide a link to the plan here, but I can’t because I don’t think it’s on the NERC website now – I received it through one of the regional entities’ mailing lists.  The best I can say is you can email me at tom.alrich@honeywell.com and I’ll send it to you.

[ii] I’m really simplifying what the V4 plan said here.  You can see it described in all its glory in my post that came out a few days later.

[iii] Although be sure to see footnote 3 on page 4 of the document.  That note points out that control centers which control blackstart resources will remain Critical Assets, even though the blackstart resources themselves won’t be.

[iv] You may well wonder, “Since CIP V4 isn’t ever going to come into effect, why would anyone choose to use the V4 criteria to identify Critical Assets under V3?  Why wouldn’t people either stick with their RBAM, or move to the V5 criteria in anticipation of V5 coming into effect?” 

I can think of a few reasons: 1) Since the V4 transition plan gave NERC entities the option of using the V4 criteria starting in April, some may have already moved along that path; 2) Given that blackstarts have been removed from the V4 criteria anyway, the V4 and V5 criteria are now fairly close in their coverage; 3) I’m told by transmission people that the V5 criteria for substations are more inclusive than the V4 ones are (i.e. more subs would be Mediums under V5 than would be Critical Assets under V4) – this means that an entity with substations might be able to reduce its immediate compliance burden by using the V4 criteria, and just implementing the V5 criteria when V5 itself (or V6) comes into effect.

[v] A literal reading of CIP-002-5 Attachment 1 will leave you confused since it refers to both Assets and Facilities.  Don’t even get me started on the wording problems with CIP-002-5; you can read about my own confusion with that deeply flawed standard here.

[vi] This is the actual name of the document I’m calling the Version 4 Transition Plan here.  This leads me to fear that NERC isn’t planning to issue a “final” version of the V5 plan; in other words, the “proposed” version I’m writing about in this post may be the last version published.  It will obviously help to provide the lessons of the V5 Transition Implementation Study to the industry when they’re available, but by not finalizing the V5 plan now, NERC won’t give the industry the option of using the V5 criteria now, in place of their V3 RBAM.  That is because the official Transition Plan remains the April one, which didn’t mention V5 criteria at all.  So I hope my fears are unfounded, and that this “proposed” plan becomes a final one soon.

[vii] Of course, this assumes my suggestion to rewrite CIP-002-5 from scratch – and the alternate version I proposed to FERC – will be ignored.  I think that’s a pretty safe bet.

[viii] I will admit that, at the moment, it’s probably not worthwhile for NERC to be producing a lot of guidance on V5, given that it isn’t known what changes FERC will require in it.  That excuse may go away when FERC issues their final order approving V5 (and most likely requiring changes in a compliance filing).  I recently learned that NERC is expecting this to happen this September.  One thing that will definitely help the transition is having the RSAW’s for V5, which NERC says they’ll provide by October.

Notice: Honeywell has produced three white papers on CIP Version 5 - what's in it and how you can comply with it.  They aren't posted yet, but to get copies, just email me at tom.alrich@honeywell.com 

All opinions expressed herein are mine, not necessarily those of Honeywell International, Inc.

Sunday, July 21, 2013

Another Dialog inspired by my post on "The Real Cost of CIP Version 4"

August 12: I just posted my analysis of what FERC's order today - extending the compliance date for CIP Version 4 - means.

8/1: I've posted yet another dialog from LinkedIn, inspired by the original "Real Cost" post, here. This is becoming a long-running franchise, like "Planet of the Apes."  I may be able to retire just by posting these dialogs.

In the NERC CIP Compliance group on LinkedIn, John Kontofela of the New York Power Authority made this comment about the previous post, "The Real Cost of CIP Verison 4"; my correspondent then replied to John's comment, through me so he retains his anonymity.  Below is the dialog.  I have broken John's comment into two parts, since my correspondent does that in answering John.

1. John writes, "Read through the provided email, and I can feel the pain. However, at the risk of opening up multiple cans of worms I question some of the approaches for '...building a world class version 5 program behind the diodes.'

Just to narrow it down to one - the enclosing the fiber optic data highway for the DCS in conduit. Opening up the PSP to include the portions of the plant where the fiber runs in trays would obviate the need for enclosing it in conduit. Where we have fiber in trays as long as they have covers or they are more than a certain number of feet off the ground we consider it in the extended PSP."

My correspondent writes, "Through several years of managing PSPs in control centers and data centers, we have found that increasing the square footage of PSPs can be very costly with regards to on-going O&M in the future. Electronic security can be automated to the point that the human element is kept to a minimum, but physical security retains more of the human element than most parts of CIP. Expanding PSPs to encompass tray areas in a Fossil plant is almost impossible due to the number of openings in the perimeters."

2. John writes, "For the electronic security there are several products out there that utilize either dark fibers in the cables or a separate fiber laid on top of the fiber cables that we use to generate an electronic alarm if the cables are disturbed. With an engineering analysis showing the difficulty of "tapping" fiber, and the documentation and testing I would not feel uncomfortable presenting this approach to auditors."

My correspondent writes "Your input is appreciated. We need to do some more benchmarking on this interpretation."

Thanks for both of you for your comments.

Thursday, July 11, 2013

Dialog Inspired by my Last Post

All opinions expressed herein are mine, not necessarily those of Honeywell International, Inc.

August 12: I just posted my analysis of what FERC's order today - extending the compliance date for CIP Version 4 - means.

After I published my post on The Real Cost of CIP Version 4, an Interested Party emailed me with two sets of questions for the beleaguered NERC entity whose travails due to CIP V4 were chronicled in that post.  I facilitated a correspondence between them (without revealing either's identity to the other).  The result was quite interesting, so I decided to make this a separate post for everyone to see. 

Regarding CIP-006 Interpretation

Interested Party
Regarding the data fiber, was it possible to implement a logical solution (e.g., network connectivity monitoring) as opposed to a physical boundary solution (as permitted by the FERC-approved interpretation[i] that was 180 degrees opposite to the interpretation that FERC remanded)?  Would that have been easier and less costly?

Beleaguered NERC Entity
In a full CIP 3 thru 9 environment with CCAs the rule as interpreted by our entity is that if an ESP network exits a PSP to traverse to another PSP then it must have six wall protection. An example is a DCS in a plant where a control system network fiber traverses a PSP in one area of the plant to another PSP area in the plant. We call these “Extended ESP networks”. Remember that this differs from the rules on networks that connect distinctly separate ESPs which do not require the same protection.

We have no evidence that connectivity monitoring would satisfy an auditor. All modern day control systems already have alarming for network connectivity in their redundant operator and control networks (at least ours do).  In version 3 audits at some control centers in the past, we got pegged for not having the conduit which raises another issue – differing interpretations and violations among the 7 regional auditors.

Interested Party
If an auditor does not accept connection monitoring as an alternative to a six-wall border, you need to contest the finding.  Take it to a hearing if need be.  The FERC-approved interpretation specifically states “For Electronic Security Perimeter wiring external to a Physical Security Perimeter, the drafting team interprets the Requirement R1.1 as not limited to measures that are “physical in nature.” The alternative measures may be physical or logical, on the condition that they provide security equivalent or better to a completely enclosed (“six-wall”) border. Alternative physical control measures may include, but are not limited to, multiple physical access control layers within a non-public, controlled space. Alternative logical control measures may include, but are not limited to, data encryption and/or circuit monitoring to detect unauthorized access or physical tampering.”  You need to be able to demonstrate that circuit failures are alarmed and quickly investigated even if the circuit comes back up after a couple of seconds.  The interpretation recognized the impracticality of physically protecting cabling in circumstances like yours.  Auditors are bound by FERC-approved interpretations.  They cannot chose to ignore what FERC has approved.

Beleaguered NERC Entity
I want to thank you for this information.  It raised some eyebrows here.

Regarding Hurry-Up Compliance Solutions

Interested Party
Are entities having to implement hurry-up band-aid solutions because of the time constraints?  Are we substituting compliance for operability (as in air gapping) because we do not have time to do something more carefully thought out and appropriate?

Beleaguered NERC Entity
I can only speak for one entity and the answer is yes in some cases. When the law is passed and you say “OK, 2 years, no problem” but from my experience the assessments and funding phases consume about half of the schedule. In our case the air-gapping was implemented at very small single combustion turbine black start units that were already air-gapped in most cases (version 3 and 4). If they weren’t we gapped them anyway because it’s not worth the expense for something you start up monthly or quarterly. Very few companies hand over tens of millions of dollars to project teams without an exhaustive approval process (which was touched on in the blog). Implementing cyber security systems on control systems is not extremely complicated but the time it takes for compliance procedures, work procedures, training, hiring, contracting vendors, writing specifications for bids required by most companies for projects this large etc. etc. etc. becomes overwhelming. Then the 2 years doesn’t seem so long.

I know of other companies that are using diodes (for compliance) in large plants but the plant had remote start capabilities for a remote black start unit. They gapped the black starts and put diodes in the large plants so yes we are in some cases definitely substituting compliance for operability. The use of diodes prohibits remote support but - only in my opinion - the level of increased security is worth it. Again, buy a plane ticket. Having the ability to access megawatt critical and protection systems from the Internet is a serious security risk in today’s environment. The logic in these systems can have a direct effect on the physical world (human lives, the environment and billions of dollars worth of equipment).

Interested Party
Is anyone communicating this concern to FERC, who wants to shorten the V5 compliance window?  What impact on reliability will we suffer as everyone takes unplanned outages to cut over to the CIP-compliant infrastructure?

Beleaguered NERC Entity
We do not have a direct pipeline to FERC. Hopefully they are reading stories like this. Shortening the compliance windows would most definitely increase cost, reduce reliability and increase the number of possible violations. I have no doubt about that. Whomever has proposed that has absolutely no idea what goes on in the day to day operations of a power utility or the budget cycles. If the timeline were reduced to the proposed durations we would have to treat our compliance spending exactly like when a hurricane destroys our transmission and distribution infrastructures. We have to instantly materialize tens or hundreds of millions of dollars and then count what we spent once we have restored power, thus the term in the blog “Storm Money”. You don’t have to be a rocket surgeon to realize how much that would increase the cost.

Interested Party
Hopefully you commented on the NOPR when you had the chance.  FERC, not the industry, is behind the idea of shortening the compliance deadline.  I think they believe much of the program is already in place from compliance with Version 3 and thus just a bit of tweaking is necessary.  We get as much as two years today to bring a new Critical Cyber Asset into compliance.  Now we are talking about many Cyber Assets at one time scattered across many locations.  That will not magically take less time, especially with a major revamping of the overall program as well.

[i] Following is the relevant text from the CIP-006 Interpretation referred to:

If a completely enclosed border cannot be created, what does the phrase, “to control physical access" require? Must the alternative measure be physical in nature? If so, must the physical barrier literally prevent physical access e.g. using concrete encased fiber, or can the alternative measure effectively mitigate the risks associated with physical access through cameras, motions sensors, or encryption? Does this requirement preclude the application of logical controls as an alternative measure in mitigating the risks of physical access to Critical Cyber Assets?

For Electronic Security Perimeter wiring external to a Physical Security Perimeter, the drafting team interprets the Requirement R1.1 as not limited to measures that are “physical in nature.” The alternative measures may be physical or logical, on the condition that they provide security equivalent or better to a completely enclosed (“six-wall”) border. Alternative physical control measures may include, but are not limited to, multiple physical access control layers within a non-public, controlled space. Alternative logical control measures may include, but are not limited to, data encryption and/or circuit monitoring to detect unauthorized access or physical tampering.

Tuesday, July 9, 2013

The Real Cost of CIP Version 4

All opinions expressed herein are mine, not necessarily those of Honeywell International, Inc.

August 12: I just posted my analysis of what FERC's order today - extending the compliance date for CIP Version 4 - means.

I recently received an email from the person in charge of CIP compliance for the generation arm of a large IOU.  This person was very frustrated by the huge costs that had been put on him, his colleagues, and his organization by the fact that CIP Version 4 was approved and looked like it would be implemented – only to be relegated to the history archives when FERC issued their NOPR in April saying they intended to approve CIP Version 5.  I must admit, I had no idea how great these costs were until my correspondence and phone discussion with him.
I will first provide – almost verbatim – this person’s eloquent email describing the problem.  I will then weigh in with what I think are the lessons to be learned from the Version 4 episode (perhaps ‘debacle’ is a better word).

The Email
“For conversation’s sake, let’s go back in time and assume that version 4 is still alive and no decision has been made for v5 to replace v4. Below is a detailed real world challenge that a utility faces. 
Last year out of nowhere, FERC approved v4; we had two years to comply. Let’s start that counter on May 1, 2012. 
First we have to perform an RBAM to determine our scope of work across Fossil. Of course there is activity in Transmission, Gen Dispatch and Distribution. Version 4 estimates alone for the company (including Nuclear NEI stuff) are over $50 million and the number can grow when assessments are complete. 
I have no control over the RBAM.[i] Transmission planning takes care of that. They take about six weeks to complete and the CIP VP signs off. The company will not let me start anything until the RBAM is fully approved in order to prevent regrettable spending. Now we are in mid-June 2012.
I have 10 plants (Critical Assets) that have to come into compliance for v4. Four of them are >1500MW, 4 are black starts, 2 are less than 1500MW.
The next step is to immediately start the assessments (walkdown, inventory and full ESP diagrams). Since this cost is un-budgeted I need emergency funding (I call it “Storm Money”). It is now late August 2012 that I have charge codes to start the assessments.
I then find out that our corporate level sponsorship mandates that we have to be complete with CIP compliance implementation 5 months before April 1, 2014, to allow 5 months of self-auditing at any site that gets full CIP-003 thru CIP-009 (lesson learned from v3 in Transmission). 2 months for data collection and a three month audit phase on all sites. My deadline is no longer April 1, 2014; it is now November 1, 2013. Once we get the assessment money in August 2012, we start an extremely aggressive assessment plan for 10 plants to be done by EOY. I had to use 4 different vendors running parallel assessments and then receive/approve all database and ESP drawings by December 10th. It was very challenging, but we pulled it off. 
Once that is done you have to develop a compliance strategy for each site. This is what we came up with: Air-Gap the black starts (they are small simple cycle CTs).[ii] Air gapping is OK here since they very seldom ever run and the need for PI data is minimal.
Plants >1500MW: We used what we call a “control system isolation” compliance strategy (our RE gave us a verbal approval on this).[iii] Even though this is not full CIP, it requires multiple projects at each plant. New secure physical networks, splitting of control system networks, installing new fuel gas control system in gas plants and air gapping them, creating new Bently Nevada networks etc. Not as big as full CIP, but a lot of work and money. As you know, you don’t just throw this stuff in. A lot of design and approval stages have to be exercised.
The <1500 MW plants get full CIP-003 thru CIP-009 since our CIP department has ruled that data diodes can't be used as a "get out of jail free" card for CIP compliance (remember CAN-0024?) and full air gap is out of the question. We need the PI data on the corporate network. That’s another long story.
Now it’s late December 2012, and we have pulled off what some thought was impossible – All assessments and compliance strategies finished in less than 4 months.
Since our company has grouped all cybersecurity funding into a single budget request (including Nuclear), this cost will by far exceed the corporate $15MM threshold that requires Risk Committee approval. I now spend about 8 to 10 weeks generating extremely detailed and accurate estimates and work scopes for every single piece of hardware, software and services, procedures, etc. that will be procured to implement CIP in Fossil.
Now we are into early March 2013 and we start the un-budgeted funding request, which takes 60 days at minimum. Remember that our 2013 budget season closed before FERC issued their NOPR in April.
By the time budget is approved, we are into May and spinning off about 30 projects at 10 different plants.[iv] They are all doable, but the full CIP-003 thru -009 implementations have to be finished in 5 months, which is almost impossible since the plants have multiple brands of control systems and we need to go platform agnostic – meaning we have to test all these agnostic tools in test beds before we install the tools on the control systems. Not to mention the procedures, work instructions, physical security systems, central monitoring, alerting etc.
So now we already have an ESP specification created to bid the ESP systems out (bid required by Supply Chain). We put it on the street to viable vendors.
While all of this is going on in May and the entire team is working 7 days per week, we find out that FERC remanded the NERC interpretation regarding six wall protection on extended ESP networks. Now this means that all of the control system network fiber that is not in conduit has to be replaced.[v] This fiber has been in the plants for years (whenever the DCS was installed), and when the fiber makes horizontal traverses within the unit it’s in cable trays most of the time and then it goes into conduit once it leaves the tray. Just in one plant alone, this is going to cost $250K and require multiple unit outages. This money wasn’t budgeted, since our budgets were made up before FERC issued their NOPR.
Now we are into late May early June and running into vendor support issues, since everyone else (other utilities) is hammering the vendors also. It looks like there is really no way we can be ready for a full self audit on November 1. We could probably make it, but quality of work would be sacrificed.  Time to re-group.
New plan – Data Diodes in the plants that need CIP-003 thru CIP-009 for version 4 compliance and then starting building a world class version 5 program behind the diodes. We would then have plants that are the most secure they can possibly be. No inbound communication and we are doing user account logging, AV/Malware, backup and restore etc. behind diodes. It doesn’t get any better than that. I am a very strong supporter of eliminating inbound connections to devices that generate megawatts or protect equipment. You wanna support me – buy a plane ticket. That’s what jets are for.
This is a very non-granular rough overview. I will spare you the numerous other challenges.
Think about all of this frantic activity and then FERC issues their NOPR in April. It looks like we won’t have to be compliant (with Version 5) until 2015 and we have just spent millions of dollars to stay on schedule. If we would have known up front, the cost would have been much less and we could have spread the expenditure across multiple fiscal years, which helps the bottom line in an industry that is suffering financially more than ever before. If you don’t know those details, then you will in the near future. Multiple utilities will be laying off thousands of employees this year and next year to be able to pay for new compliance and replacing the aging fleet.
(He added this postscript later)
My example of the extended ESP networks that traverse all over a plant is somewhat representative of many other complicated factors that I left out of my example. It would take 10 pages to convey all of the facts.  Using the $250K for the ESP fiber is a good example, though. Even though $250K is a considerable sum of money, it is irrelevant compared to this:
We did not know this cost would be required when providing the accountants with our estimates and scheduling the work. We were in mid-stream creating new networks for CIP when we got the word that this expenditure didn’t make the budget for this year. Now the $250K is un-budgeted and greatly affects our schedule. We can pull and terminate the fiber with units running, but we need outages to switch over to the new fiber on control systems and have to ask for un-budgeted funding, which makes the value much more than $250K since schedule and budget are both affected. 
It's like this – the more proactive and eager we are to comply, we pay a higher price for it every time NERC or FERC changes course. Our management starts to become apprehensive about the regrettable spending and then they start waiting until the last minute to release any additional funding (and I don't blame them) - which then puts people like me with unobtainable goals for the next fictitious version.
You don't even want to hear about the procedures issues because I would have to stab myself in the neck to tell that story. We hired a full team of procedures writers for Fossil Plant v4 procedures and then laid them all off 6 weeks later. There are a lot of unnecessary hidden costs in that situation also.”

Tom’s Opinionated Comments
The upshot of the above email is this entity spent many millions of dollars in an accelerated effort to  become compliant with CIP Version 4 on April 1, 2014.  While most of that will still be applicable to Version 5, there was a lot of money wasted because of the hurry-up nature of the project.  There was also a huge human cost, both on the existing compliance team and on others who were hired for V4 compliance then laid off after the NOPR showed they weren’t needed. 
A longer-term effect is that management now is very wary of spending anything for compliance going forward until there is absolute certainty it will be required.  And it is hard to blame them: there has  been no absolute certainty with regard to the direction of NERC CIP for about four years, and there won’t be any until sometime in 2014, if then.
Why did this person contact me about this?  And why do I find this such an important topic?  Because this story could probably be repeated across many NERC entities.  Nobody will ever know the total cost of the CIP Version 4 debacle, but it was obviously huge.  I think it’s important to try to identify the mistakes that were made that caused this to happen, so that the entities responsible for those mistakes – primarily FERC and NERC – will be careful to avoid them in the future.
What follows is my highly impressionistic take on how I think all of this unfolded.  Since most of the key decisions were made behind closed doors and never documented, there won’t be any good way to verify some of what I say here.  I was a fairly close observer on things CIP during this entire time, but I would welcome any comments or corrections from others, especially those who were actual participants.  As always, you can email me at tom.alrich@honeywell.com if you want me to publish your comments without attribution.
  1. After CIP Version 3 was approved in 2009, the CSO706 Standards Drafting Team turned their attention to Version 4.  V4 was intended to be the version that would address all of the remaining issues raised by FERC in Order 706 (which approved CIP Version 1 in January 2008).  The team settled on a radically different approach for V4.  There would be just two standards: CIP-010-1[vi] would be for identification of assets and cyber assets in scope (i.e. it would be the equivalent of CIP-002 in Versions 1-3), and CIP-011-1 would encompass everything that had to be done to those assets (i.e. the equivalent of CIP-003 through CIP-009). 
  2. The SDT held a very well-attended workshop in Dallas in May 2010 to discuss the new draft standards.  Their hope was that questions could be quickly addressed, so that Version 4 could be balloted and approved by July 2010.  At the workshop, there was lots of opposition to many areas in Version 4, a lot of it simply due to the novelty of many of the concepts.  It was clear that getting Version 4 approved would be a long slog, with probably multiple drafts and ballots required.
  3. At this point, the idea somehow came up within NERC that a new CIP version needed to come out in 2010, no matter what.[vii]  This was due to the perception that there was strong sentiment at FERC and in Congress that a new version was needed right away; so NERC couldn’t afford to wait the year or two that it would take to develop the radically new version that everyone knew was really required (in hindsight, this perception on NERC's part was probably wrong).
  4. The main reason that Congress was so upset about CIP was the fact that (in Congress’ opinion) very few assets – other than control centers – had been designated as Critical Assets, due to the fact that Versions 1-3 allowed the entity to develop its own Risk-Based Assessment Methodology (RBAM) for identifying them.  CIP-010 was going to address this issue with a set of “bright line” criteria (BLC) that would force NERC entities to designate certain assets as critical if they met the criteria (e.g. power plants over a certain MW threshold, although it was more than 1500MW at the time).  The thinking went: Why don’t we just change CIP-002-3 to replace the RBAM with the BLC, and leave the other standards (CIP-003 through -009) exactly the same as they were in Version 3?  This would be much easier to get approved, and might well result in a new CIP version sent to FERC in 2010.  Once this happened, the SDT could then turn their attention to the “real” new CIP version, Version 5.
  5. The rest of 2010 was thus spent drafting and balloting Version 4.  It was approved by the NERC Board of Trustees at the end of December and submitted to FERC for their approval in February 2011.  The SDT turned their attention to Version 5.
  6. In 2011, the SDT started making good progress on Version 5.  In fact, they became optimistic that they would get it right on the first draft – it would be balloted by the end of the year, and hopefully approved and sent to NERC in early 2012.  The question then arose: Why should we even bother with V4?  Let’s just go to Version 5!
  7. However, FERC surprised the industry by issuing a NOPR in September 2011, saying they intended to approve Version 4.  Since the optimism on CIP Version 5 was fairly high at that point, I (and others) thought that FERC was just bluffing.  I thought[viii] they were essentially using Version 4 as a club to hold over NERC’s head: “Either you approve Version 5 quickly, or we will make you comply with Version 4 and then Version 5.”
  8. But it turns out FERC wasn’t bluffing.  They actually approved Version 4 in Order 761 in April 2012; the compliance date would be April 1, 2014.  I (and others) quickly changed my tune: Since CIP Version 4 was now the law of the land, and since Version 5 was struggling a lot on the road to approval,[ix] NERC entities now needed to concentrate on getting ready for Version 4 compliance.  And entities like the one above started to do that, leading to the debacle so eloquently described.
Up until FERC approved V4 in April 2012, what mistakes were made?  I would say the biggest mistake up to that point was NERC’s, in panicking in 2010 with the idea that a new CIP version just had to come out that year.  But because of FERC's approval of Version 4, I now think FERC made the bigger mistake.  

I say this because I don’t think they were really serious about Version 4 when they issued Order 761 in April 2012.  They could see Version 5 was struggling for NERC acceptance, and they were afraid that NERC had gone back to thinking V4 would never be approved (as irresponsible bloggers like myself were saying in late 2011 and early 2012).  In FERC’s view, NERC entities were squabbling and nit-picking Version 5, now that they thought they didn’t have to worry about V4.  So Order 761 (approving V4) was a real blow to NERC’s noggin, saying “We’re very serious about this.  Get to work now on approving V5 so you can avoid V4”.  In fact, in Order 761 FERC set a deadline of March 31, 2013 to receive the NERC-approved Version 5.
The reason this could still be called a warning blow was because, if NERC got their act together and approved Version 5 on time, then Version 4 would never come into effect (since the V5 implementation plan just required that V5 be approved by FERC by April 1, 2014 for this to happen).  I think FERC felt safe in believing there would be little if any cost incurred by entities in preparing for V4 compliance, as long as the V5 approval process moved along in 2012.
The V5 approval process did move along, but – as you can see above – many entities felt they simply couldn’t take the chance that V4 would not come into effect on 4/1/2014.  And they spent a lot of money and effort rushing for compliance.[x]  Meanwhile, a huge pressure built up on FERC to make clear their intentions regarding Version 5; they did that in the NOPR issued this past April.  But there had been an entire year during which the only official word from FERC was that Version 4 was coming into effect in 2014; NERC entities had to take the most prudent course and prepare for Version 4.
To summarize, the costliest mistake regarding CIP Version 4 (meaning costliest for NERC entities) was FERC’s approval of V4 in April 2012.  I believe they issued Order 761 mainly to prod along NERC’s approval of CIP Version 5, not because they really intended for it to come into effect.  But they didn’t consider that the industry wouldn’t necessarily recognize this bluff, and in any case couldn’t afford to take the chance that it was a bluff.  We’ve just seen one example of the damage that caused.

P.S. Be sure to sign up for Honeywell’s upcoming webinar with EnergySec, “Covering your Assets in CIP Version 5”.  You can sign up for it here.  The webinar is on August 21st 10:30CDT.  If you can’t make the webinar but want to see the video, sign up anyway.  You’ll get the link to the video as soon as it is posted after the webinar.

P.P.S. (July 11) After I published this post, an Interested Party emailed me with two sets of questions for the entity who wrote most of this post.  I facilitated a correspondence between them (without revealing either's identity to the other).  The result was quite interesting, so I decided to make this a separate post for everyone to see.  You can find that post here.

P.P.P.S (July 22) Another exchange between my email correspondent and an interested party (different one) took place on LinkedIn recently.  I have posted that exchange here.

[i] (Tom speaking here) The use of “RBAM” here isn’t completely correct, since CIP Version 4 didn't require developing a Risk Based Assessment Methodology, as did Versions 1-3.   In theory, the bright-line criteria in Version 4 are supposed to be so easy to apply that a third-grader could sit down for half an hour and identify all of the entity’s Critical Assets.  In practice, that was not the case, which is why the Transmission Planning department of this entity spent six weeks applying those criteria.  I believe the same will be the case for Version 5, and there will need to be guidance provided on actually applying the V5 criteria (CIP-002-5 Attachment 1).

[ii] Of course, air-gapping plants allowed the entity to claim no Critical Cyber Assets under CIP Versions 1-4.

[iii] The author of the email is referring to the provision in Version 4 that exempts cyber assets at >1500MW plants from being CCAs, if they don’t individually affect more than 1500MW.  This provision lives on in Version 5 as Criterion 2.1 in CIP-002-5 Attachment 1.

[iv] I asked this person why they continued full speed on their V4 compliance projects, when FERC had made it clear in their NOPR that they didn’t intend to let V4 come into effect.  He pointed out that this statement couldn’t be relied on 100%, and the consequences would be disastrous if they were caught completely unprepared and V4 did come into effect next April.  It’s hard to argue with that, although see my post on the V5 transition for more perspective on this question.

[v] Since the requirement for a six-wall border goes away in Version 5, one could say that this work should have been halted when the NOPR was issued in April.  However, there are two considerations: 1) FERC may still require some specific cabling protection in the next version, and 2) this entity had already decided they couldn’t afford to take the chance that Version 4 would not become enforceable on April 1, 2014.

[vi] The suffix was “-1” since this was the first CIP-010 standard, but this was still called CIP Version 4 since it was the fourth version of CIP.  Of course, this early CIP-010 (and CIP-011) never was approved by NERC, and now the CIP-010 and CIP-011 in Version 5 also have the “-1” extension.

[vii] I was told that a group of Midwestern IOU’s reached this conclusion and drove the decision to push for an immediate Version 4, but it doesn’t really matter. 

[viii] I wasn’t blogging at the time, but I put out an open letter that you can still download here.  If you don’t want to authenticate, you can email me at tom.alrich@honeywell.com and I’ll send it to you.

[ix] Version 5 was overwhelmingly rejected on the first ballot in December 2011, and a little less overwhelmingly rejected on the second ballot in May 2012.  The third draft was approved on the third ballot in October, 2012.

[x] Some still may be preparing for V4 compliance on 4/1/2014, including the entity discussed above.  However, I think such entities should focus on compliance tasks that will be equally applicable to Version 5 as to Version 4, to avoid stranded costs.  I have suggested such tasks in this post.