Friday, March 8, 2013

Rube Goldberg in Albuquerque

                             “Those whom the gods wish to destroy, they first make mad.”
                   - Ancient Greek proverb, wrongly attributed to Euripedes

All opinions expressed herein are mine, not necessarily those of Honeywell International, Inc.

The above disclaimer really should be in boldface and underlined – what I’m about to say is not in any way Honeywell’s opinion!

I attended the NERC CIPC meeting in Albuquerque on March 5 and 6.  What I was most looking forward to was the promised presentation on NERC’s “Transition Plan” for moving from CIP Version 3 to Versions 4 and 5.  I have written much about the problems that the current uncertainty is causing, and was hoping that this plan would finally clear things up. 

I guess you know the punch line already: Not only did the plan not clear things up, it made them much murkier than ever.  In this post, I will explain why I feel this is so.  I will then issue a call for an intervention with NERC.  This situation simply can’t be allowed to continue.

In Albuquerque, Tobias Whitney of NERC presented on the transition plan, although he cautioned that it still needed legal review and wouldn’t actually be published for a couple more weeks.  His slides (which aren’t available yet) had a lot of detail that isn’t too relevant for this discussion; suffice it to say that this is one of the more complicated NERC plans I’ve ever seen, and that is saying a lot.  Rube Goldberg himself probably couldn’t have invented something more complicated.

In a nutshell, the plan is that:
  1. Entities subject to CIP Version 3 will have the option soon of implementing either the CIP Version 4 bright line criteria (BLC, from now on) or the version 5 criteria, for their Risk Based Assessment Methodology (RBAM) required in CIP-002-3.  They will need to implement the BLC in total, meaning they can’t pick and choose which criteria they want to implement.  They have to implement either all or none of them.
  2. If the entity adopts the V5 bright line criteria (which don’t use the term Critical Asset), all High or Medium impact BES Facilities will be identified as Critical Assets; Low impact facilities will not be Critical Assets.
  3. If the entity adopts the V4 BLC and identifies new Critical Assets as a result, they will need to be fully compliant for those assets on April 1, 2014 (the date they have to comply with Version 4 anyway). 
  4. If the entity adopts the V5 BLC and identifies new Critical Assets that way, they will have to be compliant on a schedule to be announced once FERC has approved CIP Version 5.
  5. There are lots of details regarding special cases: entities that lose V3 Critical Assets when they apply the V4 or V5 BLC, entities that already have received Potential Violations due to applying the BLC criteria, etc.  I’m sure the plan, when published, will address all of these cases in excruciating detail.
 So what’s wrong with this?  Here are the problems I see:
  1. At least half of the NERC Regional Entities have said that they won’t allow an entity to implement the Version 4 BLC as their RBAM (and I’m sure the idea of using the Version 5 BLC never even entered their minds until they first heard of this plan).  They will somehow need to be persuaded to change their policy, or NERC will simply have to force this down their throats (I imagine they have the authority to do that, though).[i]
  2. The Version 5 BLC are still subject to change.   FERC may very well decide that they wish to see one or more changes in V5 (including in the BLC, but in other areas of Version 5 as well).  So let’s say that an entity currently has a 1000MW plant identified as a Critical Asset under Version 3.  They adopt the V5 BLC as their RBAM, and lo and behold it is no longer critical (since it is under 1500MW).  They will probably be able to drop the plant as critical under V3 immediately (I believe that’s what Tobias said, but I can’t be certain about it), but in any case they would be able to drop it on 4/1/2014 when V3 is retired.  Then suppose that, when FERC approves V5, they lower the 1500MW threshold in the V5 BLC to 750MW (something I think is possible).  The entity will have to declare the asset a Medium impact once V5 is implemented, but in the two years between FERC approval of V5 and the implementation date, what will happen?  If they don’t have to do anything until the V5 implementation date, what is another asset owner – who also had a 1000MW Critical Asset under V3 but didn’t change their RBAM, so it remained critical – going to say about this?[ii]
  3. The biggest problem I see is what happens after 4/1/2014 for the entities that implemented the V5 BLC as their RBAM under Version 3.  Version 3 will be retired on that date, and all entities will have to comply with Version 4 – that is set in stone.  Since Version 4 includes the V4 BLC, not the V5 ones, how can an entity possibly comply with V4 unless they have adopted the V4 BLC?  Yet it obviously defeats the purpose of this whole plan if an entity can adopt the V5 BLC now and not declare a blackstart plant as critical under V3, but that same entity will have to adopt the V4 BLC on 4/1/2014 and now declare it critical (and thus subject to immediate compliance with CIP-003 through CIP-009).  So NERC is proposing that, come 4/1/2014, they will continue to audit the entities that chose the V5 BLC according to the V5 criteria, even though there is no legal basis for doing this because V5 won't be enforced (and probably not even approved) at that time!
I asked Tobias at the meeting if anything in this plan was going to require FERC approval.  He answered me – seemingly with a straight face – that they don’t need FERC approval for anything in the plan.  All I can say is, if you are willing to bet that FERC won’t put the kibosh on the idea that NERC will audit compliance with V4 using wording taken from a future standard (CIP-002-5) that hasn’t been approved and certainly hasn’t been implemented[iii], then you are extremely trusting.  I have some cheap stocks you might be interested in…..

Let’s summarize, keeping in mind of course that the plan finally released may differ from the presentation:

-          The plan is tremendously complicated.
-          It will require NERC to run roughshod over the preferences of the majority of the Regional Entities.
-          It will probably be shot down by FERC as an illegal interpretation of CIP Version 4, although not before some entities will have allowed themselves to become out of compliance with V4 by doing what NERC told them they could do.
-          It increases the current uncertainty about the new CIP versions, not decreases it.
-          Other than these minor quibbles, I think it’s a wonderful plan.

To be honest, I find this really depressing.  NERC is obviously being heavily lobbied by some entities who simply don’t want to see CIP Version 4 implemented, and they are flailing like a drowning man, trying to find anything, anything that will save them from V4[iv]. 

Now, I totally agree that it is very unfortunate that the industry will have to implement V4 followed a couple years later by V5, and I had hoped this could be avoided.  But when FERC approved V4 last April, there was no longer any possibility of just ignoring it.   Many entities started work immediately to come into full compliance[v] on 4/1/2014.  But many other entities were understandably confused by the mixed messages they were getting from NERC and some of the Regional Entities – namely, that V4 was unlikely to ever come into effect because V5 would be approved before 4/1/2014.  That possibility is very remote, and becomes increasingly so every day.  Yet NERC, instead of facing that fact and letting people know that they have now 13 months before the axe will fall, floats a complicated, unrealistic and illegal “plan” that allows entities to continue wondering what they should do – and in fact encourages them not do anything to move toward V4 compliance.

I said in a post last December that the only way NERC could even hope to avoid Version 4 coming into effect is to directly petition FERC to rescind Order 761, which approved CIP V4.  This is of course a real long shot – I don’t think FERC has ever rescinded an order – but it is probably the only way that the uncertainty will be eliminated.  If FERC just laughs in their face…then at least everyone knows that V4 is for real and they’d better get cracking on compliance with it.  But there is the (admittedly small) chance that it might actually work.  There are certainly FERC staff members who never saw the need for V4 in the first place, and who may well be quite sympathetic to NERC’s plea. NERC will never know whether this works unless they try.  However, they instead are giving the industry this “plan”.  As I said, it’s quite depressing.

Now we get to the call to action.  I’ve already called on NERC to take action and that clearly isn’t happening (not that I expected it to, of course).  So someone needs to step in and do an intervention with NERC – probably starting with Gerry Cauley, the President of NERC.  Who is the best party to do this?  I vote for the EEI (Edison Electric Institute), the trade organization of the Investor Owned Utilities, which distribute around 75% of the electric power in the US.  Let’s be honest, they have the most clout.  I believe a few high officers of EEI members should take Mr. Cauley (whom I have never met directly) to a quiet place, sit him down, and say he should do the following:

  1. Petition FERC to rescind Order 761.  If FERC does that, that’s great – then the industry just has one new CIP version to prepare for.  If they don’t, then at least the uncertainty is over.
  2. Assuming FERC doesn’t rescind the Order, NERC should petition to have the CIP Version 4 compliance date moved back from 4/1/2014 – hopefully by a year, to 4/1/2015.  I have advocated this in a separate post.  Briefly, the reason for this is that NERC’s confusion on this matter has led many entities not to make the investments they need to comply with V4 by 4/1/2014.  As a result, I think that the owners of a 1500MW+ generating station wouldn’t be able to achieve compliance by that date, even if they started today.  And the window for other assets is also likely to close soon, long before NERC stops dreaming up new schemes and simply states what some of the regions have said for a while: that V4 is real and those who believe otherwise do so at their peril.
  3. Whether or not FERC agrees to move the date back, NERC and the RE’s should immediately let all entities know they have to get moving now on V4.  And they need to make clear that, notwithstanding the ambiguity in the implementation plan documents, the intention was always that full compliance with CIP-002-4 through CIP-009-4 is due on the compliance date.
  4. NERC should also take up another suggestion of mine, that a guidance document needs to be developed for the V4 bright line criteria – and soon.  There are far too many questions about the BLC (as there would be for any set of short criteria, in an industry as convoluted and regulated as electric power) for entities not to need a lot more guidance than they’re getting now.[vi]
 P.S.  Preparing for CIP Version 4 is the topic of a webinar that Honeywell and EnergySec are presenting on March 27 at 10:30 Central Time.  We have received a huge response – we got about 120 signups in the first two hours after it was announced.  Fortunately, there are still good “seats” available.  You can register here.

[i] I think that at least some of the RE’s will be willing to accept the V4 BLC for the RBAM, since the entities will have to implement them in their entirety (meaning for example they can’t simply ignore the criterion for blackstart plants if they happen to have one of those, yet still adopt the other criteria).  I think the V5 BLC will be a completely different story – some regional auditors will have to be forced at gunpoint to allow entities to implement those as their RBAM, given that they haven’t been approved by FERC and may well be changed before that happens.  V4 is regulatory law, V5 is not.  How could this possibly work?
[ii] This is just one illustration of the many very difficult questions that are raised by this transition plan.  It will have to be very lengthy indeed to address all of these questions.  And pity the poor entity that has to try to figure out what it means – with large penalties waiting if they guess wrong!
[iii] And of course, NERC won’t even be auditing to the exact wording in CIP-002-5, since that refers to High and Medium impact BES facilities, not to Critical Assets; who will make this wording change, now that V5 has been submitted to FERC?  And the identification of High and Medium impact BES Cyber Systems associated with those facilities is quite different from the identification of Critical Cyber Assets associated with Critical Assets, as in CIP Versions 1-4.  Will that difference simply be ignored?
[iv] I think NERC’s reactions to V4 can be understood in terms of the Five Stages of Grief of Elisabeth Kubler-Ross: Denial, Anger, Bargaining, Depression and Acceptance.  They are now deep in the Bargaining phase.  Hopefully, the Depression phase will be short-lived (like maybe five seconds), and they’ll get on with Acceptance, as all adults have to do in the end.
[v] There was a big discussion at the CIPC meeting on the compliance date for CIP-003-4 through CIP-009-4, for Critical Cyber Assets that are newly identified under Version 4.  The argument was made that entities have another 12-24 months to comply with CIP-003-4 through CIP-009-4 for those CCAs that become such because of the V4 BLC.  I agree that the implementation plans for V4 are confusingly worded, but I know that both NERC and almost all of the Regional Entities are saying that full compliance with CIP-002-4 through CIP-009-4 is due on 4/1/2014, period.  For more on this, see this post.
[vi] I have talked to many entities about the V4 BLC.  I have yet to encounter a single one – other than entities that aren’t affected at all by V4 – that didn’t have at least one criterion that was very important to them, but that wouldn’t work as written for them because of this, this and this – the list always goes on for pages.  These entities have to now take a multi-million-dollar guess, in which they stand a 50/50 chance of either spending lots to comply with V4 for an asset that didn’t need it, or of getting hit with huge fines for violating all of CIP V4 because they improperly didn’t identify an asset as critical.  See the post for more on this.


  1. Hi Tom;

    Thanks for the blog discussion on the RBAM proposal by NERC. However, having been at the CIPC meeting, too, I think the only flaw in your writing is *understating* just how flabbergasted ever asset owner/operator in attendance was at the NERC proposal.

    I have never quite seen such a "pocketed" set of reactions in a single room, since the moods effectively varied with whether or not the owner was going to have something in or out of scope by applying the v5 criteria... or they were madly penciling ideas and little notes to see if they could be. I think this was the first official upheaval where everyone was sent scrambling to actually read 002-5 word-for-word.

    I'm also not sure if my observational perspective of this plan is that of:

    1. It falling into a blackhole - watching its infinitely slow descent (despite the foregone events that have long since occurred beyond the event horizon),

    ...or more along the lines of...

    2. A monkey with a hand grenade - waiting for the unfortunate but inevitable and instantaneous conclusion.

    If we were lucky it would be neither, but I'm a realist, so...

    My general thought was "well, let FERC discipline/fine/sue NERC into oblivion" if/when they balk at the plan... but this doesn't help anyone who is facing the whims of an RE in the mean time.

    Hopefully, whatever NERC does with this plan, it will come with a 20lb sledgehammer of enforcement that will require all of the REs to line up behind it, whether they like it or not.

    By the way, there's a Terry Pratchett line from 'Mort' that sums this up, too:

    Albert grunted. "Do you know what happens to lads who ask too many questions?"
    Mort thought for a moment. "No," he said eventually, "what?"
    There was silence.
    Then Albert straightened up and said, "Damned if I know. Probably they get answers, and serve 'em right.”

    1. Confounder:

      I wish to explore the physics of the first possibility you mention: that the new NERC proposal will simply disappear into a black hole. I certainly hope that is the fate of this new monster.

      But you have you considered the idea that the information in the proposal will reappear in our universe after "leaking out" of the black hole through Hawking radiation? In other words, the proposal may seem to disappear but will never really be gone. This is scary, but is unfortunately inevitable due to current physical thinking.


    2. Hi Tom;

      There's a mixed blessing to this so-called plan, as I'm sure you can appreciate.

      On the one hand you have entities who could - for all practical purposes - get a head start on v5.

      Conversely, there's the unknown quantity of FERC politics, rules, and standards that could lead to a reversal of this plan so fast it could snap NERC's neck (and leave the rest of us in the moving vehicle without a driver).

      I actually appreciate the fact that NERC is trying to get a leg up (and trying to help some folks get a head start on v5), but they're going about it all the wrong way.

      As for your point on the blackhole descent, while I appreciate that some radiation from the plan would occur, it would be nothing more that random remnant particles with no meaningful information contained therein.

      Oh wait... OK, you're right: it would be exactly like the plan is now.

    3. Thanks, confounder. The problem with getting a "head start" on V5 is that there is no assurance that the bright line criteria in V5 will be the same once FERC gets through approving V5 (in fact, I think it is likely that FERC will mandate changes to the V5 BLC before they approve V5). This fact alone makes it very unlikely that FERC would ever approve such a plan - and NERC's assertion that such approval isn't needed is highly suspect.

      Regarding the question of information leaking out of black holes, you may not know about the famous bet that Stephen Hawking lost
      He essentially conceded that information isn't destroyed in a black hole (which would require rewriting quantum mechanics, for reasons I don't understand), but can (and will) leak back out in Hawking radiation.

  2. A person who wishes to be known as "Flabbergasted" asked me to post this comment, which seems in line with Confounder's comment above:

    The problem is, there is absolutely no legal basis for accepting V5 BLC once V4 becomes effective about this time next year and every legal basis for requiring compliance with the V4 BLC once V4 becomes effective. NERC can wish and want anything, but FERC approval is what places it into the Code of Federal Regulations...and V5 ain't there yet. Woe be to the entity that believes the NERC nonsense.