All opinions expressed herein are mine, not necessarily those of Honeywell International, Inc.
Note: This is a post I originally made early last September. It is a call to action for NERC to develop a guidance document for application of the Bright-Line Criteria in CIP-002 Attachment 1 in CIP Version 4. Sadly, five months later (and five months closer to the April 1, 2014 deadline for CIP V4 compliance), we are no closer to having this. Meanwhile, literally every conversation I have had with a NERC entity since then has brought up yet another problem (or two) with applying the criteria in a particular area, or with their particular Transmission Operator, etc. I am more convinced than ever that this is needed.
I had my eyes opened at the last WECC CIP User Group meeting in June, when auditor Joe Baugh gave a presentation on “Migrating to CIP-002-4”. He set out to do something which at first sounded fairly pedestrian – discuss how WECC might audit Version 4 of CIP-002, and especially audit the entity’s application of the “bright-line” criteria for Critical Assets in Attachment 1.
This sounded almost like a wasted exercise. The whole idea of the bright-line criteria is that they are supposed to make designating critical assets a mindless process: you go down each of your assets and apply each of the criteria in Attachment 1 to it in turn. If one of the criteria fits, it’s critical. If none do, it’s not. What could be simpler?
However, the reaction to Joe’s presentation was anything but boring. Speaker after speaker lined up at the floor microphones to point out how his rules for auditing this or that criterion in Attachment 1 would never work because of x,y,z etc. And the reasons given were all over the place – not in any way pointing to some sort of common theme. I believe the discussion would have gone on all day had it not been limited by the need to go on to the next agenda item.
My first reaction to this was, “Well, Joe really blew it. He really didn’t think this through before he put his slides together.” But later I realized that there’s no way you could put together bulletproof auditing standards for a lot of the criteria. There are just so many ways they could be interpreted. Joe was the pioneer who was met by a hail of arrows as he walked into new territory. He is to be commended for doing what he did, and bringing the problem into the open. Because – as will be evident below – I think this is a problem that needs to be addressed very soon by NERC.
So is the problem with the criteria themselves? Did the Standards Drafting Team not take enough time to craft those carefully, to make sure there wouldn’t be any interpretation problems? Having attended some of the SDT meetings where they spent hours discussing a single criterion – and then reopened the discussion in the following meeting – I really don’t think this is the case.
I finally realized that the very idea of bright line criteria is flawed. I really don’t think you could ever have a set of comprehensive criteria that would be unambiguous, and for which the auditing procedures would be self-evident. Not in this industry, anyway.
I want to illustrate the problem by discussing some of these interpretation problems. I unfortunately wasn’t taking notes during the discussion at the WECC meeting, but I have gone back over Joe’s slides, and Attachment 1 of CIP-002-4, to identify what seem to me to be real interpretation problems with some criteria (although I probably should say “evidence problems” – since the issue is that there’s no way to produce unambiguous evidence for some of the criteria). And I’m not an electric industry guy by any means; I’m sure those of you who are can find many more holes than I can. Here are some of the problems I see:
Criterion 1.3 reads “Each generation Facility that the Planning Coordinator or Transmission Planner designates and informs the Generator Owner or Generator Operator as necessary to avoid BES Adverse Reliability Impacts in the long-term planning horizon.” Joe’s slide 13 calls for the entity to produce “Studies related to generation facilities and/or other rationale for including/excluding generation Facilities under this clause”.
The problem with the auditing requirement is that it doesn’t seem to relate at all to what Criterion 1.3 calls for. You would think that 1.3 would require the GO/GOP being audited to produce evidence that the PC or TP had designated the generation facility as “necessary to avoid BES Adverse Reliability Impacts in the long-term planning horizon”. But Joe is saying that the entity has to show “studies” demonstrating whether or not the facility is “necessary”.(1) Who produces those studies? The entity, the RC/TP, another party like the Regional Entity, all of the above? And if there are multiple studies with differing conclusions, which one do the auditors accept as definitive? Do you see a problem here?
Criterion 1.5 reads “The Facilities comprising the Cranking Paths and meeting the initial switching requirements from the Blackstart Resource to the first interconnection point of the generation unit(s) to be started, or up to the point on the Cranking Path where two or more path options exist, as identified in the Transmission Operator’s restoration plan.” This wording alone raises the red flag: Is it really likely that there will be an unequivocal way this can be audited – and that no disputes will arise between the entity being audited and the auditor?
Joe’s presentation says that, for Criterion 1.5, the entity will need to produce “A one-line diagram of the entity’s Transmission system and the TOP restoration plan” (the same one-line diagram is required for Criterion 1.7). If the entity and the auditor dispute whether a facility (usually a substation in this case) meets this criterion and the entity then produces their one-line diagram, is that going to resolve the issue?
One would think that the entity wouldn’t take a position that they didn’t believe was supported by their diagram. They will bring out their best engineers, who will fill the air with a lot of very technical discussion to support that position. Are the auditors (many of whom do not have an electrical engineering background) going to be able to argue with them? And who can they call on to resolve the technical dispute? Presumably someone from the 693 side at the Regional Entity. But what authority would these engineers have to make a determination regarding CIP compliance?
Criterion 1.8 reads: “Transmission Facilities at a single station or substation location that are identified by the Reliability Coordinator, Planning Authority or Transmission Planner as critical to the derivation of Interconnection Reliability Operating Limits (IROLs) and their associated contingencies”.(2) For this criterion, Joe wants the entity to produce “A list of all Transmission Facilities and evidence related to the identification of Transmission Facilities by the RC, PA, or TP as critical to the WECC interconnection”.
There is a striking difference between Criterion 1.8 and Joe’s requirement: The Criterion seems to be written with the idea that there is some sort of formal document that will always be produced when an RC, PA or TP identifies a facility (again, usually a substation) as critical to derivation of IROLs. Joe seems to admit (and I imagine he’s right) that there could be a number of different pieces of “evidence” that might constitute this act of identification. What if they contradict each other? Or what if the identification (or most likely the non-identification) was done verbally by the RC/PA/TP? What individual does the auditor need to contact at these organizations to verify this? What if the RC and the TP have different opinions on this, or even different people within the RC or TP do?
Criterion 1.12 reads: “Each Special Protection System (SPS), Remedial Action Scheme (RAS) or automated switching system that operates BES Elements that, if destroyed, degraded, misused or otherwise rendered unavailable, would cause one or more Interconnection Reliability Operating Limits (IROLs) violations for failure to operate as designed”. Joe requires “A list of SPS, RAS, and automated switching systems that would violate IROLs”.
I confess I only have a vague idea what SPS and RAS are, but it seems to me that the list the entity produces for this criterion isn’t likely to be accepted unchallenged by the auditor. Both the auditor and the entity will then have to bring out their EE’s to discuss any differences of opinion. And who decides which side won that discussion?
I think you get the idea now. There are going to be a lot of disputes about at least some of the Criteria in Appendix 1 of CIP-002-4.(3) How are those disputes going to be resolved? I see two mechanisms currently available in NERC.
The first is an audit. Of course, an entity can dispute any audit finding; but this is a tremendously inefficient and expensive way to resolve Attachment 1 differences. Say you’re a Registered Entity that believes that one of your generating plants does not meet any of the Criteria in Appendix 1; when Version 4 becomes effective on April 1, 2014, you document that you have applied the criteria and have not found it to be a Critical Asset.
In 2016 (say), you get audited and your auditor believes that Criterion 1.3 does in fact apply to this plant. You fight it but ultimately you lose. What are your potential fines? Well, you’ve not only violated CIP-002 R1, but just about every other requirement in CIP-002-4 through CIP-009-4 (since you presumably didn’t take the steps required to comply with the other requirements). Say you’ve violated 40 requirements. Your maximum fine is $40 million a day times two years…that could be fairly expensive.(4)
The only other way to resolve a dispute like this – that I know of – is to make a Request for Interpretation. There are at least three problems with that. First, RFIs take over a year to resolve and require at least one vote of the NERC membership. Second, RFI’s can’t be requested (I believe) until the standards are in effect; this doesn’t help entities that are trying to decide whether or not their assets are critical in advance of April 1, 2014. Third, the volume of RFI’s that would result from Attachment 1 would immediately completely overwhelm the system. You can be sure that virtually every entity that has any question at all whether an asset is critical or not will file probably not one but multiple RFI’s (for different criteria) for that asset.
Of course, the entity could always ask their Regional Entity for an interpretation. They might or might not get one, but there is a high probability the interpretations will be different across the regions – an unfortunate result, given that one of the big reasons for having the bright line criteria was to have uniformity of Critical Asset designation across the regions.
If you want to suggest CANs as a dispute resolution mechanism (and as a way to ensure uniformity across regions), forget those. The same thing applies to them as to RFIs: the system would be completely overwhelmed. Plus there’s no mechanism I know of by which a Registered Entity can request a CAN.
So what’s the solution? To be honest, I don’t know of a real solution, although I know this problem can’t wait to be solved until after Version 4 comes into effect and disputes start appearing in audits. I did at first think that a kind of “Attachment 1 Supreme Court” might be formed – perhaps from NERC Registered Entity representatives and Regional Entity auditors – to make decisions on particular cases and publish them for the entire NERC community. But I think even that group would be completely overwhelmed by the volume of cases.
The best I can think of – and I would be very interested in hearing others’ opinions – is that NERC put out a definitive guide to interpreting Attachment 1, much like the Critical Asset and Critical Cyber Asset identification guides that were put out a couple of years ago.(5) This would unfortunately have to be quite lengthy, since there are probably lots of different ways each criterion can be interpreted, for each asset to which it might apply. But the size of the endeavor is more than justified by the size of the mess that will result otherwise.
I think it is quite important that this be done soon. We’re already coming close to the minimum amount of time required to bring a sizable asset like a generating station into CIP Version 4 compliance by April 1, 2014. Entities that are wrongly interpreting a criterion are either a) risking sizable fines if they are found to have not classified an asset as critical that should be or b) potentially spending millions of dollars (and lots of staff time) putting in place a CIP compliance program and technologies for an asset that turns out not to be critical.
And while I’m at it, I think there should be a similar document produced for CIP Version 5, since many of the bright line criteria in that version now differ significantly from the Version 4 criteria. Of course, there’s no point in developing such a document until the V5 criteria are set in stone. And who knows when that will be? But that’s another blog post.
(1) The fact that Joe considers “studies” to be necessary seems to be an admission – probably true – that there isn’t some sort of definitive document from the PC/TP that informs the GO/GOP whether or not the generating station is “necessary to avoid BES Adverse Reliability Impacts in the long-term planning horizon”. In this way, the argument I have with this criterion is similar to the one I have with Criterion 1.8 below.
(2) There is more to it than that, but those do not affect the point of my argument.
(3) I will admit that some of the other Criteria are fairly straightforward in their interpretation, but even having just one criterion that was ambiguous could be a big problem. I also want to emphasize again that this discussion is in no way an attack on Joe Baugh. In fact, I greatly admire him for having the guts to actually tackle this question, which as I said is a thankless exercise. It certainly woke me up to this whole issue, and perhaps others as well.
(4) Obviously, the actual fine will be nowhere near this. But even if it were one ten-thousandth of that amount, it would still be over $2 million. This isn’t to say that the fine would even be that much, or that a fine wouldn’t even be waived because this was a good faith misunderstanding of the criteria. But having the audit process be the only way to resolve a dispute over the bright line criteria is pretty scary.
(5) FERC Order 761, Paragraph 41 (p.24), concludes “To address the concerns of uniform implementation, the Commission believes that responsible entities would benefit from the ERO’s guidance. “ FERC is referring here to Criterion 1.3 in Attachment 1, but I would like to generalize that recommendation to all criteria. The SDT did publish a CIP Version 4 Rationale and Implementation Reference Document in 2010 that discusses the Attachment 1 criteria, but only from the point of view of how they were derived. It won’t furnish much help in deciding how they should be interpreted.
No comments:
Post a Comment