I have had a
number of epiphanies over the course of writing this blog and
examining/reporting on CIP version 5. I
had another one last weekend after I had written my post
on the WECC CIP User Group (CIPUG) meeting in Anaheim. It concerned a couple of private discussions
of an issue that was never brought up at the meeting itself.
Here’s the
issue: In January, I had three entities with generation assets in New England
independently approach me with the same question (two earlier in the month, one
at the CIPUG itself). They had all
received an email from the ISO New England that said:
“In
accordance with Criterion 2.6 of NERC Standard CIP-002-5, ISO New England has
determined that Generation Facilities represented by your company have an AVR
and/or PSS (if equipped) that is critical to the derivation of IROLs and their
associated contingencies, as specified by FAC‐014‐2, Establish and
Communicate System Operating Limits, R5.1.1 and R5.1.3.”
For those of you who (like me) aren’t
generation gurus, AVR refers to “automated voltage regulator” and PSS is “power
systems stabilizer”. These are systems
that can regulate voltage when it gets out of a certain range required for
stability of the grid. IROLs -
“Interconnection Reliability Operating Limits” - are defined in the NERC
glossary as
“The value (such as MW, MVar, Amperes,
Frequency or Volts) derived from, or a subset of the System Operating Limits,
which if exceeded, could expose a widespread area of the Bulk Electric System
to instability, uncontrolled separation(s) or cascading outages.”
Evidently ISO NE requires most (or even all)
generators to have an AVR system, as well as an RTU to report its status back
to ISO NE. To make a long story short,
it seems that knowing the status of these systems is required for ISO NE to
derive IROLs. The email is saying that “Generation
Facilities” with AVR fall under Criterion 2.6 of Attachment 1 of CIP-002-5.1 as
Medium impact. The criterion reads:
“Generation at a single plant location or
Transmission Facilities at a single station or substation location that are
identified by its Reliability Coordinator, Planning Coordinator, or
Transmission Planner as critical to the derivation of Interconnection Reliability
Operating Limits (IROLs) and their associated contingencies.”
All three entities reached out to me because
they wanted to know what ISO NE meant would be in scope as Medium impact. Was it the whole plant? The unit?
Or just the AVR system? Two
employees of one of these entities brought this up to me at the CIPUG,
which they were attending because they also have assets in WECC. After discussing it with them separately at
dinner, we all (by chance) ended up discussing it the next day at a break, with
a high-ranking NERC staff member who was attending the meeting.
Let me be
clear before I go any further that my interest in this discussion isn’t
primarily how it turned out, although I will tell you now that all three
parties – the entity, myself and the NERC staff member –ended up in agreement
on how this email should be interpreted (and a later email from ISO NE
reinforced that conclusion). My interest
in this case (perhaps “fear and dread” is a better term) has much more to do
with the process by which this “decision” was made. The epiphany I experienced was the
realization that, at least as far as the Attachment 1 criteria go, this process
will probably not be the exception but the rule – in other words, between now
and 4/1/16 the only way of addressing questions about application of the
bright-line criteria (and perhaps in other areas of CIP v5 as well, like
CIP-002-5.1 R1 and CIP-005-5 R1) will be through ad hoc “rulings” – not Lessons Learned, FAQs, etc. This deals a further blow to any small hope I
may still
have had that CIP-002-5.1 (and perhaps one or more other v5 standards) can
ever be a clear-cut, enforceable standard.
It also shows that NERC and the regions are simply making up their
approach to CIP v5 implementation as they go along – it is now way too late to
deal with these issues in the “proper” way.
Now that
I’ve told you the punch line of this post, you’re free to go. However, if you want to know why I came to
this conclusion, you can read on.
I’m sure a
lot of entities had questions about the ISO NE email, since the ISO subsequently
sent a follow-on email. Because these
two emails were obviously sent to the same large number of entities, I have no
ethical qualms about discussing and quoting their contents. The email described a conversation among
representatives of ISO NE, NPCC (which is, of course, the Regional Entity
covering New England) and a large integrated utility. The heart of the email was this sentence:
“NPCC
indicated that its expectation is that because AVR/PSS status is the specific
component of a generator that is critical to the derivation of IROLs, Generator
Operators must protect the generator’s primary means of transmitting AVR/PSS
status to ISO-NE under CIP-02-5.1 as a Medium Impact BES Cyber Asset.”
In other
words, neither the entire plant nor even one unit is Medium impact under
Criterion 2.6; just the AVR/PSS system (and its RTU) is. At the WECC meeting, I, the entity (not the
one referred to in the above email) and the NERC staff member all agreed this
was the right approach from the standpoint of protecting the grid, as well as
not requiring a lot of unnecessary compliance expense by a large number of
generation entities.[i]
So is
everybody happy? It seems so – everybody
but me. Because I insist on asking a
silly question: Is this “ruling” actually in compliance with the wording of
CIP-002-5.1 R1, Attachment 1, and Criterion 2.6? The answer to that is “no”. I do agree with the outcome of our discussion
(and the second email from ISO NE), but it is clear to me that this outcome was
achieved in a completely ad hoc
manner, without any close attention to how the standard is written. And as I mentioned above, I fear this method
of determining questions about the bright-line criteria will become the rule,
not the exception.
Let’s go
back to the discussion between the entity, me and the NERC staff member. The entity started out by saying they thought
the original ISO NE email (neither I nor the NERC staff member had heard of the
second email at this point) meant that only the AVR system and the RTU were
Medium impact – and that they were Medium impact BES Cyber Systems.
The NERC
staff member’s initial reaction was that either the whole plant or a single
unit was Medium impact. I believe he
said that because he subscribes to the mistaken (but widely held) belief that
the Attachment 1 criteria refer to assets of the six types listed in R1
(control centers, Transmission substations, etc). Needless to say, “AVR systems” isn’t one of
those six asset types. Moreover, the
email said “ISO New England has determined that Generation Facilities represented
by your company have an AVR…” Whatever
“Generation Facilities” means, it can’t be the same as the AVR system;
otherwise, the email wouldn’t talk about Generation Facilities having an AVR. Therefore, the NERC staffer said what he did:
the plant (or the unit) is what is Medium impact.
This was met
by a horrified reaction from the entity, since if that “ruling” had stood, it
might have easily added $1MM or more to their CIP compliance costs (and to
those of every other generator that received the email). At this point, I helpfully piped up to say
that, since Criteria 2.3 – 2.8 all refer to “Facilities”, and since AVR might
meet the NERC definition of Facility[ii], then
the email made sense. In essence, the
email was saying “You have a generation Facility – AVR – that is subject to
Criterion 2.6 since it is essential to our derivation of IROLs.”
But how
would this lead ISO NE to conclude that the AVR system is a Medium impact BES Cyber Asset? Presumably because Criterion 2.6 would make
it so. But how does it do that?
Let’s look
at the context of all of the Medium impact criteria (i.e. all the 2.X
ones). They are all prefaced with the somewhat
mysterious phrase “Each BES Cyber System, not included in Section 1 above,
associated with any of the following:”
This phrase is logically preceded by requirement part 1.2, which reads “Identify
each of the medium impact BES Cyber Systems according to Attachment 1, Section
2, if any, at each asset.” In other
words, this whole chain means “The entity has to identify and classify BES Cyber
Systems that are associated with..”
(whatever Criterion 2.6 talks about). What
does 2.6 talk about, anyway?
As I
mentioned above, a lot of people – including, unfortunately, many in the NERC
regions and in NERC itself[iii] – will
tell you that the Attachment 1 criteria apply to BES “assets”[iv] that
correspond to one of the six types listed in CIP-002-5.1 R1: control centers,
Transmission substations, etc. This is
absolutely not true (I have discussed why this is so in a few posts, including this
one under the heading “Have an Apple, Adam?”
I hope to devote an entire post to this fallacy in the future). The only definitive statement that can be
made about what the bright-line criteria refer to is that they refer to their
subjects.[v] A less definitive - but more instructive - statement would be that the
criteria refer to a) “assets” (including but not necessarily limited to the six
types in R1); b) “Facilities” (a NERC defined term); or c) subjects of some of
the Medium criteria that meet the definition of Facilities (like SPS, RAS or
load-shedding systems) without using that word (I’m referring most specifically
to Criteria 2.9 and 2.10). Even better,
just don’t worry about what the criteria refer to. They refer to what they refer to, period. But they definitely refer to a lot more than
the six asset types; if you don’t believe me, read them and see for yourself.
To return to
our discussion, the “subject” of 2.6 is “Generation at a single plant location
or Transmission Facilities...” Since
we’re not talking about Transmission Facilities here, ISO NE must have been considering
“Generation at a single plant location” to be the operable part of 2.6 for the
purpose of their email notification. What
does this phrase mean?
The word
“generation” is used in two other criteria: 2.3, where it appears as “Each
generation Facility” and 2.1, where it appears as “Commissioned generation, by
each group of generating units at a single plant location…” Given this, it seems pretty clear that the
above phrase in 2.6 means either the whole plant or at least a single unit -
i.e. the interpretation the NERC staffer originally gave. If he was right that the email referred to
the whole plant or at least one unit, this would mean that every BES Cyber
System associated with the plant or unit would be Medium impact; this of course
might be quite a lot.
But since I
was convinced (not having the criterion in front of me at the time, but having
read it a few times before) that
Criterion 2.6 referred only to “Facilities”, and since I knew that a
single system could be a Facility (as in the case of some SPS and RAS systems),
I thought that the AVR “system” (meaning the hardware that actually implements
its purpose, as well as the cyber assets that control and/or monitor the
hardware) was what was Medium impact – i.e. the system was a “generation
Facility”. Of course, my position was
supported by the fact that the email referred to “Generation Facilities”[vi]. All three parties agreed this was the right
conclusion, and we returned to the NERC meeting, happy[vii] that
we had made our own small contribution to the jurisprudence of NERC CIP Version
5.
However,
when I actually reread Criterion 2.6 afterwards, I realized I had been
wrong. I had been thinking it read
something like “generation and Transmission Facilities” – in other words,
“generation Facilities and Transmission Facilities”. However, it reads “Generation at a single plant location or
Transmission Facilities…(my emphasis)”
If the SDT had wanted the first part to mean “Generation Facilities”
they would have used those words.[viii] On the other hand, “Generation at a single plant location” almost
certainly means the whole plant or at least a unit (and of course, the AVR
system can’t be considered “generation”.
By itself, it doesn’t generate anything, except perhaps compliance
uncertainty).
So the NERC staffer had been right in the
first place, and shouldn’t have changed his mind, at least as far as the
wording of the criterion is concerned.
However, given that the two employees of the registered entity that had
received the email were quite passionate that this was the wrong answer – and
given that they were both fairly big guys – it is understandable that he
changed his mind (especially when I chimed in with my opinion supporting the two guys. NERC staff members always defer to what I say
without questionJ).
I was going to deliver this bad news to the
entity the week after the CIPUG. That
is, I was going to say I had been wrong and the NERC staffer should have stuck
with his first interpretation: either the whole plant or at least one unit
would be Medium impact. However, one of
the entity employees forwarded me the second email from ISO NE – which said
unequivocally that the AVR system (with its RTU) was a Medium impact BCS[ix],
and this was the only system that would have to be so designated. That settled the matter, as far as I was
concerned. This also saved me the
embarrassment of having to tell this entity I was wrong (which rarely happens,
of course).
But the fact is that I was wrong, and the NERC staff member was right in his original
“ruling”: the whole unit is Medium impact if you go by the actual wording of
Criterion 2.6. Yet is this staff member
going to raise a stink and insist to ISO NE and to the generation entities that
this is the case? I highly doubt it, not
because he’s a coward but because a) ISO NE has issued their “ruling” in the
second email; and b) the ISO clearly made the right decision from a purely
practical point of view: What needs to
be protected to solve this IROL issue is the AVR system, nothing more.[x]
At Long
Last, the Conclusion
So what is the point of this long and
seemingly pointless narrative? After
all, I’ve said I agree that the final “decision” – made in our informal meeting
at WECC as well as in the ISO NE email – was a good one. What’s my beef?
My beef is that this was such an arbitrary
process, and one that is totally unsupported by the wording of
CIP-002-5.1. It is clear that none of
the Attachment 1 criteria apply to BES Cyber Systems directly; rather they
apply to Facilities and assets, as I said above. Once you’ve identified the asset/Facility as
Medium impact (using the criteria), then you find the BCS “associated” with it
and classify those as Medium BCS. For
ISO NE to say that the generation entity is required to identify particular
Cyber Assets (the AVR system and its RTU) as Medium BES Cyber Assets betrays –
frankly – a lack of understanding about perhaps the fundamental component[xi]
of the asset identification and classification process in CIP Version 5.[xii] As for the fact that NPCC – who presumably
approved of the second email, since they were one of the three parties at this
meeting – also showed this lack of understanding…well, I can only say I don’t ultimately
blame them either. I do blame CIP-002-5.1, because its many inconsistencies and contradictions are at the heart of this problem. And I blame NERC for continuing to forge ahead with
implementing this standard as these problems become more and more evident (see "programmable", "adversely impact", "reliability purposes", etc).
Moreover, it was obviously very arbitrary for
the NERC staff member (a very prominent member of the CIP team) to first state
the correct answer in our conversation in Anaheim, then back off when he got
opposition.[xiii] To be politically correct, I should reprimand
him for even trying to give an answer; instead, he should have said he’d take
this issue back to the NERC Transition Advisory Group, so they can put it on
their list of Lessons Learned they intend to write.
But I certainly understand why he wouldn’t
want to do that, either. It would be
many months – if not longer – before the TAG could get to this issue, and then
it would take months to draft the Lessons Learned document, post it for
comment, and finalize it. Given all of
this, there will probably only be a few months between when it’s finalized and
April 1, 2016; what good does it do anyone to release it then?[xiv] In fact, except for a few questions for which
Lessons Learned will soon be finalized, it’s probably now better for NERC to
put off any new Lessons Learned (or other general “rulings”) on CIP-002-5.1 R1
and Attachment 1, and just issue individual rulings on particular issues (even
applying to single entities, if that’s needed).
In other words, given that NERC has this
deeply flawed standard, and that it is now too late to change it without a huge
upheaval[xv],
probably the only way it can be implemented – given the many problems with the
Attachment 1 criteria – is to do exactly what NERC and the regions seem to be
doing, and will do more and more in the near future: make individual ad hoc “rulings” to resolve particular
issues. These rulings may sometimes be
for the entities in a particular ISO footprint; they may sometimes be for
entities in a certain region; and I’m sure they’ll sometimes be for particular
entities all by themselves. This is
going to happen more and more, and is already starting to happen. Of course, I’ll document those instances when
I find them; I’m sure I won’t have to look too hard (I already have one other
example that will appear in one of the next posts in this series).
Of course, it’s kind of hard to call CIP a
“standard” when its interpretation is done in an ad hoc manner. Maybe NERC
can figure out another name.
The views and opinions expressed here are my
own and don’t necessarily represent the views or opinions of Honeywell.
[i]
I have no idea how applicable this “ruling” will be in other NERC regions or
other ISO footprints. Such a discussion
is above my pay grade.
[ii]
Whether it does or not would be an interesting question, since I think a
Facility would have to be operated at high voltage and have terminals on
it. I’ve never seen an AVR system, so I
don’t know whether or not that is the case.
Fortunately, the answer doesn’t matter for this post.
[iii]
And myself, up until around this time last year.
[iv]
With “assets” being an undefined term, commonly used in the industry to refer
to “big iron”: substations, generating stations, control centers, etc.
[v]
Although someone pointed out to me that the Attachment 1 criteria don’t have subjects, since they aren’t
sentences in the first place. However,
even I have to draw the line somewhere.
I say they have subjects, and I won’t attempt to justify this. If you want to fight about this, just name
the time and place and I’ll be sure (not) to be there. Of course, the ultimate arbiter of
intellectual disputes used to be the dueling pistol. Fortunately, I don’t believe they make those
anymore.
[vi]
Of course, ISO NE shouldn’t have capitalized “Generation”, since it isn’t a
NERC defined term.
[vii]
As I said in my previous post, there was a lot of fantasy going on at that
meeting, including by yours truly. These
fantasies allowed people – NERC entities, NERC and WECC staff members, and
consultants like me - to go away feeling they’d made a lot of progress at the meeting. Whatever gets you through the day is OK with
me (with some exceptions)…
[viii]
This is supported by the fact that Criterion 2.3 refers to “Each generation
Facility”. The SDT could have easily
used exactly the same phrase in 2.6, had they wanted to mean that.
It was pointed out to me that the Guidance and
Technical Basis for CIP-002-5.1 does pretty clearly imply that the SDT meant to say “generation Facilities”,
when it says “Criterion 2.6 includes BES Cyber Systems for those Generation
Facilities…” Of course, the SDT
shouldn’t have capitalized “Generation” here. Leaving that quibble aside, it is quite
distressing that they would state so clearly in the Guidance that they meant
“generation Facilities” in 2.6, and then quite deliberately use a different
phrase in the Criterion itself.
And since I’m in a conspiratorial mindset at the
moment, and I’m in a footnote that nobody who has anything better to do will
read (this means you, by the way), I’ll expand on my theory about why the SDT
might have made this “error”. The NERC
definition of Facility reads “A set of electrical equipment that operates as a
single Bulk Electric system Element (e.g., a line, a generator, a shunt compensator,
transformer, etc.)” Since “generator” is
the only one of those terms that applies to a generating plant, this could
easily be taken to indicate that, in the context of a generating station,
“Facility” would refer to at least a unit of the plant (since a unit is the
smallest part of the plant that could be called a “generator”). Thus, had the SDT used the words “generation Facility”
in Criterion 2.6, this might have been taken to mean simply “generation
unit”. This means that notices like the
one from ISO NE might have been taken to mean a whole unit was in scope, thus
substantially increasing compliance costs for generation entities; ergo the SDT didn’t want to use the term
“generation Facilities” in 2.6.
While I’m on a roll, I’ll continue. I think it’s quite unfortunate that the SDT
used the non-defined term “generation” three times in Attachment 1, each time
to mean something different. In
Criterion 2.1, the term definitely means an entire generating plant. In 2.3, it is part of the term “generation
Facility”, and clearly means one or more individual units in a plant. Yet in 2.6, the SDT seems to have wanted to
say “generation Facility” (and did say that in the Guidance), but, perhaps because of
the consideration just mentioned, they didn’t. I think this is because they wanted to make the phrase “Generation at
a single plant location” in 2.6 apply to other generation Facilities (like
AVR), not just the units themselves. I
don’t know whether the SDT thought it was being pretty cool by using the same
term in three different ways, but it certainly introduced even more confusion
into the bright-line criteria (I’ve already complained
several times about the fact that the SDT tried to be far too parsimonious with
words, and ended up making CIP-002-5.1 R1 essentially unenforceable).
[ix]
Both the first and second notes actually said that the AVR “system” was a BES
Cyber Asset. They should really have
said BES Cyber System, since the AVR system includes the AVR computer itself
and the RTU.
[x]
Of course, some argue that the bright-line criteria are way too lenient on
generating plants, since the only plants that are clearly Mediums are those
over 1500MW. Whether or not that’s true,
it’s a different issue.
[xi]
The ISO NE email provides an even more egregious demonstration of lack of
understanding of CIP v5. The last
paragraph contains this sentence: “If an entity receives a notification that a
generator is critical to the derivation of IROLs and that notification does not
specify any particular component that is critical, then entities would have to
consider CIP protection for the entire station as a Medium Impact BES Cyber
System.” So they’re saying that an
entire generating station could be a BES Cyber System! Let’s see…a BCS is made up of BCAs, which are
themselves Cyber Assets. Where do the
turbines or the boiler fit into this? I
never thought of them as cyber assets before.
Again, I don’t particularly blame ISO NE; it simply shows the magnitude
of misunderstanding that’s out there about the v5 asset
identification/classification process, including at some entities that play key
roles in the process of implementing v5.
[xii]
I will admit there is a chance that the assertion in the email that the AVR
should be a Medium BCS under 2.6 doesn’t mean that ISO NE misunderstood how
CIP-002-5.1 R1 is supposed to work. It
may mean they have realized that the general understanding of the word
“generation” in 2.6 meant that the AVR system couldn’t fall under that
criterion – and the whole plant or unit would have to be Medium impact; so
designating the AVR a Medium BCS, however “illegal” given the wording of the
standard, was the only way to achieve the result they wanted (and as I’ve said,
even I agree that this is the best result).
I would give this possibility more credence if ISO NE hadn’t also made
the error described in the previous footnote.
[xiii]
It may seem odd that I’m reprimanding this NERC staffer for changing his
opinion to the one I was advocating in Anaheim – and which I still say is the
best decision from a reliability and resource efficiency point of view. But hey...I don’t want NERC to change its
“rulings” based on something I or anybody else says (and because of this
staffer’s position in NERC, what he said will definitely be taken by the entity
as definitive and will probably be proudly recounted by the entity in the
future, should how they handled the AVR issue be questioned in an audit). I want NERC to rule based on what the
standard says. And if the standard is
too poorly written for them to be able to properly address the issue at stake,
I want them to admit that and write a SAR for a new standard.
[xiv]
In fact, I would argue that NERC needs to stop producing all Lessons Learned
(at least on CIP-002-5.1) at some point in time before the compliance
date. This is because it can be
literally counterproductive for these documents to come out too late. For example, I’ve written
about the need for clarification on the meaning of “affect the BES” in the
definition of BES Cyber Asset. NERC
doesn’t even have this on their list of planned Lessons Learned, and at this
point I’d tell them not to bother putting it on. An entity with High or Medium impact assets
needs to be working now on identifying their BES Cyber Assets; since NERC
hasn’t come out with any guidance on this issue, entities need to “roll their
own” now – as I’ve been saying in a number of posts. Were NERC to come out with a Lessons Learned
in say November, what good would that do – except to call into question the
whole asset identification process at the entity, long after it is too late for
the entity to make any changes to the list of BES Cyber Assets/Systems? It’s now better to just wait until after
4/1/16 to start more Lessons Learned on CIP-002, unless they can be finalized
by this summer. Even better, NERC should
take up my suggestion
to push the compliance date back by a year, so that more LL’s can be developed
in time for them to actually do some good.
[xv]
I continue to assert that CIP-002-5.1 needs to be rewritten. This will require an admission of a huge
failure on NERC’s part. Organizations in
general – and NERC is certainly no exception – are always very reluctant to
take such a step. But I’m willing to bet
that in six to nine months’ time, the situation may have changed to the point
that this doesn’t look like such a crazy suggestion.
No comments:
Post a Comment