Roll Your Own, Part I: "Programmable"

Note: This post became the first in a series of posts on this issue.  You can find them all by dropping down the month folders on the right of the screen.

Prologue

I recently realized that I have progressed through all of the stages of grief regarding the problems in CIP-002-5: Denial (not seeing these problems while the standards were being drafted), Anger (when I realized the extent of the problem last year), Bargaining (lobbying FERC to order the standard be rewritten), Depression (which you can see in a number of my recent posts), and finally Acceptance (meaning I have come to realize that the more fundamental problems with the standard will never be addressed by someone with the authority to make their fix stick). 

However, if you work for a NERC entity, you probably envy me having the luxury of being able to “accept” the problems.  After all, I’m not on the hook for complying with CIP v5, but you are.  While I approach Nirvana in my blissful state of Acceptance, how are you supposed to keep out of the slammer on a charge of Aggravated CIP Violation?

This post started out as just another in my long series on the problems with CIP-002-5 (I’m sure I’ve done at least 30, probably more than that).  My subject this time is the word “programmable” in the NERC definition of Cyber Asset: “Programmable electronic devices”.  Basically, the problem is that “programmable” isn’t defined, and for some entities, the definition used could literally lead to additional millions in compliance costs.

A gentleman who I’ll discuss below told me how he is coping with this problem; it involves basically making up his own definition, documenting that, and making sure it is followed rigorously as his organization identifies Cyber Assets.  I will discuss all that, but I came to realize last week that what this person has done can be a paradigm for dealing with most of the unsolved problems in CIP-002-5.1 (and perhaps the other standards, although my guess is those problems are more easily addressed through normal “interpretation” activities than the CIP-002 problems are).

Yes folks, I’m saying you should consider the following procedure for dealing with the inconsistencies and vagueness in CIP-002-5.1:  Roll Your Own solution.  That’s why I’m calling this Part 1 in a series on how entities are making their own definitions, rewording the requirements so they make more sense, etc.  Of course, I’ll be pleased to hear how anyone else might be doing something similar (even if you won’t allow me to mention it in a post).  People have to be doing this now (perhaps inadvertently – in fact, I know many entities think they’re following the standard as written, when they’re probably not); otherwise, nobody would be making any progress at all on CIP v5.  But that’s OK; you need to move forward, even if that means Rolling Your Own.  And you’re in good company in any case.

And Now, Our Feature Story

I think a number of you are probably aware that the meaning of “programmable” is a big deal.  But I think that the people most aware of the problem are those in Generation, and especially owners/operators of large coal plants.

The problem is that these plants can have astounding numbers of electronic devices – transmitters, valve positioners, I/O cards, I/O processors, etc.  Someone who works for a large generation entity told me they have one plant that has 120,000 devices that could potentially meet the NERC definition of Cyber Asset. 

Let’s see, how long would it take to a) determine whether all of these were Cyber Assets or not; b) determine whether or not they were BES Cyber Assets; c) aggregate the BCA’s into BES Cyber Systems; and d) apply all of the requirements of CIP-003-6 (yes, 6) through CIP-011-2 (yes, 2) to these BCS?  My calculator won’t go that high, but my guess is it’s in the man-decades, if not man-centuries.  Something on the order of, “I guess we’ll have to stop generating electricity for a couple of years while everybody works on CIP version 5 compliance.”

And what makes all the difference?  The meaning of the word “programmable”.  Almost any electronic device can be “programmed” in some sense.  You might have to replace every bit of firmware in it, but by some definitions, this would constitute “programming” it.

I know there has been a lot of discussion in NERC circles about this problem, and I’m told it is one of the questions being addressed – in some way – by the new CIP Version 5 Transition Study Group (which I discussed in a recent post).  But entities really need to have this definition now (i.e. September 24, 2014 at 8:55PM Pacific Time), and many simply can’t wait any longer.

Enter the Hero.  This person is the CIP Compliance Manager for the Generation arm of a large IOU.  His job is to be compliant by April 1, 2016, not to tell everybody in his company to hold off doing any work on v5 until NERC does or doesn’t come up with a usable definition of “programmable”.  As he said in a recent email to me “Unless NERC defines a programmable logic device, the Responsible Entity should define it.”  There you go – the shot heard round the world.  Here is his definition of “programmable”:

An electronic device is normally considered as “programmable” if it has two or more of the following attributes:

a.        QWERTY or similar Keyboard or keyboard connection

b.        Management and/or programming port

c.        Serial (RS-232) or USB port

d.        Any routable protocol port, e.g. 10/100/1000BaseT port(s), 10/100/1000BaseFl port(s), WiFi, 802.x.

e.        Telephone line port

f.         Ability to configure three or more User accounts with complex passwords.

g.        Alterable memory, including firmware, that can be reprogrammed to change the function of the device.  This does not include alterable memory for the storage of configuration changes.

I won’t comment on this definition on a technical basis[i], but it does seem quite reasonable to me.  The bigger issue is, once you’ve decided on a definition[ii] for “programmable” – or any similar undefined term in the v5 standards, what do you need to do to feel comfortable actually applying it?

This gentleman did two primary things, which I recommend for anyone who wishes to follow in his footsteps (whether or not you use his definition of “programmable”, or even if this is for an entirely unrelated CIP v5 issue): He documented the definition and his rationale for wanting to use it, and he kept that as part of his compliance records.  He feels he's well prepared should an auditor challenge his definition three or four years from now.  And his answer will be simple - what else could I have done, given that there was no accepted definition to follow?

So that’s all I have to say.  There have really been two subjects to this post:

1. The definition of “programmable”
2. How at least one entity (and I know there are others doing the same thing) has decided to take matters into their own hands and develop their own definition. 

I’m implying – no, stating outright – that developing your own definitions and interpretations may be the only real “solution” to many if not most of the interpretation questions of CIP v5.  Of course, you need to follow whatever NERC and your region(s) say[iii] about these questions.  But you really can’t count on them to a) address every question, or b) provide definitive answers to the questions they do address.  You say this is unfair?  Here I quote – as I have before – the great philosopher Jimmy Carter: “Life is unfair.”

For the second post in this "Roll Your Own" series, go here.

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Honeywell. 

---

[i] There is another blog that was recently brought to my attention – “CIPsecure” by Joseph Jimenez – that provided the following definition of “programmable” (you can find the post by scrolling to near the bottom.  It is dated Aug. 31):

A device that uses digital electronic technology to run pre-configured programming logic to interact with other devices over a communications interface. Programmable devices should also include a processing unit, memory, storage for the programming logic, and a communication interface that is remotely available. The remote communication interface should also allow for the access and modification of the stored programming logic.

At first glance, I think this is operationally not too different from my friend’s definition, shown later in my post.  By the way, Joseph’s blog seems excellent.  I intend to read all of his posts in the future.

Another “definition” can be found in a presentation that Felek Abbas of NERC gave to FRCC in May.  Here is his slide:

                        What makes something “programmable”?

                        * Contains firmware

                        * Firmware is modifiable via device interface (Ethernet, serial, parallel, USB, etc.)

                         

                        Configurable is not programmable.

                        * Electro-mechanical relays

                        * Dip switches

                        * If physical removal of chip is required to program device (EPROM etc.)

                         

My initial thought is that this definition will result in more devices being deemed “programmable” than either Joseph’s or my friend’s.  However, since this can’t be taken as an “official” NERC statement in any way, you should consider it no more than one person’s opinion.

[ii] Our Hero also pointed out to me that in the proposed NERC BES Cyber Asset Survey – which was recently cancelled – the question “How would you define programmable electronic device?” was asked.  Obviously, it would have been quite interesting to see the results for that question.

[iii] The CIP V5 Transition Study Group will post their documents here.

The New CIP-002-5.1 RSAW Draft

I wrote lengthily (not that I ever write any other way) and bitterly about the first draft of the CIP-002-5.1 RSAW in this post in late June.  The second drafts of the RSAWs were released this week, so I eagerly downloaded the new CIP-002 document to see whether it would be better.  Surely, I naively thought, there must have been some big improvements.

Boys and girls, I hate to tell you this: The world doesn’t always (or even usually) follow what we may wish.   So I have good news and bad news for you.  The good news is that all of the statements that I found objectionable in the first draft have been removed.  And what’s the bad news?

The bad news is that NERC has replaced those statements with….nothing.  That’s right, nothing.  All of the statements I cited in the original post were found in a set of blue boxes, labeled “Evidence Requested”, “Compliance Assessment Approach” or “Notes to Auditor”.  The first and third boxes have simply disappeared[i].  And the Compliance Assessment Approach box now consists of nothing but a recitation of CIP-002-5.1 R1, preceded by the words “Verify that…” in several places.

It’s hard to express how depressing this is.  After originally implying that the RSAW’s would shed light on some of the problems with CIP v5, it seems NERC has now completely given up on that idea, and has reduced the CIP-002-5.1 RSAW (and I haven’t read the others yet) to simply a recitation of the requirements.  This wouldn’t be all bad if NERC were at the same time working feverishly on addressing the interpretation problems with that standard; but I see absolutely no sign of that.

Meanwhile, of course, we’re approaching October 1, exactly 18 months from the High/Medium compliance date for v5.  What are entities to do, with no guidance on these issues?  They really can’t go full bore ahead with their v5 compliance programs until they’re satisfied that they’ve identified what’s in scope correctly.  And since the only official (or even unofficial) guidance currently available is the wording of the standard itself, in all its glorious inconsistency and ambiguity, this has to be making a lot of people nervous (or they simply haven’t started their v5 process in any meaningful sense).

Of course, people will find a solution, one way or the other.  I will soon start a series of posts that will discuss how people are “rolling their own” definitions and interpretations.  What else can they do?

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Honeywell.

---

[i] This isn’t entirely true, since there still is a heading labeled “Auditor Notes”.  However, it is completely blank.  Did someone start to write some notes, run into problems, then just give up?  Are they going to add them back in a future draft?  Another in a long line of NERC mysteries.

10/16: It was pointed out to me that this end note doesn't mean anything, since the Auditor Notes are always left blank in the RSAWs.  They're for the auditors to literally write notes during their audit. Of course, that doesn't affect the argument in the body of the post.  I hope to revisit this problem shortly.

Help is on the Way?

Many of you will know I have been complaining for quite some time that NERC needs to step forward and start providing some guidance on the many interpretation issues that are found in CIP version 5, and especially in CIP-002-5.1 R1 and Attachment 1 (my particular obsession).  I wish to announce that NERC does seem to have taken one step in that direction.

I’m told that NERC has formed a “CIP Version 5 Transition Stakeholders Group”, consisting of representatives of NERC, the eight regions, the six entities that participated in the Transition Study, some drafting team members, and some “industry representatives” (my source thinks the latter may be members of the NERC CIPC).  And what is this distinguished group going to do?  Well, it seems they’re going to come up with answers to questions about CIP v5 from NERC entities (that would be you) – I’m told you just have to email those to Tobias Whitney or Steve Noess of NERC (I’ll do a post on the questions I would like to ask; anyone who has suggestions they’d like to have me include can email them to me at talrich@hotmail.com). 

What is the legal basis for this group?  That’s interesting.  I’m told it comes from Section 11 of Appendix 3a of the NERC Rules of Procedure (the page is numbered 43, but it seems to be page 174 of the actual document.  Don’t ask me why they couldn’t just number all the pages consecutively).  That section, called the “Process for Approving Supporting Documents”, discusses “documents that may be developed to enhance stakeholder understanding and implementation of a Reliability Standard.”  Six types of documents are described; the first one, “Reference”, is described as

Descriptive, technical information or analysis or explanatory information to support the understanding and interpretation of a Reliability Standard. A standard reference may support the implementation of a Reliability Standard or satisfy another purpose consistent with the reliability and market interface principles.

And who is authorized to prepare these documents?  You’ll be surprised to hear that it is “any entity”.  So as long as you’re an entity – that is, you exist – you’re authorized to prepare them (presumably, Peter Pan is excluded.  But you could argue even he’s an entity, albeit fictional).  It seems the new CV5TSG (remember, you heard that acronym here first!) is as much an entity as anyone else is, so they will take it upon themselves to prepare documents.

Section 11 goes on to say that the NERC Standards Committee “shall authorize the posting of all supporting references that are linked to an approved Reliability Standard.”  It seems that any document that meets the definition of “reference” above will be posted.  I guess that’s what the new CV5TSG will be doing – preparing references on questions regarding CIP v5.

Is this a good thing?  Definitely.  Is it going to be enough?  Well, that depends on your definition of “enough”.  If you’re looking for interpretations of the v5 standards that will serve as mandatory guides to the auditors (and therefore for the NERC entities themselves), you’ll be disappointed.  There is no way this group can do Interpretations.  As I’ve discussed previously, NERC simply cannot produce official interpretations of any standard without going through the entire Request for Interpretation process, which could easily take two years or more. There is no way that the current interpretation issues with CIP v5 can be dealt with in that time frame; they need to be addressed much more quickly.

But my guess is the group will produce well-reasoned documents that may clarify some important points.  I’m told they’ll be similar to the “Lessons Learned” documents that have already been posted, which have been well-written if not particularly earth-shattering.  Note that the quote above does say references can “support the understanding and interpretation of a Reliability Standard.”  If the CV5TSG actually produces documents that do that, this will be a significant step forward.

But there’s a catch (of course).  We’re now just over 18 months away from April 1, 2016, when High and Medium impact assets (and their owners/operators) need to be 100% compliant with CIP Version 5.  There are lots of questions that need to be answered (you could go to almost any one of my posts since the end of April 2013 and find at least three or four) quite quickly – especially on CIP-002-5 R1 and Attachment 1, which of course are the foundation for everything else in CIP version 5.  How likely is it that this team will be able to address most if not all of the significant questions soon enough for that to be of help – say by the end of this year at the latest?

And here, Dear Readers, is the bad news: I think it’s highly unlikely this will happen.  I’m going to spend a little time discussing what’s at stake here:

• Those of you who were involved with NERC CIP compliance four or five years ago know that the NERC CIPC published two very good guidelines on the CIP-002 asset identification process: one on identifying Critical Assets, the other on identifying Critical Cyber Assets (if you need a copy of these, you can email me and I’ll send them to you.  Alternatively, you could spend a couple hours looking for them on the NERC website).  These were both excellent documents, and the latter still provides good insights into issues like “external routable connectivity” that remain in CIP v5.
• I think it would be great if similar guidelines were developed now, although – given that the identification and classification of “big iron” and “little iron” are so intertwined in CIP-002-5.1 – this would need to be a single document.  This would really be the right thing to do, and it is what I have been in part requesting since 2012.  But this is simply not going to happen. I believe the two previous documents took around a year to develop, and even if the team started now, the new document would arrive way too late for it to be of help in the initial identification of BES Cyber Systems, prior to the April 1, 2016 date[i] (I keep threatening to write my own mini-version of this document as a post, and I intend to do it soon).
• So the best we can hope for is answers to specific questions.  While not being the comprehensive approach I’d prefer (at least for the CIP-002-5.1 questions), it’s certainly better than nothing.
• So what are the questions the new team will be addressing?  My contact says efforts are underway on the following topics: a) Grouping BES Cyber Assets into BES Cyber Systems; b) a definitive discussion of the “far-end” transfer-trip relay issue (this was already addressed in an email from Steve Noess, but I guess the new document would be more thoroughgoing and would carry more official weight - although I of course CAN’T call it an interpretation); c) Virtual Systems and VLANs (which has been an issue since CIP V1, so this is certainly needed); d) Disaggregation of BES Cyber Assets at a generating plant (I assume this refers to the fact that BES Cyber Systems identified through what I call the “top-down” approach need to then be disaggregated into their component cyber assets.  See this post); and “perhaps” even e) What the word “programmable” means in the definition of BES Cyber Asset (my next post will address this issue, which is a pretty big one, especially for generating plants).
• What will be the pace at which the group turns out these documents?  Remember, these are people who all have day jobs.  They’ll be meeting once a month (I doubt for more than two days).  Even though I know that work is already proceeding on some of these questions (the virtualization question has been under discussion by a group of the NERC CIPC for at least a few months), I sincerely doubt they’ll be able to turn out more than say two documents a month.  So the list above can perhaps be completely taken care of this year.
• That’s wonderful, but what about all of the other 5,689 “interpretation” questions on CIP version 5?  It won’t do most entities a lot of good to have them addressed even next year, let alone in 2017 or 2018. 
• There is another initiative I know that’s going on, which is that NERC is providing uniform training on CIP v5 for all of the auditors in the regions.  Again, that’s great and really needs to happen, but is that training really going to address a lot of the other questions that the CV5TSG won’t be able to address in the near future?  I can assure you that won’t happen. 

So what are we left with for all of the other v5 problems (and now I’m even more motivated to come up with a list of issues that I see.  I hope some people will email me their issues as well; I promise to list them all, without attribution of course)?  I’m afraid they will ultimately be dealt with using the time-tested NERC method: auditor discretion.  Isn’t that wonderful?

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Honeywell.

---

[i] This doesn’t mean it shouldn’t be done, since entities will always be coming into the CIP program, adding new Medium or High impact assets, etc.  But I certainly don’t see any movement to do that, so this is all academic.

“Transmission Facilities”

My most recent post on substations has drawn a lot of interest.  It points to a couple problems in the CIP-002-5.1 Attachment 1 criteria for which there aren’t clear answers.  I invited people to send me other examples of problems with the criteria, and Stacy Bresler of EnergySec immediately brought one up.

His is quite simple: The words “Transmission Facilities” (both capitalized) are used several times in the criteria.  Since they are capitalized, this means these are defined terms – either together or separately.  There is no NERC definition for “Transmission Facility”, but there are definitions for both “Transmission” and “Facility”.  However, Stacy points out that combining the two definitions really doesn’t produce much enlightenment about what really is a transmission facility.

I think the problem is mainly with the “Transmission” definition:

An interconnected group of lines and associated equipment for the movement or transfer of electric energy between points of supply and points at which it is transformed for delivery to customers or is delivered to other electric systems.

I believe Stacy’s concern is that many substations combine elements that would normally be called “transmission” and “distribution”.  Where exactly do you draw the line between them?  This is no idle concern, since an entity that doesn’t properly draw that line will end up either spending much more money and time than they need to protecting systems associated with purely distribution elements (lines, transformers, breakers, etc), or else they’ll end up not properly protecting their transmission elements – which can result in big fines.

Stacy does point out there is a workable definition of “Transmission Facility” that does provide the specificity that’s required.  That’s the good news.  The bad news is that this definition only applies to entities in the province of Alberta, since it is found in the glossary of the Alberta Electric System Operator (AESO) – which can be thought of as a combination of NERC and PJM for that province.

Stacy provided a link to the glossary, but here is the definition, in all its glorious specificity:

an arrangement of conductors and transformation equipment that transmits electricity from the high voltage terminal of the generation transformer to the low voltage terminal of the step down transformer operating phase to phase at a nominal high voltage level of more than 25 000 volts to a nominal low voltage level of 25000 volts or less, and includes

(i) transmission lines energized in excess of 25000 volts,

(ii) insulating and supporting structures,

(iii) substations, transformers and switchgear,

(iv) operational, telecommunication and control devices,

(v) all property of any kind used for the purpose of, or in connection with, the

operation of the transmission facility, including all equipment in a substation

used to transmit electric energy from (A) the low voltage terminal, to (B) electric distribution system lines that exit the substation and are energized at 25 000 volts or less, and

(vi) connections with electric systems in jurisdictions bordering Alberta, but does not include a generating unit or an electric distribution system.

My guess is this definition will get entities a lot closer to what they need than the anemic NERC definition of Transmission does.[i]  Does that mean everybody should start using Alberta’s definition in order to “slice and dice” the elements in their substations between Transmission and Distribution?  Well, I think this at least gives you a template for working out your own definition (or maybe NATF or some other organization can write one). 

As one of my next posts will discuss, when NERC doesn’t define a term or only vaguely defines it (and when no guidance is put out to remedy this problem, which of course is the rule with CIP v5, not the exception), I think the door is open for the entity to find something that works and use that to guide their efforts.  Just be sure to document it, and inform your Regional Entity you’re doing this…unless they have some better definition for you to use.

Of course, the best solution to this problem is for NERC to provide guidance on this question, as well as many others.  My next post will discuss a new initiative NERC is undertaking to try to provide some guidance, and the challenges involved with that effort.  But don’t start thinking miracles are on the way.

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Honeywell.

---

[i] In the Guidance and Technical Basis section of CIP-002-5.1 (page 23), the SDT did mention that entities should separate Transmission (called “BES Operations”) from Distribution Facilities in a substation:

However, in a substation that includes equipment that supports BES operations along with equipment that only supports Distribution operations, the Responsible Entity may be better served to consider only the group of Facilities that supports BES operation.

This statement seems to assume one of two things:

1.     The NERC definition of Transmission is sufficiently clear that everyone will be able to distinguish these Facilities with no problem; or

2.     There isn’t any clear line between Transmission and Distribution Facilities, meaning that whatever the entity thinks is appropriate will be fine with the auditors.  This is implied in the sentence that begins the paragraph from which the above sentence is taken: “When the drafting team uses the term ‘Facilities’, there is some latitude to Responsible Entities to determine included Facilities.”   Of course, just because the SDT said there is “some latitude” doesn’t mean you’ll be given any latitude at all when audit time comes. 

This is simply another example of the bright-line criteria requiring a lot of interpretation.  And I don’t see anybody – other than the regions on an individual basis, and even then it’s quite spotty – to do this.

What is a Substation?

I have known for a long time that the “bright-line” criteria were anything but bright, and that there were going to be a large number of questions when it came down to actually applying them.  Well, guess what?  People are applying them now, and there are a large number of questions.  This is the first in what will probably be a number of posts (not all together, of course) discussing bright-line criteria problems. 

I used to think there would be a guidance developed for using the BLC (beyond what is in the Guidance and Technical Basis section of CIP-002-5.1, which is good but simply limited), but I certainly don’t see that happening now.  I imagine these questions will be answered the way most other CIP v5 questions will be answered: by each individual region (or even each auditor within each region), with great variability across regions.  That’s why I call them the Not-so-Bright-Line Criteria.

The reason why it’s inevitable there will be many questions on the BLC is that the electric power industry is tremendously diverse.  I am always struck by the differences between facilities (e.g. one generating station vs. another, one substation vs. another), between the NERC entities themselves (obviously, coops vs. munis vs. IOUs, but even within those three categories there is huge variability in what they do, etc.  That’s why NERC has all those registrations, and why almost no two entities have the same registrations), and between the environments in which the entities operate (entities in PJM have an entirely different set of constraints than those under CAISO or SPP, for example).  There is simply no set of criteria like the BLC that could cover this diversity.

I noticed early on (when entities started considering CIP v4) that just about everyone I talked to would say, “We think we understand where we fit in the bright-line criteria, except for….”  And here they would rattle off a long story about how they have one substation or generating plant that might look like it meets one of the criteria but in reality it doesn’t because of some complicated reason that applies just in their region, etc.  I figure that, if every NERC entity has at least one “except for” situation, and there are maybe 5-800 entities subject to CIP v5, there are a whole bunch of these issues out there, waiting for me to write about.  So I expect to be done with this series of posts in about 2020, and then only because CIP v7 or v8 will be in force, not because all of the questions will have been answered.

Now on to the problem….

There are several strands to this problem, but the main one is that there is no NERC definition of “substation”.  Of course, the SDT knew this when they drafted CIP v5, and for the most part the criteria in Attachment 1 of CIP-002-5.1 are free of any dependence on the word “substation”.  However, that is not the case for criterion 2.5.

In that criterion, the Facilities that are Medium are those between 200 and 500kV.  As everybody knows by now, there is a formula to weight the different lines in the substation, and the sum of those weights has to exceed 3,000 in order for these Facilities (i.e. the 200-500kV ones) to be Medium.  If that sum is less than 3,000, then all of those Facilities, and the BES Cyber Systems associated with them, are Low impact. 

Now suppose you have a substation with two control rooms, one that controls two 345kV lines and another that controls one 245kV line (whether this is likely to happen or not, I have no idea.  I’m not a power engineer, and I don’t even play one on TV – assuming there are any TV shows about power engineers).  The two 345kV lines are worth 2600 points in the formula, while the one 245kV line is worth 700 points.  Together the substation has 3300 points, meaning that all the BES Cyber Systems associated with these three lines will be Medium impact[i].

But let’s say the two control rooms (and their respective lines) are in two separate substations.  The point total for the first substation will be 2600, while the second will be 700.  In this case, all the BES Cyber Systems in both substations will be Low impact, since they no longer meet criterion 2.5, and don’t meet any of the other Medium criteria.

You probably know where I’m going on this, but let’s go back to the original case where both control rooms are in the same substation.  Now let’s start moving the two control rooms apart (since this is a thought experiment, we can do this in the blink of an eye.  It would take somewhat longer than that – he says with a grin – to do this in real life). 

Let’s move the second control room 200 yards away from the first.  Are they still the same substation?  If we think so, let’s move them half a mile apart.  Are they still the same substation?  How about if they’re a mile apart?  Or is it the fence that makes the difference?  As long as there are two fences between the two control rooms, could they be located just twenty feet from each other and be in separate substations?

Of course, there’s no answer to these questions, since there’s no definition of “substation”.  But I predict this will become an issue as entities focus more on their compliance costs for substations subject to criterion 2.5. 

Now let’s put another turn on the screw.  For the remaining discussion in this post, I wish to acknowledge the assistance of a large transmission entity that brought this issue up to me and helped me (try to) understand what is involved.

Suppose there’s a “transfer-trip” relay associated with the 245kV line; this relay can trip one or more of the 345kV lines in certain circumstances.    In the scenario where both control rooms are indisputably in the same substation, this doesn’t change anything; the BCS in both control rooms will be Medium impact.

But let’s now throw another 245kV line into the first control room, giving that control room by itself 3300 points under criterion 2.5.  And let’s move the second control room – still containing the transfer-trip relay – 500 miles away, with 14 fences and a moat with alligators between it and the first control room; the transfer-trip relay in the second control room still can trip the relay associated with the 345kV line in the first control room (I realize this is an extremely unlikely situation in real life, and probably violates the laws of physics anyway.  But stay with me).  I don’t think anyone will dispute that the second control room is in a separate substation.  Since that substation only has 700 points under criterion 2.5, and since it doesn’t meet any of the other Medium criteria, it will be a Low substation (of course, the substation with the first control room now meets criterion 2.5, so the lines in the first control room are still Medium Facilities).

However, someone who hasn’t read my blog for the past three months might point out that Attachment 1 Section 2 of CIP-002-5.1 says that BES Cyber Systems “associated with” any assets or Facilities that meet one of the Section 2 criteria are Medium impact.  Since the transfer-trip relay in the second control room is definitely associated with one of the Facilities (a 345kV line) in the first control room, it will be Medium impact, right?

Of course, the answer to this question is “no”, given NERC’s recent “ruling” on such situations, described in this recent post; the transfer-trip relay in the second control room will be Low impact.  So now the question of what is a substation again rears its head: as we gradually move the two control centers closer to each other, at what point do the separate substations become one, meaning the transfer-trip relay will go back to being a Medium impact?

Again, I don’t have the answer to this question (that’s why I love being a blogger.  I can point out all sorts of thorny issues and make people at NERC and the regions squirm, without having any responsibility at all to be constructive and actually solve the problem), but I hope there will be one answer given at some point (presumably from NERC) rather than eight or more answers (one for each region, and perhaps more for auditors within the same region who have differing opinions on this).

Now I turn the screw again.[ii]  Let’s suppose the substation in question isn’t a criterion 2.5 one at all, but a criterion 2.4.  It has a 500+kV line (we’ll say 765kV in this case) controlled by one control room, and a 345kV line controlled by the second control room.  Once again, there is a transfer-trip relay in the second control room that can trip the 765kV line.[iii]

In this case, the situation is different from the criterion 2.5 case.  Let’s look again at what happens if both control rooms are at the same substation.  Criterion 2.4 says that “Transmission Facilities operated at 500 kV or higher” are Medium impact.  This means the 765kV line is a Medium, but the 345kV line is Low (since the substation itself doesn’t figure in criterion 2.4, as it does in 2.5).  However, since the transfer-trip relay is associated with the Medium impact (765kV) line, it will also be a Medium BCS.

We once again separate the two control rooms by 500 miles and alligators, so there are now two substations.  The second control room will without doubt be a Low impact, but since the transfer-trip relay remains associated with the 765kV line, it will be Medium impact, right?  Before you bring up NERC’s recent “ruling” referred to above, remember: that ruling only applies to criterion 2.5.  We’re now dealing with 2.4.

But if you follow my blog closely, you’ll know that the August “ruling” – by Steve Noess of NERC - that I referred to wasn’t the first “ruling” on this issue.  The first was in June by Tobias Whitney of NERC, as described in this post.  Tobias reached the same conclusion as Steve, namely that a “far-end” or transfer-trip relay located at a Low substation but associated with a Medium line was Low impact, even though it is clearly “associated with” a Medium Facility.

However, the reason Tobias gave was quite different from Steve’s.  Steve’s was based strictly on the wording of criterion 2.5 (and followed very closely the reasoning an Interested Party had provided me in a post I did the day before Tobias made his pronouncement).  On the other hand, Tobias’ reasoning was much more general: he said[iv] something to the effect that “Physical location IS a determinant factor for impact classification.”  Note there is nothing specific to criterion 2.5 in that dictum.

Let’s apply Tobias’ reasoning to the criterion 2.4 case.  Given that the transfer-trip relay in the second control room is clearly in a separate substation (and it’s 500 miles away from the first substation), I’d say that if physical location is ever going to be a “determinant factor” for classification, it should be here.  I say the transfer-trip relay has to be a Low impact one.  So Steve’s and Tobias’ reasoning lead to different results in this case.[v]  Which person should you believe?  I would say Steve, since his reasoning is written down (in an email), while Tobias' reasoning isn't (his is in a PowerPoint, but that was never officially released after the meeting, which is known as having your cake and eating it, too).   So the transfer-trip relay, which is located in a Low substation, will still be Medium impact - because the criterion we're dealing with now is 2.4, not 2.5.

Once again, let’s start moving the second control room back toward the first.  Whenever it becomes part of the original substation, then there will be no more question that the transfer-trip relay is Medium impact (under either Steve's or Tobias' reasoning), and peace will be restored at NERC.  Of course, the whole question is when the two substations do become one, which is the same as my original question: What is a substation?  That question remains as unanswered as it's always been.

So we’re back where we started, but I hope this exercise has at least cleared up what the issues are.  It certainly has for me, and I hope it will in some way lead to the two issues in this post (the meaning of "substation" and the Steve/Tobias difference) being addressed in some definitive or semi-definitive way by NERC.  This was fun, and I would like to hear of any similar issues you may have discovered in the bright-line criteria.  Contact me at talrich@hotmail.com.

Sept. 13: I've just posted a sequel to this post.

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Honeywell.

---

[i] Remember that criterion 2.5 says that “Transmission Facilities” operating between 200 and 500kV are Mediums, if they are associated with a substation that meets the 3000-point threshold.  Since all three of these lines fall in that range, all the BCS associated with them will be Mediums.  Of course, there’s an exception to this (Alrich’s Law states that no statement can be made about a NERC regulation that doesn’t have at least one exception.  So far, I have found no exceptions to this law).  That exception is the case of the “far-end” relays discussed in this post.

[ii] Of course, when I speak of turning the screw, I’m thinking of Henry James’ wonderful ghost story, “The Turn of the Screw”. 

[iii] This is the scenario – perhaps mangled in my retelling – that the entity I referred to above brought to my attention.

[iv] At a NERC CIPC meeting in Orlando.

[v] My guess is that Tobias’ reasoning, since it could conceivably apply to other criteria as well, will lead to further contradictions with Steve’s reasoning.  That is, unless someone at NERC states whose reasoning is the one to follow in this case.

50,000!

Last week, this blog passed 50,000 page views, since it started in early 2013.  I want to thank all of you for reading me, and especially for going back and reading older posts (some referred to in current posts, others seemingly discovered on your own), since I've covered different topics as I've come upon them.

I was actually thinking CIP Version 5 would be a pretty dull topic by this time; I really thought there would be general agreement on what it meant, how it would be audited, etc.  Silly me.  The way things are going, it will be another 50,000 hits before I can say that.  Meanwhile, there's still lots of very interesting material!