Discussion with an Auditor on Lessons Learned

A CIP auditor emailed me after my recent post on NERC’s Lessons Learned.  His points were all well taken, and we had a good back and forth discussion.  This post attempts to summarize the high points of that discussion.

RFIs

In the first post (bullet point number 2 near the top), I said “the NERC Rules of Procedure don’t permit any modifications to an approved standard, other than through Requests for Interpretation.”  The auditor pointed out that “a Request for Interpretation CANNOT modify the language of a Standard.  It can only clarify what the language means.  Only a modification to the Standard through the standards development process can modify a Standard.”  This is correct; I should have said that the RFI is the only “clarification” of the language that has the same force as the standards themselves; all other clarifications (including the LLs) don’t have that same force.

Disagreeing with a Lesson Learned

In the post, I said “You should definitely consider (Lessons Learned) when you are deciding how you will interpret a particular piece of wording in CIP v5…However, that doesn’t mean you should consider the LLs to be the word of God.  If you have a valid reason for believing that the definition of Programmable should be different from what is stated in the “Programmable” LL, use your version[i]!  But you do need to clearly document, for future audit,  a) that you considered all the available NERC and Regional Entity documents addressing this issue (especially any Lessons Learned), and b) the reasons why you chose the particular interpretation you used.”

The auditor’s initial response to this in his email was “If you disagree with a LL, then seek an RFI or submit a SAR.”  I replied to him “As you know, RFIs and SARs work on geologic time, and don’t do an entity any good for coming into compliance with v5” (by which I meant that submitting an RFI and getting a response three years from now doesn’t help a lot with meeting the 4/1/16 compliance date).  He responded, “That may be true, but it is still the only recourse an entity has to try to change something if they do not like the approved and published guidance.  Otherwise, live with the guidance or risk the consequences.”

I seized on “approved and published”, and pointed out “I’ll agree that if there is a finalized LL dealing with the issue in question, the entity would need to have a pretty good reason for not following it.”  But the bigger issue is that there are only two finalized LLs as of today- one on the far-end relay question, the other on ‘Impact Rating of Generation Resource Shared BES Cyber Systems’; neither of these would be considered fundamental to understanding CIP v5, although they’re both interesting to certain groups of NERC entities[ii].

But what about documents other than finalized Lessons Learned?  Specifically, a) Lessons Learned that have been drafted and submitted for comment, but not finalized (currently numbering around ten); and b) the issues that have been dealt with, or will be, in FAQs[iii]?  What is the status of these?  Does the entity need to consider these as something more than just one man’s or woman’s opinion?  If so, how much more?  Let’s say a finalized LL receives a score of 100 as Something You Need to Pay Attention To – where do these other documents fall?  Does a draft LL get a score of say 50, while a topic addressed in a FAQ is 25?  Maybe those numbers should be 47.2 and 28.9?  And isn’t it ridiculous that these would even be valid questions in the first place?

Most importantly, what about the hundreds of CIP v5 issues for which there have been no Lessons Learned or FAQs even proposed?  I listed 20 in this post last December.  I’m sure I could put together a list of 50-100 now without a lot of work.  And I’m sure there have been 500 to 1,000 total issues already identified in various forums and discussions- that number will keep growing.  The entity is obviously on its own as far as all of these issues are concerned[iv].  You – the compliance person at the entity - need to ask all the questions you can, read all the documents you can, etc.  But in the end, it’s up to you to decide these questions (including some quite fundamental ones like how you identify BES Cyber Systems).

“Programmable Electronic Devices”

This is probably the most important Lesson Learned currently in the process of being drafted / approved.  A NERC entity, before it can definitively identify and classify its BES Cyber Systems, must first identify BES Cyber Assets, and before that Cyber Assets.  Since the latter are defined as “programmable electronic devices”, this means this LL really needs to be in place before entities can finalize their plans for coming into CIP v5 compliance.  So where is the PED Lesson Learned?  Nowhere to be seen, although it is promised Real Soon Now[v].  And when do most NERC entities need to have it?  I’d say at least by January 1, 2015 (not 2016!).  Do you think NERC will meet that date?

This isn’t anything new, of course.  I discussed the need for this LL last September; it was supposedly in development back then.  Yet what does NERC say about the delivery date for the finalized PED LL?  According to the auditor, “NERC’s opinion is that even if it is not finalized for another month, the entities are on notice and they still have plenty of time to confirm (by which I believe he means ‘conform’ to the LL).”  I take this statement as an indication that whoever from NERC made it has a wonderful sense of humor, although perhaps not a well-developed sense of irony.  The auditor finds this as frustrating as I do.

This is an LL that definitely needed to be in place soon after it was proposed last year, yet it’s still not finalized.  But there is a bigger problem than the PED LL.  It’s all the LLs that haven’t even been proposed yet (that I know of), yet are sorely needed for entities to finalize their v5 compliance plans (and I don’t mean finalize their v5 compliance programs; I mean finalize their plans to reach the point where they can finalize their v5 programs).   Two very important examples of this are:

1. Methodology for identifying BES Cyber Systems.  I have written many posts on this topic, including this recent one.  I pointed out in this recent post that a CIPC group that was developing a LL on a particular issue that involved BCS identification recently gave up that task, pending some guidance from NERC on the question of BCS identification in general.  So it’s unlikely there can be any real guidance on any aspect of CIP-002-5.1 R1 until this issue is addressed.
2. How to resolve the inherent contradiction in the definition of BES Cyber System without bringing on the need to develop an RBAM as in CIP v1-3.  This was the subject of my post last week

Of course, there are many other LLs that were needed months ago that haven’t even been proposed.  As I said above, the only choice for entities – rather than wait indefinitely to start their v5 compliance efforts, in the vain hope that all of these issues will be quickly addressed – is to gather the best advice possible from all sources, then use their judgment on each of these issues.  And, of course, to document, document, document their entire reasoning process and conclusions.  If you get issued a PV three years from now for not following a Lesson Learned that came out, say, at the end of 2015 or early 2016 – when you had to finalize your v5 compliance plans in January of 2015 – I suggest you vigorously fight that PV. See the last section of this post for what might be an appropriate remedy to apply should this happen.

Another v5 Compliance Resource

In the course of debating about the Programmable Electronic Device Lesson Learned, the auditor pointed out to me a new compliance resource I’d forgotten about:

“If you want to roll your own, I suggest you dig deeply into the official record.  What is that, you ask?  Why, it is the petition to FERC for approval of the CIP V5 Standards.  The filing was in four parts due to its size.  The record of note is Exhibit F, two of the four parts.  Exhibit F is the complete record of the industry comments submitted during the comment and balloting periods along with the required Standards Drafting Team response.  Surprise, surprise…guess where the definition of PED was debated and guess where the SDT firmly defined what PED meant.  What a shame they did not make a Glossary definition[vi].  And, yes, you guessed it, FERC read that record and therefore took that definition into consideration when they approved the Standards.  When you see the next round of guidance on PED, expect to see a definition that follows the official record.  What a shame no one thought about looking there months ago when the study participants tried to develop a definition independently.  But, you know, hindsight is 20x20.  Want to find the record?  Go to the NERC website, click on the filings tab, click on the 2013 archive, and go to January 31, 2013.  Guaranteed to be a titillating thriller of a read!  Might be something entities should read for the very first time!”

I’m reproducing this paragraph from the auditor’s email as a public service to the NERC community.  The document referred to is certainly something that everyone needs to know about.  Indeed, it’s been on my reading list since it was submitted in 2013, and it will most likely remain there for some time.  While I agree that entities should read this (not every word of the 2,000 or so pages, of course), I have these concerns:

1. As I said in this post last year, any attempt to discern “the intent of the SDT” is suspect – unless it is confined to the two official records the SDT voted on and left us.  These are the CIP v5 standards themselves (and the new definitions related to them), and the Guidance and Technical Basis included with each standard.  Anything outside of that will be interesting reading, and may raise some points you may want to consider, but in the end it is merely someone else’s opinion, even if that “someone else” happens to be an SDT member who was assigned the task of drafting this particular section of a document.
2. If NERC (or one of the regions) wants to use some parts of the v5 petition as a background for audits, they have the right to do that.  But this needs to be stated explicitly to the NERC entities involved, and they need to be told which sections of the 2000-page document are relevant.  You can’t just say, “Read the whole thing.  Quiz tomorrow.”

Defenestration

I concluded the first post with this memorable sentence:

“What should an entity do with an auditor who suggests that the Lessons Learned (or the Guidance and Technical Basis in the standards, or the FAQs, or the Bible) provide some sort of mandatory interpretation of the standards?  I suggest defenestration.”

Since to defenestrate someone means to throw them out of the window, the auditor – rightly – took me to task for suggesting physical violence against auditors.  I agree that I should have made clear that the defenestration I was talking about was a moral one: If an auditor tells you that the Lessons Learned provide mandatory guidance, you should make it quite clear to him/her that you do not believe this is the case (even if the LL in question has been finalized, sealed in blood and deposited in the National Archives next to the Constitution).  And here’s something that may help you: At the NPCC meeting on CIP v5 that I attended last week, it was stated quite clearly that the Lessons Learned are merely guidance, not mandatory.  Of course, other regions don’t necessarily agree.  Your mileage may vary.

However, notwithstanding the above, the auditor did tell me that he intends to do all of his audits on the first floor from now on.

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Honeywell.

---

[i] However, see my footnote vi below regarding “programmable”.

[ii] I also need to say, regarding the far-end relay LL, that either it is badly written or it is being broadly misinterpreted.  I will have a post out on this soon.

[iii] The auditor pointed out that “34 FAQ questions and answers were posted for a 45-day industry comment period this afternoon (meaning April Fool’s Day).  Comments are due May 15, 2015.”  This is good, of course - although I have just noticed a real problem with one of these FAQs that I hope to address in my next post.

[iv] And whatever you do, don’t even suggest to me that the Small Group Advisory Sessions are a way to address all of these remaining issues.  They aren’t public and the results aren’t going to be published, even in a sanitized form.  They are certainly “guidance”, but only for individual entities.  If NERC CIP were really just a set of highly “personalized” requirements, tailored to the unique situation of each entity, the SGAS would be wonderful.  Unfortunately, I can’t find anything in Section 215 of the Electric Power Act of 2005 (which set up the current FERC/NERC regulatory structure for the electric power industry) that indicates Congress intended for FERC and NERC to issue and enforce such personalized requirements.  I think Congress wanted requirements that would be the same for each entity complying with them, just as laws and ordinances are meant to apply to everybody.  But maybe I’m just too caught up in these old-fashioned ideas about laws…you know, John Locke and all that.

[v] Note that these paragraphs were written before the auditor provided me the information discussed in footnote vi below.  I left it in because I’m an ornery SOB.

[vi] The auditor did update these comments two days ago.  Here is what he said:

“Look for a handful of key guidance documents to be posted in the next two weeks.  Expected to be included is a final, definitive definition of PED (programmable electronic devices).  The source of the definition can be found in the SDT response to comments submitted by the industry during the V5 balloting process.  You will find it in Exhibit F, Part 2 of 2, on Page 214 of the PDF file.  The response states:

Some commenters asked about “programmable” with respect to electronic device. The SDT notes that it is an electronic device which can execute a sequence of instructions loaded to it through software or firmware, and configuration of an electronic device is included in “programmable.” Depending on the scheme used, these instructions may include data, or data that must be processed to execute these instructions.

“A SAR will be required to change the definition.  An RFI will simply point to this memorialization of the SDT consideration of comments.

“Unlike the recollections of an exhausted SDT member, this formal response to the comments raised by the industry was memorialized in the official record and provides the most complete and definitive definition of PED.  It did not say the Chair believes, or some named drafter believes; the response says the SDT believes…  I suspect that a judge overseeing a hearing or an appeal would find this official record to be helpful and compelling when compared to an entity’s protestations to the contrary.  And even if entities did not read every word of every comment, they should have at least read the SDT response.  Like a FERC order, where 'the Commission determines…', there are parts of a record that are much more important.  The SDT response, which is required by the Standards Development Process, should be mandatory reading for any entity desiring to vote on the Standards.  That is why the process gives entities weeks to cast their votes.”

My comments on this:

1.       This is the first I've heard that NERC will create a SAR (Standards Authorization Request) to develop a definition of “programmable electronic device”.  However, since it will take a while for the definition to be finalized (including I believe drafting and comment periods, a ballot of the membership, as well as NERC BoT and FERC approval), NERC is giving a pretty broad hint that the definition that results from this process will follow very closely the passage from their filing of CIP v5 in 2013, quoted above.

2.       Unlike the Lessons Learned, etc, this definition will be the law of the land, just like the requirements themselves and the other definitions (BCS, Cyber Asset, etc) that were approved with v5.  Of course, it won’t really be law until FERC approves the final product, which could easily be 1-2 years from now.  If an entity has already “rolled their own” definition and built their whole v5 compliance program on that, I certainly don’t think they have to throw everything away and start over again, using this new definition.  However, if you are just getting serious now about v5 compliance, I would definitely recommend using this new definition for PED.  

3.       I will write a new post on this development – since I’ve just realized how important this is.  It deserves a lot more than a footnote in a post on another topic.