Night of the Living RBAM

Parental Warning: Young children may find this post very disturbing.  For that matter, older children will as well – especially those who have been involved in NERC CIP compliance for a few years.

There was very little lamentation or gnashing of teeth when NERC CIP Version 5 did away with the RBAM - Risk-Based Assessment Methodology – and replaced it with the bright-line criteria.   Of course, FERC hated the RBAM because it allowed the entity to develop their own criteria for identifying assets that were critical to the BES (which is why they ordered NERC to develop bright-line criteria in Order 706).  And while I think some entities liked the – ahem – freedom that the RBAM gave them in identifying their Critical Assets, I think the majority were happy that there were now definite (or at least that was the idea) criteria for identifying “big iron” in scope for CIP.   There can be no more complaints about utilities that shirked their duty to secure their important assets; from now on, the entity just has to say, “We are just following the criteria in Attachment 1”.

However, I have some disturbing news: It seems we should have not only buried the RBAM, but driven a stake through its heart and sealed it in a lead-lined coffin.  For it appears the RBAM may not be really dead after all and may have returned from its grave, ready to wreak havoc on the living as they try to comply with CIP Version 5.  Below is the text of a frantic call I received late one night from a compliance person at a major electric utility.  The call ended very abruptly, and I have not been able to contact that person since then.  I am quite worried for him, and have been double-locking my door in the fear that whatever happened to him might happen to me as well.

“I have been reading your posts religiously, and I have taken to heart your constant repetition of the idea that, while there are two main approaches to BES Cyber System Identification – top-down and bottom-up - the only method that is actually mandated[i] in CIP-002-5.1 R1 is the bottom-up one.  In that method, the entity needs to start by identifying which Cyber Assets meet the definition of BES Cyber Asset; they then need to identify BES Cyber Systems that include one or more BCAs – and every BCA must be included in at least one BCS.  Note there is nothing mentioned about the BES Reliability Operating Services (BROS) in this approach.

“The other approach is the top-down one, which starts with identifying the BROS that are fulfilled by the asset or Facility in question, then identifying the systems that fulfill one or more BROS for the asset or Facility.  The systems on this list that have a 15-minute impact on the BES are BES Cyber Systems. The only problem with this approach is that it is in no way required by CIP-002-5.1 R1, while the bottom-up approach is.

“Unfortunately, it seems that many in the industry – and probably the majority of auditors, at least at the moment – believe that the top-down approach is the only way to identify BCS.  I have tried to challenge these people by asking them where in R1 it says to use that approach; until recently, nobody has been able to do this (although in the first draft of CIP v5 in 2011, the BROS were included in the BCA definition, and thus were the only approach allowed by R1; they were moved to the Guidance section with the second draft and were no longer part of R1).  But at the same time, nobody wants to stop believing that use of the BROS is required by R1.

“Until recently, I thought these people’s refusal to agree with me was caused simply by stubbornness.  However, I recently had a discussion with someone who is quite knowledgeable about R1, yet still holds this view.  He referred me to the definition of BES Cyber System: ‘One or more BES Cyber Assets logically grouped by a responsible entity to perform one or more reliability tasks for a functional entity.’  He pointed out that the second part of this definition seems to indicate that performing a ‘reliability task’ is what defines a BCS; and, of course, the BROS are definitely ‘reliability tasks’.

“I must confess that I have been following your lead on reading this definition, Tom.   You have always focused on the first part: that a BCS consists of one or more BCAs ‘logically grouped’ by the entity.  Neither you nor NERC has considered the second part to be important.  For example, in NERC’s Lesson Learned on ‘Grouping  BES Cyber Assets’, nothing is mentioned about the second part of the BCS definition; that is, nowhere does the LL say that a system needs to perform a reliability task in order to be a BCS.  Yet that is what my friend is saying is the case.

“And I must admit, I can’t counter this argument.  After all, his interpretation is there in the definition of BCS.  You can see that more clearly if you reword the definition: ‘Every BES Cyber Asset must be included in one or more BES Cyber Systems that perform one or more reliability tasks.’  A further implication of this is: ‘A BCS that includes BCAs must perform a reliability task.’  Otherwise, the system isn’t a BCS and the BCA can’t be included in it – rather, the entity needs to find another system that includes the BCA in question, that does perform a reliability task.  This system is a BCS.

“I see no way to refute what my friend says.  The definition of BES Cyber System includes the requirement that the BCS fulfill a ‘reliability task’ (which could be one of the BROS, but could be other tasks as well).  But the first part of the definition includes a different requirement: that all BCAs must be included in one or more BCSs.  This leads to the question: Is there a contradiction between these two ‘requirements’?  In other words, will a Cyber Asset that meets the BCA definition always fulfill a reliability task, so it can be included in a BCS?  Given the wording of the BCS definition, there doesn’t seem to be a way that a BCA that doesn’t fulfill a reliability task could ever be part of a BCS – that is, assuming there are Cyber Assets that meet the BCA definition but don't perform a reliability task.

“At this point, I remembered that you have identified two types of systems that would be identified as BCS if the entity performed the full bottom-up approach, but that don’t themselves fulfill any reliability tasks.  The first is an example used in an SPP workshop last year:  A Stack Emissions Monitoring System (SEMS) in a 1500MW+ coal plant will immediately alert operators when there has been an environmental excursion – that is, when emissions of certain gases have exceeded the levels permitted by the EPA for that plant.  Suppose the plant’s management has laid down a hard-and-fast rule that, if an excursion can’t be addressed within 10 minutes, the plant needs to shut down (this may be dictated by the plant’s EPA permit).

“Were a cyber attacker to penetrate this system and falsely make it appear to the operator that an excursion has occurred and that it has lasted ten minutes, the operator would shut the plant down.  This means that one or more components of this system would meet the definition of BES Cyber Asset – i.e. their misuse would adversely impact one or more BES Facilities (here meaning the units monitored by the SEMS, which will presumably be the ones shut down), which if rendered unavailable would affect the reliable operation of the BES within 15 minutes.  Using the bottom-up approach, the entity would need to create a BCS called SEMS that included the components of this system, in order to fulfill the first part of the BCS definition – that all BCAs be included in a BCS.

“However, would the top-down approach identify the SEMS as a BCS?  In other words, does the SEMS fulfill a BROS?  The purpose of this system is emissions monitoring.  While this is of course an important function, it is not a reliability function (at least as I understand it – there is of course no NERC definition of ‘reliability function’ or even ‘reliability’).  The BES would be just as reliable if there were no emissions controls at all; of course, we might all be dead from breathing the gases emitted, but the lights would stay on nevertheless.  The SEMS doesn’t fulfill a BROS[ii], although its components definitely meet the BCA definition.

“Your other example of a system that would meet the BCA definition but doesn’t perform a reliability function is the fire suppression system in a substation.  Obviously, if that system fails to operate when needed (i.e. in the event of a fire), there will be an immediate impact on the BES.  But fire suppression isn’t a reliability function (again, as I understand the phrase), and it certainly isn’t one of the BROS – no matter how important it may be to suppress a fire when it occurs.

“I’m sorry to be long-winded on this (and I’m taking my cue from you, Tom), but what I’m saying with these two examples is there is a contradiction between the two parts of the BCS definition.  The first part says that every BCA must be in a BCS, but the second part says that a BCS must fulfill a reliability function (which my friend is interpreting to mean ‘BROS’).  Since I’ve just shown that there can be BCAs that don’t fulfill a BROS, this means they wouldn’t be part of a BCS due to the second part of the definition - even though they are required to be included in a BCS by the first part.

“But another friend pointed out that the phrase ‘reliability function’ in the BCS definition doesn’t have to be limited to the BROS; there could well be other reliability functions than just the ones that the SDT identified as BROS in the Guidance to CIP-002-5.1.  In fact, since ‘reliability function’ isn’t a defined term, you could simply say it means anything that can have a BES impact if misused, etc. – in other words, you could say that the definition of BCA defines what it means to be a system that performs a reliability function.  This solves the contradiction in the BCS definition, since there will no longer be any possibility that a Cyber Asset could meet the definition but not also perform a reliability function – they’re the same definition!  So the SEMS and the fire suppression system are now BCS.  I must admit this is a pretty neat trick, and not something that can be refuted purely by logic.

“However, can my friend’s assertion be refuted in the context of CIP v5?  First, one can assume that, if the SDT had really wanted the definition of reliability function to be the same as the BCA definition, they would have said that it was; of course, they didn’t.  But let’s say NERC were to put out a Lesson Learned saying this is how “reliability function” should be defined.  In that case, I would have to agree that entities would be well advised to follow that LL.

“So what does this mean for how a NERC entity should identify their BES Cyber Systems, and how they should be audited on that task?  It’s clear that, if NERC does put out this LL, every system that performs a “reliability function” (i.e. meets the BCA definition) should be identified as a BCS.  And the auditors will determine whether the entity has properly identified its BCS by making sure there are no systems performing reliability functions (in the expanded definition) that haven’t been designated as BCS.

“But the problem with going beyond the BROS is that there are no longer any objective criteria for identifying a reliability function.  So let’s suppose that, if the A/C failed in a generating station on a very hot day, the temperature in the plant would rise to a level where it was dangerous for the workers to continue working; they would therefore have to shut the plant down immediately and leave (this is just one example, of course).  Do you need to declare the HVAC system a BCS?  More importantly, if you decide it’s not a BCS but your auditor thinks differently, who’s to say who is right?  No other body is going to come in and make a final decision on this matter – you basically have to fight this out with your region and NERC. 

“And the even bigger issue is: How is the auditor going to know whether or not you have done a complete assessment of the risks to the BES posed by your facility – so that he/she can determine whether or not you have missed some potential BCS?  I don’t think NERC is going to come up with a comprehensive “expanded BROS” list that includes every possible function that could be performed at an asset/Facility.  The only good way to address this problem is to do what has been done in similar situations with both the previous CIP versions and with CIP v5: The entity needs to develop a methodology for assessing these risks and identifying the systems that are critical for mitigating those risks, then implement that methodology.  The result of this process then constitutes the ‘final’ list of BCS, since once this is done there is no more need for further steps for BCS identification (i.e. this methodology will combine the top-down and bottom-up approaches, and every BCS that would be identified by either approach will be identified in this one).

“So what should we call this methodology for assessing risk to the BES?  Let me see, how about…I know, how about Risk Based Assessment Methodology! 

“To summarize what I’ve just said, there is a fundamental contradiction between the two parts of the BES Cyber System definition.  The best way to fix this problem (and many others with CIP-002-5.1) is to rewrite CIP-002-5.1 R1.  However, even if this is done, it will take two to three years, meaning there needs to be some intermediate option.  The intermediate option would be for NERC to a) issue a Lesson Learned saying that a Cyber Asset that performs a ‘reliability function’ is the same thing as a Cyber Asset that meets the BCA definition; and b) require every entity with Medium or High impact assets or Facilities to develop an RBAM and apply it to each of their assets/Facilities.  

“However, my guess is that nothing official will be done about this problem at all; it will be one of a myriad of problems with CIP-002-5.1 R1 (and Attachment 1) that will be left up to the ‘free market’ to resolve – in other words, it will be up to each region and really each auditor to determine how they will address the fundamental contradiction in the definition of BES Cyber System.  Some auditors will simply ignore the contradiction (and I have never seen it mentioned in any document before – from NERC or a region), of course. 

“But I’m also sure that some auditors will require that the entity demonstrate to them that they have considered the different risks that a facility poses for the BES, and identified all the systems that mitigate those risks as BCS.  And this, folks, is an RBAM.  The RBAM has indeed risen from the dead, and is now stalking the land, waiting to attack unsuspecting NERC CIP compliance professionals.  I urge all who read this to be wary, for at any minute…..NO!  How did you get in here!?  What do you want?...Please don’t hurt me – I’ve always liked RBAMs.  Some of my best friends….”

And here the call broke off.  I immediately tried to call back, but just got voicemail.  In fact, I have gotten nothing but voicemail since then.   I tried to call the police in the city where this person lives, but I just got a message saying that, due to a sudden and severe power outage, there could be no further communication with that city – and this was a week ago.

So let me say what I want, and I’ll be brief (there’s a first for everything, I guess).  Mark my words well: The RBAM has come back and has already produced lots of offspring.  They are lumbering into NERC compliance departments and NERC Regional Entities across North America.  There is no way to stop them, except for NERC to develop a definitive clarification of how the BCS identification and classification process should work in R1 (developing a SAR for a new version is also required longer term, although that won’t stop the immediate RBAM invasion).

And since I see about a zero chance of this happening, the only thing I can say is…They’re coming!  Warn your friends that….OMG, something is breaking in...where’s the Upload button?...They’re coming!

Editor’s Note: It seems Mr. Alrich did find the upload button just in time.  We have repeatedly tried to contact him since this post was uploaded, but have had no success.  Our efforts are hindered by the fact that a complete electrical blackout seems to have coincidentally occurred in his home town of Evanston, Illinois within a minute after he uploaded this post; the blackout persists today, four days after he initially uploaded it.  We can make no contact with anybody in that town.  We are deeply concerned about him, as he was about his friend who dictated this post. Meanwhile, we have heard of several other strange cases that seem to match these two.  We have no idea when or if posts on this blog will be resumed.

We regret any inconvenience this may cause you.  We suggest you peruse this list of other Blogspot blogs that you may find take the place of Mr. Alrich’s interesting, yet decidedly long-winded, posts.  Have a nice day.

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Honeywell.

---

[i] Of course, to say this approach is ‘mandated’ by R1 is something of a stretch.  The fact is that nowhere in CIP-002-5.1 R1 or Attachment 1 is the entity told to identify BES Cyber Systems.  The word ‘identify’ is used in R1.1 and R1.2, but its real meaning there is ‘classify’ – i.e. the entity is told to use Attachment 1 to classify those BCS that are High or Medium impact.  It is simply assumed that the entity will figure out beforehand which systems are in fact BCS.  The entity needs to use the BCS and BCA definitions to do this – which is the bottom-up approach.

Note from Tom: I think it’s quite remarkable that this caller was able to include footnotes in his call.  To be honest, I don’t know how he did it.

[ii] Another party indicated that the SEMS could really be said to be fulfilling the Situational Awareness BROS, since it is monitoring a condition of the plant.  However, providing awareness of the plant’s emissions, which the SEMS is doing, doesn’t have anything to do with reliability, but rather environmental compliance.  It’s true that, given the rule put in place by the plant manager, the plant will be shut down if there is an excursion for ten minutes.  However, if you removed this rule the SEMS would operate the same as it always has.  Its function doesn’t depend at all on whether or not the rule is in place; yet the only way in which the SEMS can impact the BES is through this rule.

We Still Haven’t Learned Our Lessons

I’ve heard a lot about a discussion at last week’s CIPC meeting in which Scott Mix and Tobias Whitney of NERC tried to “clarify” what the Lessons Learned were and ended up causing more confusion, not reducing it. 

This shouldn’t be any surprise, since NERC is basically trying to square the circle with the LL’s.  I do have some sympathy for them.  They are really between a rock and a hard place when it comes to CIP v5, namely:

1. There are a lot of problems with CIP v5: missing definitions, ambiguity, outright contradictions, etc.  These are most apparent with CIP-002-5.1, but they aren’t limited to that standard.  NERC entities, the NERC regions, and opportunistic bloggers are clamoring for more guidance, given the looming 4/1/16 compliance date.
2. At the same time, the NERC Rules of Procedure don’t permit any modifications to an approved standard other than through Requests for Interpretation, which require a process almost as involved as that for drafting and approving the standards in the first place.  First, the Interpretation needs to be drafted (there was even an Interpretations Drafting Team during the reign of CIP v3).  Then it needs to be balloted by the membership (with possible multiple drafts and ballots) and approved by the NERC Board.  Finally, it needs to be approved by FERC (although FERC doesn’t have to approve it.  In fact, they remanded the last two v3 Interpretations, setting the process back to zero for both of them).  This is easily a 2 - 3 year process, even when NERC starts accepting RFIs on v5; and I’m told they haven’t accepted any yet.  Obviously, RFI’s aren’t the tool NERC needs to address the problems in CIP v5 by 4/1/16.
3. Yet a total rewrite of CIP v5 is unthinkable.   V5 took close to five years to develop; absolutely nobody has the stomach to start again from scratch.  But even if we do that (and I am of course pushing to rewrite CIP-002), the question becomes: What do we do in the 3-4 years it will take to develop a new version?  Go back to v3? 

So what’s Plan B?  Clearly, NERC has to figure out a way to provide guidance without actually saying they’re doing Interpretations.  And that’s where the Lessons Learned came in.  Originally, they were pitched as a way to provide guidance to NERC entities based on the experience of the five entities that participated in the v5 Transition Study.  Of course, the LLs have moved way beyond that charter, and are now simply a way for a committee of entities and NERC/Regional staff members to develop quasi-interpretations, which are then vetted through a comment process.

I don’t have a problem with the LL’s in principle; in fact, I welcomed them when I first heard about them last fall.  But I do have a problem when NERC staff members try to say that the LLs have some sort of status other than simply being reasoned opinions that have been put through a public comment period.

Some attendees at the CIPC meeting understood Tobias and Scott to be saying that the regions were somehow going to be “bound” to audit based on the LLs.   There was something said to the effect of “The guidance (i.e. the Guidance and Technical Basis included with each standard, as well as the Lessons Learned and FAQs) is our understanding of the requirement.  If you don’t meet our understanding, you can get a PV.”  I also understand that Tobias and Scott went back and forth on this, so I won’t say this is some sort of final pronouncement.  But it does show why there is so much confusion over the LLs.

Folks, take it from me, Trustworthy Tom: The Lessons Learned are simply documents that express the consensus of a number of NERC community members who have examined the issue in question.  You should definitely consider these when you are deciding how you will interpret a particular piece of wording in CIP v5 (for instance, you should consider the Lesson Learned on the meaning of “Programmable” as you identify Cyber Assets).  

However, that doesn’t mean you should consider the LLs to be the word of God.  If you have a valid reason for believing that the definition of Programmable should be different from what is stated in the LL, use your version!  But you do need to clearly document, for future audit,  a) that you considered all the available NERC and Regional Entity documents addressing this issue (especially any Lessons Learned), and b) the reasons why you chose the particular interpretation you used.  As long as you build a solid case for your interpretation, and as long as the requirement itself hasn’t been revised or a contrary Interpretation hasn’t been approved by both NERC and FERC, I find it highly unlikely that you will be issued a Violation (whether you’re issued a Potential Violation is another story, since I can’t speak for what individual auditors may do; but I see no way that a PV could make it to a full violation, as long as your entity has done its best to comply with the requirement in question.  I have written about two auditors that have endorsed this idea: here and here).

However, all this discussion about how much validity the Lessons Learned have is in some way missing the bigger point: there are only two completely finalized LLs available now, and it is very likely there will be no more than two finalized before April 1 (which, of course, marks exactly a year before full compliance with CIP v5 for High and Medium impact BES Cyber Systems is due).  And what are these two LLs?  Why, two blockbusters: Impact Rating of Generation Resource Shared BES Cyber Systems and Impact Rating of Relays (Far-End Relay).   Both of these are interesting, but they can hardly be considered fundamental to the understanding of CIP v5.

What about some of the LLs that actually address topics on which there is a lot of uncertainty – and for which guidance is needed right now?  I can think of no better example of this than the Lesson Learned on the meaning of “programmable”.   If any LL is needed right now, it’s this one.  Without knowing what “programmable” means, you can’t identify your Cyber Assets.  If you can’t identify those, you can’t identify BES Cyber Assets.  If you can’t identify BCAs, you can’t identify BES Cyber Systems.  And if you can’t identify BCS, you might as well consider quitting your job and exploring careers in the circus.  Especially today, one year and twelve days before the High / Medium compliance date.

So what is the state of the LL on “programmable”?  I had thought it was pretty settled and that NERC would be finalizing it any day now.  But in Kevin Perry’s presentation at the SPP meeting last week, he discussed his preferred definition – which seems to me (illiterate blogger that I am) to be quite at odds with the definition found in the current LL draft.  Now, Kevin is on the committee that is in charge of the LLs, so his opinion will definitely carry weight in whatever ends up being finalized.  I of course don’t know what the outcome of this debate will be, but I do know it will probably be 2-3 months (or even more) before the LL on “programmable” is finalized.  And that’s only 9-10 months before the compliance date.  Folks, if you wait until then to start your CIP v5 asset identification process, you’re SOL (a new NERC Functional Model classification that I’m proposing).  You have no choice but to roll your own definition now and see what comes out when the LL is finalized.   You need to get the job done now.

Unfortunately, this argument can be repeated for other LLs.  Interactive Remote Access sounds like it’s a long way from being finalized.  And virtualization?  Well, the LL on that hasn’t even been started yet.  The only question in my mind is whether it will be out this year or next.  So do I recommend you completely put off IRA or virtualization work until these LLs come out? No, I don’t.  You need to try to the best of your ability to understand these issues and come up with your best judgment on what they mean for your own CIP v5 compliance situation.  And you need to document the approach you ultimately decided on, as well as the different alternative viewpoints you considered.  If a LL comes out two months from now when it’s too late to change your process, then too bad for the LL.  You just need to make sure you document the fact that the LL was developed too late for it to do you any good.

And don’t even bring up the subject of the LL on Functional Obligations and Control Centers when you’re around a compliance person from a municipal or a coop utility; you’ll get an earful for the next eight hours.  This is a huge issue for them, which they're fighting in any forum they can.  We’ll be lucky if we ever have a final LL on this topic, let alone by 4/1/16.

There’s a further question: What about all the Lessons Learned that haven’t even been started yet (as far as I know)?  Of these, my favorite – natch – is a LL on Identification of BCS.  I recounted in a recent post how a CIPC subgroup, charged with developing a particular LL on CIP-002, had decided they couldn’t fulfill their mandate (which had to do with a particular case of BCS identification) until the issues with BCS identification in general are addressed.   And there are a number of other potential LLs that I might suggest right now, were it not for the fact that I would be wasting my time: they all depend on having some document[i] clarifying how the BCS identification process should work – which obviously ain’t coming anytime soon.

Let’s sum up here, and ask some important questions:

1. Should NERC continue writing Lessons Learned?  Absolutely, they are important no matter when they come out.
2. Should NERC entities wait for LLs before they kick off their CIP v5 compliance programs in earnest?  Absolutely not.  They need to roll their own definitions or interpretations, using any available LLs as guidance.  In the end, they’ll get a PV if they haven’t even begun to comply with a particular requirement.  They won’t get a violation (or hopefully even a PV) if they have done their best to understand what a requirement or term means, have documented their reasoning, then have implemented based on the solution they just documented.
3. Should NERC try to tell entities they will be audited in some way based on the LLs?  No, because it’s not true.
4. What should an entity do with an auditor who suggests that the Lessons Learned (or the Guidance and Technical Basis in the standards, or the FAQs, or the Bible) provide some sort of mandatory interpretation of the standards?  I suggest defenestration.

  

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Honeywell.

---

[i] In CIP v1-v3, there were documents prepared by subcommittees of the CIPC that provided guidance on identification of Critical Assets, as well as of Critical Cyber Assets.  They were certainly late in coming, of course.  But the need for them had at least been acknowledged early on.  I haven’t seen NERC even come to that realization yet - that comprehensive guidance is needed on the question of BCS identification and classification.  

EnergySec’s New NERC CIP Newsletter

I’m pleased to report that EnergySec has just introduced an excellent newsletter on NERC CIP.  To be published two to three times a month, it looks like it’s aimed to keep the reader apprised of all the important developments in CIP – not just limited to the CIP standards per se, but including the whole “CIP environment”.  For example, the introductory newsletter includes insightful commentary by Steve Parker (EnergySec’s president), notes and links on FERC developments, links to upcoming meetings and webinars (as well as past webinar recordings), links and discussion of recent CIP Notices of Penalty, and discussion on current Lessons Learned.

You can receive the newsletter for free if you work for an EnergySec member organization (and if you don’t, maybe you should talk to someone about joining).  You can get a free 30-day trial here (the trial is for both of their newsletters: the weekly one on cyber security developments of interest to the industry, as well as the new CIP newsletter).

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Honeywell.

CIP Version 5: The Endgame

Things fall apart; the centre cannot hold;
Mere anarchy is loosed upon the world,
The blood-dimmed tide is loosed, and everywhere
The ceremony of innocence is drowned;
The best lack all conviction, while the worst
Are full of passionate intensity…

…And what rough beast, its hour come round at last,
Slouches towards Bethlehem to be born?

- W.B. Yeats, The Second Coming

It’s been said, “Just because you’re paranoid doesn’t mean everyone isn’t out to get you.”  Since I’m known as somewhat of a Chicken Little, I’ll modify that: “Just because you’re Chicken Little doesn’t mean the sky isn’t falling.”  Folks, the sky isn’t falling quite yet, but I’m beginning to find some suspicious pieces of something in my front yard (now that the snow is finally melting here in Chicago).  I think CIP v5 is now in the endgame, by which I mean there will soon be no more possibility that, starting 4/1/16, the v5.5 standards will be enforceable standards.   Either the compliance date will be pushed back, or the Registered Entities will be given a “holiday” – either intentionally or unintentionally – from being assessed Potential Violations, as long as they’ve made a good faith effort to comply.

Here are some of those pieces, in no particular order[i]:

Item:  I heard last week that some of the NERC regions are now encouraging entities not to choose to be audited on CIP v5 if they have an audit scheduled this year; instead, they are telling them to stick with v3.  This is not a good sign; the auditors in these regions clearly don’t think they understand v5 well enough to audit it yet.  But this means these regions won’t get v5 audit experience, and their entities won’t get experience trying to comply with v5, until the date when they have to be fully compliant, April 1, 2016.  The whole idea of the transition program was that entities and auditors were going to get valuable experience with v5 before the compliance date.

Item:  I heard about another region that was doing just the opposite: strongly encouraging some entities to be audited on v5 this year, even if they can’t also maintain their v3 compliance program (which they are supposed to do, according to NERC’s v3-v5 Transition Plan).  Since v5 is now only being audited on a “just for fun” basis (and since v3 will effectively be as well, given what the region is telling the entity), this may indicate that this region just wants to focus on having their auditors and entities learn about v5 compliance together, without gumming that up with an adversary auditing experience.  To be honest, I like that approach a lot; however, it doesn’t give much encouragement to the idea that CIPs v5 will be an enforceable set of standards next year.

Item:  I participate in one of the CIPC working groups that is working on compliance issues.  They have been working on a Lesson Learned having to do with whether communications equipment should be considered as BES Cyber Systems, where there is no ESP in a substation.  However, as we pursued the discussion, it became clear there can be no resolution to this issue until larger issues with CIP-002-5.1 R1 are addressed – specifically , the question of how you are supposed to identify BES Cyber Systems in general (which I just addressed in a post this past weekend).

Everyone pretty quickly agreed with this conclusion, but the question is what to do next.  As you may know, I think the only way to fix CIP-002-5.1 R1 is to send it to meet its Maker and start over again with a SAR for a new standard (which of course will probably take three years to come to be developed).  The group clearly wasn’t quite ready to agree with that, although they also couldn’t come up with any other way to actually fix the requirement (hint: There is none.  The only other binding option is to do a Request for Interpretation.  But those take almost as long to bear fruit as rewriting CIP-002 will.  The same goes for all the other issues that are tied to the meaning of CIP-002-5.1 R1 – including more fundamental ones like how one identifies a BES Cyber System; none of these will be properly addressed for years, until the requirement is rewritten from scratch.  Therefore, entities have to identify their cyber assets in scope for v5 pretty much using their own interpretations of R1, which is probably the most confusing and contradictory requirement ever written by NERC.

Item:  I just said above that NERC hasn’t come out with a Lesson Learned on BCS identification.  That isn’t quite true, because they did include a lot of discussion of BCS identification in their recent filing with FERC on the results of the BES Cyber Asset survey.  While I think that overall this is a good document, it is quite clear to me that the people that wrote it didn’t understand what CIP-002-5.1 R1 and Attachment 1 say about BCS identification; this isn’t good, considering that NERC is supposed to be teaching everyone else about this fundamental process - not wandering around in the same wilderness that everyone else is in.  I will put out a post on this in the near future, God willing and the creek don’t rise.

Item:  At the WECC CIP User Group meeting in Anaheim that I attended in January, Tobias Whitney of NERC announced there would be 15 Lessons Learned developed by April 1.  However, I listened (remotely) to a good[ii] presentation by Kevin Perry of SPP this week, delivered at SPP’s Spring Compliance Workshop in Little Rock, AK.  In it, he pointed out that, while there would be about 15 LL’s posted by 4/1, there will be only two that are final: Generation Segmentation and Far-End Relay.  All of the others are in various stages of initial post for comment, etc.

So think about it.  There are literally hundreds of issues in CIP v5 (especially in the bright-line criteria) that need to be addressed by NERC.  And by April 1, exactly a year before the compliance date (and about 6 or 7 months after the Lessons Learned effort was initiated), NERC will have “definitively” addressed only two.[iii]  I’m sure they’ll step up that pace soon, but there are two big problems:

• Many if not most of the big issues with CIP v5 have to do with determining what’s in scope (i.e. CIP-002-5.1 R1 and Attachment 1, as well as CIP-005-5 R1).  But these are precisely the issues that entities need addressed before they can fully move ahead with their CIP v5 compliance programs.  They really needed these issues to be settled last year, so for NERC to say now that they won’t even be addressed by April 1 is simply a cruel joke.[iv] 
• As NERC entities finally start to seriously move forward on v5 (and some of them are, of course, having no choice in the matter), more and more interpretation issues will come up.  So if there are say 300 issues now, there will easily be 600 in six months (I publicly suggested to Tobias Whitney at the WECC CIPUG in January that NERC at least needs to collect a database of all known issues.  Of course, he rejected that idea, and also said that NERC entities should put aside their confusion about what the requirements mean and just forge ahead on compliance.  I presume he said that to show he has a great sense of humor).  No matter how many Lessons Learned and FAQs NERC puts out between now and 4/1/16, it is inevitable that there will be more unaddressed issues on that day than there are today. 

Item:  At the WECC meeting, Tobias also proudly announced “CIP University” – kind of an online course catalog showing you different teaching events provided by the regions.  I was skeptical of that because a) there really hasn’t been a huge amount of CIP v5 outreach by the regions (at least not compared to the need for it); b) most entity compliance staff members don’t have the travel budgets to go flying around to every regional meeting; and c) most people are going to be fairly skeptical about the relevance of compliance advice given by a different region than theirs (perhaps unfairly.  I have attended a lot of regional meetings and have seen very little in the way of local “parochial” content – at least when they devote a day or two just to CIP).

Well, at the SPP meeting yesterday, Kevin Perry announced that CIP U has been replaced by the “CIP Workshops and Curriculum Calendar”.  To find that, you go to the NERC v5 Transition Program site and look for these words.  The page linked does give you a calendar of regional workshops.  But to be honest, I don’t think it's likely that many NERC entities will rack up frequent flyer miles as they try to slake their thirst for CIP v5 understanding by drinking at all these regional founts of knowledge.  There is also a Resources tab that lists a grand total of two webinar recordings[v] given in early 2014; excuse me for not cheering about that.

Most importantly, I can assure you there is a lot of inconsistency among regional discussions of CIP v5 (which is probably why there have been so few such discussions – the regions decided to lay low until they fully understood v5.  Unfortunately, most Registered Entities can’t wait ‘til 2030 for their region to finally understand v5; they have to comply next year).   Even if a compliance person had a lot of time and money to spend, what good would it do him or her to keep flitting from meeting to meeting, hearing different things in each one?

My last item is one I discussed in a post last week: the Small Group Advisory Sessions (SGAS) that NERC is now holding in Atlanta.  In Kevin Perry’s presentation this week, he touted these as something that SPP entities should take advantage of ; he went on to specifically point out that I had waxed “apoplectic” about these in my post, saying they were illegal and immoral.

Now, I know Kevin quite well and have lots of respect for him; he has taught me a lot of what I know about CIP v5.  But I have to give him a PV for his statement, since he just didn’t read my post carefully enough.  In the post, I pointed out that the SGAS were definitely illegal (according to the NERC Rules of Procedure, which say nothing about NERC having private meetings with entities to answer compliance questions) and probably immoral (since the information provided to one entity wouldn’t be provided to others); but I said I could get over both of those hurdles.  In fact, there are no longer any completely “legal” ways that NERC can provide guidance on CIP v5 before the 2016 compliance date[vi].  I’ve been advocating since last year that NERC needs to forget about legality and provide guidance in any way they can.

 

No, my objection to the SGAS is that in these sessions NERC staff members are almost certainly going to provide interpretations of the standards, of some sort, for particular entities – and more importantly, that these aren’t going to be made public to the NERC community.  IMHO, this undermines the whole foundation for CIP v5 as an enforceable set of standards. 

In his presentation, Kevin said NERC would provide “advice” to entities on compliance – he was of course very careful not to say “interpretations”.  Is there a real difference between these terms?

If you mean by “interpretation” a long, philosophical discussion of the meaning of a particular requirement (such as you get in this blog, or in the Lessons Learned –although I’ve easily got the LL’s beat for length), I’m sure the SGAS won’t provide that; but I don’t mean “interpretations” in that sense.  What you will see in the SGAS is “advice” on complying with particular requirements, as Kevin said.  He actually used the example (in a later email to me) of an entity telling NERC how they intend to comply with a particular requirement, and NERC telling them whether that was a good idea or not.  How is that different from NERC’s “interpreting” the requirement for the entity?

For example, suppose an entity mentions that they considered the fire suppression system in a Medium substation not to be a BES Cyber System, since it doesn’t fulfill a BES Reliability Operating Service – and it doesn’t, since fire suppression isn’t listed as a BROS.  The NERC person will helpfully point out that, even though it doesn’t fulfill a BROS, the system (or the cyber assets within it) does meet the definition of a BES Cyber Asset. This is because, if the system isn’t available when needed to fight a fire, the substation will burn down, probably in well under 15 minutes; that will have a big impact on the BES.  Therefore, the system needs to be a BCS, and a Medium one at that.

March 17:  Kevin emailed today to point out that people might think that the above example was the one he'd mentioned in his email.  It isn't - I chose this example because I thought it was a good illustration of what I'm trying to say here.  Kevin's example would have probably worked just as well.

So what about an entity that didn’t have an SGAS – or maybe they had one, but the fire suppression system was never brought up?  The NERC people therefore never were able to give them the same advice they gave the first entity (remember, the SGAS are only 60-90 minutes long.  An entity of any size will never be able to get every single question answered in that time).  The second entity may pay a big price come audit time, due entirely to the fact that they weren’t lucky enough to bring up the fire suppression system in their SGAS, while the first entity was.  

Kevin calls this "advice", but “advice” vs. “interpretation” is a distinction without a difference, in my book.  Any time NERC talks directly with an entity about a compliance question, what they say is in some way an interpretation (just as the Lessons Learned are interpretations); NERC should share these interpretations with all other entities (of course, the document would have to be properly “sanitized” so that the entity that brought up the question initially can’t be identified.  This isn’t hard to do).   But NERC hasn’t stated any intention to publicly release the advice they provide in the SGAS.[vii] 

Of course, there’s a very good explanation for why NERC has come up with the SGAS – and you can find it in what I’ve already written in this post: There are far too many interpretation issues with CIP v5 for NERC to be able to address them through Lessons Learned or other semi-official means.  If CIP v5 is going to be enforceable on 4/1/16, NERC has to take extraordinary measures.  They’re certainly doing that, but the measures they’re taking – especially the SGAS – are rapidly making the standard unenforceable [viii], on 4/1/16 or any subsequent date.  Kind of ironic, don't you think?

I plan to elaborate on this – and discuss what I think will ultimately happen to CIP v5 – in my next two or three posts (or maybe 10-20 posts.  This is probably one of those tip-of-the-iceberg issues).

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Honeywell.

---

[i] Since I’m getting tired of referencing the same posts over and over, I won’t be including as many links to previous posts as I normally do.  If I mention that I’ve said something before and you don’t know which post to look for, email me at talrich@hotmail.com.

[ii] With the exception of one thing he said, which I’ll discuss below.

[iii] And to call the Lessons Learned “definitive” is a real stretch.  I will have a new post on this soon.

[iv] I was interested to note in Kevin Perry’s presentation today that the Lesson Learned on the meaning of “programmable” isn’t one of the two LL’s that will be finalized as of 4/1.  So a year before the compliance date, entities still don’t have a firm guidance on how they identify Cyber Assets (whose definition depends almost entirely on the meaning of “programmable”).  If they can’t properly identify Cyber Assets, they can’t properly identify BES Cyber Assets.  If they can’t properly identify BCAs, they can’t properly identify BES Cyber Systems.  And if they can’t properly identify BCS, they can never be sure they’re properly complying with CIP v5 at all; nor can their auditors.  There’s an awful lot riding on the definition of one word.  Entities have no choice but to “roll your own” definition.  However, this just goes to show the folly of pushing ahead with the 4/1/16 compliance date.

[v] Kevin said yesterday that the webinar he’d done in February 2014 (which was good) would be available, but it’s not – at least not yet.

[vi] Even the Lessons Learned live in a kind of shadowland of legality.  They’re based on an obscure section of an obscure appendix to the NERC Rules of Procedure, which simply says that any “entity” can prepare “documents that may be developed to enhance stakeholder understanding and implementation of a Reliability Standard.”  If the NERC Standards Committee approves, these can be posted.  Does this sound like a statement that these “documents” will be almost as valid as actual Interpretations?  Yet some at NERC are clearly trying to have people believe that.  I’ll have a post out on this issue soon.

[vii] Kevin also defended the SGAS by pointing out that compliance staff from the entity’s region will usually be present, and they give advice all the time to their entities.  This is certainly true, but if the purpose of the meetings is just to give the regions a place to meet with their entities (and the announcement says nothing about the regions being present), why hold them at NERC’s headquarters in the first place?  NERC develops the standards.  It isn’t supposed to be giving compliance advice to particular entities that it doesn’t share with all others, period.

[viii] Certainly, SGAS isn’t the only “extraordinary” means that NERC is employing nowadays.  Staff members have given opinions on standards, both in public presentations and informal conversations (such as the one I recount in this post); these opinions have sometimes had to be taken back as further thought was given to the matter.  But I’d rather have NERC staff members spouting off their opinions in open forums than in closed meetings with one entity.

BES Cyber System Identification Methodology – Now, Less Filling!

Sept. 1, 2015: I made some important changes to this post just now, since I want to reference it in a new post I'm writing. So don't be surprised that this post references posts that weren't written until afterwards. The only ESP I possess is the kind that surrounds BES Cyber Systems.

I am a great believer in recycling.  So today, when I ended up writing – for a different purpose – a kind of “Lite” version of my previous post on CIP-002-5.1 R1 compliance methodology (“R1”), I decided – gee, wouldn’t the blog readers just love to see this?  Plus, since I receive about $5 a word for what I put up, I’m always interested in having more words to post (you always suspected I was paid by the word, didn’t you?).

Note: The previous post just referenced lays out a "methodology" for the entire process of complying with R1.  In this post, I am just addressing the BES Cyber System Identification part - which you will conduct for assets and Facilities that meet High or Medium impact criteria.

Seriously, I do like this post, since it’s really the first time I’ve written down a high-level R1 methodology that is actually readable.  However, I need to make the same qualifications I did in the previous post:

1. This is a high-level methodology.  If you want a little lower-level view, go to the previous post.
2. But even that post makes clear there is no such thing as a complete methodology for R1 compliance.  To write out a methodology that covered every possible option, you would produce something longer than the Bible, and the effort would have to be continued by your children and grandchildren.
3. As you go through what I write below, keep in mind that at some point I’ve probably done a whole post on each sentence you read.  I don’t feel like putting all those links in (a lot of them will be found in the post linked above), but if you want clarification on something, email me at talrich@deloitte.com.

Just to make you feel bad, compare what I write below (or even better, in the post linked above) to the methodology for Critical Cyber Asset identification in CIP v1-3 (and this is the complete methodology, believe it or not):

1. Identify your Critical Assets using your RBAM;
2. Identify the Critical Cyber Assets that are essential[i] to the operation of those Critical Assets;
3. Done!

Makes you want to cry, doesn’t it?  Well, let’s get on with this dirty business. 

There are two main approaches to Cyber Asset identification.  I call these “top-down” and “bottom-up”.  In practice, I (and a few others I’ve worked this out with) think substations and generating stations that meet Criteria 2.3 and 2.6 should just do bottom-up.  Control centers and Criterion 2.1 generating stations should do top-down, which includes bottom-up as a reality check (as you’ll see below).  Bottom-up is what is “required” by CIP-002-5.1 R1 (although R1 never explicitly states the entity has to do this), but the top-down approach (which includes some of the bottom-up approach) will save a lot of time when it’s applicable (it is likely to involve unnecessary work when it's applied to substations, though, which is why I recommend the bottom-up for those).

Bottom-up

Once you have decided what are likely to be your High and Medium impact assets/Facilities (and I’m skipping a huge discussion on what “Facilities” are), you need to identify all Cyber Assets at the asset (for High control centers), or associated with the asset or Facility (for Medium assets/Facilities).  You next need to compare these to the definition of BES Cyber Asset to identify your BCAs (This requires your having some interpretation of "adverse impact" in the BCA definition. See this post for more on that).

Finally, you group the BCAs into BCS.  A BCS can consist of one or more BCAs, but you can also include Cyber Assets that aren't BCAs as well; however, you should make sure that every Cyber Asset you include in a BCS needs to be included, since all components of the BCS will be subject to the CIP v5 requirements that apply to BCS.  There is no obligation to group BCAs in a particular way, and you shouldn’t get distracted by the use of the words “reliability tasks” in the BCS definition.  Since every BCA you’ve identified performs a reliability task per the BCA definition, you don’t need to repeat that exercise when you group them into BCS. 

Top-down

In the top-down approach, you basically follow the discussion of the BROS in the CIP-002-5.1 Guidance and Technical Basis.  I won’t go through all the steps outlined in the Guidance, but in general you need to identify the BROS that are fulfilled by the asset / Facility in question and then identify the systems that fulfill one or more BROS.  This is your preliminary list of BES Cyber Systems. You don’t now need to go on and consider whether each Cyber Asset that is a component of a preliminary BCS meets the BCA definition or not; all components of the BCS will be subject to the v5 requirements, whether or not they are BCAs.  

The next step is to perform a "reality check" on the preliminary BCS list, to remove those systems that don't have a 15-minute impact on the BES - since the components of these systems wouldn't meet the definitions of BES Cyber Asset.

However, there could be other BCS that haven't been identified so far. These are composed of Cyber Assets that meet the BCA definition but don’t fulfill a BROS.  For example, the components of a CEMS system in a generating plant (which monitors for various gases in the emissions, and issues a warning or shuts the unit down in the event of an excursion) don't fulfill any BROS (environmental monitoring isn't a reliability service. The plant is just as reliable even if it's spewing dirty gases into the air). But if the CEMS system is programmed to shut the plant down within 15 minutes of an excursion, then its components probably do meet the BCA definition. So CEMS is a BCS in this case, even though it wouldn't be identified by the top-down procedures described so far.

This means that, once you have your preliminary BCS list, you need to run a modified bottom-up approach. However, you get a pretty big break when you do this: For each Cyber Asset that is a component of one of the BCS on your preliminary list, you don’t need to consider whether it is a BCA or not.  As I just said, as a BCS component, it’s already subject to all of the BCS requirements and there’s no reason to determine whether or not it as a BCA.  So, from the original list of all Cyber Assets, you subtract all those that are components of systems on the preliminary BCS list.  You then apply the BCA definition to the remaining Cyber Assets to identify BCAs (for example, the components of the CEMS discussed above should come up as BCAs at this point).  Finally, you make sure each BCA is included in at least one BCS (you might include them in BCS that are already on the preliminary list, or you can group them into new BCS).

You now add the BCS identified through this (limited) bottom-up approach to those identified through top-down.  This is your final BCS list.[ii]  Congratulations, you’re done!

Simple, isn’t it?

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte Advisory.

---

[i] I realize “essential to the operation of…” was never defined in the previous CIP versions.  However, people seem to have gotten over that, whereas gaps like the definition of “programmable” are causing major fits in v5.

[ii] There is an exception to this process: Criterion 2.1 generating stations may have to be creative about how they do the bottom-up check, since they sometimes have tens of thousands of potential BES Cyber Assets.  Entities owning such a plant need to figure out a way to identify Cyber Assets and then BES Cyber Assets en masse rather than one-by-one. I'm told this can definitely be done.

More on Windows XP and CIP v5

Since my post yesterday about Windows XP and CIP Version 5, I’ve corresponded with two auditors (different regions) and one NERC entity about two interesting topics.  I’ll address the shorter one first.

Topic 1

An auditor (not the same one I quoted in the original post) made this very good point:

“One comment about your recent blog post – the topic of Windows XP on BESCAs/PACS/EACMS/PCAs came up in a seminar recently, and the response given there was similar to the one you had, except the presenter (CIP auditor) stated that when that information was pulled in as part of the pre-audit scoping processes related to the shiny new NERC risk-based auditing framework, it would heighten the entity’s risk profile and thus expose them to an audit of additional CIP standards/requirements, more samples of performance evidence requested, etc.

 

“IMHO - I’d expect similar treatment for folks with ‘Applicable Systems’ running Open VMS, Solaris, OS/2, OS/400, backlevel Cisco IOS/CatOS…basically any deprecated or EOL version of an OS.”

Of course, when the auditor refers to the new “risk-based auditing framework”, he means what most of the rest of us call RAI – the Reliability Assurance Initiative (that name has officially been retired, since it’s no longer just an initiative).  Essentially, this means that an entity with a lot of XP might be considered more risky.  Therefore, that entity might not be moved as quickly to the new risk-based auditing framework as one that had very little or no XP (of course, this doesn’t apply if the reason they don’t have XP is that they haven’t yet been able to upgrade from Windows NT!  I hope there aren’t any entities in that situation).

The auditor who commented in the original post had this to say, when I asked his opinion: “Yes, running on obsolete, unsupported systems would be a risk factor that could, not necessarily would, increase the level of scrutiny directed to the entity’s compliance program.  That is an auditor discretion issue.”

Topic 2

The second topic arose when a compliance person at a large IOU wrote in with this question:

“Tom, a couple of questions were raised when I read your last post regarding patching of XP. The first is around Entities that have bought the extended service from Microsoft and thus still receive XP patches; are you required to patch? I take at this point you must evaluate those patches since they are available - but what about Entities that don't subscribe to that service; patches are available sort of, it's just they don't want to pay for them?

This gets a bit more complicated with this question. So you have certain systems from a vendor like GE, Honeywell, or Emerson that are running XP but those vendors are not releasing patches for their XP systems (not saying that all these vendors are not releasing XP patches). Due to some contract agreements, the Entity is not allowed to patch those systems independently - rather the vendor must provide the patch for those systems...and if the vendor isn't (providing patches) because they don't subscribe to the service - it puts the Entity in a precarious position. From one standpoint they evaluate all patches from the vendor that are available and install them (it's just there are none); yet then there are those patches they are receiving from Microsoft...but the Entity really can't install those either.”

The First Auditor’s Response:

I first ran this by the auditor from the original post, who said:

“In my view (and I cannot guarantee all auditors will take the same stance), there is nothing in the CIP standards requiring an entity to subscribe to post-end-of-life for-fee support.  There are no more XP patches publicly available from a vendor that routinely makes its patches publicly available for supported systems.  I would not demand that an entity pay huge dollars for continued, limited security support.  But if they did, then they are expected to monitor for and install any applicable patches that might be released under the support agreement.

“As far as the GE, Emerson, etc., these are typically turnkey system deliveries.  As such, especially with a contractual requirement that the customer not try to implement patches out of band, not provided by the vendor, the entity appropriately relies upon the third-party turnkey system provider for all patches and updates.

“Finally, as we are in the CIP V5 Transition period, the entity can assert compliance with the CIP V5 requirement, designate the patch source, and proceed accordingly.”

This last is a good point.  Under CIP v5, this issue wouldn’t even come up, since the entity chooses, for each system and software package, which vendor it will follow for patches.   So if your vendor is XYZ and they’re no longer releasing XP patches because Microsoft isn’t providing them for free anymore, there’s no obligation to go beyond what XYZ offers.

The Second Auditor’s Response:

Since I had the second auditor’s attention (with the first issue), I decided to press my luck and ask him to respond to the NERC entity’s questions.  I’m very glad I did, since he provided some very good guidance not only on this specific issue but on patch management in CIP v5 in general.  Here are his comments; note that he inserted them into the text of the entity’s original email, which he italicized (of course, I didn’t send either auditor the actual email, so they don’t know who sent it to me.  And I won’t reveal that information, even if Dick Cheney himself waterboards me).

“Tom, a couple of questions were raised when I read your last post regarding patching of XP. The first is around Entities that have bought the extended service from Microsoft and thus still receive XP patches; are you required to patch?

• You’re required to execute your CIP-007-5 R2.2-2.4 processes to perform the following:
•  R2.2 – Is this patch:
•  Applicable to one of the “Applicable Systems” for your entity; if it is an application specific patch (say for Microsoft Windows Media Player for XP), is that application installed on the “Applicable System”?
• Is it a security patch?  Smart folks will treat every patch as a security patch until the patch source (vendor) explicitly tells them otherwise or it is explicit in the README or release notes with the patch (especially anything that is a “Service Pack” for multiple issues).  If it’s ambiguous, treat it as a security patch and start processing it – you can always inquire in parallel to the vendor to get clarification, but you can’t turn the 35 day hourglass back over if you find out on day 30 it was a security patch all along. 
• R2.3/R2.4
• Apply the patch to all “Applicable Systems” within 35 calendar days, or
• Create a mitigation plan to mitigate the vulnerabilities addressed by each security patch and follow it (R2.4).  
• The most common delay in patching is due to operational concerns or constraints (have to take an outage on all units controlled by the DCS, need to test it on one smaller plant before rolling it out to all plants in the fleet) then the expectation is probably that there would be something put in place in the meantime to “mitigate” (maybe an additional IDS/IPS signature, more frequent review of logs, etc) but that the patch would ultimately be applied. 
• If the patch is deferred “forever”, there would be some evidence expected as to why the patch could not be applied (testing showing failure is the easiest and most common piece).  Simply expressing a fear of patching due to an ambiguous danger to reliability without evidence to back that up would be a dicey play.  Even if your RE didn’t write you up, (you might end up by) having a long mitigation plan with multiple milestones to trip you up if you don’t “implement” it on time (R2.4).  Did I mention having to have TFEs for deferred patches?  In the V3-V5 Transition Guidance NERC released, they identified R2.4 as a TFEable requirement.  All in all, this is a short term gain, long term loss.

This gets a bit more complicated with this question. So you have certain systems from a vendor like GE, Honeywell, or Emerson that are running XP but those vendors are not releasing patches for their XP systems (not saying that all these vendors are not releasing XP patches). Due to some contract agreements, the Entity is not allowed to patch those systems independently - rather the vendor must provide the patch for those systems...and if the vendor isn't because they don't subscribe to the service - it puts the Entity in a precarious position.

• Some of this was why in V5 the Registered Entity is allowed to define (and they should date, define, and document this in writing so that it is above board) their patch source(s) in R2.1.  So if they identify that their DCS vendor is their patch source instead of Microsoft because of their contractual obligation, then they would be able to rely on the vendor for patches. 
• However if they then did not pay for the support agreement/subscription to receive those patches from the same source they had identified in R2.1(whether it was Microsoft or a DCS/EMS/PLC vendor), I don’t think they’d have a leg to stand on – I’d see a PV on CIP-007-5 R2, Part 2.2.  To me, it’s not a far cry from saying that they didn’t receive any patch notifications from their vendors because to save money they didn’t pay the bill for their corporate internet connection for email.  I understand that those agreements aren’t cheap, but this isn’t a cost/benefit model at this stage – that was in the stage where FERC issued the NOPR. Positive note:  It’s something to consider in the procurement cycle for a lot of these systems – if they are going to have a long upgrade cycle and the ongoing cost is considerable or the support agreement is thorny, factor it in to your selection.”

  

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Honeywell.