Two Holiday Presents for you – Present Number Two

This is the second of two posts that contain unadulterated good news for entities subject to the NERC CIP requirements. You make think my motivation for doing these posts is that I was visited by three ghosts on Christmas Eve who told me to mend my ways. It isn’t that, but rather the realization that I always have a big queue of things I want to write about, and I should try to prioritize the good things to post around holiday season.

In November, I wrote this post about the NOPR that FERC issued in October, stating their intention to approve CIP-003 version 7. This is the version of CIP-003 that was much debated and approved last year (i.e. in 2016), which eliminated the definitions of LERC and LEAP and incorporated those concepts into the requirement itself (and if you’re hazy about what this debate was all about, this post and this one may refresh your memory. My memory certainly needed refreshment!). Of course, this whole discussion has to do with Low impact assets (mostly substations) that have some sort of routable connection to the outside world.

As discussed in my November post, what I found most surprising in the NOPR was that FERC clearly stated their intention to push NERC to go well beyond what is in CIP-003 v7. That standard, which will come into effect 18 months after FERC issues the Order they committed to issuing in their NOPR, requires that an entity owning a Low impact asset, for which there is at least one routable communications stream with a cyber asset outside of the boundary of the asset itself, have in place some means of mitigating the risk posed by that communications. It could be a device like a firewall (formerly known as a LEAP), but it could be something else like network separation, a unidirectional gateway, etc.

In the NOPR, FERC said they will approve v7 as written. However, they also asked for comments on going beyond what is in v7 to require further steps for Low assets. Specifically, they pointed to authentication and password complexity as two of the four items[i] they would now like included in the requirement for securing BES Cyber Systems located at Low impact assets that contain some form of external routable connectivity. They might also ask for more than those four items. Of course, since NERC’s Rules of Procedure don’t allow changes to be made to a standard once it has been approved by the NERC Board of Trustees (and all new standards are approved by the BoT before they’re even submitted to FERC for their approval), these changes will be in a new version of CIP-003, which will be version 8 (and that version won’t be effective for 3-4 years from when FERC orders it).

In my November post, I pointed out at the end that I thought it was clear that the changes FERC is proposing will doom what has been a bedrock principle of NERC’s Low impact compliance program since CIP v5: that no inventory of Low impact BES Cyber Systems will be required. My reason for saying this (not stated in the post) was that I simply didn’t see any way that this principle could be preserved if CIP-003 is going to require authentication and password complexity for Low impact BCS. It seemed to me that there would be no way these requirements could be audited, absent an inventory of Low impact BCS.

However, a couple weeks after that post, an auditor wrote in to me with critiques of several of the points I had made in the post, including the one I just cited. I will quote in full what he said on this topic:

“(Auditing the new requirements that FERC is considering can be done without requiring the entity have) an enumerated list of Low Impact BCS or their component Cyber Assets.  Now, as an auditor, I may ask the entity to demonstrate that the controls have been implemented, so at that time I may ask for a list of the relays in a sampled substation.  Or, knowing that there will be a breaker relay in the substation for a circuit breaker on a Transmission Line or bus, I might ask for the Station 1-Line diagram or SCADA substation display, point to a breaker drawn on that diagram, and ask for evidence associated with that breaker’s relay.  I might even visit a randomly selected substation, point to a SEL-421 distance protection relay and inquire how it is managed.  Remember that relaying engineers typically do not do anything without a work order and all sorts of authorizations, so being able to come up with the work order to change a password on that very device might not be an insurmountable challenge requiring additional records keeping not already being done.  And if they are not managing the passwords on the relays, then shame on them.  Then again, if they implemented the SEL-3620 or equivalent, I don’t need to look at any controls on the relays because access is well managed at the gateway.  The point is that there is still no mandate that the entity identify every Cyber Asset in a Low Impact environment, identify the subset of those Cyber Assets that are Low Impact BCAs, and group them into a documented list of Low Impact BCS, and I can envision how the new requirements can be implemented and audited without requiring a list of Low Impact BCS.”

To paraphrase this quotation, the auditor points out at the end why many NERC entities are so worried about possibly needing to maintain an inventory of Low BCS. To do this will require doing pretty much everything that needs to be done to identify BCS at Medium and High impact assets now: identify every Cyber Asset at the Low impact asset, use the BES Cyber Asset definition to determine which of these are BCAs, and finally group BCAs (and possibly other Cyber Assets) into BES Cyber Systems. Then repeat this on a regular basis, so as to include new Cyber Assets that may have been added.

But he makes it clear that he thinks I’m wrong in (implicitly) stating that requiring authentication and password complexity on Low BCS will inevitably require an inventory. I assumed this was inevitable because there would be no way to audit these requirements without an inventory, but he thinks I’m simply wrong in this assumption. He points out several ways he could audit these requirements without demanding that the entity produce an inventory of all of their Low impact BCS.

And now that I read his words again, I realize I was making a false analogy from Medium and High impact BCS. While these BCS are also required to have authentication and password complexity, these and other such requirements aren’t the reason why an inventory of Medium and High impact BCS is required under CIP v5/v6. The reason an inventory is required is because CIP-002 R1 requires it, period. And CIP-002 R1 not only doesn’t require an inventory of Low BCS, it explicitly states that such an inventory won’t be required.

All of this isn’t to say that, once CIP-003 version 8 comes into effect, NERC entities won’t be under even more pressure from their regions to maintain an inventory of Low impact BCS. Some of the regions have already stated that they would like to see their entities do this, and they will be able to make that statement with even more justification once CIP-003-8 comes into effect. But until the statement that an inventory of Low BCS isn’t needed is actually removed from the CIP standards – and now it’s found in CIP-003-6 and CIP-003-7, as well as in CIP-002 R1 and Attachment 1 – I think NERC entities can rest assured that nothing fundamental has changed in this regard, no matter what requirements end up in CIP-003-8. 

The views and opinions expressed here are my own, and do not reflect those of any organization I work with. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i][i] For a list of the four items FERC is proposing to require, see bullet point 4 toward the end of the November post.

Two Holiday Presents for you – Present Number One

I’m hereby starting a long-standing tradition of writing one or two (or more, if possible) posts around the holiday season containing unadulterated good news. While it certainly isn’t true – as I’m sure a few people believe - that I’m a doom-and-gloom blogger, for whom every glass is half empty, I probably do talk more about disturbing developments (or discoveries I’ve made) than about good stuff. So here’s the first of two attempts to redress the balance a little.

At the December meeting of the NERC CIPC in Atlanta two weeks ago, an important FERC staff member made the statement that – while he in no way speaks for the Commission – he doesn’t believe that FERC will stand in the way of a reasonable effort to make whatever changes need to be made to the CIP standards to allow NERC entities to locate entire BES Cyber Systems in the cloud. In other words, cloud services like outsourced SCADA, whose use has already been embraced by other industries besides electric power, but which haven’t so far been used much by the power industry simply because of concerns about how this would be possible given the current CIP standards, should one day be open to power industry participants.

Let’s review how this concern about CIP and the cloud came about:

1. There is nothing in CIP-002 R1, which determines which Cyber Assets need to be included in BES Cyber Systems and classifies BCS as Low, Medium and High impact, that says BCS can’t be physically located at cloud providers. However, I – as well as many others – have always believed that, in the case of Medium and High impact BCS, the demands that the entity would need to place on the cloud provider in order to ensure compliance with some of the Medium and High impact requirements, especially CIP-004 R3 to R5, would be such that the cloud provider would never agree to them.
2. This in itself doesn’t mean that BCS couldn’t be outsourced to the cloud while the current CIP standards are in effect. Whatever you may have heard about NERC auditors always auditing to the letter of the requirement, there are many cases now of practices that are tolerated even though they don’t comply with what is actually written in the requirement[i].
3. However, the trump card, in my opinion, has always been the fact that NERC and FERC – especially FERC – were fundamentally opposed to the idea that BES Cyber Systems could be outsourced to the cloud. Without their active support for doing this, there would be no formal or informal way to get around the problems posed by certain CIP requirements. With their active support, these CIP problems could be expected to just melt away (see number 2 above).
4. Where did I get the idea that FERC was opposed to outsourcing BCS to the cloud? Certainly from hearing one or two FERC staff members speak at public meetings (even though they always acknowledged that they don’t speak for the Commission), as well as from interpreting statements made by NERC staff members (especially at the Emerging Technologies roundtable discussion that was held after the June CIPC meeting in San Diego). But the main reason I have always assumed FERC was opposed to this is the fact that I have never heard anyone from FERC say anything different. Until two weeks ago.

This is why, when I wrote a post in March that wrapped up a series of posts on the cloud and CIP, I stated (in bullet point 7 toward the end of the post):

“Regarding the second fundamental question, whether High or Medium impact BCS themselves (not just information about them) can be stored in the cloud, the answer is: Not if you want to have any friends at NERC or your Regional Entity. Regardless of whether this is compliant or not, I know that both NERC and FERC are very much against “real-time operations in the cloud” – as I pointed out in this post in January….”

However, after having heard the statement from the FERC staff member at the CIPC meeting two weeks ago, I can no longer say that outsourcing Medium and High impact BCS to the cloud will never be allowed by NERC and FERC. On the other hand, if I were you I wouldn’t rush out tomorrow and outsource all your Medium and High BCS to the cloud[ii]. NERC will need to offer some sort of guidance document that will make clear both what is allowed in this regard and what kind of evidence will be required from the cloud provider in order for a CIP potential violation not to be identified.

But at least this is a good step in the right direction.

The views and opinions expressed here are my own, and do not reflect those of any organization I work with. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] For example, almost every NERC entity – and NERC auditor – will tell you that Attachment 1 of CIP-002 exists to classify assets like substations, generating plants, etc. as High, Medium or Low impact. But a plain reading of Attachment 1 leaves little doubt that what is being classified are BES Cyber Systems, not the assets themselves. Moreover, Attachment 1 makes clear that Medium BCS can be found at other locations than the asset they are located at (an unfortunate side effect of which I recently discussed in this post). This means that an entity with Medium impact assets (there, I’ve just committed the same sin!) in theory needs to look all through their system to find the BCS associated with a particular Medium asset (especially Medium control centers). Yet I strongly doubt any NERC entity has ever received a PNC for not doing this, or ever will for that matter. It is a settled practice now that NERC entities only need to look for BCS associated with a Medium or High impact asset at the asset itself, nowhere else.

[ii] Low impact BCS are another story. Given that the only CIP requirement that applies to them is CIP-003 R2, and given that there is nothing in that requirement that isn’t already being done in spades by every cloud provider, I don’t see anything in CIP now that should prevent Low BCS from being outsourced to the cloud. However, as with all questions about CIP requirements, you should talk with your NERC Regional Entity before doing this.

Does Auditing Matter?

Five of my last six posts have dealt with plan-based requirements, such as CIP-013 R1, CIP-014 R5, CIP-010 R4, and (parts of) CIP-003 R2. These requirements have become very popular among the four NERC CIP drafting teams that have done their work after the CIP version 5 standards were developed, approved and implemented. In fact, almost all of the requirements, and both of the new standards, that have been developed since CIP v5 have been plan-based. And I see no sign that this process will let up anytime soon. The new standard currently being developed, CIP-012, is definitely plan-based as well.

The last four of those five posts have all addressed the question of auditability - namely, how can a plan-based requirement be audited? In the last two posts, I have made the point that, in my opinion, for a plan-based requirement to be auditable it needs to include a list of threats that must be addressed[i]. For example, in this post I pointed out that CIP-014 R5 requires that the plan specifically address the threats and vulnerabilities identified in the threat and vulnerability analysis required by R4. And in this post I discussed CIP-010 R4, which does a very good job of outlining the threats to be addressed, and is the best example I’ve seen so far of an auditable plan-based requirement.

And then there’s CIP-013 (especially CIP-013 R1). In the last three posts in this series, I’ve stated that it is only auditable in a limited sense. R1.1 lists three threat types that need to be addressed in the plan: threats from procuring vendor equipment and software, installing vendor equipment and software, and transitions between vendors. The only list of threats is in R1.2, and those are only a subset of the threats that would apply to just one of the three threat types (they’re all threats from procurement). Given that the great majority of the threats that would appear in a comprehensive list are not shown, I rate CIP-013 R1 as mostly un-auditable.

However, in each of the last three posts in this series, I’ve gone on to raise another question: Does it matter that CIP-013 R1 (and by implication R2) is un-auditable? More specifically, in what ways does it matter and not matter?

Let’s step back and ask a larger question: What is the purpose of auditing in the NERC standards environment, anyway? For the prescriptive requirements (which I will soon be calling “means-based” requirements, since that is the term that people who write about these things use) – including most of the 693 (O&P) requirements and many of the CIP requirements - the answer is quite simple: to determine whether the entity did or didn’t do what they were supposed to do or not do.

But why do we need to audit the prescriptive standards in the first place? Because they’re mandatory. And why are they mandatory? Because following them is deemed important for the reliable operation of the Bulk Electric System. And this is the fundamental reason why prescriptive standards are audited.

But how about the plan-based requirements (which I will soon be calling “management-based”)? Given that the ultimate goal is the protection of the BES, what is the best way to accomplish that goal? We shouldn’t engage in a false analogy here and just say “Well, if auditing is the way that the BES is protected with respect to prescriptive requirements, then it’s obviously the way to protect the BES with respect to plan-based requirements.” Is this really the case? Let’s look at the nature of prescriptive and plan-based requirements.

First, why are many NERC requirements prescriptive? Let’s use the example of FAC-003, the infamous tree-trimming standard whose violation was one cause of the 2003 Northeast blackout. Table 2 of this standard shows minimum distances that must be maintained between a high-voltage transmission line and any tree directly under it. It is prescriptive, because all it takes is one overgrown tree to trip such a line – and having a few such lines tripped in one day, coupled with generation outages, can lead to a cascading outage (or at least it did in 2003). Since any one tree can cause a line trip, there is really no choice in the matter: the requirement has to be prescriptive, and it must apply to every tree in scope. Here, the audit’s role is to confirm that the utility is really complying with the requirement and trimming all of the trees that they need to.[ii]

Now, why are some NERC CIP requirements plan-based? Because a plan-based requirement is designed to address an objective that isn’t measurable.[iii] In the CIP world, the objective of a requirement is always the mitigation of the risk to the BES posed by one or more cyber or physical threats. As with prescriptive requirements, the protection of the BES is still the primary goal here. But what’s the best way to protect the BES, when a requirement is plan-based?

I don’t know about you, but I think the best way to protect the BES, in the case of a plan-based requirement, is for the NERC entity to have a good plan and to implement it well. Note that this isn’t a binary choice, as in the case of prescriptive requirements. This isn’t a question of verifying whether the entity took particular actions or not. Rather, it’s a case of whether the plan they developed, and their implementation of it, advanced the objective of the plan (in the case of CIP-013, the objective is supply chain security. In CIP-014, it’s physical security of key substations; etc).

But if this is true, what good does it do to simply audit after the fact? For example, suppose an entity has no idea how to write a supply chain security plan, and puts together a document that includes a lot of miscellaneous directives that don’t help BES supply chain security at all. Or suppose they put together a good plan, but do a terrible job of implementing it, so that – again – supply chain security isn’t improved. How does it help the BES for auditors to simply come in after the fact, stroke their chins, and say “Yup, this was a terrible plan, all right” or “These guys wrote a great plan, but they put an idiot in charge of implementing it, so they might as well not have bothered with the whole exercise”?

The answer to this question is it doesn’t help the BES at all for the auditors to determine after the fact whether the plan is good and whether it has been well implemented. If the entity is struggling with their plan, they should be able to get help at the time they’re struggling, not years later when they’re told their plan is no good. The same with implementation: As they start implementing the plan, someone needs to check on them to determine whether they’re on the track to success or not. If they aren’t, they need to receive some advice on how they can get on track.

Two questions come up here. First, if the entity is struggling with their plan or with implementation, why can’t they turn to a consulting firm? The answer to that is they certainly should be getting help from consultants if they need it, but in the NERC world the “consultants” who really matter are the auditors. You can find a good example of that in my recent post on CIP-014, where I mentioned a NERC entity that had received a PNC (potential non-compliance) notice because they had taken the requirements in that standard literally when they say that the entity is to protect against attacks on the substation itself, not on any of the individual Facilities (buses, transformers, etc) in the substation. The auditors thought they should be protecting the Facilities as well (especially the transformers).

As I also pointed out in that post, I don’t blame the auditors for saying the entity’s plan should have included protection for the transformers as well as for the whole substation, although I think they should just have identified this as an Area of Concern (AoC), as another set of auditors had done with another entity in the same region in a similar situation. But regarding my point on consulting firms, this entity had worked closely with a very respected NERC CIP compliance firm to do the threat and vulnerability assessment and develop the plan, yet the plan was still wrong as far as the auditors were concerned. It’s a sad fact but true: as far as CIP compliance is concerned (not just CIP-013), the only correct interpretation of the requirement is the one your auditor follows.

This utility was actually quite lucky that they got audited when they did – shortly after they had developed their Physical Security Plan for CIP-014 but before they had implemented it. Had their audit date fallen a couple of years later, they would have not only developed the plan but implemented it. For them to hear at that time that the plan was all wrong would have been much more serious, since it would mean they should have spent their implementation money differently than they actually did spend it, over the past couple of years.

Now to the second question: Given that the auditors (or at least the region) are the ones who really need to judge the plan and its implementation, what is to keep them from giving the plan an informal review after it’s developed, and providing helpful advice on how it could be improved? And why couldn’t the auditors, when the entity has started their implementation of the plan and has some questions about whether they’re doing it right, simply take a look at how the implementation is going so far and point out any problems they see?

You may have guessed the answer: the principle of auditor independence. According to auditing standards and especially to the Generally Accepted Government Auditing Standards (GAGAS), which NERC auditors are all trained on, the auditor can never provide compliance advice to an entity they will audit, prior to the audit itself.

I agree that auditor independence makes sense in the case of prescriptive requirements, where there really is a binary answer to compliance questions. But what about in the case of plan-based requirements, where the entity – as we have just seen – could go way off course and spend a lot of money implementing a plan that isn’t what the auditors want to see? I have a great example of that. It’s from this post – also about CIP-014 – where I described a different entity that had asked their region to give them advice on a big physical security investment (about $80 million, in fact) they were thinking of making at their critical substations; they wanted to know whether that investment would increase the likelihood that their Physical Security Plan would be found to be adequate when they were audited. Given the amount of money involved and the fact that there were a lot of alternative uses for it, management didn’t want to make this investment without some clear signal that it would help, and especially not hurt, their CIP-014 compliance posture.

I don’t think it will surprise you when I say that the answer they received was no, that the auditors couldn’t provide an answer to this question because it would violate – ta da! – auditor independence. Again, I can’t blame the auditors. Given that they are effectively government auditors and they have to follow GAGAS, how could they give any different answer?

But instead of shrugging our shoulders, why don’t we go back to the question I asked above: Why is this standard mandatory? The answer: Like with the other CIP standards, the protections it affords are deemed to be very important to the BES. Then let’s ask another question: Given that the region’s refusal to answer the entity’s question may well result in an $80 million physical security investment in critical substations not being made, and given that it’s indubitable that this investment would have enhanced the physical security of the BES (otherwise, why would a large, sophisticated utility have even considered it in the first place?), how does the possible cancellation of this project make the BES more secure?

The answer to this last question is simple: It doesn’t. The potential cancellation of this project (and I have no knowledge whether it might be cancelled or not) will definitely result in a less-secure grid than would its implementation. So if our ultimate concern is really the security of the BES (which it is), not guarding the principle of auditor independence, what can we do about this?

In my opinion, this is how plan-based standards should be managed (I don’t want to use the word “audited”) by NERC and the regions:

1.       Any requirement to develop a plan will need to include a list of threats to be addressed in the plan, as well as some mechanism for updating that list on a regular basis (hopefully by an industry body, not just by the entity itself). The requirement will also need to specify that the plan must include a timeline for implementation.

2.       For every requirement to develop a plan, there must be another requirement to implement it. Part of that requirement will specify that any changes in timeline or objectives of the plan will be documented and explained, so that when the region reviews the implementation they will always be looking at the current plan.

3.       The entity develops their plan, either on their own or using a third party like a consulting firm.

4.       The entity submits the plan to their region (not necessarily to the auditors, of course). The region reviews it and points out any deficiencies they see.

5.       The entity revises their plan and re-submits it. If it’s now good to go, the region approves it. If it’s not, the region sends it back a second time, with a new set of pointers (and perhaps a stern admonition if they think the entity hasn’t been paying close enough attention to what they said).

6.       If needed, there’s another cycle, but ultimately the plan will be approved. The entity then starts the implementation process, which will follow the timeline shown in the plan.

7.       At any point during implementation, and with or without a request from the entity, the region can step in and review how the implementation is going. They’ll compare the entity’s progress to the timeline, and will require explanations of any deviations from that timeline (in fact, any deviation should already have been documented). Of course, if they think the project is off course in any way, they’ll provide advice on how to fix that.

8.       If at any point in this process the entity seems to be acting in bad faith, or simply ignores advice the region gives them, then – and only then – can they be issued a PNC.

Don’t you think the BES would be more secure if this were how the region operated, rather than auditing CIP-014 like it’s FAC-003, so that a judgment needs to be made on whether the entity has complied with each requirement and requirement part – but at the price of the region not being able to provide any guidance at all before the actual audit? I do, too. This is why I’m saying that, for plan-based requirements, audits really don’t matter very much. What matters is the guidance the regions can give to the entities. If we had to give up auditing of these requirements altogether and essentially make them voluntary standards, the BES would be more secure, not less.

How does this problem get fixed? In the long term, it will be a big deal. For one thing, the requirements themselves will need to be rewritten as I described in the first two bullet points. For another, NERC’s CMEP (Compliance Monitoring and Enforcement Plan) and perhaps the Rules of Procedure will also have to be modified. Once that is done, the “management” steps I just described will allow plan-based requirements to be “audited”, but in a different sense than audits of prescriptive requirements.

However, there is also a shorter-term fix, which involves doing all of the above steps with different people than the actual auditors and only issuing AoC’s, not PNC’s. This would also be a big change, but it wouldn’t require rewriting the standards, CMEP or the RoP. And in fact, I think it is effectively how plan-based requirements will be “audited”, until the longer-term fixes can be implemented. I’ll describe this in an upcoming post, but there are a few that I want to write before I go back to this topic. So stay tuned.

Have a good holiday!

The views and opinions expressed here are my own, and do not reflect those of any organization I work with. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] I now realize that this list doesn’t have to be provided in the requirement itself. Rather, it can be developed through a local threat and vulnerability assessment. This would only be appropriate in a standard like CIP-014, where the physical security threats to one substation will inevitably be substantially different from threats to another substation, due to variability in terrain, population, etc. Most cyber threats aren’t specific to one facility,

 one NERC entity, or even one region of the country, which is why I think the threats to be addressed in the plan for requirements like CIP-013-1 should normally be included in the requirement itself. And as I’ve also pointed out, there should be a provision for updating this list regularly, since new threats come up all the time. So far, only CIP-013 has a provision for updating threats, and it doesn’t do this the way that I would.

[ii] Here’s a fun exercise for the reader. Compare the argument I just made for a prescriptive tree-trimming requirement to CIP-007 R2 patch management, the most prescriptive CIP requirement. Does the same argument apply there? If not, how is the situation different? If you want to email me your papers, I’ll send back your grade. But please, no looking at your neighbor’s paper! For my answer, you’ll have to wait for the book I’m working on now, but it’s likely I’ll touch on this topic in the blog sometime before then. I have probably discussed it at least a few times previously, but don’t ask me to look through all those old musty posts to try to find where I said that! One person can only take so much punishment…

[iii] By giving this quick-and-dirty definition, I’m going beyond what I said about plan-based requirements in the first post in this series. I thought up until a few days ago that plan-based requirements were just a subset of objectives-based ones. But I’ve now learned that true objectives-based requirements need to have a measurable objective. The plan-based requirements in CIP aim for non-measurable objectives like supply chain security, physical security, etc. As I already said, plan-based requirements are really one type of “management-based” requirements. In management-based requirements (found in many compliance regimes, not just NERC or NERC CIP), the entity needs to put together a good program (i.e. plan) to pursue the objective, but with the understanding that there is no such thing as obtaining an objective like supply chain security. If you thought it was actually possible to attain that objective, then I’m sorry to have to disabuse you of that notion. By the way, there’s no Santa Claus, either. 

Continuing Dialog with an Auditor on Auditing CIP-013

In my last post, I engaged in a dialog with an auditor regarding how CIP-013 will be audited (although this discussion applies to other plan-based standards and requirements, including CIP-014, CIP-010 R4 and CIP-003 R2). The auditor had emailed me to object to the main point I had made in my previous post: that a requirement that mandates that the entity have a plan needs to have some specific high-level criteria in the requirement itself in order for it to be auditable. CIP-013 R1 only provides six criteria that apply to only one of the three main areas that the supply chain cyber security risk management plan needs to address; therefore, I consider R1 to be mostly un-auditable.

In his email, the auditor said I had overlooked the fact that there are both “objective” and “subjective” audits. A prescriptive requirement like CIP-007 R2 can only be audited “objectively” – that is, by checking the box on each part of the requirement; did the entity do it, or didn’t they? But the auditor said a requirement like CIP-013 R1 needs to be audited subjectively. And what is a subjective audit? It is one that determines whether “the approach used was reasonable and sound, and whether the entity sufficiently achieved the objective to not merit a PNC.  That makes the audit subjective and it places a greater burden on the ability of the auditor to make such a determination.”

I then pointed out (I’m still talking about my last post, of course!) that I didn’t think this was good enough to audit a plan-based requirement; I still believed there need to be some objective criteria in the requirement itself (and in my opinion the criteria listed in Attachment 1 for CIP-010 R4 are the best example of providing criteria in a plan-based requirement).  My reasoning came down to this: “If the plan is a reasonable one and adequately takes account of the threats that need to be addressed in the plan, the entity should presumably receive a passing grade on the audit.” And you can consider the “criteria” in the requirement to be essentially a listing of the threats that need to be addressed in the plan.

Here’s an example of what I’m talking about: CIP-010 R4 is a plan-based requirement. It requires the entity to develop and implement a plan for securing Transient Cyber Assets and Removable Media. But it doesn’t just mandate that the entity develop the plan, it provides a whole set of criteria for what must be addressed in the plan; these are listed in Attachment 1 of CIP-010-2.

There are three sections to Attachment 1 (TCA’s managed by the Responsible Entity, etc), each of which contains between two and five headings. The headings show domains that must be addressed within that section: “Software Vulnerability Mitigation”, “Malicious Code Mitigation”, etc. These headings can be thought of as threats whose risk needs to be mitigated: the threat of vulnerabilities in software installed on TCA’s, the threat of malicious code carried on removable media, etc. Under each heading is a list of processes or technologies that can be used to mitigate those threats; the entity must use “one or a combination” of these to mitigate the risk posed by the threat indicated in the heading.[i]

To sum up what I’ve just said, I think that a plan-based requirement needs to include a list of threats that must be addressed in the plan (which I called “criteria” until just now). If the requirement doesn’t include such a list, then I don’t think the auditor can make a decision on whether the plan is adequate or not; in other words, I don’t think the requirement is auditable. CIP-013 R1 only lists six items that need to be included in the plan. They only partially address one of the three overall threat areas that R1.1 requires to be addressed: procuring vendor equipment and software; installing vendor equipment and software; and transitions between vendors. Therefore, I believe CIP-013 R1 (and by extension R2) is only partially auditable. On the other hand, CIP-010 R4 Attachment 1 does a good job of listing threats that need to be addressed, as well as giving the entity options for how they mitigate those threats; I believe that this requirement is auditable.

After I put up this post, the auditor emailed back to me and said the following[ii]: “I am arguing that there do(es) not have to be defined criteria in the Requirement, specifying what must be in a “plan” in order to render the requirement, and thus the plan, auditable.  In fact, the minute criteria appear in the requirement, two things happen.  First, the requirement becomes prescriptive.  The plan must contain these elements or else the plan will be found non-compliant.  Second, many entities will write a plan that only includes those elements.  They will not do the deep thinking, as you suggested in your previous post, to determine what needs to be in a really good plan.  And as circumstances change, there is no incentive to modify the plan because it already hit the required elements…it is technically compliant.”

My response to what the auditor has just said is the following: I am advocating that plan-based requirements include a listing of the threats[iii] that need to be addressed in the plan, not of how those threats need to be mitigated. The fact that the entity still can choose how they mitigate the threat means that the requirement hasn’t been turned into a prescriptive one.[iv] Your second point, that entities will write a plan “that only includes those elements”, should now be reworded as “entities will write a plan that only addresses the threats listed in the requirement.”

Theoretically, raising this as an issue would make sense if we are assuming that NERC entities all have the capability to look out over the whole threat landscape and decide for themselves what are the important threats that need to be addressed in the area covered by the plan-based requirement in question (of course, in the case of CIP-013, those threats are supply chain threats. In the case of CIP-014 R5, the threats are physical threats to an important substation). It is certainly likely that some entities do have this capability (mostly the larger ones, of course). However, I think that a lot of NERC entities (even ones with High and/or Medium impact BES Cyber Systems) would have a hard time articulating how they would do this.

And indeed, I really don’t think it should be up to the entities to survey the threat landscape and decide what are the most important threats that apply to them. Or at least, they shouldn’t be required to do this, if they are potentially going to face a Potential Non-Compliance finding for not being able to identify all of the threats that – in the auditor’s opinion, of course – should be addressed in the plan. So I think there should be a list of threats to be addressed included in the requirement for the plan. This is the case with CIP-010 R4, but it isn’t the case with CIP-013 R1, which is again why I say that requirement is only partially auditable.

A Slight Digression

(You can skip this section without missing anything of the overall argument. In fact, it might be better to skip this section the first time around, so you can keep the larger argument in mind).

The auditor goes on to discuss CIP-013 R1.[v] He says “You argue that Part 1.1 does not specify what must be in the procurement planning processes defined in the plan and therefore Part 1.1 is not auditable.  I disagree.  Part 1.1 specifically states that the processes used in planning for the procurement of BES Cyber Systems must “identify and assess cyber security risk(s) to the Bulk Electric System from vendor products or services.”  Part 1.1 goes on to mandate that two types of risks must be addressed: “(i) procuring and installing vendor equipment and software; and (ii) transitions from one vendor(s) to another vendor(s).”  Part 1.1 does not presume to know what those risks may be.  It is explicitly up to the entity to identify those risks and to assess those risks that have been identified.  Once assessed, the Part expects one or more processes to be developed to address those risks.  The plan and/or procedures will have to include how the risks are identified and assessed, since that is a key expectation of the Requirement.  The entity might have (a) beautifully documented plan and process, defining a completely ineffective methodology for getting where the entity needs to be.  And, as implied by the intended vagueness or non-prescriptiveness of the Part, the entity needs to continue to identify, assess, and mitigate evolving threats.”

You will notice the auditor is talking about “risks” here, not threats. That is because CIP-013 also talks about risks, not threats. However, I believe that threats need to be identified in the requirement, not risks. And what – ask you – is the difference between risks and threats? The Oxford Dictionaries define a threat as “A person or thing likely to cause damage or danger.” Notice there is nothing quantitative about this. For example, a cyber threat might be “software vulnerabilities”.

A risk, on the other hand, is quantitative[vi]. I define it as the probability that a threat will be realized, times its impact. I also believe that, at least in cyber security, it is close to impossible to even estimate the absolute values of particular risks. However, I do believe that it is possible to speak about the different levels of risk that apply to particular threats; this then leads to the possibility of ranking threats according to the degree of risk they pose.[vii]

So if we substitute “threat” for “risk” in what the auditor says above, as well as in the text of CIP-013 itself, I don’t think we’ll do too much harm to the meaning. The auditor can now be understood as simply giving an example of what he said in the first paragraph that I quoted of his email: That it should be up to the entity to identify the threats that need to be addressed in a particular plan-based requirement, and up to the auditor to determine whether the entity has properly identified those threats. As I’ve already said, I disagree with that position. I think the requirement should include in it the threats that need to be addressed in the plan, and that it isn’t auditable unless this is done.

Continuing with our Story

The auditor concludes with this paragraph: “And that is a key point here.  Non-prescriptive requirements by their very nature have implied, “prescriptive” requirements.  CIP-013 R1 is not a “one and done” effort.  As I just mentioned, there is an expectation of continuous identification and assessment of new risks as they arise.  Another implied requirement is that the defined processes that were developed as part of the overall plan are expected to be implemented.  In the past, FERC felt it necessary to require Standards to be modified to explicitly require implementation, otherwise entities might, in FERC’s view, argue that failure to implement the process or control was not a violation because the Standard did not explicitly require it.  FERC even went so far as to declare a CIP-008 Incident Response Plan that was not performed (implemented) in the event of a cyber security incident was to be found as a violation.  In the age of non-prescriptive requirements, the underlying expectations have not changed; they are just not explicitly stated.  You cannot have it both ways.  If you want prescriptive requirements, then fine.  Just don’t then complain about the minutiae.  If you do not want prescriptive requirements, then accept that you will have to read between the lines and meet the intention of the Standard and its requirements.  And accept that the auditor is adept at reading between the lines as well and will evaluate what you did against the intention of the requirement and its underlying expectations, to determine if you did “good enough.”  The audit team will then apply its collective expertise and professional judgment to determine what type of finding might be warranted, given the facts and circumstances of the instant case.” (my emphasis)

I have three comments on this. The first is that a requirement may well be implied by another requirement (and I talked a lot about implicit requirements in CIP v5 in the long-ago days), but the implied requirement can never be audited by itself (although I don’t disagree with the auditor that the entity could be in violation of the original requirement if they haven’t followed a requirement that is clearly implied by it. On the other hand, this is no way to write mandatory standards! All requirements should be explicitly stated. This is one of the big problems with CIP v5 – the fact that full compliance requires following so many implied requirements). The auditor uses the example of a requirement to develop a plan but not to implement it. I know that was a problem in the CIP v1-v3 days, but I don’t now know of any current CIP requirement to develop a plan, that isn’t accompanied by a requirement for the entity to implement the plan.

Second, I want to call attention to what the auditor said about “continuous identification and assessment of new risks as they arise”. Of course, I would substitute “threats” for “risks” here, but otherwise I totally agree that there needs to be a process to identify new threats as they arise. As I discussed in this post, this is what CIP-013 R3 does. But this also shouldn’t be the entity’s job. There should be a centralized body that continually reviews new supply chain threats, as well as new techniques for mitigating them, and regularly updates NERC entities on these.

But my biggest issue is with the sentences I highlighted. It seems to me that the auditor is making a false dichotomy here. He seems to be saying that if a requirement isn’t prescriptive (i.e. it’s an objectives-based or plan-based requirement), then entities just have to accept that there will be implied requirements they will be held accountable for. I think this is almost exactly backwards. Implicit requirements only come into play when there are prescriptive standards that aren’t worded properly. Since a plan-based requirement (which BTW I will soon start calling management-based, while I’ll start calling prescriptive requirements means-based. There’s a reason for this, which I’ll discuss in a separate post soon. For the moment, I’m sticking with my current terms) doesn’t prescribe any actions, there shouldn’t be any implicit requirements associated with it. Especially since I’ve now clarified my wording, and said that all plan requirements should include a list of threats that need to be addressed in the plan, not a list of “criteria”. The auditor feared that criteria could easily become prescriptive requirements, and I agree with him about that. But threats can’t become requirements in themselves.

Finally, the End

I started this post with my assertion (from two previous posts) that plan-based requirements should have a list of threats that need to be addressed in the plan, in order to be auditable. Furthermore, I stated that, since CIP-013 R1 only provides a partial list of the threats to be addressed in the supply chain cyber security risk management plan (SCCSRMP, in case you forgot the acronym. In another year, this acronym will be on every NERC compliance professional’s lips, it’s so musical. I ought to trademark it), R1 is mostly un-auditable. So I am ending the post by making the same two assertions that I started it with, after having considered the auditor’s excellent points that he raised in his email. I have learned a lot by writing this post, and I have clarified my positions. Just because I ended up where I started doesn’t mean the journey wasn’t worthwhile!

               

Of course, I have yet to answer the question I’ve said I’ll address in my next post twice already: namely, whether it matters that CIP-013 R1 is mostly un-auditable (spoiler alert: I’m going to say it really doesn’t matter that much). Maybe next time I’ll actually do that, although I won’t bet on it.

The views and opinions expressed here are my own, and do not reflect those of any organization I work with. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] I’m sure some people questioned the fact that in none of these sections is the entity free to find another mitigation that is equally effective as the ones listed; these are the only options the entity has. My guess is the drafting team decided that there was too much risk that an entity would grab onto ineffective means of mitigating the risk, rather than the tried-and-true ones listed. I won’t try to weigh in on this argument, since the requirement is now in effect and I haven’t heard any grousing about the fact that the entity doesn’t have any other options besides those listed (like they do in CIP-003-7 Attachment 1, Section 3.1, where the entity is just required to “implement electronic access controls” to “Permit only necessary inbound and outbound electronic access” to Low impact assets with external routable connectivity. Of course, CIP-003-7 is another plan-based requirement).

[ii] There were actually two email exchanges this time. What I’m quoting is all form the second exchange.

[iii] And I will readily admit that my thinking has evolved even since my last post, when I was simply saying that there need to be “criteria” in the requirement. I wasn’t thinking that the criteria would be simply threats that need to be addressed, but that they could indeed be something like requirement parts. If I hadn’t changed my thinking, I would completely agree with the auditor that these criteria could well have turned into prescriptive requirements. However, by clarifying that the requirement needs to include a listing of threats, not “criteria”, I think I have addressed the potential problem of turning a plan-based requirement into a prescriptive one.

[iv] If you’ve been paying attention, you may think this is a contradiction to the praise that I just heaped on CIP-010 R4 Attachment 1. This requirement lists threats to be mitigated, but it only provides certain options for mitigation; the entity doesn’t have complete freedom to mitigate in a way that makes the most sense for them. While I don’t necessarily agree that this is the right approach, I think that just having these options means this hasn’t turned into a prescriptive requirement.

[v] Part of what he says isn’t germane to the main argument of this post, but is nevertheless quite good. Here it is:

“The purpose of the Standard, and thus the collective purpose of the requirements is to “to mitigate cyber security risks to the reliable operation of the Bulk Electric System (BES) by implementing security controls for supply chain risk management of BES Cyber Systems.”  Everything that the entity does with respect to the requirements is expected to achieve the overall purpose of the Standard.  In plain language, the entity needs to “do something”, i.e., implement one or security controls to mitigate risks.  That, therefore, implies that the entity needs to identify and understand the risks so that it can design effective security controls to mitigate those risks.  Yes, I know that the purpose statement is not an approved, auditable requirement.  But, it serves to inform the intention of the Standard and its requirements, thus establishing certain expectations of performance.

“Requirement R1 states “each Responsible Entity shall develop one or more documented supply chain cyber security risk management plan(s) for high and medium impact BES Cyber Systems.”  The requirement goes on to specify certain elements that must be in the plan.  There are two specific elements: (Part 1.1) one or more process(es) used in planning for the procurement of BES Cyber Systems, and (Part 1.2) one or more process(es) used in procuring BES Cyber Systems.

“I agree that Part 1.2 prescribes six specific elements of the procurement process that are auditable.  Either the processes collectively address the six elements or they do not.  Part 1.2 is prescriptive and can be objectively audited in terms of whether the elements are present.  Part 1.2 can also be audited subjectively in terms of whether the elements were adequately addressed.  The subjective aspect of the audit of Part 1.2 primarily applies to Parts 1.2.5 and 1.2.6.  Those are the two Parts that afford entity the most discretion in how the entity approaches the expectations.  For example, Part 1.2.5 requires “verification of software integrity and authenticity of all software and patches provided by the vendor for use in the BES Cyber System.”  While hashing downloaded files and verifying the hash results against information provided by a vendor is an excellent way to accomplish this expectation, not all vendors will provide hashes of their software.  The process will have to address how the entity plans to perform the expected verification in the absence of provided hash values (in this example).  That suggests that as the plan and its required processes are developed, the entity is coincidently evaluating what its vendors do today in order to identify and develop appropriate verification schemes that accommodate vendor practice.  And, if the vendor changes its practices in the future, the processes might need to be changed to maintain alignment.

[vi] I will readily admit that the definition of risk that I’m giving here isn’t from any dictionary. Rather, it is the definition that makes my worldview of industrial cyber security work. I’ve said before – in fact, probably in mid-2016 – that I’m working on a book, with two co-authors, on how NERC CIP can be “reformed” to address the major problems it has, and make it sustainable into the future (which it is not now, in our opinion). I’m glad to say that we’ve finally progressed beyond the talking stage and are doing serious writing now, although I’m not going to even talk about a publishing date yet. In any case, this “definition” of risk is the one we are using in our book and it works for our purposes. Other definitions would work for other purposes.

[vii] This observation – that it is possible to at least give a rough ranking of the relative degree of risk that each cyber threat poses – is the key to making our upcoming book work. I hate to tease you with this idea and leave it there, but I can’t really explain why this is the case without quoting a good portion of the book. And the book isn’t written yet!

Still Available!

When I first left my former employer at the end of October, I was quite optimistic about finding a new position right away, and thought I would have found something new by now. Unfortunately, that hasn’t happened, although there are still good possibilities.

So if you know of something – either full time or on contract – that you think I would be good for, I’d appreciate your letting me know! The email address below is the right one to reach me at.

But the blog will keep coming!

The views and opinions expressed here are my own, and do not reflect those of any organization I work with. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. 

An Auditor Weighs in on Auditing Plan-based Requirements

In my last post, I raised the question whether (or rather, how much, since this isn’t a binary question) CIP-013 was auditable. In the previous post, I had discussed this same question with respect to CIP-014, and decided it was for the most part auditable. However, for CIP-013 I decided that only one requirement part, R1.2, was fully auditable; the other requirements and requirement parts in the standard are either un-auditable or undetermined.

However, at the very end of my last post, I also pointed out that I like CIP-013 a lot; in fact, I think it’s as close to what I would call the ideal standard as any of the CIP standards are. Why would I say this? Is it due to early-onset senility, or is there some other explanation? I’m perhaps not the best person to decide this question, but I can assure you that I believe there’s another explanation.

I hinted at that explanation in the last paragraph of that post: I no longer think that auditability is the most important criterion by which a CIP standard should be judged, or at least one that is based on a plan (and I’ve been discussing plan-based standards and requirements in the previous three posts). My last sentence of the post said I didn’t want to make that post any longer, so I would discuss this idea in my next post. This is that post.

I was all set to continue with the idea in this post, when an auditor – who has contributed to many of my posts over the years – wrote in with his comments on the last post. As sometimes happens, we had a long email exchange, with one of us inserting comments in one color, then the other inserting his in another color (in the past we’ve once or twice gone beyond all the primary colors and started looking through paint catalogs to find more colors; we didn’t get that far this time). I will just summarize what I think were the most important points he made.

The main point of my last post was that, in order to be auditable, a plan-based requirement would need to include criteria for what should be in the plan. For example, I pointed out that CIP-014 R5 specifies four general areas that must be addressed in the physical security plan, which made it auditable in my opinion. But CIP-013 R1 just says the entity should develop a supply chain cyber security risk management plan. It provides no criteria for what should be in the plan, other than six specific items listed in R1.2. This was why I said this requirement was “mostly un-auditable”.

Moreover, as I’ve pointed out at least a couple times, the six items in R1.2 are only there because FERC specifically ordered they be addressed in the new standard. R1.1 makes clear that the plan should address risks in three general areas, but the six items all have to do with only one of those three areas (vendor risk) – and they don’t come anywhere near addressing all the risks in that one area.

The auditor asserted that I was wrong to say that a plan-based standard needed criteria in the requirement in order to be auditable. He made the distinction between objective and subjective audits. Objective audits are only possible with prescriptive requirements. Since these requirements (ideally) specify particular actions that an entity should take, they can be easily audited: a) Did the entity take the actions? If so, they complied with the requirement; b) Did the entity not take the actions? Then they didn’t comply.

However, this is only realistic when the requirement is unambiguous, in both a) what it applies to and b) the steps that need to be taken. Unfortunately, there are a number of CIP requirements where there is ambiguity in one or both of these areas. I don’t particularly blame this on the teams that drafted the requirements, or even on the NERC entities that approved them. Rather, it’s simply due to the nature of the cyber security domain – as opposed to the deterministic, physics-based domain of the other NERC standards (called the “693” standards and also Operations and Planning) – that ambiguities are inevitable. A lot of the task of writing prescriptive cyber security standards can be characterized as trying to nail Jell-O ™ to the wall. IMHO, this is one reason why prescriptive cyber standards don’t work, whether for banking, manufacturing or electric power.

To get back to the auditor’s email, how does he characterize “subjective” audits? He says they come into play when you have objectives-based requirements. I agree with the auditor that these are the two categories into which NERC CIP requirements fall: prescriptive (think CIP-007 R2) and objectives-based (think CIP-007 R3). If a requirement simply mandates that you achieve a particular objective, but doesn’t specify how you are to do that, it is objectives-based. If it simply lists a set of steps that you must take in order to be compliant, then it’s a prescriptive requirement.

If you’re wondering where plan-based requirements fit in with this schema, they are one type of objectives-based requirement. CIP-010 R4 is a plan-based requirement because it mandates the entity develop a plan to obtain the objective of securing the use of Transient Cyber Assets and Removable Media at Medium and High impact BES assets. However, CIP-007 R3 is also objectives-based (the objective being to mitigate the threat of malicious code), but it isn’t plan-based, since it doesn’t require a plan.[i]

The auditor uses CIP-007 R3 to illustrate his point about subjective audits. He points out that, when auditing this requirement, “It now becomes a question of whether the approach used was reasonable and sound, and whether the entity sufficiently achieved the objective to not merit a PNC.  That makes the audit subjective and it places a greater burden on the ability of the auditor to make such a determination[ii].”

In other words, the auditor is saying I’m using the word “auditable” improperly. He is effectively saying that I’m using it in a sense that only applies to prescriptive requirements, where an objective audit is possible. When an objectives-based[iii] requirement like CIP-007 R3 comes along, a subjective audit is not only possible but necessary. This is a different kind of audit, but it is still an audit. So objectives-based requirements are also auditable.

I agree with him, but only when it comes to non-plan-based objectives-based requirements like CIP-007 R3. When it comes to plan-based requirements like CIP-003 R2 (or at least part of that requirement), as well as CIP-013 R1, CIP-014 R5, CIP-010 R4 and CIP-011 R1[iv], I believe something more is required of the auditor. When the requirement is for a plan, it seems to me that the audit in essence turns into at least partially an objective one. And this is precisely because of the situation I described in the second paragraph of end note i below (which I pointed to as perhaps the main reason why CIP drafting teams have gravitated toward plan-based requirements ever since CIP v5 was drafted): If the plan is a reasonable one and adequately takes account of the threats that need to be addressed in the plan, the entity should presumably receive a passing grade on the audit, even if it turns out that unforeseen circumstances prevented the plan from actually achieving its objective. This isn’t anywhere near as possible for a non-plan-based objectives-based requirement like CIP-007 R3.

But how is the auditor to know that the plan “is a reasonable one and adequately takes account of the threats that need to be addressed in the plan”? As discussed in the previous post, I believe that the requirement should provide some criteria for what should be included in a good plan. The criteria should in principle cover the universe of what needs to be in the plan; they shouldn’t just be some subset of that universe, as in the case of the six items listed in CIP-013 R1.2. But if these criteria are included in the requirement, then IMHO the requirement is auditable. In my previous post, I said that CIP-014 R5 provides such criteria, but CIP-013 R1 doesn’t. This is why I said the former is an auditable requirement, while the latter isn’t.

To sum up my argument so far (I don’t know about you, but if I don’t do this, I’ll have no idea what I said!):

1. In my last post, I said that plan-based requirements that don’t provide criteria for what should be in the plan aren’t auditable.
2. The auditor wrote in to tell me that both prescriptive and objectives-based requirements (and plan-based requirements are a subset of the latter) are auditable, although the former require an “objective” audit and the latter a “subjective” one.
3. I agreed with him in the limited sense that non-plan-based, objectives-based requirements can be subjectively audited (and by the way, I think his choice of the term “subjective” in this case is unfortunate, because a lot of people will tell you that “subjective audit” is an oxymoron like “British cuisine” or “jumbo shrimp”. I would prefer to use the term “effectiveness audit”, taking a clue from something that Lew Folkerth of RF said at a meeting last spring, which I wrote about in this post). However, I pointed out just above that, when the requirement in question is plan-based, it requires a different type of audit, which is more like the “objective” audits that the auditor says – and I agree – are the right way to audit prescriptive requirements. This is why I stand by my assertion that a requirement to develop a plan needs to provide criteria for what should be in the plan, in order to be (objectively) auditable. Q.E.D.

Of course, I still haven’t even gotten to the question of whether it even matters whether a plan-based requirement is auditable or not (no matter what kind of audit it is). That will have to wait for a new post (although I can’t guarantee it will be the next one). I hope you can stand the suspense for a few days.

  

The views and opinions expressed here are my own, and do not reflect those of any organization I work with. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] The auditor at one point suggested to me that there wasn’t any real difference in principle between objectives-based requirements that are plan-based and those that aren’t plan-based. I considered this idea but then rejected it. I think there is a fundamental difference between a requirement that requires the entity to develop a plan to achieve an objective, and one that simply requires the entity to achieve the objective. It seems to me that, when you develop a plan, you take a much more comprehensive look at the objective of the plan than you do if you’re just required to meet the objective. In the latter case, you might just head down one route to achieve the objective, and if it turns out you can achieve it that way, you will naturally persevere until you’ve done that. In the former case, you need to first consider all the ways to achieve the objective and choose the best one – so you’re required to survey the landscape thoroughly before plunging ahead along one path.

But the biggest reason that I think plan-based requirements and standards have become so popular among NERC Standards Drafting Teams  - to the extent that, since CIP v5 was drafted in 2011-2012, there has been no major new CIP standard or requirement that isn’t plan-based, and this trend gives no sign of diminishing – is that they’re less risky. In a non-plan-based, objectives-based requirement like CIP-007 R3, the entity pretty much has to achieve the objective to be compliant; if they don’t achieve the objective (for example, in the case of CIP-007 R3, the entity might not be deemed to have achieved the objective because there was a serious malware-caused incident that happened right before the audit), they will have a tough time explaining to the auditor why they should still be considered compliant. But when the entity is just required to have a plan, if they can convince the auditor that the plan was reasonable and should have worked except for some particular circumstance that couldn’t have been foreseen, they are more likely to experience the auditor’s mercy. This is my guess for why plans are all the rage among CIP SDT’s nowadays.

[ii] He goes on to illustrate what he means: “Here I can use a CIP-007/R3 example.  The entity is required to ‘deploy method(s) to deter, detect, or prevent malicious code.’  As written, this is an objective based requirement.  It has not prescribed the use of any specific software such as anti-virus installed on the host Cyber Assets.  Let’s assume that the entity chooses to “detect” malicious code.  And let’s assume the entity chooses to do so through an anti-malware plugin or an IDS plugin on the firewall (at the EAP).  No other solution has been implemented.  The entity is relying on the firewall plugin functionality to meet the objective.  The theory is that the firewall will see the malicious traffic as it roams around the network and will alert upon detection.  Mission accomplished, right?  Maybe yes, maybe no.  It all depends on the network configuration and how traffic moves around the network.  If the network is supported by Layer 2 or Layer 3 switches and two Cyber Assets within the ESP can communicate between themselves with the traffic never leaving the switch, then the firewall might not see the malicious traffic.  The solution, while looking good on paper, falls short of meeting the objective and a PNC will likely ensue.  There was nothing in the requirement that mandated a firewall solution and there was nothing in the requirement that mandated all inter-ESP traffic go through the firewall.  The requirement is not prescriptive.  The auditor will have to not only look at the firewall to see that the plugins are present and properly configured, but also that the network design compels all traffic to hit the firewall.  In other words, the auditor must evaluate what the entity has done and determine if the entity solution is sufficiently effective.  The same concepts apply to CIP-013.  There are some well-known risks that any reasonable person will anticipate and plan for.  The auditor will expect to see those risks sufficiently addressed (not necessarily every single one, but predominately).  The auditor will also expect to see a process in the plan for evaluating new risks as they are identified.  This is where the subjective opinion of the audit team comes into play.  Did the entity do enough to warrant a No Finding or, maybe, an AOC or Recommendation?  It is not black and white, but certainly not so vague and unmeasurable that a PNC cannot ever be found.”

[iii] You may notice that the word “objective” has two different uses in this post. First, I’m discussing objectives-based requirements like CIP-007 R3 and all of the plan-based requirements. But the auditor is also using it as one of two types of audits. He asserts that prescriptive requirements can be audited objectively, while objectives-based requirements (including plan-based ones) can only be audited subjectively. I hate to see this potential for confusion, but I don’t want to mess around with the terms at this point.

[iv] CIP-011 R1 doesn’t require a “plan” but it does require a “program”. I believe that this amounts to virtually the same thing, although I’ll admit I’m still open to persuasion that it isn’t. It is definitely an objectives-based requirement, in any case.