How do you Audit a “Plan”?

My previous post tried to explain why almost every new CIP standard or requirement since CIP v5 has been based on a “plan”. Clearly, these standards/requirements are the wave of the future with NERC CIP. In this post, I will discuss how – or rather, if – such standards and requirements can be audited by NERC.

For clarity, I’m going to start with a simple hypothetical example. Let’s say we develop a new standard for cyber security of programmable thermostats. We want it to be objectives-based (non-prescriptive) and we’re purists, so we just write two requirements:

R1: Develop a plan for securing your house’s programmable thermostat.

R2: Implement that plan.

Now suppose that, several years later, an auditor shows up to audit your compliance with this standard. Starting with R1, he or she asks to see your plan, and you show it to them. The first thing they will look for is whether you actually have a plan, as opposed to say a bunch of random words generated to fill up a few pages. If you have the latter, the auditor will most likely issue you a Potential Non-Compliance finding on the spot.

Now the hard part comes in. Suppose the auditor takes a read through your plan and thinks it really isn’t a good one? For example, you may have completely omitted the topic of how you will protect unauthorized individuals (either in the house or coming in through the Internet) from accessing the thermostat. The auditor believes (with good reason) that this is a very serious omission, perhaps a fatal one. But what are the auditor’s options? The requirement just says you have to have a plan, not a good one; it doesn’t say anything about what should be in the plan. In this case, I would say that the auditor has no option but to pass you. They might also offer you some friendly advice (which translates into an Area of Concern in NERC-land) about correcting this omission. They will also recommend that you correct it before the next audit; however, as with NERC Areas of Concern, you can’t be held in violation even if the next audit shows you still haven’t corrected this omission.

The same consideration will hold true when the auditor considers R2. If you have implemented what you said you would implement in your plan, they have to pass you. It doesn’t matter that your thermostat may still be riddled with security holes; as long as you have done what you said you would do, you have to pass. I think you’ll agree with me that the concept of “audit”, in the case of a standard like the one I just described, is close to meaningless; the standard provides no criteria by which to judge a good from a bad plan. Whoever drafted this standard may have felt that there shouldn’t be any constraints placed on your plan; but in doing this, they made the standard effectively un-auditable.

So let’s say a new drafting team drafts version 2 of this standard. As part of their preparation, they ask auditors about their experience auditing version 1. The auditors point out that they really can’t audit it in any meaningful sense; the requirement must list some sort of criteria that need to be met by a good plan (not just a separate guidance document, since those are never binding on the auditor or the entity, even if they happen to be called Implementation Guidance).

So now version 2 of the standard reads:

R1: Develop a plan for securing your house’s programmable thermostat. The plan must include:

·         Controls on physical access to the programming interface; and

·         Controls on remote access to the programming interface; and

·         Etc.

R2: Implement that plan.

When the auditor comes back the next time, he or she is much more likely to feel good about their job. They will no longer be providing a meaningless “pass” to everybody they audit, who has even the most minimal plan. Instead, they will comb through the plan to determine whether it contains each of the bulleted items listed in R1. If it is missing one or more of these, the auditor will probably issue a PNC. They may also order a mitigation plan (enforceable this time), so that you will have to include these in your plan within say one year.

Clearly, there need to be criteria for what should be in the plan. Even more importantly, these criteria need to be listed in the requirement or standard itself, not in a separate guidance document. This is why CIP-010 R4 Attachment 1 (discussed in the previous post) goes into so much detail on what should be included in the plan for TCAs and Removable Media. Each of the items listed needs to be in the plan, or the entity risks a PNC.

But there’s a limit to this. Obviously, general criteria like “Authorize use of Removable Media by both user and location” don’t allow for any nuance in how the entity interprets them. Suppose the entity decides that anybody in the entire IT department should have authorization to use removable media in the Control Center, and the auditor thinks that is far too broad. Just as was the case in version 1, the auditor will have to swallow his or her objection and give the entity a pass, while most likely issuing an Area of Concern. There’s nothing in “Authorize use of Removable Media by both user and location” that would rule out what the entity has done.

For the next version of the standard, the auditor might provide advice to the drafting team to the effect that it would be better to perhaps add the words “based on need”, or something like that. And maybe another auditor would decide the next version should really include a provision for removing access after the person has left the company, perhaps within 24 hours or even instantaneously. As a result, this standard could really become a set of backdoor prescriptive requirements. There always needs to be a balance struck between auditability and prescriptivity.

But what if these auditors decide they don’t want to wait for the SDT to change the requirement, and they simply start acting as if these criteria were in it now? What if an auditor starts issuing PNCs for entities that don’t within 24 hours revoke use of Removable Media by users who have changed jobs or left the company? Clearly, at that point they have overstepped their authority. They shouldn’t be allowed to do that.

How does CIP-014 fit into this?

A recent post described the experiences of two entities with CIP-014 audits. How do those experiences relate to the imaginary standard I’ve just described? Both of these entities fell afoul of CIP-014 auditors by making a similar mistake: They noted that CIP-014 R1, R4 and R5 are completely focused on threats to, and vulnerabilities of, the substation as a whole. These requirements never mention threats to individual elements like transformers.

Both of these entities built their threat and vulnerability assessment (required by R4) on the assumption that the only threats and vulnerabilities it needed to consider were those to the whole substation. And the one entity that had developed a physical security plan (required by R5) had built that on the same assumption. However, both of them were reprimanded by their auditors for not considering threats to their individual transformers. Plus they both were initially threatened with PNCs for not doing this. In one of the cases, the entity did receive a PNC.

As I pointed out in the post, I think the auditors were right in saying that the entities should consider threats to individual transformers. But this is clearly not in any of the requirements. If the auditors want to admonish the entities, they should issue an Area of Concern and ask the entity to fix the omission in their plan. I would be very surprised if the PNC that was issued survives to become an actual Violation.

Aside from this, how auditable is CIP-014? Let’s look at the individual requirements. R1 requires a risk assessment, which establishes whether or not a particular substation is in scope for this requirement. The criterion for deciding this is relatively clear: The substation has to be such that “if rendered inoperable or damaged could result in instability, uncontrolled separation, or Cascading within an Interconnection.” I believe the risk assessment is auditable.

I’ll skip over R2 and R3, both of which seem completely auditable to me. Let’s look at R4. As I’ve just said, R4 requires an assessment of threats and vulnerabilities to the substation. As I already pointed out, the requirement just talks about threats and vulnerabilities to the whole substation, so any attempt by an auditor to issue a PNC to an entity for not including transformers (or other elements like buses) in their assessment should be met with a firm “Please show me where in the requirement it says this.”

But setting this aside, how auditable is this requirement? Fortunately, the requirement lists three criteria that must be considered in the assessment: the unique characteristics of the substation, history of attacks on similar facilities, and intelligence received from the E-ISAC and other organizations. So the assessment can be audited to make sure these three criteria were considered. All three are fairly broad, so a lot of considerations would fall into them. But any other criteria that the auditor might like to see in the assessment would have to be dealt with as an Area of Concern.

R5 requires the entity to develop and implement a physical security plan to mitigate the threats and vulnerabilities. There are four specific (yet comprehensive) criteria for what needs to be in the plan, including a timeline for completing it. This seems to me to make the plan auditable.

How about implementation of the plan? Is that auditable? The requirement says that the plan should be “executed according to the timeline specified in the physical security plan.” In my opinion, this makes implementation auditable, although as I noted in this post, one region’s auditors were trying to go beyond that and make up their own criteria to judge how well the entity was making progress on implementing its plan. Of course, it’s certainly their prerogative to look at other criteria than simply the timeline that the entity listed in their plan[i], but anything beyond that timeline should strictly be a matter for an Area of Concern.

Overall, I think CIP-014 is quite auditable, in the sense that there are specific criteria that make it possible for auditors to make specific determinations of whether the entity has complied with the requirements. Were this one of the prescriptive CIP standards, that’s all that would matter. However, with objectives-based standards like CIP-014 and CIP-013 (and CIP-012, now in development), I think there is a bigger consideration than auditability: it’s partnership. How can NERC and the regions partner with the entities to develop and implement good plans, rather than be locked in an eternal hands-off stance that benefits nobody, especially not those of us who rely on the electric grid for our daily existence? I will return to this theme in a later post.

My next post will discuss whether CIP-013 is auditable. I suspect that answer may be different from what it was for CIP-014.

Note from Tom: The follow-on to this post is here. In starting to write the follow-on, I realized my discussion of CIP-014 in this post was incomplete, so I continued it in that post.

The views and opinions expressed here are my own, and do not reflect those of any organization I work with. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] One lesson from this is that, if an entity is falling behind the implementation schedule listed in its physical security plan, it should amend the plan to reflect the new schedule, and document why this change was made.