Friday, November 10, 2017

The Second Lesson from CIP-014


Two weeks ago, I wrote a post that discussed the first of two lessons that can be learned from NERC’s experience so far with CIP-014, the physical security standard that applies to certain key substations. I think these lessons should have implications for CIP-013, since both of these standards are objectives-based (non-prescriptive) and risk-based. After a follow-up post to the first one, this post discusses the second lesson.

I learned both of these lessons from a friend who is in charge of CIP physical security compliance for a large utility (I need to point out that he wasn’t trying to teach me lessons! I learned them from what he told me). He described a meeting on CIP-014 between some auditors from his NERC region and some of the entities in the region. In the meeting, the auditors were asked how they would audit the entity’s implementation of the physical security plan for important substations. They asked this question because CIP-014 R5 requires the entity to develop and implement a physical security plan.

My friend was quite surprised when he heard the auditors’ answer. They said they were going to look at various measures of how far along the entity was in implementing the plan at the time of the audit. They would look at (among other things) the number of physical security devices put into place but not connected, the number connected but not activated, and the number fully connected and activated. They would then presumably compare these results with some measures (and how they would come up with those is anybody’s guess) of where the entity ought to be at this stage of the implementation process. Also presumably, they might issue a Potential Non-Compliance finding if the entity was too far behind where they thought they should be at this stage.

My friend – and others at the meeting – rightly raised the question what all of this has to do with compliance with CIP-014. As the auditors described it, they weren’t going to audit the entities on how well they’ve fulfilled the requirements – which, to be succinct, require the entity to develop and implement a physical security plan. To properly audit the requirements, the auditors would have to determine whether or not the entity’s plan is a good one, and whether they properly implemented it.

And why weren’t the auditors going to audit compliance with the actual requirements? I’m assuming this is because there aren’t any criteria in the requirements themselves for what would be a good plan vs. a bad one, or a good implementation vs. a bad one. Of course, CIP-013 is the same way: The entity is only required to develop and implement a supply chain cyber security risk management plan. The requirements themselves say nothing about what that plan should contain (except for six particular items listed in R1.2, which were ordered by FERC. These have to be included in the plan, but they in no way constitute the whole plan).

Instead, the auditors were saying that the entity would be judged on some artificial constructs that can definitely be measured, but have no relation to the question whether the plan is a good one or whether or not the entity is doing a good job of implementing it – which is what the standard requires. Naturally, my friend found this idea kind of problematic!

This relates to a problem I brought up (at great length, I’ll warn you) in this post from about a year ago: that drafting teams are often under pressure to put measurable requirements in the standards they’re developing, even when the quantities being measured aren’t really very germane to the actual objective of the requirement. And, since the CIP-014 drafting team seems to have resisted this pressure, evidently the auditors at the meeting my friend attended were pushing this one step further. I’ll trace out what I think their logic was:

  1. The requirements in CIP-014 don’t provide any sort of measurable criteria to audit.
  2. The auditors could try to figure out how to effectively audit the requirements of CIP-014. But this would require them to use judgment (how else can you determine whether the entity has developed a good plan, or whether they’ve done a good job implementing it?), and use of auditor judgment on this scale isn’t countenanced in CMEP or GAGAS[i].
  3. Therefore, they developed some measurable criteria to audit, even though these have nothing to do with the central purpose of CIP-014: developing and implementing a good physical security plan for key substations.

I want to point out that I have no idea whether what the auditors said they would do at this meeting will ever see the light of day, in the region in question or anywhere else. This is because: a) I’m getting this whole story second hand; b) I have no idea whether the auditors dropped these ideas after that meeting; and c) I don’t know whether these ideas were shared with other regions or not. And you may notice I’ve gone out of my way not to clue you in on which region was involved here.

But I’m not criticizing the auditors anyway. They are between the proverbial rock and a hard place. On the one hand, they’re faced with a standard that presents no measurable criteria to audit, meaning an application of judgment is required. On the other hand, the documents that govern what they’re supposed to do – CMEP and GAGAS, as well as the NERC Rules of Procedure – adamantly reject the idea of the auditor’s exercising judgment in a situation like this one.

In my opinion, the only thing for the auditors to do in this case (and the only thing that ultimately will be done, I’m sure) would be to review the physical security plans and use their own judgment to determine whether they’re good or bad. Of course, if they don’t like the plan, they can’t issue a PNC unless the entity had simply not developed a serious plan at all. But they can at least issue an Area of Concern that points out deficiencies in the plan or its implementation; the entity would then be well advised to address the AoC, even though they couldn’t strictly speaking be found in violation if they didn’t do that.

But, since GAGAS and CMEP absolutely prohibit an “audit” that consists of nothing more than an application of the auditor’s educated judgment, the auditors – and NERC themselves – would have to acknowledge that CIP-014 R5 (the requirement to develop and implement a plan) isn’t really auditable at all, except in the case of gross disregard for the requirement.

What do I think will happen with CIP-014 enforcement? I am pretty sure that, when it comes to the actual audit, the auditors won’t invent criteria like the ones described above. They will do what the auditors for CIP-013 will do, as I described in this post: They will use their own judgment, guided hopefully by an ample set of guidance documents from NERC as well as other parties. If they find deficiencies in either the physical security plan or its implementation, they will issue an Area of Concern. This isn’t like a Potential Non-Compliance finding, which can ultimately lead to a violation finding. The entity will need to remediate this AoC, but in the end they can’t be held in violation for not doing so.

I would write a lot more on this idea, except I already wrote it all in my last post; so please read (or re-read) that. I’ll just point out that entities subject to CIP-014, as they approach the time when audits start, should be on their guard against attempts by the regions to enforce spurious audit criteria (I somewhat doubt this will happen, but ya’ never know!). And now that I think of it, the same goes for entities subject to CIP-013!


The views and opinions expressed here are my own, and do not reflect those of any organization I work with. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

[i] For a discussion of these two acronyms, see my previous post.

No comments:

Post a Comment