Two weeks
ago, I wrote a post
that discussed the first of two lessons that can be learned from NERC’s
experience so far with CIP-014, the physical security standard that applies to
certain key substations. I think these lessons should have implications for
CIP-013, since both of these standards are objectives-based (non-prescriptive)
and risk-based. After a follow-up post
to the first one, this post discusses the second lesson.
I learned
both of these lessons from a friend who is in charge of CIP physical security
compliance for a large utility (I need to point out that he wasn’t trying to
teach me lessons! I learned them from what he told me). He described a meeting on
CIP-014 between some auditors from his NERC region and some of the entities in
the region. In the meeting, the auditors were asked how they would audit the
entity’s implementation of the physical security plan for important
substations. They asked this question because CIP-014 R5 requires the entity to
develop and implement a physical security plan.
My friend
was quite surprised when he heard the auditors’ answer. They said they were
going to look at various measures of how far along the entity was in
implementing the plan at the time of the audit. They would look at (among other
things) the number of physical security devices put into place but not
connected, the number connected but not activated, and the number fully
connected and activated. They would then presumably compare these results with
some measures (and how they would come up with those is anybody’s guess) of
where the entity ought to be at this stage of the implementation process. Also
presumably, they might issue a Potential Non-Compliance finding if the entity
was too far behind where they thought they should be at this stage.
My friend –
and others at the meeting – rightly raised the question what all of this has to
do with compliance with CIP-014. As the auditors described it, they weren’t
going to audit the entities on how well they’ve fulfilled the requirements – which,
to be succinct, require the entity to develop and implement a physical security
plan. To properly audit the requirements, the auditors would have to determine
whether or not the entity’s plan is a good one, and whether they properly
implemented it.
And why
weren’t the auditors going to audit compliance with the actual requirements? I’m
assuming this is because there aren’t any criteria in the requirements
themselves for what would be a good plan vs. a bad one, or a good implementation
vs. a bad one. Of course, CIP-013 is the same way: The entity is only required
to develop and implement a supply chain cyber security risk management plan.
The requirements themselves say nothing about what that plan should contain (except
for six particular items listed in R1.2, which were ordered by FERC. These have
to be included in the plan, but they in no way constitute the whole plan).
Instead, the
auditors were saying that the entity would be judged on some artificial
constructs that can definitely be measured, but have no relation to the
question whether the plan is a good one or whether or not the entity is doing a
good job of implementing it – which is what the standard requires. Naturally,
my friend found this idea kind of problematic!
This relates
to a problem I brought up (at great length, I’ll warn you) in this
post from about a year ago: that drafting teams are often under pressure to put
measurable requirements in the standards they’re developing, even when the
quantities being measured aren’t really very germane to the actual objective of
the requirement. And, since the CIP-014 drafting team seems to have resisted
this pressure, evidently the auditors at the meeting my friend attended were
pushing this one step further. I’ll trace out what I think their logic was:
- The requirements in CIP-014 don’t provide any sort of
measurable criteria to audit.
- The auditors could
try to figure out how to effectively audit the requirements of CIP-014.
But this would require them to use judgment (how else can you determine
whether the entity has developed a good plan, or whether they’ve done a
good job implementing it?), and use of auditor judgment on this scale
isn’t countenanced in CMEP or GAGAS[i].
- Therefore, they developed some measurable criteria to
audit, even though these have nothing to do with the central purpose of
CIP-014: developing and implementing a good physical security plan for key
substations.
I want to
point out that I have no idea whether what the auditors said they would do at
this meeting will ever see the light of day, in the region in question or
anywhere else. This is because: a) I’m getting this whole story second hand; b)
I have no idea whether the auditors dropped these ideas after that meeting; and
c) I don’t know whether these ideas were shared with other regions or not. And
you may notice I’ve gone out of my way not to clue you in on which region was
involved here.
But I’m not
criticizing the auditors anyway. They are between the proverbial rock and a
hard place. On the one hand, they’re faced with a standard that presents no
measurable criteria to audit, meaning an application of judgment is required.
On the other hand, the documents that govern what they’re supposed to do – CMEP
and GAGAS, as well as the NERC Rules of Procedure – adamantly reject the idea
of the auditor’s exercising judgment in a situation like this one.
In my
opinion, the only thing for the auditors to do in this case (and the only thing
that ultimately will be done, I’m
sure) would be to review the physical security plans and use their own judgment
to determine whether they’re good or bad. Of course, if they don’t like the
plan, they can’t issue a PNC unless the entity had simply not developed a
serious plan at all. But they can at least issue an Area of Concern that points
out deficiencies in the plan or its implementation; the entity would then be
well advised to address the AoC, even though they couldn’t strictly speaking be
found in violation if they didn’t do that.
But, since
GAGAS and CMEP absolutely prohibit an “audit” that consists of nothing more
than an application of the auditor’s educated judgment, the auditors – and NERC
themselves – would have to acknowledge that CIP-014 R5 (the requirement to
develop and implement a plan) isn’t really auditable at all, except in the case
of gross disregard for the requirement.
What do I
think will happen with CIP-014 enforcement? I am pretty sure that, when it
comes to the actual audit, the auditors won’t invent criteria like the ones
described above. They will do what the auditors for CIP-013 will do, as I
described in this
post: They will use their own judgment, guided hopefully by an ample set of
guidance documents from NERC as well as other parties. If they find
deficiencies in either the physical security plan or its implementation, they
will issue an Area of Concern. This isn’t like a Potential Non-Compliance
finding, which can ultimately lead to a violation finding. The entity will need
to remediate this AoC, but in the end they can’t be held in violation for not
doing so.
I would write
a lot more on this idea, except I already wrote it all in my last
post; so please read (or re-read) that. I’ll just point out that entities
subject to CIP-014, as they approach the time when audits start, should be on
their guard against attempts by the regions to enforce spurious audit criteria
(I somewhat doubt this will happen, but ya’ never know!). And now that I think
of it, the same goes for entities subject to CIP-013!
The views and opinions expressed here are my own, and do
not reflect those of any organization I work with. If you would like to comment
on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.
[i]
For a discussion of these two acronyms, see my previous post.
No comments:
Post a Comment