In my last post,
I raised the question whether (or rather, how much, since this isn’t a binary
question) CIP-013 was auditable. In the previous post,
I had discussed this same question with respect to CIP-014, and decided it was
for the most part auditable. However, for CIP-013 I decided that only one
requirement part, R1.2, was fully auditable; the other requirements and
requirement parts in the standard are either un-auditable or undetermined.
However, at
the very end of my last post, I also pointed out that I like CIP-013 a lot; in
fact, I think it’s as close to what I would call the ideal standard as any of
the CIP standards are. Why would I say this? Is it due to early-onset senility,
or is there some other explanation? I’m perhaps not the best person to decide
this question, but I can assure you that I believe there’s another explanation.
I hinted at
that explanation in the last paragraph of that post: I no longer think that
auditability is the most important criterion by which a CIP standard should be
judged, or at least one that is based on a plan (and I’ve been discussing
plan-based standards and requirements in the previous three posts). My last
sentence of the post said I didn’t want to make that post any longer, so I
would discuss this idea in my next post. This is that post.
I was all
set to continue with the idea in this post, when an auditor – who has
contributed to many of my posts over the years – wrote in with his comments on
the last post. As sometimes happens, we had a long email exchange, with one of
us inserting comments in one color, then the other inserting his in another
color (in the past we’ve once or twice gone beyond all the primary colors and
started looking through paint catalogs to find more colors; we didn’t get that
far this time). I will just summarize what I think were the most important
points he made.
The main
point of my last post was that, in order to be auditable, a plan-based
requirement would need to include criteria for what should be in the plan. For
example, I pointed out that CIP-014 R5 specifies four general areas that must
be addressed in the physical security plan, which made it auditable in my
opinion. But CIP-013 R1 just says the entity should develop a supply chain
cyber security risk management plan. It provides no criteria for what should be
in the plan, other than six specific items listed in R1.2. This was why I said
this requirement was “mostly un-auditable”.
Moreover, as
I’ve pointed out at least a couple times,
the six items in R1.2 are only there because FERC specifically ordered they be
addressed in the new standard. R1.1 makes clear that the plan should address
risks in three general areas, but the six items all have to do with only one of
those three areas (vendor risk) – and they don’t come anywhere near addressing
all the risks in that one area.
The auditor
asserted that I was wrong to say that a plan-based standard needed criteria in the requirement in order to be
auditable. He made the distinction between objective and subjective audits.
Objective audits are only possible with prescriptive requirements. Since these
requirements (ideally) specify particular actions that an entity should take,
they can be easily audited: a) Did the entity take the actions? If so, they
complied with the requirement; b) Did the entity not take the actions? Then
they didn’t comply.
However,
this is only realistic when the requirement is unambiguous, in both a) what it
applies to and b) the steps that need to be taken. Unfortunately, there are a
number of CIP requirements where there is ambiguity in one or both of these areas.
I don’t particularly blame this on the teams that drafted the requirements, or
even on the NERC entities that approved them. Rather, it’s simply due to the
nature of the cyber security domain – as opposed to the deterministic,
physics-based domain of the other NERC standards (called the “693” standards
and also Operations and Planning) – that ambiguities are inevitable. A lot of
the task of writing prescriptive cyber security standards can be characterized
as trying to nail Jell-O ™ to the wall. IMHO, this is one reason why
prescriptive cyber standards don’t work, whether for banking, manufacturing or
electric power.
To get back
to the auditor’s email, how does he characterize “subjective” audits? He says
they come into play when you have objectives-based requirements. I agree with
the auditor that these are the two categories into which NERC CIP requirements
fall: prescriptive (think CIP-007 R2) and objectives-based (think CIP-007 R3). If
a requirement simply mandates that you achieve a particular objective, but
doesn’t specify how you are to do that, it is objectives-based. If it simply
lists a set of steps that you must take in order to be compliant, then it’s a prescriptive
requirement.
If you’re
wondering where plan-based requirements fit in with this schema, they are one
type of objectives-based requirement. CIP-010 R4 is a plan-based requirement
because it mandates the entity develop a plan to obtain the objective of securing
the use of Transient Cyber Assets and Removable Media at Medium and High impact
BES assets. However, CIP-007 R3 is also objectives-based (the objective being
to mitigate the threat of malicious code), but it isn’t plan-based, since it
doesn’t require a plan.[i]
The auditor
uses CIP-007 R3 to illustrate his point about subjective audits. He points out
that, when auditing this requirement, “It now becomes a question of whether the
approach used was reasonable and sound, and whether the entity sufficiently
achieved the objective to not merit a PNC. That makes the audit
subjective and it places a greater burden on the ability of the auditor to make
such a determination[ii].”
In other
words, the auditor is saying I’m using the word “auditable” improperly. He is
effectively saying that I’m using it in a sense that only applies to prescriptive
requirements, where an objective audit is possible. When an objectives-based[iii]
requirement like CIP-007 R3 comes along, a subjective audit is not only
possible but necessary. This is a different kind of audit, but it is still an
audit. So objectives-based requirements are also auditable.
I agree with
him, but only when it comes to non-plan-based objectives-based requirements
like CIP-007 R3. When it comes to plan-based requirements like CIP-003 R2 (or
at least part of that requirement), as well as CIP-013 R1, CIP-014 R5, CIP-010
R4 and CIP-011 R1[iv],
I believe something more is required of the auditor. When the requirement is
for a plan, it seems to me that the audit in essence turns into at least
partially an objective one. And this is precisely because of the situation I
described in the second paragraph of end note i below (which I pointed to as
perhaps the main reason why CIP drafting teams have gravitated toward
plan-based requirements ever since CIP v5 was drafted): If the plan is a
reasonable one and adequately takes account of the threats that need to be
addressed in the plan, the entity should presumably receive a passing grade on
the audit, even if it turns out that unforeseen circumstances prevented the
plan from actually achieving its objective. This isn’t anywhere near as
possible for a non-plan-based objectives-based requirement like CIP-007 R3.
But how is
the auditor to know that the plan “is a reasonable one and adequately takes
account of the threats that need to be addressed in the plan”? As discussed in
the previous post, I believe that the requirement should provide some criteria
for what should be included in a good plan. The criteria should in principle
cover the universe of what needs to be in the plan; they shouldn’t just be some
subset of that universe, as in the case of the six items listed in CIP-013
R1.2. But if these criteria are included in the requirement, then IMHO the
requirement is auditable. In my previous post, I said that CIP-014 R5 provides
such criteria, but CIP-013 R1 doesn’t. This is why I said the former is an
auditable requirement, while the latter isn’t.
To sum up my
argument so far (I don’t know about you, but if I don’t do this, I’ll have no
idea what I said!):
- In my last post, I said that plan-based requirements that
don’t provide criteria for what should be in the plan aren’t auditable.
- The auditor wrote in to tell me that both prescriptive and
objectives-based requirements (and plan-based requirements are a subset of
the latter) are auditable, although the former require an “objective”
audit and the latter a “subjective” one.
- I agreed with him in the limited sense that
non-plan-based, objectives-based requirements can be subjectively audited
(and by the way, I think his choice of the term “subjective” in this case
is unfortunate, because a lot of people will tell you that “subjective
audit” is an oxymoron like “British cuisine” or “jumbo shrimp”. I would
prefer to use the term “effectiveness audit”, taking a clue from something
that Lew Folkerth of RF said at a meeting last spring, which I wrote about
in this
post). However, I pointed out just above that, when the requirement in
question is plan-based, it requires a different type of audit, which is
more like the “objective” audits that the auditor says – and I agree – are
the right way to audit prescriptive requirements. This is why I stand by
my assertion that a requirement to develop a plan needs to provide
criteria for what should be in the plan, in order to be (objectively) auditable.
Q.E.D.
Of course, I
still haven’t even gotten to the question of whether it even matters whether a
plan-based requirement is auditable or not (no matter what kind of audit it
is). That will have to wait for a new post (although I can’t guarantee it will
be the next one). I hope you can stand the suspense for a few days.
The views and opinions expressed here are my own, and do
not reflect those of any organization I work with. If you would like to comment
on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.
[i]
The auditor at one point suggested to me that there wasn’t any real difference
in principle between objectives-based requirements that are plan-based and
those that aren’t plan-based. I considered this idea but then rejected it. I
think there is a fundamental difference between a requirement that requires the
entity to develop a plan to achieve an objective, and one that simply requires
the entity to achieve the objective. It seems to me that, when you develop a
plan, you take a much more comprehensive look at the objective of the plan than
you do if you’re just required to meet the objective. In the latter case, you
might just head down one route to achieve the objective, and if it turns out
you can achieve it that way, you will naturally persevere until you’ve done
that. In the former case, you need to first consider all the ways to achieve
the objective and choose the best one – so you’re required to survey the
landscape thoroughly before plunging ahead along one path.
But the biggest reason that I think plan-based
requirements and standards have become so popular among NERC Standards Drafting
Teams - to the extent that, since CIP v5
was drafted in 2011-2012, there has been no major new CIP standard or
requirement that isn’t plan-based,
and this trend gives no sign of diminishing – is that they’re less risky. In a
non-plan-based, objectives-based requirement like CIP-007 R3, the entity pretty
much has to achieve the objective to be compliant; if they don’t achieve the
objective (for example, in the case of CIP-007 R3, the entity might not be
deemed to have achieved the objective because there was a serious malware-caused
incident that happened right before the audit), they will have a tough time
explaining to the auditor why they should still be considered compliant. But
when the entity is just required to have a plan, if they can convince the
auditor that the plan was reasonable and should have worked except for some
particular circumstance that couldn’t have been foreseen, they are more likely
to experience the auditor’s mercy. This is my guess for why plans are all the
rage among CIP SDT’s nowadays.
[ii]
He goes on to illustrate what
he means: “Here I can use a CIP-007/R3 example.
The entity is required to ‘deploy method(s) to deter, detect, or prevent
malicious code.’ As written, this is an
objective based requirement. It has not
prescribed the use of any specific software such as anti-virus installed on the
host Cyber Assets. Let’s assume that the
entity chooses to “detect” malicious code.
And let’s assume the entity chooses to do so through an anti-malware
plugin or an IDS plugin on the firewall (at the EAP). No other solution has been implemented. The entity is relying on the firewall plugin
functionality to meet the objective. The
theory is that the firewall will see the malicious traffic as it roams around
the network and will alert upon detection.
Mission accomplished, right?
Maybe yes, maybe no. It all
depends on the network configuration and how traffic moves around the
network. If the network is supported by
Layer 2 or Layer 3 switches and two Cyber Assets within the ESP can communicate
between themselves with the traffic never leaving the switch, then the firewall
might not see the malicious traffic. The
solution, while looking good on paper, falls short of meeting the objective and
a PNC will likely ensue. There was
nothing in the requirement that mandated a firewall solution and there was
nothing in the requirement that mandated all inter-ESP traffic go through the
firewall. The requirement is not
prescriptive. The auditor will have to
not only look at the firewall to see that the plugins are present and properly
configured, but also that the network design compels all traffic to hit the
firewall. In other words, the auditor
must evaluate what the entity has done and determine if the entity solution is
sufficiently effective. The same
concepts apply to CIP-013. There are
some well-known risks that any reasonable person will anticipate and plan
for. The auditor will expect to see
those risks sufficiently addressed (not necessarily every single one, but
predominately). The auditor will also
expect to see a process in the plan for evaluating new risks as they are
identified. This is where the subjective
opinion of the audit team comes into play.
Did the entity do enough to warrant a No Finding or, maybe, an AOC or
Recommendation? It is not black and
white, but certainly not so vague and unmeasurable that a PNC cannot ever be
found.”
[iii]
You may notice that the word “objective” has two different uses in this post.
First, I’m discussing objectives-based requirements like CIP-007 R3 and all of
the plan-based requirements. But the auditor is also using it as one of two
types of audits. He asserts that prescriptive requirements can be audited
objectively, while objectives-based requirements (including plan-based ones)
can only be audited subjectively. I hate to see this potential for confusion,
but I don’t want to mess around with the terms at this point.
[iv]
CIP-011 R1 doesn’t require a “plan” but it does require a “program”. I believe
that this amounts to virtually the same thing, although I’ll admit I’m still
open to persuasion that it isn’t. It is definitely an objectives-based
requirement, in any case.
No comments:
Post a Comment