Thursday, December 14, 2017

An Auditor Weighs in on Auditing Plan-based Requirements


In my last post, I raised the question whether (or rather, how much, since this isn’t a binary question) CIP-013 was auditable. In the previous post, I had discussed this same question with respect to CIP-014, and decided it was for the most part auditable. However, for CIP-013 I decided that only one requirement part, R1.2, was fully auditable; the other requirements and requirement parts in the standard are either un-auditable or undetermined.

However, at the very end of my last post, I also pointed out that I like CIP-013 a lot; in fact, I think it’s as close to what I would call the ideal standard as any of the CIP standards are. Why would I say this? Is it due to early-onset senility, or is there some other explanation? I’m perhaps not the best person to decide this question, but I can assure you that I believe there’s another explanation.

I hinted at that explanation in the last paragraph of that post: I no longer think that auditability is the most important criterion by which a CIP standard should be judged, or at least one that is based on a plan (and I’ve been discussing plan-based standards and requirements in the previous three posts). My last sentence of the post said I didn’t want to make that post any longer, so I would discuss this idea in my next post. This is that post.

I was all set to continue with the idea in this post, when an auditor – who has contributed to many of my posts over the years – wrote in with his comments on the last post. As sometimes happens, we had a long email exchange, with one of us inserting comments in one color, then the other inserting his in another color (in the past we’ve once or twice gone beyond all the primary colors and started looking through paint catalogs to find more colors; we didn’t get that far this time). I will just summarize what I think were the most important points he made.

The main point of my last post was that, in order to be auditable, a plan-based requirement would need to include criteria for what should be in the plan. For example, I pointed out that CIP-014 R5 specifies four general areas that must be addressed in the physical security plan, which made it auditable in my opinion. But CIP-013 R1 just says the entity should develop a supply chain cyber security risk management plan. It provides no criteria for what should be in the plan, other than six specific items listed in R1.2. This was why I said this requirement was “mostly un-auditable”.

Moreover, as I’ve pointed out at least a couple times, the six items in R1.2 are only there because FERC specifically ordered they be addressed in the new standard. R1.1 makes clear that the plan should address risks in three general areas, but the six items all have to do with only one of those three areas (vendor risk) – and they don’t come anywhere near addressing all the risks in that one area.

The auditor asserted that I was wrong to say that a plan-based standard needed criteria in the requirement in order to be auditable. He made the distinction between objective and subjective audits. Objective audits are only possible with prescriptive requirements. Since these requirements (ideally) specify particular actions that an entity should take, they can be easily audited: a) Did the entity take the actions? If so, they complied with the requirement; b) Did the entity not take the actions? Then they didn’t comply.

However, this is only realistic when the requirement is unambiguous, in both a) what it applies to and b) the steps that need to be taken. Unfortunately, there are a number of CIP requirements where there is ambiguity in one or both of these areas. I don’t particularly blame this on the teams that drafted the requirements, or even on the NERC entities that approved them. Rather, it’s simply due to the nature of the cyber security domain – as opposed to the deterministic, physics-based domain of the other NERC standards (called the “693” standards and also Operations and Planning) – that ambiguities are inevitable. A lot of the task of writing prescriptive cyber security standards can be characterized as trying to nail Jell-O ™ to the wall. IMHO, this is one reason why prescriptive cyber standards don’t work, whether for banking, manufacturing or electric power.

To get back to the auditor’s email, how does he characterize “subjective” audits? He says they come into play when you have objectives-based requirements. I agree with the auditor that these are the two categories into which NERC CIP requirements fall: prescriptive (think CIP-007 R2) and objectives-based (think CIP-007 R3). If a requirement simply mandates that you achieve a particular objective, but doesn’t specify how you are to do that, it is objectives-based. If it simply lists a set of steps that you must take in order to be compliant, then it’s a prescriptive requirement.

If you’re wondering where plan-based requirements fit in with this schema, they are one type of objectives-based requirement. CIP-010 R4 is a plan-based requirement because it mandates the entity develop a plan to obtain the objective of securing the use of Transient Cyber Assets and Removable Media at Medium and High impact BES assets. However, CIP-007 R3 is also objectives-based (the objective being to mitigate the threat of malicious code), but it isn’t plan-based, since it doesn’t require a plan.[i]

The auditor uses CIP-007 R3 to illustrate his point about subjective audits. He points out that, when auditing this requirement, “It now becomes a question of whether the approach used was reasonable and sound, and whether the entity sufficiently achieved the objective to not merit a PNC.  That makes the audit subjective and it places a greater burden on the ability of the auditor to make such a determination[ii].”

In other words, the auditor is saying I’m using the word “auditable” improperly. He is effectively saying that I’m using it in a sense that only applies to prescriptive requirements, where an objective audit is possible. When an objectives-based[iii] requirement like CIP-007 R3 comes along, a subjective audit is not only possible but necessary. This is a different kind of audit, but it is still an audit. So objectives-based requirements are also auditable.

I agree with him, but only when it comes to non-plan-based objectives-based requirements like CIP-007 R3. When it comes to plan-based requirements like CIP-003 R2 (or at least part of that requirement), as well as CIP-013 R1, CIP-014 R5, CIP-010 R4 and CIP-011 R1[iv], I believe something more is required of the auditor. When the requirement is for a plan, it seems to me that the audit in essence turns into at least partially an objective one. And this is precisely because of the situation I described in the second paragraph of end note i below (which I pointed to as perhaps the main reason why CIP drafting teams have gravitated toward plan-based requirements ever since CIP v5 was drafted): If the plan is a reasonable one and adequately takes account of the threats that need to be addressed in the plan, the entity should presumably receive a passing grade on the audit, even if it turns out that unforeseen circumstances prevented the plan from actually achieving its objective. This isn’t anywhere near as possible for a non-plan-based objectives-based requirement like CIP-007 R3.

But how is the auditor to know that the plan “is a reasonable one and adequately takes account of the threats that need to be addressed in the plan”? As discussed in the previous post, I believe that the requirement should provide some criteria for what should be included in a good plan. The criteria should in principle cover the universe of what needs to be in the plan; they shouldn’t just be some subset of that universe, as in the case of the six items listed in CIP-013 R1.2. But if these criteria are included in the requirement, then IMHO the requirement is auditable. In my previous post, I said that CIP-014 R5 provides such criteria, but CIP-013 R1 doesn’t. This is why I said the former is an auditable requirement, while the latter isn’t.

To sum up my argument so far (I don’t know about you, but if I don’t do this, I’ll have no idea what I said!):

  1. In my last post, I said that plan-based requirements that don’t provide criteria for what should be in the plan aren’t auditable.
  2. The auditor wrote in to tell me that both prescriptive and objectives-based requirements (and plan-based requirements are a subset of the latter) are auditable, although the former require an “objective” audit and the latter a “subjective” one.
  3. I agreed with him in the limited sense that non-plan-based, objectives-based requirements can be subjectively audited (and by the way, I think his choice of the term “subjective” in this case is unfortunate, because a lot of people will tell you that “subjective audit” is an oxymoron like “British cuisine” or “jumbo shrimp”. I would prefer to use the term “effectiveness audit”, taking a clue from something that Lew Folkerth of RF said at a meeting last spring, which I wrote about in this post). However, I pointed out just above that, when the requirement in question is plan-based, it requires a different type of audit, which is more like the “objective” audits that the auditor says – and I agree – are the right way to audit prescriptive requirements. This is why I stand by my assertion that a requirement to develop a plan needs to provide criteria for what should be in the plan, in order to be (objectively) auditable. Q.E.D.

Of course, I still haven’t even gotten to the question of whether it even matters whether a plan-based requirement is auditable or not (no matter what kind of audit it is). That will have to wait for a new post (although I can’t guarantee it will be the next one). I hope you can stand the suspense for a few days.

  
The views and opinions expressed here are my own, and do not reflect those of any organization I work with. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.


[i] The auditor at one point suggested to me that there wasn’t any real difference in principle between objectives-based requirements that are plan-based and those that aren’t plan-based. I considered this idea but then rejected it. I think there is a fundamental difference between a requirement that requires the entity to develop a plan to achieve an objective, and one that simply requires the entity to achieve the objective. It seems to me that, when you develop a plan, you take a much more comprehensive look at the objective of the plan than you do if you’re just required to meet the objective. In the latter case, you might just head down one route to achieve the objective, and if it turns out you can achieve it that way, you will naturally persevere until you’ve done that. In the former case, you need to first consider all the ways to achieve the objective and choose the best one – so you’re required to survey the landscape thoroughly before plunging ahead along one path.

But the biggest reason that I think plan-based requirements and standards have become so popular among NERC Standards Drafting Teams  - to the extent that, since CIP v5 was drafted in 2011-2012, there has been no major new CIP standard or requirement that isn’t plan-based, and this trend gives no sign of diminishing – is that they’re less risky. In a non-plan-based, objectives-based requirement like CIP-007 R3, the entity pretty much has to achieve the objective to be compliant; if they don’t achieve the objective (for example, in the case of CIP-007 R3, the entity might not be deemed to have achieved the objective because there was a serious malware-caused incident that happened right before the audit), they will have a tough time explaining to the auditor why they should still be considered compliant. But when the entity is just required to have a plan, if they can convince the auditor that the plan was reasonable and should have worked except for some particular circumstance that couldn’t have been foreseen, they are more likely to experience the auditor’s mercy. This is my guess for why plans are all the rage among CIP SDT’s nowadays.

[ii] He goes on to illustrate what he means: “Here I can use a CIP-007/R3 example.  The entity is required to ‘deploy method(s) to deter, detect, or prevent malicious code.’  As written, this is an objective based requirement.  It has not prescribed the use of any specific software such as anti-virus installed on the host Cyber Assets.  Let’s assume that the entity chooses to “detect” malicious code.  And let’s assume the entity chooses to do so through an anti-malware plugin or an IDS plugin on the firewall (at the EAP).  No other solution has been implemented.  The entity is relying on the firewall plugin functionality to meet the objective.  The theory is that the firewall will see the malicious traffic as it roams around the network and will alert upon detection.  Mission accomplished, right?  Maybe yes, maybe no.  It all depends on the network configuration and how traffic moves around the network.  If the network is supported by Layer 2 or Layer 3 switches and two Cyber Assets within the ESP can communicate between themselves with the traffic never leaving the switch, then the firewall might not see the malicious traffic.  The solution, while looking good on paper, falls short of meeting the objective and a PNC will likely ensue.  There was nothing in the requirement that mandated a firewall solution and there was nothing in the requirement that mandated all inter-ESP traffic go through the firewall.  The requirement is not prescriptive.  The auditor will have to not only look at the firewall to see that the plugins are present and properly configured, but also that the network design compels all traffic to hit the firewall.  In other words, the auditor must evaluate what the entity has done and determine if the entity solution is sufficiently effective.  The same concepts apply to CIP-013.  There are some well-known risks that any reasonable person will anticipate and plan for.  The auditor will expect to see those risks sufficiently addressed (not necessarily every single one, but predominately).  The auditor will also expect to see a process in the plan for evaluating new risks as they are identified.  This is where the subjective opinion of the audit team comes into play.  Did the entity do enough to warrant a No Finding or, maybe, an AOC or Recommendation?  It is not black and white, but certainly not so vague and unmeasurable that a PNC cannot ever be found.”

[iii] You may notice that the word “objective” has two different uses in this post. First, I’m discussing objectives-based requirements like CIP-007 R3 and all of the plan-based requirements. But the auditor is also using it as one of two types of audits. He asserts that prescriptive requirements can be audited objectively, while objectives-based requirements (including plan-based ones) can only be audited subjectively. I hate to see this potential for confusion, but I don’t want to mess around with the terms at this point.

[iv] CIP-011 R1 doesn’t require a “plan” but it does require a “program”. I believe that this amounts to virtually the same thing, although I’ll admit I’m still open to persuasion that it isn’t. It is definitely an objectives-based requirement, in any case.

No comments:

Post a Comment