Continuing Dialog with an Auditor on Auditing CIP-013

In my last post, I engaged in a dialog with an auditor regarding how CIP-013 will be audited (although this discussion applies to other plan-based standards and requirements, including CIP-014, CIP-010 R4 and CIP-003 R2). The auditor had emailed me to object to the main point I had made in my previous post: that a requirement that mandates that the entity have a plan needs to have some specific high-level criteria in the requirement itself in order for it to be auditable. CIP-013 R1 only provides six criteria that apply to only one of the three main areas that the supply chain cyber security risk management plan needs to address; therefore, I consider R1 to be mostly un-auditable.

In his email, the auditor said I had overlooked the fact that there are both “objective” and “subjective” audits. A prescriptive requirement like CIP-007 R2 can only be audited “objectively” – that is, by checking the box on each part of the requirement; did the entity do it, or didn’t they? But the auditor said a requirement like CIP-013 R1 needs to be audited subjectively. And what is a subjective audit? It is one that determines whether “the approach used was reasonable and sound, and whether the entity sufficiently achieved the objective to not merit a PNC.  That makes the audit subjective and it places a greater burden on the ability of the auditor to make such a determination.”

I then pointed out (I’m still talking about my last post, of course!) that I didn’t think this was good enough to audit a plan-based requirement; I still believed there need to be some objective criteria in the requirement itself (and in my opinion the criteria listed in Attachment 1 for CIP-010 R4 are the best example of providing criteria in a plan-based requirement).  My reasoning came down to this: “If the plan is a reasonable one and adequately takes account of the threats that need to be addressed in the plan, the entity should presumably receive a passing grade on the audit.” And you can consider the “criteria” in the requirement to be essentially a listing of the threats that need to be addressed in the plan.

Here’s an example of what I’m talking about: CIP-010 R4 is a plan-based requirement. It requires the entity to develop and implement a plan for securing Transient Cyber Assets and Removable Media. But it doesn’t just mandate that the entity develop the plan, it provides a whole set of criteria for what must be addressed in the plan; these are listed in Attachment 1 of CIP-010-2.

There are three sections to Attachment 1 (TCA’s managed by the Responsible Entity, etc), each of which contains between two and five headings. The headings show domains that must be addressed within that section: “Software Vulnerability Mitigation”, “Malicious Code Mitigation”, etc. These headings can be thought of as threats whose risk needs to be mitigated: the threat of vulnerabilities in software installed on TCA’s, the threat of malicious code carried on removable media, etc. Under each heading is a list of processes or technologies that can be used to mitigate those threats; the entity must use “one or a combination” of these to mitigate the risk posed by the threat indicated in the heading.[i]

To sum up what I’ve just said, I think that a plan-based requirement needs to include a list of threats that must be addressed in the plan (which I called “criteria” until just now). If the requirement doesn’t include such a list, then I don’t think the auditor can make a decision on whether the plan is adequate or not; in other words, I don’t think the requirement is auditable. CIP-013 R1 only lists six items that need to be included in the plan. They only partially address one of the three overall threat areas that R1.1 requires to be addressed: procuring vendor equipment and software; installing vendor equipment and software; and transitions between vendors. Therefore, I believe CIP-013 R1 (and by extension R2) is only partially auditable. On the other hand, CIP-010 R4 Attachment 1 does a good job of listing threats that need to be addressed, as well as giving the entity options for how they mitigate those threats; I believe that this requirement is auditable.

After I put up this post, the auditor emailed back to me and said the following[ii]: “I am arguing that there do(es) not have to be defined criteria in the Requirement, specifying what must be in a “plan” in order to render the requirement, and thus the plan, auditable.  In fact, the minute criteria appear in the requirement, two things happen.  First, the requirement becomes prescriptive.  The plan must contain these elements or else the plan will be found non-compliant.  Second, many entities will write a plan that only includes those elements.  They will not do the deep thinking, as you suggested in your previous post, to determine what needs to be in a really good plan.  And as circumstances change, there is no incentive to modify the plan because it already hit the required elements…it is technically compliant.”

My response to what the auditor has just said is the following: I am advocating that plan-based requirements include a listing of the threats[iii] that need to be addressed in the plan, not of how those threats need to be mitigated. The fact that the entity still can choose how they mitigate the threat means that the requirement hasn’t been turned into a prescriptive one.[iv] Your second point, that entities will write a plan “that only includes those elements”, should now be reworded as “entities will write a plan that only addresses the threats listed in the requirement.”

Theoretically, raising this as an issue would make sense if we are assuming that NERC entities all have the capability to look out over the whole threat landscape and decide for themselves what are the important threats that need to be addressed in the area covered by the plan-based requirement in question (of course, in the case of CIP-013, those threats are supply chain threats. In the case of CIP-014 R5, the threats are physical threats to an important substation). It is certainly likely that some entities do have this capability (mostly the larger ones, of course). However, I think that a lot of NERC entities (even ones with High and/or Medium impact BES Cyber Systems) would have a hard time articulating how they would do this.

And indeed, I really don’t think it should be up to the entities to survey the threat landscape and decide what are the most important threats that apply to them. Or at least, they shouldn’t be required to do this, if they are potentially going to face a Potential Non-Compliance finding for not being able to identify all of the threats that – in the auditor’s opinion, of course – should be addressed in the plan. So I think there should be a list of threats to be addressed included in the requirement for the plan. This is the case with CIP-010 R4, but it isn’t the case with CIP-013 R1, which is again why I say that requirement is only partially auditable.

A Slight Digression

(You can skip this section without missing anything of the overall argument. In fact, it might be better to skip this section the first time around, so you can keep the larger argument in mind).

The auditor goes on to discuss CIP-013 R1.[v] He says “You argue that Part 1.1 does not specify what must be in the procurement planning processes defined in the plan and therefore Part 1.1 is not auditable.  I disagree.  Part 1.1 specifically states that the processes used in planning for the procurement of BES Cyber Systems must “identify and assess cyber security risk(s) to the Bulk Electric System from vendor products or services.”  Part 1.1 goes on to mandate that two types of risks must be addressed: “(i) procuring and installing vendor equipment and software; and (ii) transitions from one vendor(s) to another vendor(s).”  Part 1.1 does not presume to know what those risks may be.  It is explicitly up to the entity to identify those risks and to assess those risks that have been identified.  Once assessed, the Part expects one or more processes to be developed to address those risks.  The plan and/or procedures will have to include how the risks are identified and assessed, since that is a key expectation of the Requirement.  The entity might have (a) beautifully documented plan and process, defining a completely ineffective methodology for getting where the entity needs to be.  And, as implied by the intended vagueness or non-prescriptiveness of the Part, the entity needs to continue to identify, assess, and mitigate evolving threats.”

You will notice the auditor is talking about “risks” here, not threats. That is because CIP-013 also talks about risks, not threats. However, I believe that threats need to be identified in the requirement, not risks. And what – ask you – is the difference between risks and threats? The Oxford Dictionaries define a threat as “A person or thing likely to cause damage or danger.” Notice there is nothing quantitative about this. For example, a cyber threat might be “software vulnerabilities”.

A risk, on the other hand, is quantitative[vi]. I define it as the probability that a threat will be realized, times its impact. I also believe that, at least in cyber security, it is close to impossible to even estimate the absolute values of particular risks. However, I do believe that it is possible to speak about the different levels of risk that apply to particular threats; this then leads to the possibility of ranking threats according to the degree of risk they pose.[vii]

So if we substitute “threat” for “risk” in what the auditor says above, as well as in the text of CIP-013 itself, I don’t think we’ll do too much harm to the meaning. The auditor can now be understood as simply giving an example of what he said in the first paragraph that I quoted of his email: That it should be up to the entity to identify the threats that need to be addressed in a particular plan-based requirement, and up to the auditor to determine whether the entity has properly identified those threats. As I’ve already said, I disagree with that position. I think the requirement should include in it the threats that need to be addressed in the plan, and that it isn’t auditable unless this is done.

Continuing with our Story

The auditor concludes with this paragraph: “And that is a key point here.  Non-prescriptive requirements by their very nature have implied, “prescriptive” requirements.  CIP-013 R1 is not a “one and done” effort.  As I just mentioned, there is an expectation of continuous identification and assessment of new risks as they arise.  Another implied requirement is that the defined processes that were developed as part of the overall plan are expected to be implemented.  In the past, FERC felt it necessary to require Standards to be modified to explicitly require implementation, otherwise entities might, in FERC’s view, argue that failure to implement the process or control was not a violation because the Standard did not explicitly require it.  FERC even went so far as to declare a CIP-008 Incident Response Plan that was not performed (implemented) in the event of a cyber security incident was to be found as a violation.  In the age of non-prescriptive requirements, the underlying expectations have not changed; they are just not explicitly stated.  You cannot have it both ways.  If you want prescriptive requirements, then fine.  Just don’t then complain about the minutiae.  If you do not want prescriptive requirements, then accept that you will have to read between the lines and meet the intention of the Standard and its requirements.  And accept that the auditor is adept at reading between the lines as well and will evaluate what you did against the intention of the requirement and its underlying expectations, to determine if you did “good enough.”  The audit team will then apply its collective expertise and professional judgment to determine what type of finding might be warranted, given the facts and circumstances of the instant case.” (my emphasis)

I have three comments on this. The first is that a requirement may well be implied by another requirement (and I talked a lot about implicit requirements in CIP v5 in the long-ago days), but the implied requirement can never be audited by itself (although I don’t disagree with the auditor that the entity could be in violation of the original requirement if they haven’t followed a requirement that is clearly implied by it. On the other hand, this is no way to write mandatory standards! All requirements should be explicitly stated. This is one of the big problems with CIP v5 – the fact that full compliance requires following so many implied requirements). The auditor uses the example of a requirement to develop a plan but not to implement it. I know that was a problem in the CIP v1-v3 days, but I don’t now know of any current CIP requirement to develop a plan, that isn’t accompanied by a requirement for the entity to implement the plan.

Second, I want to call attention to what the auditor said about “continuous identification and assessment of new risks as they arise”. Of course, I would substitute “threats” for “risks” here, but otherwise I totally agree that there needs to be a process to identify new threats as they arise. As I discussed in this post, this is what CIP-013 R3 does. But this also shouldn’t be the entity’s job. There should be a centralized body that continually reviews new supply chain threats, as well as new techniques for mitigating them, and regularly updates NERC entities on these.

But my biggest issue is with the sentences I highlighted. It seems to me that the auditor is making a false dichotomy here. He seems to be saying that if a requirement isn’t prescriptive (i.e. it’s an objectives-based or plan-based requirement), then entities just have to accept that there will be implied requirements they will be held accountable for. I think this is almost exactly backwards. Implicit requirements only come into play when there are prescriptive standards that aren’t worded properly. Since a plan-based requirement (which BTW I will soon start calling management-based, while I’ll start calling prescriptive requirements means-based. There’s a reason for this, which I’ll discuss in a separate post soon. For the moment, I’m sticking with my current terms) doesn’t prescribe any actions, there shouldn’t be any implicit requirements associated with it. Especially since I’ve now clarified my wording, and said that all plan requirements should include a list of threats that need to be addressed in the plan, not a list of “criteria”. The auditor feared that criteria could easily become prescriptive requirements, and I agree with him about that. But threats can’t become requirements in themselves.

Finally, the End

I started this post with my assertion (from two previous posts) that plan-based requirements should have a list of threats that need to be addressed in the plan, in order to be auditable. Furthermore, I stated that, since CIP-013 R1 only provides a partial list of the threats to be addressed in the supply chain cyber security risk management plan (SCCSRMP, in case you forgot the acronym. In another year, this acronym will be on every NERC compliance professional’s lips, it’s so musical. I ought to trademark it), R1 is mostly un-auditable. So I am ending the post by making the same two assertions that I started it with, after having considered the auditor’s excellent points that he raised in his email. I have learned a lot by writing this post, and I have clarified my positions. Just because I ended up where I started doesn’t mean the journey wasn’t worthwhile!

               

Of course, I have yet to answer the question I’ve said I’ll address in my next post twice already: namely, whether it matters that CIP-013 R1 is mostly un-auditable (spoiler alert: I’m going to say it really doesn’t matter that much). Maybe next time I’ll actually do that, although I won’t bet on it.

The views and opinions expressed here are my own, and do not reflect those of any organization I work with. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] I’m sure some people questioned the fact that in none of these sections is the entity free to find another mitigation that is equally effective as the ones listed; these are the only options the entity has. My guess is the drafting team decided that there was too much risk that an entity would grab onto ineffective means of mitigating the risk, rather than the tried-and-true ones listed. I won’t try to weigh in on this argument, since the requirement is now in effect and I haven’t heard any grousing about the fact that the entity doesn’t have any other options besides those listed (like they do in CIP-003-7 Attachment 1, Section 3.1, where the entity is just required to “implement electronic access controls” to “Permit only necessary inbound and outbound electronic access” to Low impact assets with external routable connectivity. Of course, CIP-003-7 is another plan-based requirement).

[ii] There were actually two email exchanges this time. What I’m quoting is all form the second exchange.

[iii] And I will readily admit that my thinking has evolved even since my last post, when I was simply saying that there need to be “criteria” in the requirement. I wasn’t thinking that the criteria would be simply threats that need to be addressed, but that they could indeed be something like requirement parts. If I hadn’t changed my thinking, I would completely agree with the auditor that these criteria could well have turned into prescriptive requirements. However, by clarifying that the requirement needs to include a listing of threats, not “criteria”, I think I have addressed the potential problem of turning a plan-based requirement into a prescriptive one.

[iv] If you’ve been paying attention, you may think this is a contradiction to the praise that I just heaped on CIP-010 R4 Attachment 1. This requirement lists threats to be mitigated, but it only provides certain options for mitigation; the entity doesn’t have complete freedom to mitigate in a way that makes the most sense for them. While I don’t necessarily agree that this is the right approach, I think that just having these options means this hasn’t turned into a prescriptive requirement.

[v] Part of what he says isn’t germane to the main argument of this post, but is nevertheless quite good. Here it is:

“The purpose of the Standard, and thus the collective purpose of the requirements is to “to mitigate cyber security risks to the reliable operation of the Bulk Electric System (BES) by implementing security controls for supply chain risk management of BES Cyber Systems.”  Everything that the entity does with respect to the requirements is expected to achieve the overall purpose of the Standard.  In plain language, the entity needs to “do something”, i.e., implement one or security controls to mitigate risks.  That, therefore, implies that the entity needs to identify and understand the risks so that it can design effective security controls to mitigate those risks.  Yes, I know that the purpose statement is not an approved, auditable requirement.  But, it serves to inform the intention of the Standard and its requirements, thus establishing certain expectations of performance.

“Requirement R1 states “each Responsible Entity shall develop one or more documented supply chain cyber security risk management plan(s) for high and medium impact BES Cyber Systems.”  The requirement goes on to specify certain elements that must be in the plan.  There are two specific elements: (Part 1.1) one or more process(es) used in planning for the procurement of BES Cyber Systems, and (Part 1.2) one or more process(es) used in procuring BES Cyber Systems.

“I agree that Part 1.2 prescribes six specific elements of the procurement process that are auditable.  Either the processes collectively address the six elements or they do not.  Part 1.2 is prescriptive and can be objectively audited in terms of whether the elements are present.  Part 1.2 can also be audited subjectively in terms of whether the elements were adequately addressed.  The subjective aspect of the audit of Part 1.2 primarily applies to Parts 1.2.5 and 1.2.6.  Those are the two Parts that afford entity the most discretion in how the entity approaches the expectations.  For example, Part 1.2.5 requires “verification of software integrity and authenticity of all software and patches provided by the vendor for use in the BES Cyber System.”  While hashing downloaded files and verifying the hash results against information provided by a vendor is an excellent way to accomplish this expectation, not all vendors will provide hashes of their software.  The process will have to address how the entity plans to perform the expected verification in the absence of provided hash values (in this example).  That suggests that as the plan and its required processes are developed, the entity is coincidently evaluating what its vendors do today in order to identify and develop appropriate verification schemes that accommodate vendor practice.  And, if the vendor changes its practices in the future, the processes might need to be changed to maintain alignment.

[vi] I will readily admit that the definition of risk that I’m giving here isn’t from any dictionary. Rather, it is the definition that makes my worldview of industrial cyber security work. I’ve said before – in fact, probably in mid-2016 – that I’m working on a book, with two co-authors, on how NERC CIP can be “reformed” to address the major problems it has, and make it sustainable into the future (which it is not now, in our opinion). I’m glad to say that we’ve finally progressed beyond the talking stage and are doing serious writing now, although I’m not going to even talk about a publishing date yet. In any case, this “definition” of risk is the one we are using in our book and it works for our purposes. Other definitions would work for other purposes.

[vii] This observation – that it is possible to at least give a rough ranking of the relative degree of risk that each cyber threat poses – is the key to making our upcoming book work. I hate to tease you with this idea and leave it there, but I can’t really explain why this is the case without quoting a good portion of the book. And the book isn’t written yet!