Does Auditing Matter?

Five of my last six posts have dealt with plan-based requirements, such as CIP-013 R1, CIP-014 R5, CIP-010 R4, and (parts of) CIP-003 R2. These requirements have become very popular among the four NERC CIP drafting teams that have done their work after the CIP version 5 standards were developed, approved and implemented. In fact, almost all of the requirements, and both of the new standards, that have been developed since CIP v5 have been plan-based. And I see no sign that this process will let up anytime soon. The new standard currently being developed, CIP-012, is definitely plan-based as well.

The last four of those five posts have all addressed the question of auditability - namely, how can a plan-based requirement be audited? In the last two posts, I have made the point that, in my opinion, for a plan-based requirement to be auditable it needs to include a list of threats that must be addressed[i]. For example, in this post I pointed out that CIP-014 R5 requires that the plan specifically address the threats and vulnerabilities identified in the threat and vulnerability analysis required by R4. And in this post I discussed CIP-010 R4, which does a very good job of outlining the threats to be addressed, and is the best example I’ve seen so far of an auditable plan-based requirement.

And then there’s CIP-013 (especially CIP-013 R1). In the last three posts in this series, I’ve stated that it is only auditable in a limited sense. R1.1 lists three threat types that need to be addressed in the plan: threats from procuring vendor equipment and software, installing vendor equipment and software, and transitions between vendors. The only list of threats is in R1.2, and those are only a subset of the threats that would apply to just one of the three threat types (they’re all threats from procurement). Given that the great majority of the threats that would appear in a comprehensive list are not shown, I rate CIP-013 R1 as mostly un-auditable.

However, in each of the last three posts in this series, I’ve gone on to raise another question: Does it matter that CIP-013 R1 (and by implication R2) is un-auditable? More specifically, in what ways does it matter and not matter?

Let’s step back and ask a larger question: What is the purpose of auditing in the NERC standards environment, anyway? For the prescriptive requirements (which I will soon be calling “means-based” requirements, since that is the term that people who write about these things use) – including most of the 693 (O&P) requirements and many of the CIP requirements - the answer is quite simple: to determine whether the entity did or didn’t do what they were supposed to do or not do.

But why do we need to audit the prescriptive standards in the first place? Because they’re mandatory. And why are they mandatory? Because following them is deemed important for the reliable operation of the Bulk Electric System. And this is the fundamental reason why prescriptive standards are audited.

But how about the plan-based requirements (which I will soon be calling “management-based”)? Given that the ultimate goal is the protection of the BES, what is the best way to accomplish that goal? We shouldn’t engage in a false analogy here and just say “Well, if auditing is the way that the BES is protected with respect to prescriptive requirements, then it’s obviously the way to protect the BES with respect to plan-based requirements.” Is this really the case? Let’s look at the nature of prescriptive and plan-based requirements.

First, why are many NERC requirements prescriptive? Let’s use the example of FAC-003, the infamous tree-trimming standard whose violation was one cause of the 2003 Northeast blackout. Table 2 of this standard shows minimum distances that must be maintained between a high-voltage transmission line and any tree directly under it. It is prescriptive, because all it takes is one overgrown tree to trip such a line – and having a few such lines tripped in one day, coupled with generation outages, can lead to a cascading outage (or at least it did in 2003). Since any one tree can cause a line trip, there is really no choice in the matter: the requirement has to be prescriptive, and it must apply to every tree in scope. Here, the audit’s role is to confirm that the utility is really complying with the requirement and trimming all of the trees that they need to.[ii]

Now, why are some NERC CIP requirements plan-based? Because a plan-based requirement is designed to address an objective that isn’t measurable.[iii] In the CIP world, the objective of a requirement is always the mitigation of the risk to the BES posed by one or more cyber or physical threats. As with prescriptive requirements, the protection of the BES is still the primary goal here. But what’s the best way to protect the BES, when a requirement is plan-based?

I don’t know about you, but I think the best way to protect the BES, in the case of a plan-based requirement, is for the NERC entity to have a good plan and to implement it well. Note that this isn’t a binary choice, as in the case of prescriptive requirements. This isn’t a question of verifying whether the entity took particular actions or not. Rather, it’s a case of whether the plan they developed, and their implementation of it, advanced the objective of the plan (in the case of CIP-013, the objective is supply chain security. In CIP-014, it’s physical security of key substations; etc).

But if this is true, what good does it do to simply audit after the fact? For example, suppose an entity has no idea how to write a supply chain security plan, and puts together a document that includes a lot of miscellaneous directives that don’t help BES supply chain security at all. Or suppose they put together a good plan, but do a terrible job of implementing it, so that – again – supply chain security isn’t improved. How does it help the BES for auditors to simply come in after the fact, stroke their chins, and say “Yup, this was a terrible plan, all right” or “These guys wrote a great plan, but they put an idiot in charge of implementing it, so they might as well not have bothered with the whole exercise”?

The answer to this question is it doesn’t help the BES at all for the auditors to determine after the fact whether the plan is good and whether it has been well implemented. If the entity is struggling with their plan, they should be able to get help at the time they’re struggling, not years later when they’re told their plan is no good. The same with implementation: As they start implementing the plan, someone needs to check on them to determine whether they’re on the track to success or not. If they aren’t, they need to receive some advice on how they can get on track.

Two questions come up here. First, if the entity is struggling with their plan or with implementation, why can’t they turn to a consulting firm? The answer to that is they certainly should be getting help from consultants if they need it, but in the NERC world the “consultants” who really matter are the auditors. You can find a good example of that in my recent post on CIP-014, where I mentioned a NERC entity that had received a PNC (potential non-compliance) notice because they had taken the requirements in that standard literally when they say that the entity is to protect against attacks on the substation itself, not on any of the individual Facilities (buses, transformers, etc) in the substation. The auditors thought they should be protecting the Facilities as well (especially the transformers).

As I also pointed out in that post, I don’t blame the auditors for saying the entity’s plan should have included protection for the transformers as well as for the whole substation, although I think they should just have identified this as an Area of Concern (AoC), as another set of auditors had done with another entity in the same region in a similar situation. But regarding my point on consulting firms, this entity had worked closely with a very respected NERC CIP compliance firm to do the threat and vulnerability assessment and develop the plan, yet the plan was still wrong as far as the auditors were concerned. It’s a sad fact but true: as far as CIP compliance is concerned (not just CIP-013), the only correct interpretation of the requirement is the one your auditor follows.

This utility was actually quite lucky that they got audited when they did – shortly after they had developed their Physical Security Plan for CIP-014 but before they had implemented it. Had their audit date fallen a couple of years later, they would have not only developed the plan but implemented it. For them to hear at that time that the plan was all wrong would have been much more serious, since it would mean they should have spent their implementation money differently than they actually did spend it, over the past couple of years.

Now to the second question: Given that the auditors (or at least the region) are the ones who really need to judge the plan and its implementation, what is to keep them from giving the plan an informal review after it’s developed, and providing helpful advice on how it could be improved? And why couldn’t the auditors, when the entity has started their implementation of the plan and has some questions about whether they’re doing it right, simply take a look at how the implementation is going so far and point out any problems they see?

You may have guessed the answer: the principle of auditor independence. According to auditing standards and especially to the Generally Accepted Government Auditing Standards (GAGAS), which NERC auditors are all trained on, the auditor can never provide compliance advice to an entity they will audit, prior to the audit itself.

I agree that auditor independence makes sense in the case of prescriptive requirements, where there really is a binary answer to compliance questions. But what about in the case of plan-based requirements, where the entity – as we have just seen – could go way off course and spend a lot of money implementing a plan that isn’t what the auditors want to see? I have a great example of that. It’s from this post – also about CIP-014 – where I described a different entity that had asked their region to give them advice on a big physical security investment (about $80 million, in fact) they were thinking of making at their critical substations; they wanted to know whether that investment would increase the likelihood that their Physical Security Plan would be found to be adequate when they were audited. Given the amount of money involved and the fact that there were a lot of alternative uses for it, management didn’t want to make this investment without some clear signal that it would help, and especially not hurt, their CIP-014 compliance posture.

I don’t think it will surprise you when I say that the answer they received was no, that the auditors couldn’t provide an answer to this question because it would violate – ta da! – auditor independence. Again, I can’t blame the auditors. Given that they are effectively government auditors and they have to follow GAGAS, how could they give any different answer?

But instead of shrugging our shoulders, why don’t we go back to the question I asked above: Why is this standard mandatory? The answer: Like with the other CIP standards, the protections it affords are deemed to be very important to the BES. Then let’s ask another question: Given that the region’s refusal to answer the entity’s question may well result in an $80 million physical security investment in critical substations not being made, and given that it’s indubitable that this investment would have enhanced the physical security of the BES (otherwise, why would a large, sophisticated utility have even considered it in the first place?), how does the possible cancellation of this project make the BES more secure?

The answer to this last question is simple: It doesn’t. The potential cancellation of this project (and I have no knowledge whether it might be cancelled or not) will definitely result in a less-secure grid than would its implementation. So if our ultimate concern is really the security of the BES (which it is), not guarding the principle of auditor independence, what can we do about this?

In my opinion, this is how plan-based standards should be managed (I don’t want to use the word “audited”) by NERC and the regions:

1.       Any requirement to develop a plan will need to include a list of threats to be addressed in the plan, as well as some mechanism for updating that list on a regular basis (hopefully by an industry body, not just by the entity itself). The requirement will also need to specify that the plan must include a timeline for implementation.

2.       For every requirement to develop a plan, there must be another requirement to implement it. Part of that requirement will specify that any changes in timeline or objectives of the plan will be documented and explained, so that when the region reviews the implementation they will always be looking at the current plan.

3.       The entity develops their plan, either on their own or using a third party like a consulting firm.

4.       The entity submits the plan to their region (not necessarily to the auditors, of course). The region reviews it and points out any deficiencies they see.

5.       The entity revises their plan and re-submits it. If it’s now good to go, the region approves it. If it’s not, the region sends it back a second time, with a new set of pointers (and perhaps a stern admonition if they think the entity hasn’t been paying close enough attention to what they said).

6.       If needed, there’s another cycle, but ultimately the plan will be approved. The entity then starts the implementation process, which will follow the timeline shown in the plan.

7.       At any point during implementation, and with or without a request from the entity, the region can step in and review how the implementation is going. They’ll compare the entity’s progress to the timeline, and will require explanations of any deviations from that timeline (in fact, any deviation should already have been documented). Of course, if they think the project is off course in any way, they’ll provide advice on how to fix that.

8.       If at any point in this process the entity seems to be acting in bad faith, or simply ignores advice the region gives them, then – and only then – can they be issued a PNC.

Don’t you think the BES would be more secure if this were how the region operated, rather than auditing CIP-014 like it’s FAC-003, so that a judgment needs to be made on whether the entity has complied with each requirement and requirement part – but at the price of the region not being able to provide any guidance at all before the actual audit? I do, too. This is why I’m saying that, for plan-based requirements, audits really don’t matter very much. What matters is the guidance the regions can give to the entities. If we had to give up auditing of these requirements altogether and essentially make them voluntary standards, the BES would be more secure, not less.

How does this problem get fixed? In the long term, it will be a big deal. For one thing, the requirements themselves will need to be rewritten as I described in the first two bullet points. For another, NERC’s CMEP (Compliance Monitoring and Enforcement Plan) and perhaps the Rules of Procedure will also have to be modified. Once that is done, the “management” steps I just described will allow plan-based requirements to be “audited”, but in a different sense than audits of prescriptive requirements.

However, there is also a shorter-term fix, which involves doing all of the above steps with different people than the actual auditors and only issuing AoC’s, not PNC’s. This would also be a big change, but it wouldn’t require rewriting the standards, CMEP or the RoP. And in fact, I think it is effectively how plan-based requirements will be “audited”, until the longer-term fixes can be implemented. I’ll describe this in an upcoming post, but there are a few that I want to write before I go back to this topic. So stay tuned.

Have a good holiday!

The views and opinions expressed here are my own, and do not reflect those of any organization I work with. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] I now realize that this list doesn’t have to be provided in the requirement itself. Rather, it can be developed through a local threat and vulnerability assessment. This would only be appropriate in a standard like CIP-014, where the physical security threats to one substation will inevitably be substantially different from threats to another substation, due to variability in terrain, population, etc. Most cyber threats aren’t specific to one facility,

 one NERC entity, or even one region of the country, which is why I think the threats to be addressed in the plan for requirements like CIP-013-1 should normally be included in the requirement itself. And as I’ve also pointed out, there should be a provision for updating this list regularly, since new threats come up all the time. So far, only CIP-013 has a provision for updating threats, and it doesn’t do this the way that I would.

[ii] Here’s a fun exercise for the reader. Compare the argument I just made for a prescriptive tree-trimming requirement to CIP-007 R2 patch management, the most prescriptive CIP requirement. Does the same argument apply there? If not, how is the situation different? If you want to email me your papers, I’ll send back your grade. But please, no looking at your neighbor’s paper! For my answer, you’ll have to wait for the book I’m working on now, but it’s likely I’ll touch on this topic in the blog sometime before then. I have probably discussed it at least a few times previously, but don’t ask me to look through all those old musty posts to try to find where I said that! One person can only take so much punishment…

[iii] By giving this quick-and-dirty definition, I’m going beyond what I said about plan-based requirements in the first post in this series. I thought up until a few days ago that plan-based requirements were just a subset of objectives-based ones. But I’ve now learned that true objectives-based requirements need to have a measurable objective. The plan-based requirements in CIP aim for non-measurable objectives like supply chain security, physical security, etc. As I already said, plan-based requirements are really one type of “management-based” requirements. In management-based requirements (found in many compliance regimes, not just NERC or NERC CIP), the entity needs to put together a good program (i.e. plan) to pursue the objective, but with the understanding that there is no such thing as obtaining an objective like supply chain security. If you thought it was actually possible to attain that objective, then I’m sorry to have to disabuse you of that notion. By the way, there’s no Santa Claus, either.