The Prisoner’s Dilemma

I remember reading a few news stories about a long-time prisoner who is released and finds that life on the outside is much harder for him than it was in prison. I don’t think too many of these people actually ask to be re-admitted to prison, but it certainly stands to reason that someone who has lived almost his whole adult life there might view his complete lack of any lifestyle choice at all while behind bars to be in many ways preferable to the need to make a myriad of choices, which all of us who live outside of prison have to do every day.

This is perhaps a little bit of an overdrawn analogy, but I have more than once thought of this “prisoner’s dilemma[i]”as akin to that of NERC compliance professionals who object to non-prescriptive, results-based standards for cyber security (such as the CIP-013 standard for supply chain security) on the grounds that they will not work in practice, compared with good ol’-fashioned prescriptive standards.

Why do some NERC CIP compliance professionals feel that non-prescriptive requirements won’t work? If you haven’t been in the trenches of CIP compliance for the past say three or four years at least, you may not understand why anyone would prefer prescriptive requirements, at least when it comes to CIP. For example, who would prefer CIP-007 R2.2 - which mandates that every 35 days the entity must, for every device in its ESPs, check with the patch source to determine whether a new patch has been released - to CIP-007 R3.1, which simply directs the entity to “Deploy method(s) to deter, detect, or prevent malicious code”?

Well, it seems a lot of people do in effect prefer R2.2. The fact is that many NERC compliance professionals – in particular ones who have had to deal with CIP for a number of years – shrink back in horror from the idea that all of the CIP requirements should be like CIP-007 R3.1, not R2.2. I was reminded of this very forcefully when I recently attended a meeting of the team drafting CIP-013, the new supply chain security standard, in San Antonio.

The team met just after their first draft of that standard was soundly rejected by the NERC ballot body. All of the requirements in that draft are non-prescriptive[ii], and some commenters indicated they were very worried about giving CIP auditors too much “discretion” in how they conduct their audits. And there certainly have been a lot of stories – more during the CIP v1-v3 days than currently – of auditors who have made their own interpretations of some of the CIP requirements, and identified Potential Violations in cases where other auditors would have found compliance.

In other words, these people seem to be saying “We need to keep these auditors on a very tight leash, so they don’t have any discretion in how they audit the CIP requirements. If all the NERC CIP requirements are made non-prescriptive, the auditors will no longer be under anyone’s control, and will be able to issue PVs for anything they perceive to be a bad cyber security practice, whether or not it’s allowed in the requirement they’re auditing.”

I certainly do have a lot of sympathy for these people, since I know that at least some of their horror stories are real and have made their lives miserable, at least for a time. However, I think these people are basing their arguments on the wrong premises. In fact, auditors are now virtually required to use discretion; they no longer have an option not to do so. Let me elaborate.

As I discussed in a post last year (not too coincidentally, one which also referenced discussions about CIP-013), many if not most of the prescriptive CIP requirements include particular details – like “within 35 days” in CIP-007 R2 – that aren’t dictated by any principles of cyber security or electrical engineering, but which simply are there because the drafters thought they needed some number - in order to have a requirement that could be “unambiguously” audited.  So there’s a strange logic here that goes something like

1. We need prescriptive requirements so that auditors can’t use any discretion.
2. Prescriptive requirements can only be audited if they have precise criteria, preferably numerical.
3. Therefore, the requirements should include specific numbers (or other goals), even if there is no compelling reason why any particular number should be chosen.

To word this another way, a lot of the details in prescriptive requirements are there simply because the requirements are prescriptive, not because they are dictated by say cyber security practices or the laws of physics.[iii] Lots of money and effort are being expended on meeting these arbitrary deadlines and numerical targets, when entities could deploy their limited cyber security budgets much more cost-effectively if they were given a goal to achieve and then allowed to choose – and defend to an auditor – whatever means of achieving that goal makes the most sense in their environment. And this means that NERC CIP compliance professionals who won’t support non-prescriptive requirements are in effect saying they’d much prefer to put up with the additional headache and expense of meeting these arbitrary targets, even when they contribute very little toward increased cyber security, rather than take the chance that an auditor might decide to judge a non-prescriptive requirement in an arbitrary way.

However, these people are missing a very important point: The auditors are already imposing their own (or their region’s) judgment regarding the meaning of particular CIP requirements, because of the many gaps, ambiguities and outright contradictions in the requirements. And this trend has simply increased with CIP versions 5 and 6. For example, in order to determine whether an electronic device is a Cyber Asset, the entity needs to know what the word “Programmable” means in the Cyber Asset definition; that word is undefined and is still heavily debated. And to determine whether a Cyber Asset is a BES Cyber Asset, the entity needs to understand what “impact the BES” means in the BCA definition – this phrase is also undefined[iv]. So the only way for the auditor to determine whether the entity is compliant with CIP-002 R1 (the requirement in question here) is to exercise judgment on whether the entity’s own interpretation of these terms was reasonable or not (although if they’re lucky their NERC Region will have provided them some guidance on this and other similar questions. But the regions don’t usually publicize their guidance, and different regions sometimes provide different guidance, if they provide any at all).

So which would you rather have: a prescriptive requirement like CIP-002 R1 that seems to be very precise but in fact requires a lot of under-the-table judgment, or a non-prescriptive requirement like CIP-007 R3, which states that the entity must “Deploy method(s) to deter, detect, or prevent malicious code”? Granted, the latter leaves much room for the auditor to exercise their own judgment, but this is judgment about cyber security issues – in this case, whether the methods deployed by the entity to prevent malicious code are appropriate, given the cyber assets being protected. And if, in the auditor’s judgment, the entity hasn’t deployed the right methods, they won’t simply issue a PV (unless, of course, the entity hasn’t deployed any method at all, or has deployed a method that could never possibly meet the objective of the requirement). Rather, the auditor will most likely issue an Area of Concern, stating that – while the entity did technically meet the requirement since they did deploy a “method” to prevent malicious code – the entity should strongly consider another method which, in the auditor’s opinion, would be more effective.

But consider the auditor’s judgment in auditing CIP-002 R1, discussed earlier. In interpreting what “Programmable” and “affect the BES” mean, the auditor isn’t applying his or her cyber security knowledge or experience, but simply guessing what the words mean, based on the context of the definition, how it is used, etc. This is judgment about the semantics of legal terms. Yet I don’t think a legal semantics background is required of CIP auditors, while a cyber security background certainly is required. So in which area would you prefer to have the auditor exercise judgment: semantics or cyber security? In auditing the prescriptive CIP requirements, the auditor is not supposed to exercise cyber security judgment, but he or she is in fact often required to exercise semantic judgment. In auditing non-prescriptive requirements like CIP-007 R3, the auditor is only required to exercise cyber security judgment – and they all should be able to do that by now.

Of course, you may point out that some NERC CIP auditors aren’t the world’s best cyber security experts. Or they may know cyber but they’re weak on industry knowledge. Or both. I’m sure this is the case, but these problems can be addressed through measures like longer on-the-job training periods and mentorship. But requiring cyber security auditors to exercise judgment about cyber security matters is a much better bet than asking them to exercise judgment on ambiguous (or in some cases missing) definitions and requirements.

This is one reason why I think the non-prescriptive route is the only good one for NERC from now on, regarding development of new CIP standards and requirements. And NERC (with prodding from FERC) seems to have made this decision already, since no new prescriptive NERC standard or requirement has been developed in the past three years (CIP-014, CIP-013 and CIP-003 R2 Section 3 are all non-prescriptive. I believe these are the only ones that have been developed new - although CIP-003 R2 Section 3 was a rewrite of a prescriptive requirement part).

But just making sure all new requirements are non-prescriptive isn’t enough. Some of the existing CIP v5 requirements are highly prescriptive. CIP-007 R2 is my poster child for a prescriptive requirement, but CIP-010 R1 isn’t too far behind. A lot of the other requirements are prescriptive as well. The fact that they are prescriptive imposes a big burden on NERC entities (one entity told me that literally half of all the compliance documentation they produce in their control centers is for CIP-007 R2 alone). More importantly, I believe there can be no further meaningful extension of the existing CIP standards – to areas like virtualization or the cloud – until all of the existing requirements are made non-prescriptive. More on this in my next post.[v]

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte.

---

[i] And in using this phrase, I’m not in any way referring to the famous “game” called the Prisoner’s Dilemma, which is much discussed in game theory.

[ii] “Results-based” would probably be a better phrase than “non-prescriptive”, since it emphasizes what cyber security requirements should do (require the entity to achieve a certain goal) rather than what they shouldn’t do (prescribe particular steps to achieve that goal). However, I hesitate to use the former phrase, since NERC uses it to describe the current CIP requirements, even though some of them (such as CIP-007 R2 and CIP-010 R1) are very far from being truly results-based.

[iii] Please note that I think almost all of the non-CIP NERC standards (i.e. the Operations and Planning, aka “693”, standards) probably do need to be prescriptive. For most of those standards, the specific goals and numerical targets are dictated in some way by the laws of physics. For some of those standards, a failure to do a certain thing at a certain time could very well lead to a cascading outage, as was the case in the 2003 Northeast blackout. But cyber security is statistical. Missing the availability of a security patch for one system for one month is not inevitably going to cause any problem, although not patching any systems for say a year or two will very possibly lead to a BES event. The entity needs to determine what makes the most sense in their environment and with their budget (since I don't know any NERC entity that has been given a blank check to cover anything they feel like spending on cyber security. They need to prioritize their spending so it yields the greatest possible security per dollar spent – as discussed in this post). Non-prescriptive requirements are the only way to do this.

[iv] Both of these issues are on the table for the CIP Revisions drafting team to address. By the time new definitions are drafted, balloted and commented on, redrafted, re-balloted and commented on, re-re-drafted and re-re-balloted and commented on, and then approved (or not) by FERC (assuming they have a quorum sometime in the not-to-distant future), it will be probably 3-4 years from now. In the meantime, auditors and entities need to muddle through, as they’ve been doing all along.

[v] I cringe a little when I say what will be in my “next post”, since more often than not some other topic comes up that seems more compelling, and the “next post” actually appears five or six posts later. In any case, a post on this topic is coming, whether it’s the next one or not.