The Biggest Problem with NERC CIP

Since at least last February, I have been pushing the idea that the CIP standards are on an unsustainable course and need to be radically rewritten. I have mentioned several reasons for this, but I have said the most important is cost, since the amount that NERC entities are spending on CIP compliance grew significantly with the advent of CIP v5 and v6. Plus it will inevitably increase even more significantly due to the new Supply Chain standard (CIP-013), to CIP v7 and to other scope increases still likely to come, which include virtualization, the cloud, machine-to-machine communications and who knows what else.

However, just the fact that NERC entities are spending a lot of money on CIP – with a lot more to come – isn’t in itself the issue; after all, the whole idea of CIP was to get entities to invest in cyber security to an extent they couldn’t be counted on to do without facing a mandatory requirement. The problem – as I’ve discussed in several posts, including the one just referenced - is that so much of this spending (including “spending” on staff time, of course) isn’t going to cyber security but to what I’ve been calling “compliance paperwork”. The estimates I have received of the portion being so spent seem to center on 50%, meaning literally half of what entities are spending on CIP compliance nowadays is not contributing to cyber security. Even if the actual figure is less than that, it is still a big number when you consider that total spending (including employee time invested) on CIP v5/v6 compliance has been well into the billions of dollars.[i]

I haven’t ever really discussed what I mean by “compliance paperwork”. For one, it definitely includes the documentation that an entity has to do to prove they did things for compliance purposes – as opposed to the documentation that entities would do in the absence of a mandatory requirement, simply because it is a good business and security practice. It also includes a lot of the self-reports, mitigation plans, compliance certifications, etc. that are required to comply with CIP. And it certainly includes a lot of the effort that is required to prepare for and get through an audit.

The idea that so much of current CIP spending isn’t going to security leads directly to the question “Could a better design of the standards reduce the amount of spending going solely to compliance ‘paperwork’ – meaning the same amount of total spending on CIP would lead to a much larger investment in actual cyber security?” My answer to this has been an emphatic yes, assuming the standards were made much less prescriptive. In other words, I have been saying it is possible to write much less-prescriptive CIP standards whose implementation will lead to a much greater amount of investment in cyber security, even if the overall level of CIP spending were unchanged.

However, a recent conversation made me realize that, while this is a very important reason to rewrite CIP, there is a more important one. To understand this new reason (which I have probably alluded to previously but not explicitly called out, mainly because I had been thinking it was really a part of the “compliance paperwork” reason just discussed), I need you to do a “thought experiment[ii]” with me: Say that you annually have available to you an amount CYBER, which is your organization’s current expenditures on cybersecurity. In addition, you have another amount CIP, which is your expenditures on NERC CIP compliance. Now suppose the NERC CIP standards suddenly go away, but the total amount available to you to spend remains the same, as long as you spend it all on cyber security[iii]. In other words, you can now spend the entire amount you would have spent on CIP on cyber security, in addition to the amount you were planning to spend on (non-CIP) cyber anyway; let’s call this total amount SUM(CIP+CYBER). How would you allocate this total sum? I’m guessing you would do it the way hopefully most organizations do for all of their security spending: You would first identify the threats that affect your organization and estimate the costs to mitigate each one (which of course isn’t the same as eliminating those threats. That’s impossible, at least for most threats). Then you would rank these threats by their potential impact. Finally, you would budget the available funds so that they mitigated the most important threats.[iv]

At this point, you have a list that reads “Threat A has the largest potential impact and it would cost $Y to mitigate; we’ll definitely address that, since this is well below our total budget. Threat B has the second largest impact, and it would cost $Z to mitigate. $Y + $Z is still below our total budget, so we will do both of these…..” You would continue this process until you have exhausted your budget for the year (while still leaving some for contingencies, of course!). Of course, your total budget for cyber security is SUM(CIP+CYBER), since there are no longer any CIP requirements.

Now let’s say NERC CIP suddenly reappears (Don’t ask me why. I told you this was just a thought experiment!), and you have to comply with the current CIP standards, as well as address cyber security threats. What happens to your calculations? All of a sudden, you have a bunch of new compliance threats that need to be mitigated: the threat of non-compliance with CIP-004 R3, non-compliance with CIP-007 R2, etc. But your original cyber threats are still there and also need to be mitigated. How do you now prioritize your spending (and remember, you still have SUM(CIP+CYBER) available to mitigate both CIP and non-CIP cyber threats)? Can you look at a CIP compliance threat, like say the risk of non-compliance with CIP-010 R1, and rank that ahead of say the threat posed by phishing attacks – a cyber risk that isn’t addressed at all in CIP?

Here’s the correct answer: You certainly can do that. In fact, I believe most NERC entities would almost never prioritize any non-CIP cyber threat over the threat of non-compliance with a CIP requirement. Not only can there be huge fines for ignoring CIP requirements, but most NERC entities will do almost anything to avoid being fined at all because of damage to reputation, impact on rate cases, etc. In fact, I think there are few NERC entities that would passively accept a CIP fine, even if the “penalty” were a trip to Disney World. This means that the threats of non-compliance with CIP requirements will assume a special priority ahead of probably all the other cyber threats. In essence, your organization will need to make sure it has addressed all of its CIP non-compliance threats before it spends very much to address non-CIP cyber threats.[v]

So why is this bad? As I’ve just said, some significant percentage of what your organization spends on CIP compliance doesn’t go to security at all but simply to proving compliance. This alone results in the amount you are spending on CIP being less effective in advancing cyber security than would be the same amount of spending in the absence of CIP. So the fact that you have to prioritize CIP spending means that a lot of your total spending - that is, some percentage of the quantity SUM(CIP+CYBER) – goes to compliance paperwork, rather than cyber security. This might not be 50 percent, but it is certainly significant.

I used to think this was the biggest problem with CIP, but I no longer do. To see the biggest problem, let’s go back to our thought experiment and see what happened to your spending priorities when CIP was re-imposed. Before that happened (during the period of time when CIP went away but you still had the money that had been allocated for CIP, along with the non-CIP cyber budget), you were allocating your entire cyber security budget based strictly on the impact level of each threat: the threats with the largest impact were first in line to receive mitigation money, those with the next-largest impact were next in line, etc.

When CIP was re-imposed, you had to change that. You essentially “prioritized” the threat of non-compliance with each CIP requirement to put these compliance threats ahead of the “pure” cyber threats. For example, your number one cyber threat (say, phishing, which was recently said to be the immediate cause of 91% of cyber attacks) might get pushed from number one down to number 44 (or so) on the list of threats to be addressed – because the threats of non-compliance with each of the CIP requirements all took their place ahead of it. The bottom line: you will be able to address far fewer non-CIP cyber threats than you could have before CIP was re-imposed.

A Rude Interruption

Here you might rudely interrupt me and say “Even though we’ve had to prioritize CIP expenditures over other cyber expenditures, CIP is just cyber anyway. This means we’re still devoting our entire budget (which equals SUM(CIP+CYBER), of course) to cyber security. If we put aside for the moment the fact that a certain percentage of CIP compliance spending is “wasted” on compliance paperwork,[vi] there is really no impact due to the fact that CIP was re-imposed; we’re still spending 100% of the budget on cyber security.”

What’s wrong with this statement? It ignores the fact that what you are spending your money on has drastically changed, in two ways. The first is that the threats you prioritized during the period when there was no CIP have now been pushed way down the list of threats to be addressed, since the threats of non-compliance with the different CIP requirements have all been pushed ahead of them. In fact, probably a lot of the non-CIP cyber security threats you would like to mitigate will now receive no funding. For example, even though phishing was your number one threat to address before CIP was re-imposed, it is now your number 44 threat and you’re going to have to fight hard to get even a modest amount of money allocated to it. In other words, even though the total amount you’re spending hasn’t gone down, it no longer is based on your prioritization of the threats. While this effect can’t be quantified, it is certainly quite significant – it acts something like a tax on your cyber security program.

Let's move to the second way in which what you’re spending money on has changed with the re-imposition of CIP; the best way to describe this is with an example. I think everyone agrees that the threat posed by security vulnerabilities in software should rank among the most important cyber threats; this means that patch management should form a big part of every entity’s cyber security program. Of course, CIP-007 R2 is the patch management requirement in CIP v5 and v6. In our thought experiment, when CIP is reimposed the threat of non-compliance with CI-007 R2 (and all of the other CIP requirements) will be prioritized ahead of all the non-CIP cyber threats like phishing.

But wait! You point out that one of the top cyber threats on your list, during the time when CIP went away, was software vulnerabilities. Patch management is probably the most important way to mitigate this threat; it also happens to be what is required by CIP-007 R2. Thus, it doesn’t matter if this threat gets pushed back by the CIP compliance threats; your organization will still have a patch management program since it has to comply with CIP-007 R2.

This is true, but look at what CIP-007 R2 requires you to do: First, every 35 days you need to check with your patch source for every piece of software installed in your ESP to determine whether there is a new security patch available; you have to do this regardless of whether the vendor is likely to issue a security patch, or even whether they have ever issued such a patch. Then you have to determine whether each patch is applicable. Next, for every applicable patch, you have 35 days to either install it (on every ESP device to which it applies) or develop a mitigation plan to address the vulnerability in a different way, pending installation of the patch. Finally, you have to track each mitigation plan and, if it isn’t completed within the timeframe in the plan, get approval of the CIP Senior Manager or delegate for a revision or extension. And, of course, you have to document whatever you did at every step of the way. This means documenting that all of this was done for all the software installed on every device within the ESP.

In practice, this one requirement has proved to be a huge source of potential violations of CIP, especially for large organizations with hundreds or thousands of devices in their ESPs. Unless they have a wonderful system in place to automate a lot of this (and not all of this process can be automated, especially when it comes to “one-off” software packages installed on just a few systems within the ESP), there will almost inevitably be a lot of patches that don’t get identified, installed or mitigated on one or more systems. I know that some organizations are devoting a huge amount of resources to trying to reduce instances of non-compliance with CIP-007 R2 to as low a number as possible, even though they realize they will never eliminate them.

Contrast this with the patch management program your organization would put in place if you didn’t have to comply with CIP-007 R2. You will obviously want to check regularly to determine whether new security patches are available. But do you need to do it every 35 days for every piece of software installed in your ESP, regardless of whether or not the vendor is likely to have a new patch available – or indeed, regardless of whether they have ever released a security patch? And if you only install patches quarterly for some lower-importance software, will the world come to an end?

The point of this example is that, even though it is almost indisputable that you will want to have a patch management program for devices in your ESPs, it is not at all indisputable that you will want to invest the same amount of resources in that program as are required to comply with CIP-007 R2. The difference between the cost of the program you would follow if you didn’t have to comply and the program you are in fact following (or should be following…ahem!) to comply with CIP-007 R2, constitutes another way in which having mandatory, prescriptive CIP standards imposes a “tax” on your cyber security program, making it less effective than it would be in the absence of the current CIP standards. Of course, it would be a huge, very imprecise, exercise to try to determine what that difference would be, for a particular organization. But I think it would be fairly substantial for most organizations.

However, please note that I’m not saying the additional amount you’re spending due to having to comply with CIP-007 R2 – over what you would spend in the absence of this requirement - is “wasted”. It is clearly better for your organization’s cyber security if you are tracking new patches every 35 days and if you are applying all patches (or otherwise mitigating the vulnerabilities) within another 35 days, than if you are accomplishing each of these steps in say three months.

But here’s the point: You could say the same thing about almost anything that can be done in cyber security. If you take more measures and you do them more frequently and for more devices, there will always be some improvement in security. But where does this stop? Instead of 35 days, maybe you should look for new patches every hour, for each of the software packages installed on each device in your ESP. And maybe this requirement should be extended to the IT network as well since – as the original Ukraine attacks showed – a single compromise in the IT network can ultimately lead to the OT network being “owned” by the attackers. This would undoubtedly lead to even better security than does the current wording of CIP-007 R2. Why don’t we change the requirement to read that way?

Of course, if you had unlimited funds to spend on cyber security, you wouldn’t care how frequently you checked for new security patches, or for how many devices. In fact, you would spend money on anything that could lead to even a modest improvement in your security posture. In this case, it doesn’t matter whether CIP prescribes three months, 35 days or one hour as the interval for checking patch availability. With unlimited funds at your disposal, the patch management program you would have in the absence of the current CIP requirements would be no different (or less resource-intensive) than your program to comply with CIP-007 R2. For you, CIP doesn’t act as a “tax” on your cyber program.[vii]

Unfortunately, I don’t know any NERC entity with an unlimited budget for CIP compliance, cyber security, or anything else. In the real world, the second way in which the current (prescriptive) CIP standards impose a “tax” on your cyber security spending, preventing your cyber expenditures from having the positive impact they would have in the absence of the current CIP standards, is the fact that you are required to take steps for patch management that you wouldn’t take were CIP-007 R2 not a mandatory requirement. To put it differently, if you didn’t have to comply with CIP-007 R2, you could still include patch management in your cyber security program, but you wouldn’t have to do it in as expensive a fashion as is required by CIP. This would free up money that could be spent on addressing other areas in which the threats posed are more serious. This same reasoning applies to the other prescriptive CIP requirements, like CIP-010 R1.

Here’s the moral of our story: In the absence of unlimited budgets, the existence of mandatory prescriptive CIP requirements inevitably means that cyber security programs will be less effective than in their absence. This is because a) the entity doesn’t have the funds to address the threats they feel are most important, since they have no choice but to first address CIP compliance, and b) the steps the entity has to take to address threats like software vulnerabilities require a lot of resources that the entity would prefer to spend on higher-priority cyber threats – were it not for the CIP standards.

Can we quantify this “tax” imposed by prescriptive CIP requirements? I don’t think so, either on the level of individual NERC entities or of all of them. But it has to be considerable. And I believe it is much greater than what I used to consider the Number One problem with CIP: the fact that some large percentage of the amount spent goes to “compliance paperwork”, not cyber security. By severely distorting the way an entity would allocate cyber security funds in the absence of prescriptive CIP requirements, this “tax” constitutes the primary problem with the current CIP standards.

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte Advisory.

---

[i] Please keep in mind that any particular CIP compliance activity – say, developing a mitigation plan when a patch can’t be applied, per CIP-007-6 R2 – will always be a mixture of what I call “pure compliance paperwork” and activities that actually do contribute to cyber security, and it will be almost impossible just to look at a description of what was done and say how much of that went to security and how much to pure compliance. So any estimate of the amount being spent on pure compliance will always be very subjective, whether on the level of an individual entity or for the aggregate of NERC entities.

[ii] In doing this, I’m in good company. Einstein was a great user of thought experiments to prove the validity of his theories. In fact, his greatest achievement, the theory of General Relativity, was almost instantly accepted as soon as it was published, even though the first experimental proof only came a few years later, when the British navy measured a slight deviation in a star’s apparent position next to the sun during a total solar eclipse. Einstein supported his theory with some compelling thought experiments; that was all that was required to convince the scientific community of its validity.

[iii] I know some of you are going to jump out of your seats and tell me this is a stupid assumption, since if there were no NERC CIP, you would never be given the same amount of money to spend on cyber security. Where I’m going with this argument is the idea that re-writing the CIP standards in a non-prescriptive format would remove the problems discussed in this post, but it would mandate that NERC entities make a significant investment in cyber security.

[iv] This is a simplification, of course. I think most organizations will want to make sure they have “adequately” addressed each major security domain: patch management, vulnerability assessments, access management, etc. They will do this regardless of the importance of the particular threats affecting that domain. Only after they have done this will they look at the currently identified threats and spend their remaining funds mitigating the most important of those threats. But there should always be some prioritization of expenditures based on threat rankings.

[v] Once again, this is a big simplification. For one thing, it ignores the fact that most organizations address CIP compliance threats from a separate budget than the one for non-CIP cyber threats, so in theory there is never really a conflict between the two. But I contend that at some level – perhaps just the CEO or the Board – someone has to be balancing all of the spending numbers. And they will almost certainly be making trade offs like “Gee, we’ve had to spend a lot more on CIP this year. Since CIP is for cyber security, this means we shouldn’t have to spend so much in our separate cyber budget.” Conversely, if there were no CIP, this person would almost certainly be willing to spend more on cyber security.

[vi] It may seem odd that I’m now assuming away a point I made at great effort earlier in this post. When you do thought experiments, you can make all kinds of strange assumptions. I’m doing this because I want to show that there is a negative consequence of CIP that is even greater than the fact that a large amount of spending goes to compliance paperwork.

[vii] I want to point out now that this example would be quite different if we substituted the anti-malware requirement, CIP-007 R3, for CIP-007 R2. R3 is one of only a few non-prescriptive requirements in CIP v5/v6. It states that entities must “Deploy method(s) to deter, detect, or prevent malicious code.” It doesn’t say you have to use antivirus vs. another method like application whitelisting. It doesn’t say you have to take particular steps to implement or maintain whatever solution(s) you choose. It is up to the entity to decide what is the best way to address the threat posed by malicious code, given their particular environment and the fact that they have a lot of other cyber threats they need to address, not just this one – while they don’t have unlimited funds at their disposal.