“Compliance Paperwork”

I have been saying for a year that the NERC CIP standards, in their current prescriptive format, are unsustainable. Until my last post my number one reason for saying this was that a large portion – perhaps even half - of the effort that NERC entities have to expend in order to comply with CIP goes toward activities that have no security benefit.[i] In my opinion, instituting a non-prescriptive, threat-based approach to CIP would be one way to increase the portion of CIP spending going to security, without requiring a net increase in spending to achieve this result.

In saying this, I always referred to “compliance paperwork” as by far the largest (but not the only) component of this “non-productive” effort. In other words, my proposed solution to CIP’s unsustainability problem would result in a large reduction in paperwork, although it wouldn’t eliminate it, since some compliance paperwork would still be required.

However, the problem with this argument was that I had to admit there is no good way to tell, simply by looking at a particular paperwork activity, whether it is “good” paperwork – which contributes to security and thus would be retained under my proposal – or “bad” paperwork, which doesn’t contribute to security at all. Given this, an entity would have no objective criterion for determining how much of their CIP compliance effort contributes to security; they would just have to take a guess, based on their experience. So I was basing my argument on something that might be called an “inherently unverifiable” fact: This is a fact that can never be proven true or false.

In my last post, I demoted this reason for CIP’s unsustainability from Number One to Number Two. You can read about my new Number One reason in the post already cited, but in short the reason is that the prescriptive CIP requirements force entities to allocate their cyber security spending (both spending of dollars and “spending” of employee time) to activities that provide less security benefit – and often much less – than activities they would otherwise prioritize. In demoting the previous Number One reason to Number Two (but still saying it was a valid reason), I was in effect saying that, even if an entity’s priorities for cyber security would – if CIP were suddenly made non-mandatory - align exactly with the activities mandated by CIP v5 and v6 (of course the chance of this happening is zero), they would still be wasting a lot of effort on activities that had no effect at all on security.

Last week, I spoke in front of the CIP users’ group for one of the NERC Regional Entities about the problems with CIP and my tentative proposal to fix them.[ii] There were a lot of really good questions, and we had a great discussion, in which I probably learned a lot more than my audience did.[iii]

During this discussion, someone expressed skepticism that any CIP compliance paperwork has zero security value; after all, documenting what you do is a good practice – and often required for internal audit purposes – in any activity related to computer systems and networks. I at first replied with my standard answer described above, that there is no way that, simply by looking at a paperwork task, an outside observer could determine that it did or didn’t contribute to security; only longtime compliance or cyber security staff members at the entity itself could make this determination – and that would only be based on gut feel. So this determination will always be inherently unverifiable.

But as soon as I said this, I felt quite uneasy. This was perhaps because, during the week I made this presentation, there was a raging debate in the national press about whether the idea of “alternative facts” was a valid one, or just another way of saying “lies”. And here I was going one step further by asserting that certain facts were true but just could never be verified. If the person who invented the phrase “alternative facts” had instead asserted my concept of “inherently unverifiable” facts, she might not have received all the flak that she encountered – if anything, the members of the press would have started looking through the literature on epistemology to see if “inherently unverifiable facts” might be a valid concept (i.e., can there be a fact that could never be verified? It’s an interesting question. It actually is a big debate in physics today, where proponents of string theory, and also the idea that there are an infinite number of universes, readily admit that these ideas can never be definitively proven true or false).

I was really not comfortable continuing to assert that there is no way to identify paperwork that is required for compliance but doesn’t contribute at all to security. But then I realized there is no reason to continue to make this assertion, since the result is virtually the same - whether these activities don’t contribute at all to security or whether they do contribute but only minimally. The result in both cases will be that a lot of the paperwork required by CIP contributes very little to security. So let me stipulate from here on out that every activity required by CIP contributes in some way to security, although often in a very small way.

Once I admitted that, I realized my Number Two reason why CIP is unsustainable had now gone away and been subsumed into Reason Number One, without requiring that I change how I articulate that reason at all. As I said above (and in my last post), the Number One problem with the CIP requirements is that they cause entities to use their limited cyber security budgets to carry out security mitigation activities that would otherwise have a very low priority – if the entity were free to do what it thought was best.[iv] Since no NERC entity – at least none that I know of – has an unlimited cyber security budget, this results in the most important cyber threats (based on the current threat landscape in North America,[v]) going either unmitigated or inadequately mitigated.

To summarize this post, I no longer believe that there are activities – which I’ve previously called “pure compliance paperwork” - that are required by the CIP standards but contribute nothing to cyber security. Every activity required by CIP contributes in some way to security, but a lot of these activities make a very small contribution. I am making a proposal that would rewrite CIP to require that NERC entities prioritize the activities that contribute the most to BES[vi] cyber security, without prescriptively saying that certain activities are required, no matter how little they advance the goal of securing the bulk electric system.

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte Advisory.

---

[i] I based this statement on informal discussions I’ve had with various NERC entities, not on any sort of formal poll.

[ii] I prefaced my remarks by pointing out that I am working, with two co-authors, on a book that will lay out this proposal, among other things. We expect to have it out later this year.

[iii] I will probably have another post inspired by this discussion soon.

[iv] You may cringe when you hear me say that the CIP standards shouldn’t unnaturally constrain NERC entities from allocating their limited cyber security budgets as they “think best”. You may point out that a) a lot of, or even most, organizations still believe that what is best as far as cyber security goes is to spend as little on it as possible; and b) even if an entity realizes it must spend a substantial amount on cyber, it won’t necessarily spend it in an optimal way, due perhaps to a lack of understanding of cyber security principles and practices.

Both of these objections can be answered by pointing out that my “proposal” for rewriting CIP will require the entity (or a third party) to assess its security posture with respect to various security domains (software vulnerability mitigation, anti-phishing, change control, etc.) and develop a plan for mitigating the most important deficiencies identified. This plan will have to be reviewed by a competent outside party, which might be a consulting firm or the entity’s NERC Region; this process is similar to the one now mandated by CIP-014.  I am currently leaning toward the idea that the Regions themselves should do this review. I realize they don’t currently have the manpower to review all of these plans. That will hopefully change, but even then the Regions will probably still have to hire outside resources, at least to address temporary overloads. But since otherwise the entities would have to engage their own consultants for this task, and there would be the potential for some consulting firms to go easier on the entity in exchange for being engaged to do the not insubstantial job of implementing the mitigation plan (in fact, this is the biggest problem I see with the PCI standards for payment card security, since the PCI standards are audited by assessors paid by the retailer being audited, who are then allowed to be engaged to mitigate the problems that they identify. They have lots of incentive to downplay the problems in the official report, since they know it will make the retailer look good), I still think it’s better for the Region to do it. While having the Regions do it will probably require an increase in the assessments paid by each entity, the entities will hopefully see that this simply replaces an amount they would otherwise have to spend themselves.

Having the Region review an entity’s assessment and mitigation plan will address both of the objections shown above. If the entity happens to think that their cyber security posture is just great and there’s no need to spend much more money on cyber, or if the entity’s mitigation plan will spend too much on unimportant tasks and too little on important ones, the Region will be able to order the entity to revise all or parts of their plan. And they will be regularly audited (perhaps even once a year) on how well they are carrying out that plan.

[v] My proposal for rewriting CIP – and specifically the one I and my co-authors will outline in our upcoming book – will require that the team that drafts the new standards identify the primary cyber threats to the North American bulk electric system. The entity will be required to address each of those threats in some way, either to mitigate deficiencies in their defenses that are identified in an assessment, or to document why a particular threat doesn’t apply to it. However, since the threat landscape changes very rapidly (e.g., phishing came out of nowhere about five years ago to become probably the most serious cyber threat today, and the origin of more than half of successful cyber attacks in recent years), there needs to be some way of continually updating this threat list. I am proposing that there be a NERC committee which meets at least quarterly to a) assess new threats and determine whether or not they should be added to the list; b) determine whether any threats currently on the list should be removed; and c) write and update guidance on best practices for mitigating these threats.

In addition, since some threats only apply to particular types of entities or particular regions of the country, there will always be threats that an entity faces, that aren’t included in the “NERC-wide” list just described. It will be up to the entity to make sure these particular threats are also addressed, and it will be up to the NERC Region to verify that the entity’s mitigation plan adequately addresses these threats.

[vi] Note that, in my proposal, the CIP standards will still be focused entirely on BES security. Every NERC entity has other cyber security goals: protecting financial data, protecting customer information, etc. These also need to be addressed, but CIP has no bearing on these. In other words, under my proposal the entity will need two cyber security budgets: the budget to address BES threats and the budget to address all other cyber threats.  

The Biggest Problem with NERC CIP

Since at least last February, I have been pushing the idea that the CIP standards are on an unsustainable course and need to be radically rewritten. I have mentioned several reasons for this, but I have said the most important is cost, since the amount that NERC entities are spending on CIP compliance grew significantly with the advent of CIP v5 and v6. Plus it will inevitably increase even more significantly due to the new Supply Chain standard (CIP-013), to CIP v7 and to other scope increases still likely to come, which include virtualization, the cloud, machine-to-machine communications and who knows what else.

However, just the fact that NERC entities are spending a lot of money on CIP – with a lot more to come – isn’t in itself the issue; after all, the whole idea of CIP was to get entities to invest in cyber security to an extent they couldn’t be counted on to do without facing a mandatory requirement. The problem – as I’ve discussed in several posts, including the one just referenced - is that so much of this spending (including “spending” on staff time, of course) isn’t going to cyber security but to what I’ve been calling “compliance paperwork”. The estimates I have received of the portion being so spent seem to center on 50%, meaning literally half of what entities are spending on CIP compliance nowadays is not contributing to cyber security. Even if the actual figure is less than that, it is still a big number when you consider that total spending (including employee time invested) on CIP v5/v6 compliance has been well into the billions of dollars.[i]

I haven’t ever really discussed what I mean by “compliance paperwork”. For one, it definitely includes the documentation that an entity has to do to prove they did things for compliance purposes – as opposed to the documentation that entities would do in the absence of a mandatory requirement, simply because it is a good business and security practice. It also includes a lot of the self-reports, mitigation plans, compliance certifications, etc. that are required to comply with CIP. And it certainly includes a lot of the effort that is required to prepare for and get through an audit.

The idea that so much of current CIP spending isn’t going to security leads directly to the question “Could a better design of the standards reduce the amount of spending going solely to compliance ‘paperwork’ – meaning the same amount of total spending on CIP would lead to a much larger investment in actual cyber security?” My answer to this has been an emphatic yes, assuming the standards were made much less prescriptive. In other words, I have been saying it is possible to write much less-prescriptive CIP standards whose implementation will lead to a much greater amount of investment in cyber security, even if the overall level of CIP spending were unchanged.

However, a recent conversation made me realize that, while this is a very important reason to rewrite CIP, there is a more important one. To understand this new reason (which I have probably alluded to previously but not explicitly called out, mainly because I had been thinking it was really a part of the “compliance paperwork” reason just discussed), I need you to do a “thought experiment[ii]” with me: Say that you annually have available to you an amount CYBER, which is your organization’s current expenditures on cybersecurity. In addition, you have another amount CIP, which is your expenditures on NERC CIP compliance. Now suppose the NERC CIP standards suddenly go away, but the total amount available to you to spend remains the same, as long as you spend it all on cyber security[iii]. In other words, you can now spend the entire amount you would have spent on CIP on cyber security, in addition to the amount you were planning to spend on (non-CIP) cyber anyway; let’s call this total amount SUM(CIP+CYBER). How would you allocate this total sum? I’m guessing you would do it the way hopefully most organizations do for all of their security spending: You would first identify the threats that affect your organization and estimate the costs to mitigate each one (which of course isn’t the same as eliminating those threats. That’s impossible, at least for most threats). Then you would rank these threats by their potential impact. Finally, you would budget the available funds so that they mitigated the most important threats.[iv]

At this point, you have a list that reads “Threat A has the largest potential impact and it would cost $Y to mitigate; we’ll definitely address that, since this is well below our total budget. Threat B has the second largest impact, and it would cost $Z to mitigate. $Y + $Z is still below our total budget, so we will do both of these…..” You would continue this process until you have exhausted your budget for the year (while still leaving some for contingencies, of course!). Of course, your total budget for cyber security is SUM(CIP+CYBER), since there are no longer any CIP requirements.

Now let’s say NERC CIP suddenly reappears (Don’t ask me why. I told you this was just a thought experiment!), and you have to comply with the current CIP standards, as well as address cyber security threats. What happens to your calculations? All of a sudden, you have a bunch of new compliance threats that need to be mitigated: the threat of non-compliance with CIP-004 R3, non-compliance with CIP-007 R2, etc. But your original cyber threats are still there and also need to be mitigated. How do you now prioritize your spending (and remember, you still have SUM(CIP+CYBER) available to mitigate both CIP and non-CIP cyber threats)? Can you look at a CIP compliance threat, like say the risk of non-compliance with CIP-010 R1, and rank that ahead of say the threat posed by phishing attacks – a cyber risk that isn’t addressed at all in CIP?

Here’s the correct answer: You certainly can do that. In fact, I believe most NERC entities would almost never prioritize any non-CIP cyber threat over the threat of non-compliance with a CIP requirement. Not only can there be huge fines for ignoring CIP requirements, but most NERC entities will do almost anything to avoid being fined at all because of damage to reputation, impact on rate cases, etc. In fact, I think there are few NERC entities that would passively accept a CIP fine, even if the “penalty” were a trip to Disney World. This means that the threats of non-compliance with CIP requirements will assume a special priority ahead of probably all the other cyber threats. In essence, your organization will need to make sure it has addressed all of its CIP non-compliance threats before it spends very much to address non-CIP cyber threats.[v]

So why is this bad? As I’ve just said, some significant percentage of what your organization spends on CIP compliance doesn’t go to security at all but simply to proving compliance. This alone results in the amount you are spending on CIP being less effective in advancing cyber security than would be the same amount of spending in the absence of CIP. So the fact that you have to prioritize CIP spending means that a lot of your total spending - that is, some percentage of the quantity SUM(CIP+CYBER) – goes to compliance paperwork, rather than cyber security. This might not be 50 percent, but it is certainly significant.

I used to think this was the biggest problem with CIP, but I no longer do. To see the biggest problem, let’s go back to our thought experiment and see what happened to your spending priorities when CIP was re-imposed. Before that happened (during the period of time when CIP went away but you still had the money that had been allocated for CIP, along with the non-CIP cyber budget), you were allocating your entire cyber security budget based strictly on the impact level of each threat: the threats with the largest impact were first in line to receive mitigation money, those with the next-largest impact were next in line, etc.

When CIP was re-imposed, you had to change that. You essentially “prioritized” the threat of non-compliance with each CIP requirement to put these compliance threats ahead of the “pure” cyber threats. For example, your number one cyber threat (say, phishing, which was recently said to be the immediate cause of 91% of cyber attacks) might get pushed from number one down to number 44 (or so) on the list of threats to be addressed – because the threats of non-compliance with each of the CIP requirements all took their place ahead of it. The bottom line: you will be able to address far fewer non-CIP cyber threats than you could have before CIP was re-imposed.

A Rude Interruption

Here you might rudely interrupt me and say “Even though we’ve had to prioritize CIP expenditures over other cyber expenditures, CIP is just cyber anyway. This means we’re still devoting our entire budget (which equals SUM(CIP+CYBER), of course) to cyber security. If we put aside for the moment the fact that a certain percentage of CIP compliance spending is “wasted” on compliance paperwork,[vi] there is really no impact due to the fact that CIP was re-imposed; we’re still spending 100% of the budget on cyber security.”

What’s wrong with this statement? It ignores the fact that what you are spending your money on has drastically changed, in two ways. The first is that the threats you prioritized during the period when there was no CIP have now been pushed way down the list of threats to be addressed, since the threats of non-compliance with the different CIP requirements have all been pushed ahead of them. In fact, probably a lot of the non-CIP cyber security threats you would like to mitigate will now receive no funding. For example, even though phishing was your number one threat to address before CIP was re-imposed, it is now your number 44 threat and you’re going to have to fight hard to get even a modest amount of money allocated to it. In other words, even though the total amount you’re spending hasn’t gone down, it no longer is based on your prioritization of the threats. While this effect can’t be quantified, it is certainly quite significant – it acts something like a tax on your cyber security program.

Let's move to the second way in which what you’re spending money on has changed with the re-imposition of CIP; the best way to describe this is with an example. I think everyone agrees that the threat posed by security vulnerabilities in software should rank among the most important cyber threats; this means that patch management should form a big part of every entity’s cyber security program. Of course, CIP-007 R2 is the patch management requirement in CIP v5 and v6. In our thought experiment, when CIP is reimposed the threat of non-compliance with CI-007 R2 (and all of the other CIP requirements) will be prioritized ahead of all the non-CIP cyber threats like phishing.

But wait! You point out that one of the top cyber threats on your list, during the time when CIP went away, was software vulnerabilities. Patch management is probably the most important way to mitigate this threat; it also happens to be what is required by CIP-007 R2. Thus, it doesn’t matter if this threat gets pushed back by the CIP compliance threats; your organization will still have a patch management program since it has to comply with CIP-007 R2.

This is true, but look at what CIP-007 R2 requires you to do: First, every 35 days you need to check with your patch source for every piece of software installed in your ESP to determine whether there is a new security patch available; you have to do this regardless of whether the vendor is likely to issue a security patch, or even whether they have ever issued such a patch. Then you have to determine whether each patch is applicable. Next, for every applicable patch, you have 35 days to either install it (on every ESP device to which it applies) or develop a mitigation plan to address the vulnerability in a different way, pending installation of the patch. Finally, you have to track each mitigation plan and, if it isn’t completed within the timeframe in the plan, get approval of the CIP Senior Manager or delegate for a revision or extension. And, of course, you have to document whatever you did at every step of the way. This means documenting that all of this was done for all the software installed on every device within the ESP.

In practice, this one requirement has proved to be a huge source of potential violations of CIP, especially for large organizations with hundreds or thousands of devices in their ESPs. Unless they have a wonderful system in place to automate a lot of this (and not all of this process can be automated, especially when it comes to “one-off” software packages installed on just a few systems within the ESP), there will almost inevitably be a lot of patches that don’t get identified, installed or mitigated on one or more systems. I know that some organizations are devoting a huge amount of resources to trying to reduce instances of non-compliance with CIP-007 R2 to as low a number as possible, even though they realize they will never eliminate them.

Contrast this with the patch management program your organization would put in place if you didn’t have to comply with CIP-007 R2. You will obviously want to check regularly to determine whether new security patches are available. But do you need to do it every 35 days for every piece of software installed in your ESP, regardless of whether or not the vendor is likely to have a new patch available – or indeed, regardless of whether they have ever released a security patch? And if you only install patches quarterly for some lower-importance software, will the world come to an end?

The point of this example is that, even though it is almost indisputable that you will want to have a patch management program for devices in your ESPs, it is not at all indisputable that you will want to invest the same amount of resources in that program as are required to comply with CIP-007 R2. The difference between the cost of the program you would follow if you didn’t have to comply and the program you are in fact following (or should be following…ahem!) to comply with CIP-007 R2, constitutes another way in which having mandatory, prescriptive CIP standards imposes a “tax” on your cyber security program, making it less effective than it would be in the absence of the current CIP standards. Of course, it would be a huge, very imprecise, exercise to try to determine what that difference would be, for a particular organization. But I think it would be fairly substantial for most organizations.

However, please note that I’m not saying the additional amount you’re spending due to having to comply with CIP-007 R2 – over what you would spend in the absence of this requirement - is “wasted”. It is clearly better for your organization’s cyber security if you are tracking new patches every 35 days and if you are applying all patches (or otherwise mitigating the vulnerabilities) within another 35 days, than if you are accomplishing each of these steps in say three months.

But here’s the point: You could say the same thing about almost anything that can be done in cyber security. If you take more measures and you do them more frequently and for more devices, there will always be some improvement in security. But where does this stop? Instead of 35 days, maybe you should look for new patches every hour, for each of the software packages installed on each device in your ESP. And maybe this requirement should be extended to the IT network as well since – as the original Ukraine attacks showed – a single compromise in the IT network can ultimately lead to the OT network being “owned” by the attackers. This would undoubtedly lead to even better security than does the current wording of CIP-007 R2. Why don’t we change the requirement to read that way?

Of course, if you had unlimited funds to spend on cyber security, you wouldn’t care how frequently you checked for new security patches, or for how many devices. In fact, you would spend money on anything that could lead to even a modest improvement in your security posture. In this case, it doesn’t matter whether CIP prescribes three months, 35 days or one hour as the interval for checking patch availability. With unlimited funds at your disposal, the patch management program you would have in the absence of the current CIP requirements would be no different (or less resource-intensive) than your program to comply with CIP-007 R2. For you, CIP doesn’t act as a “tax” on your cyber program.[vii]

Unfortunately, I don’t know any NERC entity with an unlimited budget for CIP compliance, cyber security, or anything else. In the real world, the second way in which the current (prescriptive) CIP standards impose a “tax” on your cyber security spending, preventing your cyber expenditures from having the positive impact they would have in the absence of the current CIP standards, is the fact that you are required to take steps for patch management that you wouldn’t take were CIP-007 R2 not a mandatory requirement. To put it differently, if you didn’t have to comply with CIP-007 R2, you could still include patch management in your cyber security program, but you wouldn’t have to do it in as expensive a fashion as is required by CIP. This would free up money that could be spent on addressing other areas in which the threats posed are more serious. This same reasoning applies to the other prescriptive CIP requirements, like CIP-010 R1.

Here’s the moral of our story: In the absence of unlimited budgets, the existence of mandatory prescriptive CIP requirements inevitably means that cyber security programs will be less effective than in their absence. This is because a) the entity doesn’t have the funds to address the threats they feel are most important, since they have no choice but to first address CIP compliance, and b) the steps the entity has to take to address threats like software vulnerabilities require a lot of resources that the entity would prefer to spend on higher-priority cyber threats – were it not for the CIP standards.

Can we quantify this “tax” imposed by prescriptive CIP requirements? I don’t think so, either on the level of individual NERC entities or of all of them. But it has to be considerable. And I believe it is much greater than what I used to consider the Number One problem with CIP: the fact that some large percentage of the amount spent goes to “compliance paperwork”, not cyber security. By severely distorting the way an entity would allocate cyber security funds in the absence of prescriptive CIP requirements, this “tax” constitutes the primary problem with the current CIP standards.

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte Advisory.

---

[i] Please keep in mind that any particular CIP compliance activity – say, developing a mitigation plan when a patch can’t be applied, per CIP-007-6 R2 – will always be a mixture of what I call “pure compliance paperwork” and activities that actually do contribute to cyber security, and it will be almost impossible just to look at a description of what was done and say how much of that went to security and how much to pure compliance. So any estimate of the amount being spent on pure compliance will always be very subjective, whether on the level of an individual entity or for the aggregate of NERC entities.

[ii] In doing this, I’m in good company. Einstein was a great user of thought experiments to prove the validity of his theories. In fact, his greatest achievement, the theory of General Relativity, was almost instantly accepted as soon as it was published, even though the first experimental proof only came a few years later, when the British navy measured a slight deviation in a star’s apparent position next to the sun during a total solar eclipse. Einstein supported his theory with some compelling thought experiments; that was all that was required to convince the scientific community of its validity.

[iii] I know some of you are going to jump out of your seats and tell me this is a stupid assumption, since if there were no NERC CIP, you would never be given the same amount of money to spend on cyber security. Where I’m going with this argument is the idea that re-writing the CIP standards in a non-prescriptive format would remove the problems discussed in this post, but it would mandate that NERC entities make a significant investment in cyber security.

[iv] This is a simplification, of course. I think most organizations will want to make sure they have “adequately” addressed each major security domain: patch management, vulnerability assessments, access management, etc. They will do this regardless of the importance of the particular threats affecting that domain. Only after they have done this will they look at the currently identified threats and spend their remaining funds mitigating the most important of those threats. But there should always be some prioritization of expenditures based on threat rankings.

[v] Once again, this is a big simplification. For one thing, it ignores the fact that most organizations address CIP compliance threats from a separate budget than the one for non-CIP cyber threats, so in theory there is never really a conflict between the two. But I contend that at some level – perhaps just the CEO or the Board – someone has to be balancing all of the spending numbers. And they will almost certainly be making trade offs like “Gee, we’ve had to spend a lot more on CIP this year. Since CIP is for cyber security, this means we shouldn’t have to spend so much in our separate cyber budget.” Conversely, if there were no CIP, this person would almost certainly be willing to spend more on cyber security.

[vi] It may seem odd that I’m now assuming away a point I made at great effort earlier in this post. When you do thought experiments, you can make all kinds of strange assumptions. I’m doing this because I want to show that there is a negative consequence of CIP that is even greater than the fact that a large amount of spending goes to compliance paperwork.

[vii] I want to point out now that this example would be quite different if we substituted the anti-malware requirement, CIP-007 R3, for CIP-007 R2. R3 is one of only a few non-prescriptive requirements in CIP v5/v6. It states that entities must “Deploy method(s) to deter, detect, or prevent malicious code.” It doesn’t say you have to use antivirus vs. another method like application whitelisting. It doesn’t say you have to take particular steps to implement or maintain whatever solution(s) you choose. It is up to the entity to decide what is the best way to address the threat posed by malicious code, given their particular environment and the fact that they have a lot of other cyber threats they need to address, not just this one – while they don’t have unlimited funds at their disposal.

Are you Attending Distributech?

If you’re planning on attending Distributech in San Diego the week of January 30^th, I hope you’ll visit with Deloitte. We’ll be showcasing the extensive work we do in Grid Modernization (what the Smart Grid is called nowadays, in case you didn’t get that memo), and in securing power transmission, distribution and generation environments. We’re in booth 1807.

If you will be in San Diego on Monday night the 30^th, I hope you’ll attend our reception that evening and enjoy some microbrews. You can register for it here (please also drop me an email at talrich@deloitte.com so I’ll know to look for you there). And if you would like a free show floor pass, send me an email and I’ll send you the link to get one.

Hope to see you there!

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte Advisory.

A Tale of Two SDTs

For the first time, there are now two – count ‘em! – NERC drafting teams working on new or revised CIP standards at the same time; in December I attended meetings of both teams on adjacent weeks. The Supply Chain Security SDT, which is drafting a new standard labeled CIP-013, met near Philadelphia during the first week of December, while the “CIP Modifications” SDT (which I insist on also calling the CIP v7 SDT) met in Orlando the following week. While I didn’t anticipate this, it turns out that the contrast between the two meetings provided a valuable lesson on the future of the CIP standards.

I attended both meetings because I hadn’t been at a meeting of either team since I attended a v7 SDT meeting in August (the v7 SDT has met monthly since the spring of 2016, although the Supply Chain SDT has only had three face-to-face meetings so far, one of which was actually a Technical Conference in November). The two drafting teams are very different, and they have very different agendas. I want to briefly characterize the two teams; then I want to ask you a question.

Our Two Protagonists

I start with Team One, the CIP v7 drafting team: This team is composed of very experienced cyber security and compliance professionals from a cross section of NERC entities (although almost all are from fairly large entities). Close to half the team members have experience on previous CIP drafting teams, and at least two members also served on the two teams that developed CIP versions 2, 3, 4, 5 and 6 – meaning these two individuals have been on CIP drafting teams for close to eight years each. All of the team members seem to understand both cybersecurity and how the NERC standards development process works, and they are keenly aware of what is in their mandate and what isn’t. Their mandate includes a number of weighty items, and the team is well behind their original schedule for having first drafts of all the changes or additions they will be proposing, but I think most team members – as well as most knowledgeable observers in the NERC community – would say this SDT has things well under control. After all, they in theory aren’t required to break any radically new ground in their work (which is why they’re called the “CIP Modifications” team). I have not heard any of the team members say directly that they see any major problems looming that would prevent their being able to fulfill their mandate.

Now Team Two: The Supply Chain drafting team is composed primarily of supply chain professionals from large entities who have little experience in the often inscrutable ways of NERC and FERC, and who don’t all know a lot about the current and previous CIP standards. Besides that, they face some huge obstacles. The first obstacle is the fact that they are developing probably the first non-defense mandatory supply chain security standard ever, and FERC provided fairly minimal guidance on what they wanted to see in it. The second is that FERC gave this SDT only one year to develop the standard, get it approved by the NERC ballot body (which normally seems to always take four ballots with a minimum of 3-4 months per ballot) and the NERC Board of Trustees, and finally put it on FERC’s desk. It is highly unlikely that a request for an extension of time would be granted, absent some unforeseen act of God.

Here’s my question: Which of these two teams do I believe is likely to fulfill the tasks in their Standards Authorization Request (SAR) very easily, and which do I see as probably never finishing a lot of what’s in their SAR? Of course, the answer is….Team Number Two should fulfill their tasks quite easily despite their very short deadline, but I have a serious doubt that Team Number One will ever finish the tasks in their SAR, even though all but one or two of those tasks don’t have any FERC-supplied deadline.

Of course, this assertion seems fairly counter-intuitive, given how I just described the two teams. Why do I say this? I do want to point out up front that this is in no way a reflection on the members of either drafting team. All of the members of both teams seem to me to be very dedicated and hardworking, and ready to do whatever it takes to accomplish the task in front of them. But I’ve come to the conclusion that the CIP v7 team has been given a Sisyphean task that they will never finish, while the Supply Chain team – despite the woefully inadequate time they have been given to draft and finalize the new standard – will almost certainly finish on schedule.

The reason the prognoses for the two teams are so different can be found in the nature of the standards they’re revising or writing. In Order 829, FERC made it clear that they wanted the new supply chain security standard to be non-prescriptive[i]; in other words, it would address just the objectives to be achieved, not the specific means of achieving them (the “what”, not the “how”). For this reason, the requirements of CIP-013 primarily lay out objectives, such as

1.1.1.       Identify and assess risk(s) during the procurement and deployment of vendor products and services; and

1.1.2.       Evaluate methods to address identified risk(s).[ii]

In contrast, the CIP Modifications SDT has to make changes (and in some cases additions) to the existing requirements of CIP v5 and v6. Many of these existing requirements tell entities how to comply with their objectives, such as this part of CIP-007-6 R2 (patch management):

                           2.2        At least once every 35 calendar days, evaluate security patches for

                                           applicability that have been released since the last evaluation from the

                                           source or sources identified in Part 2.1.

The result of this difference is that the Supply Chain Security SDT’s job is much simpler than the CIP Modifications SDT’s job. The Supply Chain team doesn’t have to draft and ballot requirements that describe the specific means that must be used to achieve the objectives in their standard[iii], while the CIP Modifications SDT will have to spend a lot of time drafting, balloting and commenting on precisely these issues – that is, which particular means entities will need to use to fulfill the goals of the prescriptive requirements the SDT is writing or revising, and why those particular means were chosen rather than others.

Of course, the Supply Chain SDT will have to provide guidance on means that a NERC entity can use to achieve the objectives in their (non-prescriptive) requirements; they will do this in the Guidance and Technical Basis discussion that will accompany CIP-013. But that’s all this will be – guidance. In the end, it will be up to the entity to determine the best way for it to meet the objectives listed in the requirements of CIP-013, and for the regional auditor to determine how well they have met those objectives.

The CIP-013 requirements themselves won’t tell the entity specifically what they need to do to meet the objectives. As a result, the primary source of debate in the original CIP v5 and v6 drafting teams, and to a large degree in the current v7 team, is for the most part absent in the meetings of the Supply Chain team. There is no need to debate prescriptive requirement language, when FERC itself has said they don’t want prescriptive requirements.

Now let’s go back to Team One, the CIP v7 team. Their job is not to create new standards de novo; it is to modify the existing CIP v5 and v6 standards[iv] to meet a certain set of objectives  set out in their Standards Authorization Request (SAR). Three of these objectives were set by FERC in Order 822; the remainder were set by NERC when they finally realized there was no way they could provide binding “interpretation” on ambiguous areas of CIP v5 and v6 other than by either revising the standards, which requires a bare minimum of three years before the revised standards will come into effect, or responding to a Request for Interpretation, which can take almost as long.

So if Team One only has to modify the existing CIP v5 and v6 standards, why is their job so much harder than that of Team Two, which has to create a new standard literally from scratch (and without any previous standards from NERC or any other organization - other than perhaps DoD - to look to for guidance)? It is because almost all of the requirements of CIP v5 and v6 are prescriptive. If they weren’t prescriptive, most of the changes Team One is charged with making could be made just by modifying guidance (and in some cases also adding some new definitions, as in the case of the team’s mandate to incorporate virtualization into CIP). If the v5 and v6 requirements all read like the draft CIP-013 requirement parts quoted above or like CIP-007-6 R3 or CIP-010-2 R4 (both of which are non-prescriptive requirements in CIP v6), Team One’s job would be much easier than it is.

An Example of What I’m Talking About

For example, consider the change the CIP Modifications SDT successfully balloted last year in Section 3.2 of Attachment 1 of CIP-003-6. In Order 822, FERC mandated that NERC clarify the meaning of the word “direct” in the LERC definition. The v6 version of Section 3.2 prescriptively required that, in cases where LERC was present, a LEAP needs to be implemented. FERC was concerned that the ambiguity regarding “direct” was allowing some entities to take an overly broad view of the measures they could adopt to “break” LERC. As it turns out, the SDT incorporated the LERC definition (without the word “direct”) into the requirement itself and made the requirement non-prescriptive, so that it now simply directs the entity to take appropriate measures to mitigate the risk posed by the presence of external routable connectivity to BES Cyber Systems at Low impact assets; the guidance includes many examples of measures that can be taken to accomplish this objective.[v]

Making this change has taken most of the SDT’s attention since at least last May, even though there is a lot more on its plate. The SDT had to initially just focus on LERC because FERC set a deadline of next March for this change to be drafted, balloted, approved by the NERC Board and submitted to FERC. This is the only change ordered by FERC in Order 822 that has a deadline associated with it; naturally, the SDT had to put off everything else until this was done. As a result, I believe almost all of the time spent in the full SDT meetings since about June (and a significant portion of the time spent in the intervening phone meetings) has been devoted to drafting and re-drafting the new requirement, as well as working on guidance and responding to comments.

It isn’t completely true to say that the SDT has spent eight or nine months just addressing the LERC issue and getting the revised requirement approved by the NERC ballot body. There has been some subcommittee work on other areas of their mandate such as virtualization, but all of this is simply in preparation for when the full team can turn their attention back to these topics. It is now very likely that the SDT will not be able to turn to these other topics in earnest until their February meeting.[vi] This is in spite of the fact that the team originally planned to have finished the first draft of the revised standards, addressing all of the items in their SAR, by the end of 2016.

I will freely admit that I found the fact that the SDT had to spend so much time on the LERC issue – including having to draft the new requirement twice and answer a huge set of questions on both drafts – to be discouraging. What was most discouraging about this was that the requirement that was finally approved by the NERC ballot body didn’t require NERC entities to change how they comply with this requirement at all (as I discussed in this post). On the contrary, the revised requirement allows much more flexibility in how entities can comply, without taking away any options they had under the original wording. Yet it still took about eight or nine months of the SDT’s time to draft and sell the new approach to the NERC membership[vii]! How long will it take to sell a revised requirement that actually does require the entity to change what they do to comply, or to sell a new requirement that imposes a whole new compliance obligation on them?

I will discuss that question in a moment. However, please keep in mind that I’m not arguing that the fact that the CIP v7 SDT had to spend much longer than they’d hoped on the LERC issue is why they are unlikely ever to be able to fulfill the full mandate in their SAR. A setback of say 3 or 4 months (assuming that the LERC issue should just have taken about four months of their time, not seven or eight) is unfortunate but certainly no reason to despair. What I am saying is that, if changing one requirement can take this long, changing say 15 or 20 requirements will take something of the order of ten years or more – which is definitely a huge threat to the SDT ever fulfilling their mandate. And I believe that just one of the items on the SDT’s plate – virtualization – could very well take that long.

How Long Will it Take to Address Virtualization?

I’ve written several posts (such as this and this one) on the problem of virtualization and NERC CIP (as well as participated in one webinar on that topic). The big problem is that the CIP standards (and associated definitions) don’t take any account of virtualization now – either server, switch or storage virtualization; that is, they don’t provide any guidance at all on how NERC entities can safely deploy virtualization in their ESPs. This might be interpreted by some as a license to do whatever you want, since technically, for example, a VM isn’t a Cyber Asset (defined as a programmable electronic device) and thus can’t be a BCA/BCS. Since CIP v5/v6/v7 apply only to BES Cyber Systems and related devices, this means that strictly speaking VMs are of no concern at all to CIP[viii], and therefore NERC entities should be able to pretty much do what they want when it comes to implementing virtualization. In practice, entities who are afraid to push the envelope have simply not deployed virtualization within their ESPs, thus depriving themselves (and their ratepayers) of the many benefits that can be gained from the technology.

Recognizing this problem, the CIP v5 Transition Advisory Group included virtualization among a list of 5 or 6 topics that were included in the SAR for the current SDT, developed early in 2016. However, as I discussed in this post, the v5TAG only indicated that revisions would be required for CIP-005 and the definitions of Cyber Asset and ESP. I thought that this was being overly optimistic and suggested one other change that had been suggested by a TRE webinar. But I also believed that it would not require a lot of changes to requirements to properly incorporate virtualization into the current CIP framework. As it turns out, both the v5TAG and I were wrong.

One of the first things the current SDT did when they started meeting last spring was to appoint a sub team to study what needs to be changed in CIP v5 and v6 to accommodate virtualization. This group decided there were potentially a lot of requirements or definitions that either need to be developed from scratch or revised. But rather than just identify these willy-nilly, the group rightfully decided they should go back and review the various security standards and frameworks, as well as probably other sources, to determine what are the security threats that can be posed by the use of virtualization in control system environments. Once they have done that, they will be able to determine what requirements or definitions need to be either developed or modified.

I know this sub team has developed a spreadsheet listing the different threats that virtualization poses, as well as what might be new or modified requirements or definitions that would address those threats; I know they’re eagerly awaiting the full SDT’s attention (after they have finally put LERC to bed), so that they can begin discussing these. I’ve heard they will first need to get the full team’s agreement on certain new concepts, which will have to be drafted – and voted on multiple times, of course - as new NERC Glossary terms. Once there is agreement on those concepts, then the work on the individual requirements – new or revised - can begin.

So here’s the punchline: I saw a preliminary version of this group’s spreadsheet last summer, and I can say there might easily be 15-20 (or even more) new or revised definitions or requirements needed to incorporate server, switch and storage virtualization into NERC CIP. Now let’s go back to the LERC discussion above: As I said, just this issue – i.e. a change to one requirement - took up more than six months of the SDT’s time, and it really didn’t involve any change to what entities need to do to comply with the requirement in question. Given how new the whole virtualization issue is to CIP, it is likely that each of the new or revised definitions and requirements, that will be required to incorporate server, storage and switch virtualization into CIP, will be controversial. Each one could easily take six months or more of the SDT’s time, by the time they are drafted, posted and balloted, redrafted (given that almost nothing is likely to pass on the first ballot, and perhaps not the second ballot, either) and finally submitted for approval.

So let’s do some math. If there are 20 new or revised requirements or definitions needed to accommodate virtualization in CIP, and if each of those takes a minimum of six months to draft, post, respond to comments, re-draft, re-post, respond to new comments – and potentially repeat this cycle one or two more times – we are talking about a minimum of ten years just to deal with virtualization. And by the way, virtualization isn’t the only thing remaining on the SDT’s agenda[ix]!

Something is going to have to give here. One possibility is that the SDT will decide their mandate is just to address the three items that FERC ordered NERC to address in Order 822. Since one of those is LERC (already approved, at least by the membership) and another is transient cyber assets used at Low impact assets (which will also probably be approved soon), this “only” leaves the mandate to develop requirements to protect communications between control centers on the SDT’s agenda (although see end note ix regarding this mandate). That may cut down the time the SDT needs to address its FERC mandate to maybe about three years, but it does pose the problem that virtualization (and the other items that were added to the SAR by the NERC v5TAG) will be left in a kind of CIP no-man’s-land. Virtualization will continue to be a set of technologies that NERC entities would love to implement in their ESPs (especially in control centers), but only the bravest will feel comfortable actually implementing it – i.e. just like things stand now.

And Now, Our Exciting Conclusion

The bottom line of this very long discussion (one of my longest posts, but not the longest) is that I don’t see any way that virtualization will be officially incorporated into NERC CIP. It might be unofficially “incorporated” through some guidance document that NERC might put out, but as I’ve said recently, no guidance document will have the status of actual modifications to the CIP standards.

But I’ll go further: I don’t see any way that the existing CIP standards can be extended to incorporate any new technology from now on; the standards development process is far too cumbersome to allow this. This applies especially to the cloud, which is once again a technology that only the bravest of NERC entities have implemented in their ESPs so far (and NERC entities need to be much more cautious about implementing the cloud than they have been even about implementing virtualization, as discussed in this recent post). It also applies to other new areas for CIP, like phishing (the initial cause of 91% of successful cyber attacks, according to one recent article, yet something that isn’t addressed in CIP at all. Indeed, I don’t think the term was even invented when the SDT that drafted CIP v2 through v5 first met in the fall of 2008).

What’s the way out of this problem? We clearly can’t leave CIP in its current never-never land. NERC entities can become more efficient and save ratepayer dollars if they virtualize, but for now this can only be accomplished through an elaborate system of winks and nods with NERC and your regional entity.

You can find the answer to this question (regarding the way out of this problem) in the phrase “existing CIP standards” in the first line of the second paragraph above this one. The existing CIP standards need to be rewritten so that all requirements are non-prescriptive (rather than just 3 or 4 requirements being non-prescriptive, as is the case now). At that point, extending the standards to accommodate new areas (which will always be required, since both technologies and threats will always change) will be something that can actually be accomplished in a reasonable amount of time. As it stands now, the time that is required for this is completely unreasonable.

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte Advisory.

---

[i] To quote FERC’s words (Order 829, p. 2), “In making this directive, the Commission does not require NERC to impose any specific controls, nor does the Commission require NERC to propose “one-size-fits-all” requirements. The new or modified Reliability Standard should instead require responsible entities to develop a plan to meet the four objectives, or some equally efficient and effective means to meet these objectives, while providing flexibility to responsible entities as to how to meet those objectives.”

The last time FERC ordered a new standard – which became CIP-014 for physical security at substations – they also required that it not be “one-size-fits-all”. I hereby predict that FERC will never order a new CIP standard that is prescriptive. As you will know if you’ve been reading my posts regularly, I think prescriptive standards just don’t work for security (which is inherently a statistical, not a deterministic, process). It seems FERC came to that conclusion long before I did; I wish NERC would also come to that conclusion, and order that the existing CIP standards be rewritten in a non-prescriptive format (of course, the details of doing that are anything but simple, as I and two co-authors have discovered as we write a book on this topic).

[ii] These requirement parts come from the most recent unofficial draft of CIP-013-1 that has been sent out to the SDT’s Plus List. It is dated December 19, 2016. The official first draft is due to be posted in January and balloted starting in February.

[iii] I do need to point out that probably most of the Supply Chain SDT members – as well as a lot of the observers at the meeting – didn’t quite get what a non-prescriptive standard means, and frequently lapsed back into discussions where someone would ask whether the team should write the requirement so that the Responsible Entity would need to do X to comply. In the December meeting, I found myself frequently pointing out that, in drafting non-prescriptive requirements, discussion of means to achieve the objective of a requirement is strictly to be included in the guidance, not in the requirements themselves. I joked that I should have just recorded that statement and played it every time I felt compelled to make this point. But I will admit I myself fell into the old way of thinking sometimes. When you’ve been dealing with CIP for a while, you sometimes begin to think that prescriptive standards are the only truly “enforceable” ones (in fact, this is a common objection  I hear when I expound my idea that CIP should be rewritten non-prescriptively). Fortunately, since FERC ordered that the supply chain standard be non-prescriptive, this shouldn’t be a matter of debate for the Supply Chain team, just as it wasn’t for the team that drafted CIP-014. FERC made it clear in both cases that they wanted a non-prescriptive standard.

[iv]Of course, the modified standards will bear a new version number. This will normally be “-7”, although modifications to CIP-002, -005 or -008 will be numbered “-6”, since those standards were never revised with the other standards when “CIP v6” was created. With the exception, of course, of CIP-010 and -011, which are now both numbered “-2” and will be numbered “-3” when changed and approved. And oh, by the way, one or more of the modified standards may be numbered “-8”, since more than one new version of those standards may be drafted and approved by FERC, before the “v7” team is done. This is likely to be the case for CIP-003. The standard in effect now is numbered “-6”, but a new version numbered “-7”, including the recently-approved changes regarding electronic access controls and (probably) Transient Cyber Assets for Low impact assets will be submitted to FERC by March and presumably come into effect by 2018. But this SDT will very likely have to make further changes in CIP-003 in order to fulfill their other mandates, such as the one regarding virtualization. When those changes are approved by NERC and FERC, CIP-003 will then be numbered “-8”. And by the way, the Supply Chain CIP standard will of course be numbered “-1”. Got all this? I agree; it isn’t confusing at all!

I used to think that the CIP standards should all be “revved” whenever even just one was changed (this was done when FERC ordered a change in just one of the CIP v2 standards, but they all were revved - so that all the standards that were in effect before CIP v5 were numbered “-3” and known as CIP v3, rather than having seven “-2” standards and one “-3” standard). I lost that battle when NERC called the v6 team the “CIP Version 5 Revisions” SDT, and the team deliberately only revved the standards that had been changed, leaving the rest as v5 standards. I suggest the industry just start focusing on the individual standards and make sure they are using the current version of each standard. In other words, it no longer makes a lot of sense to talk about the version number of “CIP” itself, only the version number of each particular CIP standard. This will be harder to do, but hey, it’s now the world we live in.

However, in the best tradition of religious leaders and politicians who exhort their flocks to follow a certain path while themselves doing just the opposite, I don’t intend to stop using the terms v5, v6 and v7 until the CIP standards carry such a hodgepodge of version numbers that these terms no longer have any real meaning at all. We haven’t reached that point yet, thank God for small favors. But we will in a few years.

[v] In other words, the CIP Modifications SDT took a prescriptive requirement – that entities with LERC implement a LEAP – and made it non-prescriptive. Of course, if the requirement had been non-prescriptive to begin with, the modifications could mostly have been made in the guidance, which would have been much easier. It is interesting to note, however, that – almost instinctively – the SDT gravitated toward the non-prescriptive approach when they re-drafted this requirement, since the complexity of doing what they needed to do with a new prescriptive requirement would have been much greater. I was fortunate enough to be at the meeting where they drafted the new requirement, and wrote about the process in this post.

This shows that FERC isn’t the only player in the CIP arena that has recognized that only non-prescriptive requirements will work going forward. In the cases of both FERC and the CIP Modifications SDT, the impetus for going non-prescriptive was the realization that trying to write prescriptive standards or requirements to accomplish the required tasks would take far too much time. That is certainly one important reason for using the non-prescriptive approach, although it is far from the only one. I am willy-nilly bringing up other reasons in various blog posts now, but I and my co-authors will try to comprehensively describe all of those reasons in our upcoming book.

[vi] As an example of how the SDT has had to put everything off to focus on the LERC issue, I was hoping that – since the revised requirement and implementation plan had passed the second ballot (as announced last week) – the SDT would be able to talk about those other areas at the meeting in Orlando in December. But no, they had to devote the entire time to fine-tuning the requirement, the implementation plan and the RSAW, and especially to answering the comments they received along with the second ballot. They had to do this even though the second draft had been approved; NERC’s Rules of Procedure require that the SDT respond to comments, even after approval of the standard.

[vii] Part of the reason this was such a hard “sell” was actually due to that increased flexibility. I have found that a significant number of people in the NERC CIP community are automatically suspicious of any change toward less prescriptive standards, because of fears about how auditors will use their new freedom (since the flip side of not having a prescribed way of meeting an objective is not having a prescribed way for the auditor to determine whether or not the entity met the objective). I hope to have a post out addressing these fears (which I believe are unwarranted for several good reasons) soon.

[viii] This is probably an exaggeration, since the Cyber Asset definition says “Programmable electronic devices, including the hardware, software and data in those devices”. One could certainly make the point that VMs would fall under “software and data”, and thus are themselves Cyber Assets. This was clearly not the intent of the v5 SDT, and even if it were, this one consideration wouldn’t suddenly bring all of virtualization into CIP. For one thing, it would only bring virtual servers into scope, not virtual LANs or virtual storage.

[ix] In fact, I believe the sub team that is looking at FERC’s mandate to write requirements protecting communications between control centers has had to go back to fundamentals, just like the virtualization team has had to do. I don’t think this other sub team will need quite as many changed or new requirements or definitions as the virtualization team will, but – given the fundamental issues involved in their task – I’m sure there will be at least say five items that have to be balloted and approved. So assume this task takes “just” two and a half years, and maybe everything else on the SDT’s agenda takes “just” another 2 ½ years. Adding in at least ten years for virtualization, we’re still talking about over 15 years for the current SDT to accomplish their assigned tasks. Does anyone think it is even remotely likely that the team will be willing to continue meeting for say 6 or 7 years, let alone 15?