“We Need Something to Measure”

I have been trying to keep up with the new Standards Drafting Team that is tasked with developing the CIP Supply Chain Security standard that FERC ordered in July. Last Friday, they had a short phone meeting to set the stage for their first face-to-face meeting in Atlanta this week (which I unfortunately had to miss because it conflicts with NERC’s GridSecCon). I won’t summarize the phone meeting, but I do recommend that anyone with any interest in the new standard should get on the SDT’s email list – called the “Plus List” – in order to see documents, learn about upcoming meetings, and in general learn what’s going on (I do want to point out something that some people don’t seem to understand about NERC meetings in general: almost all of them are open to anybody who is a user of electricity. And if you’re not an electricity user, how on earth are you reading this blog post?).

What inspired this post was a single comment made by someone on last Friday’s SDT call, as the discussion was focusing on the preliminary version of the new standard (which is currently called CIP-013-1, proving that the SDT members are not superstitious). This person said something to the effect of, “We need something to measure.” The meaning seemed to me to be, “You can’t have an enforceable requirement unless it can be rigorously audited. You can’t audit compliance with a requirement without having some black-or-white criterion for whether the entity has complied with it.”

I want to say now that I don’t mean in any way to pick on the person who said this (and I honestly don’t know who it was, nor do I care). I will also say that I have heard this assertion made many times. Finally, I want to mention that I and one or two other people on the call quickly pointed out to this person that this will be the wrong approach to use for CIP-013, if for no other reason than because FERC said explicitly in their Order 829 (pages 30 and 31) that the new standard should not be prescriptive. So my guess is this issue has already been laid to rest as far as the Supply Chain Security SDT is concerned.

That being said, there’s something more I want to point out: The majority of the existing CIP requirements are written with precisely this thought in mind, namely that the only good standard is a prescriptive one. And folks, in my humble opinion this is the fundamental problem with the current NERC CIP standards. Because they are so prescriptive, they are unsustainable and will collapse of their own weight if not modified.

I have been making this point in various posts, starting with this one early in the year. And the more I talk to people in the industry and follow developments in CIP, the more I am convinced it is true. In fact, I am now working on a book with two co-authors that will not only discuss the problems with NERC CIP but lay out what we see as a solution to that problem (which could be adapted to other critical infrastructure domains, not just electric power). However, it will be late 2017 at the earliest that the book will be available, so I’m not going to tell you to wait for it to learn why I made the above assertion; I am going to explain why  I am saying that. On the other hand, I’m not going to attempt to write the book (or even a chapter of it) in this post. In this post, you will see a high-level view of the argument we will make to support this point, but not a lot of details to support our case. For those details, you will have to wait for the book.

Here is roughly the argument we will make:

1. Costs for NERC CIP compliance have ballooned with CIP versions 5 and 6, They are only going to continue to balloon due to CIP v7, CIP-013 (the supply chain standard), and a host of future versions that will be required to address security of a lot of areas so far left unaddressed by the current CIP standards (virtualization, the cloud, phishing attacks, and Distribution systems, just to name a few). I know there are at least a few large NERC entities that easily spent 25-50 times as much on implementing compliance with CIP v5 as they did with v1. And guess what? They’ll have to spend some multiple of that amount to address the new and revised CIP standards that will be coming down the road.
2. This situation might be justifiable if the bulk of this money (say 90%) were going to improve security. However, I have been conducting a totally informal and unscientific poll of NERC entities, asking them how much of every dollar they spent on implementing CIP v5 compliance actually goes to increasing security, as opposed to all of the paperwork, etc. that is required to prove compliance with those highly prescriptive standards. The highest estimate I received was 70%. This itself is appalling, since it says that “only” 30% of what this entity spent on v5 was “wasted” on pure compliance paperwork! The low estimate I heard was about 25%, and I’d say the median was 50% (a couple other knowledgeable observers also say that 50% is probably the industry average, although I admit there is no ­way this could be objectively verified. Different individuals will have different opinions on whether a particular dollar spent on CIP went to security or purely to compliance activities with no security benefit)
3. When you combine these two assertions, that the cost of CIP compliance is ballooning and will continue to do so for a long time, and that about half of those costs are not going to securing the Bulk Electric System, you come out with what I call an Intolerable Situation. Either one by itself might be tolerable, but the two together are intolerable. We can’t let things go on like this, until a significant portion of the US GNP is going to NERC CIP compliance, without any proportionate increase in security. Here are some numbers: I believe that at least $2 billion was spent by the industry on implementing compliance with CIP versions 5 and 6, including security products purchased, staff time, consultant time, etc. Let’s be charitable and say that between 30 and 50% of that went only to compliance costs, not security. This works out to $600 million to $1 billion. This is more than I earn in a year, and is a lot of money to waste.
4. This isn’t to say that the CIP standards should be “frozen” as they are, to keep costs from ballooning even more than they have already; there is just about universal agreement in the cyber security community that there is a lot more that needs to be addressed in CIP. But it is to say that we need to find out why so many in the CIP community feel that one of every two dollars they spend on compliance is going down the rat hole and fix that problem, hopefully before new increases in the scope of CIP lead to another doubling or tripling of the amount that utilities have to spend on complying (30-50% of which larger number will also not go to security).
5. In other words, what we need to do is find a way to fix the whole CIP compliance regime (which is a lot more than just rewriting the standards) so that a much greater percentage of every dollar spent on implementing and maintaining compliance actually goes to security. To go back to the previous illustration, if 80% of CIP v5-v6 spending went to security rather than 50-70%, this would be an effective increase of $200 to $600 million in industry spending on cyber security – without NERC entities being required to spend an additional dollar. If we could get to 90% (which I believe is possible), that would be an increase of $400 to $800 million.
6. But as I’ve just said, the scope of CIP is going to keep increasing, and entities will still end up spending more money, even if CIP is rewritten. However, if CIP is rewritten it will make the increased spending much more palatable. If entities feel that most of what they spend on CIP will actually increase their security and that of the BES, they are much more likely to support such scope changes than they are now, when they know that close to half of the increased cost will simply be wasted, from the point of view of promoting cyber security.

Why do I think we have this problem? It is because the CIP standards are prescriptive to a fault (I do want to emphasize the disclaimer found on all of my posts: that any opinions expressed in this blog – and certainly in this post – are mine alone, not those of my employer or for that matter any other entity or person). And why are they prescriptive? Because of the idea – which is still prevalent in NERC circles – that the only way to write enforceable mandatory requirements is to make them prescriptive; prescriptive standards lead to measurable outcomes, so that in theory it should be completely clear whether or not the entity has complied (no auditor discretion allowed or required). But this prescriptivism leads to NERC entities spending large amounts of money on activities that don’t increase security but do help them avoid getting cited for non-compliance with the prescriptive requirements. Even more importantly, prescriptive standards greatly inhibit NERC’s ability to expand CIP’s scope to address new domains like the cloud, without engaging in a painful, expensive, years-long standards development process.

I will first illustrate this point with the Tale of Two Requirements: CIP-007-6 R2 and R3. R2 (Security Patch Management) is the bad guy here. For example, take the requirement part (R2.2) that says the entity needs to, every 35 days, determine whether there are new security patches available from the vendor of every piece of software installed on even one BES Cyber System or Protected Cyber Asset, and evaluate whether the patch applies to their systems or not. It doesn’t matter if the software is installed on one or 1,000 systems. It doesn’t matter how important the software is to the entity's operations. It doesn’t matter whether the vendor has never released a security patch and probably never will. This has to be done every 35 days, for every piece of software on a BCS or PCA.

From the point of view of measurability, this requirement is a champ. It is in theory very easy for an auditor to review the entity’s documentation to determine whether 35 days or less elapsed between the patch release date and the date the evaluation was done. However, NERC entities – especially larger ones – are finding this a tremendously burdensome requirement to comply with, due in part to the difficulty of contacting loads of smaller software vendors once a month to see if they have made any security patches available. But they have to do this, and they have to do it within 35 days (that is of course just the first step in complying with CIP-007 R3. There are a couple other specific deadlines after that). And most important of all, they have to have good documentation that they have done this, for every software product for every month – and have it readily retrievable when the auditor asks to see the evidence that all security patches were identified and evaluated for these particular systems in this particular month.

But where does the 35 day deadline come from? Is there some principle of cyber security – or computer science or electrical engineering – that dictates that security patches need to be identified and evaluated within 35 days or all hell will break loose (as conceivably could be the case with other NERC standards, where failure to meet a particular parameter or deadline could perhaps lead to a cascading outage of the BES)? Of course not. This is an arbitrary deadline, chosen simply because the CIP v5 SDT felt they had to choose some deadline in order to – of course! – have something measurable. But entities then have to spend a lot of effort and money designing processes and installing systems to make sure this deadline is never exceeded, for all of the perhaps hundreds of software packages installed within their ESP. Measurability has a high cost, and shouldn’t be invoked just to make audits easier.[i]

Now let’s look at CIP-007-6 R3, Malicious Code Prevention. This is one of two non-prescriptive requirements currently found in the CIP standards (the other is CIP-010-2 R4. Plus, one requirement is in the process of being switched from prescriptive to non-prescriptive by the CIP v7 SDT: CIP-003-7 “R3.1”[ii]). The heart of this requirement is part 3.1, which reads “Deploy method(s) to deter, detect or prevent malicious code.” That’s it. There is no requirement to deploy anti-virus software with no other options (as in the previous CIP versions), no requirement to check for new signatures daily, etc. By the same token, the entity doesn’t have to put in place a lot of procedures and systems to comply with arbitrary deadlines, as well as train and monitor a host of staff members who have lots of other important tasks on their plates. And most importantly, the entity doesn’t have to generate reams of documentation showing that they complied with arbitrary deadlines for say A/V signature deployment every day for every system within the ESP.

I could go on and on regarding this topic (and we will discuss it thoroughly in the book!), but you hopefully get the point. I hold prescriptive requirements responsible for the Intolerable Situation mentioned above. The obligation to meet arbitrary deadlines and other targets - without consideration of the risk posed by a particular system, software package, vendor, etc. – drives a lot of the huge and increasing cost of NERC CIP compliance.

And this situation will just get worse as CIP is expanded to cover more domains, like supply chain and the cloud. I think FERC realizes this, which is why, for the last two major expansions of CIP that they have ordered – substation physical security in CIP-014 and supply chain security in CIP-013 – they have gone out of their way to say they don’t want NERC to take a prescriptive approach in writing the standards. At some point, I predict they will have to order NERC to rewrite all of CIP in a non-prescriptive fashion.

I do want to say now that I don’t blame NERC, FERC, or any individuals employed by or associated with those organizations, for this situation. Nobody set out to achieve this end. In my opinion (again, this will be elaborated and justified in the book), the situation is due totally to the fact that, until CIP-014, which is entirely non-prescriptive, NERC was in the business of writing prescriptive standards. And for their original mission – protecting against single acts of commission or omission that can have catastrophic physical effects on the BES – that is exactly what they should be doing.

But cyber security is different. There is no way an entity can mitigate (or even identify) every cyber vulnerability that affects even a limited group of systems like BCS. But the CIP standards (with the exception of CIP-014 and the two other CIP requirements mentioned above) implicitly assume that this is a goal that can be achieved. And, given the size of the potential fines that NERC entities face for non-compliance with even one requirement in any NERC standard, they have lots of incentive to devote their entire security budgets to CIP compliance, and leave nothing for anything that isn’t required. That they don’t in fact do this – but definitely do spend a lot of money on cyber security beyond what CIP requires – is testament to the fact that these entities truly are concerned with security, not just with compliance.

To sum up my argument, in my personal opinion the perceived need to have “measurable” requirements has led to the situation where a huge – and growing – amount of money is being spent on NERC CIP compliance without anywhere near a proportional increase in security of the electric grid. This situation will only get worse if the entire CIP compliance regime isn’t re-thought and re-written.

What’s the solution? The easy answer is that I and my co-authors will go into that in great detail in the book. The not-so-easy answer is that we are currently still having a lot of discussions, both among ourselves and with others, about this question. The general direction is clear, but that is all that’s clear at the moment. It won’t suffice simply to rewrite all of the current CIP requirements in a non-prescriptive format, as is currently being done with CIP-003-7 “R3.1” (which I described above). The CIP standards need to non-prescriptively address all threats to the cyber security of the cyber assets that run the BES, whether they come in through the IT network (as in the Ukraine attacks), through the cloud, through “machine-to-machine” communications from outside the ESP, etc.

On the other hand, it’s not possible to simply have a CIP standard that says “Make sure you’re cyber secure”. The entities will need to be told the different areas they need to address (patch management, securing virtual systems, etc), and provided with guidance on best practices for addressing those areas. They will then be audited based on how well they addressed each area; and yes, the auditors will need to be cyber security professionals who can determine how well the entity did address each area, without having to resort to a checklist with arbitrary boxes labelled “35 days”, etc.

Even more importantly, the CIP framework needs to be able to rapidly incorporate new threats that will pop up in the future that are currently unheard of, without having to go through a multi-year standards development process. So there will need to be some sort of governing body that will regularly meet to review the primary threats to the cyber security of the BES, and add or subtract areas from the list of what the entities need to address (as well as write guidance for new areas). None of these goals can be achieved by simply rewriting the current standards; there has to be a new compliance regime.

But I’m not going to force you to wait for the book to hear about my ideas for what should replace the current CIP standards. In this blog, I will keep you up to date with what I am thinking in this regard. And I’d be very interested in hearing your ideas as well.

  

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte Advisory.

---

[i] I am not trying to pick on the CIP v5 SDT here! The next paragraph describes a very non-prescriptive requirement that they also wrote. Plus times have changed. I attended a number of the SDT’s meetings (in 2010-12), and I freely admit I never once even thought about raising the issue of prescriptive vs. non-prescriptive requirements. It just wasn’t something I even considered important until about a year ago. Now it’s all I think about.

[ii] I use the quotes because it is technically Section 3.1 of Attachment 1 of CIP-003-7 not requirement R3.1.