Some Heretical Thoughts

Note: This post originally started out to be part II of II on lessons learned so far from the CIP-014 experience – that post covered the first lesson, and this post was intended to cover the second lesson. I started this post with what I thought would just be a paragraph or two expanding on what I’d written in the first post – and you can see what happened. But have no fear, I have most of the post regarding the second lesson already written, and it will appear very shortly.

Two weeks ago, I wrote a post that was intended to be the first of two, on lessons that can be learned from NERC’s experience so far with CIP-014, the physical security standard that applies to certain key substations. Why am I worried about CIP-014, since I’ve written very little about it previously? Because CIP-014 is an objectives-based (non-prescriptive), risk-based standard like CIP-013, the upcoming supply chain security standard[i]; any lessons that can be learned from the CIP-014 experience so far are very relevant to CIP-013. And I also think the other CIP standards should be replaced by objectives-based, non-prescriptive ones, so these lessons will be relevant when that long-hoped-for (at least by me. I’m not sure if anyone else hopes for them, although I believe I’ll be able to persuade my dog in the near future) day arrives.

My main concern regarding objectives-based, risk-based standards is how they will be enforced by NERC. I realized earlier this year that simply rewriting the CIP standards isn’t enough. Because the whole NERC compliance regime, as embodied in the Compliance Monitoring and Enforcement Plan aka CMEP, is based on auditing very prescriptive standards, to properly enforce objectives-based, risk-based standards will require a huge adjustment in the mindset of the auditors, as well as everybody else involved in enforcement at NERC and the regions. And it will probably require a rewrite of CMEP, or perhaps a separate CMEP for the CIP standards, vs. the other NERC standards.[ii]

The lesson from CIP-014 that I wrote about two weeks ago was that there really needs to be some sort of review by NERC or the regions of the entity’s plan, before they implement it. In the case of CIP-014, this is the plan for improving physical security of a key high-voltage substation. In the case of CIP-013, it is the entity’s Supply Chain Cyber Security Risk Management Plan. In that post, I discussed how a large NERC entity would probably never implement a planned $80MM physical security system at their key substations, because the region refused to say whether this deployment would help (or hurt) their compliance position on CIP-014.

Of course, having NERC (or in this case, the Regional Entity) review the plan before it is implemented goes against everything in CMEP – plus, as an auditor pointed out to me after he read that post, it also goes against GAGAS, the government guidelines for auditors that all NERC auditors follow. According to those two documents, auditors need to maintain their independence, and can’t provide compliance advice in advance to the entities they will audit. And I totally agree with them that this is the case.

Yet this also poses a huge problem for objectives-based, risk-based standards. Think about it: In a world of prescriptive, non-risk-based requirements (as is basically true for CIP v5/v6, although there are a number of requirements that are objectives-based, or at least non-prescriptive. On the other hand, there are no requirements in CIP v5 or v6 that are risk-based), the auditors absolutely shouldn’t be providing compliance advice up front to the entities they audit.[iii]

But objectives-based, risk-based standards don’t lend themselves to the type of audits that prescriptive standards do; in fact, it’s hard to see how the word “audit”, in the normal sense of somebody with a clipboard going through and checking a Yes or a No box after each of a set of questions, of the form “Did the entity do X?” is at all applicable for these standards. In CIP-013 and CIP-014, the entity is simply required to develop and implement a plan of some sort. The only criteria by which they can be judged to have complied or not are the two questions: 1) Did they develop a good plan? and 2) Did they implement the plan well? How would it be possible for an auditor to answer either of those questions except by using a tremendous amount of judgment, no matter how informed that judgment is by a host of guidance emanating from NERC, the regions, and other parties (and right now, of course, there is very little official NERC guidance on CIP-013 and not enough on CIP-014)?

Strictly speaking, if NERC were going to seriously audit CIP-013 and CIP-014 according to CMEP and GAGAS, they would have to make the whole enforcement process a giant gamble. The entity would have to, completely on their own or at least relying on no guidance that comes directly from NERC or the regions, come up with their own ideas on how to develop and implement their plan, and then how to implement it. At the end of this whole process, they would be subject to a single audit, where the auditors would just check Yes or No for each of the two questions in the previous paragraph. The fate of the entity’s entire CIP-013 and CIP-014 compliance programs would rest entirely on whether the auditors checked Yes or No for these two questions. If you think you’re under pressure before your CIP audit now, just imagine what this would be like!

I point this out not because I think this scenario will come to pass, but because I’m sure it won’t. It simply makes no sense to do this. How does it make sense not to provide any guidance to entities as they develop and implement their plans? What possible good would be achieved by not doing this? Just think of the example from the first post in this series: the utility that probably won’t make a particular big physical security investment, since they don’t know whether it might help or possibly even hurt their chances of being determined compliant with CIP-014? And I recently met with a utility that said they were having a hard time getting management at all interested in starting to prepare for CIP-013 compliance (despite the benefits of doing so now) because there is very little real guidance from NERC or the regions about how to develop and implement a supply chain cyber security risk management plan (more on this in a subsequent post).

Face it, objectives-based (and probably risk-based) CIP standards are here to stay. Since CIP v5, all of the new CIP requirements and standards that have been developed (this includes CIP-010-2 R4, CIP-003-6 and CIP-003-7, CIP-014, and CIP-012) have been objectives-based. This is partly because FERC ordered both CIP-013 and CIP-014 to be objectives-based (although FERC’s term for it was different – something like “not one-size-fits-all”), but also because nobody on a NERC CIP standards drafting team nowadays has an appetite for developing prescriptive standards (as I found out when I attended what turned out to be a key SDT meeting in June of 2016). The problem of a mismatch between NERC CIP standards and the NERC auditing regime is only going to get worse as time goes on.

So how will this mismatch be resolved? In the longer run, it will be resolved both by rewriting the CIP standards and by hopefully developing a separate version of CMEP for the CIP standards (since the current CMEP is fine for the O&P standards). That will of course be a huge, wrenching change for NERC and the regions. You might thus dismiss this as a pipe dream, and I would agree with you if I thought NERC had another option; but in the long run I don’t think they do. There is pressure building among the public and in Congress for the government to take more steps to secure the electric grid from cyber attack (and I wrote about how this pressure might come about in this  post)[iv]. I believe that in the next few years NERC is going to be faced with the choice of either developing a sustainable set of mandatory cyber security standards, and a sustainable compliance framework to go with them, or the responsibility for cyber security of the Bulk Electric System will be taken away from NERC (and FERC) and invested in some other agency like the Department of Energy, the Department of Homeland Security, or even a proposed Department of Cyber Security.

But in the short run, it is clear to me that the problem of the current CMEP not allowing CIP-013 and CIP-014 to be properly enforced will lead to the result that….they won’t be “enforced” at all. Auditors will review both the entity’s plan and how the plan was implemented; then they will provide advice on how the entity could improve on what they did. But this advice won’t be in the form of a Potential Non-Compliance finding (which can lead to an actual Violation), but rather in the form of an Area of Concern finding. As you probably know, AoC’s are issued now whenever an auditor finds some security practice that they think is deficient, but which is not actually in violation of any CIP requirement. The entity is under moral pressure to correct the problem, but they can’t be found in violation if they don’t.

Am I complaining about the fact that CIP-013 and CIP-014 aren’t really “enforceable”? No, I think this is much preferable to the doomsday audit scenario I described above.  But I also think there have to be mandatory CIP requirements that are actually enforced; I discussed my reason for saying this in another recent post. In brief, NERC entities will simply not get the same budgets for cyber security spending if their managements begin to perceive that the standards are increasingly being enforced on a “voluntary” basis. There really should be substantial monetary penalties when an entity clearly develops an inferior plan for CIP-013 or CIP-014, and/or doesn’t implement it properly or at all. Until a new compliance regime is in place at NERC, this simply can’t happen.

I wish to close with one of my favorite quotes from Lewis Carroll (from Through the Looking Glass, the companion book to Alice in Wonderland). Humpty Dumpty has just used a word in a very non-normal way. Alice objects that he isn’t using the word in the proper way.

'When I use a word,' Humpty Dumpty said, in rather a scornful tone, 'it means just what I choose it to mean — neither more nor less.'

'The question is,' said Alice, 'whether you can make words mean so many different things.'

'The question is,' said Humpty Dumpty, 'which is to be master — that's all.'

I would rephrase Humpty Dumpty’s response as “The question is, which is to be master – you or the words?” And so I ask you, which is to be master in the case I just described, CMEP and GAGAS, or the need to have an enforcement framework that is appropriate for objectives-based, risk-based standards like all of CIP should be and like CIP-013 and CIP-014 are now? I promise I’ll have a lot more to say on this in the future. You have been warned.

The views and opinions expressed here are my own, and do not reflect those of any organization I work with. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] In general, I think CIP-013 handles risk better than CIP-014, but that may be because the subject matters of the two standards are so different: purchasing policies and procedures vs. physical security of certain high-voltage substations.

[ii] My general idea is that there is a very big difference between the CIP cyber security standards and the other NERC standards, which are usually called the Operations and Planning (O&P) or “693” standards. The latter are ultimately based on the laws of physics: If you do or don’t do X, Y will happen. This type of standard has to be prescriptive, and needs to be audited in a very prescriptive manner.

But cyber security practice is statistical. If you don’t patch one server for one day, it’s unlikely you’ll suffer adverse consequences. If you don’t patch any of your servers for one year, you will be much more likely to be successfully attacked, but even then it isn’t a certainty. However, there would undoubtedly still be some small security benefit to patching your servers every day. Does this mean that CIP-007 R2 (my poster child for a prescriptive, non-risk-based requirement) should be expanded so that NERC entities now will need to patch their servers daily? Most NERC entities – and auditors – would shudder at the mere thought of that, since it would require a massive effort and many more resources. Yet, in the prescriptive framework of NERC standards, with no allowance for risk at all, there really is no way to say what the stopping point should be – that is, on a risk-adjusted basis, the point at which any possible security benefits are outweighed by the cost. The only logical stopping point would be when there were so many prescriptive CIP standards in place that the risk of cyber attack was effectively zero. Even if that point could ever be reached – which it can’t, of course – if we got there, we would have long ago passed the point where electric utilities would have to suspend distribution of electric power and devote their entire staffs to NERC CIP compliance!

Thus, I think cyber security standards in general need to be objectives-based and risk-based. And to be honest, I don’t know of any other mandatory cyber security standards that are prescriptive and non-risk-based, like CIP-002 through CIP-011 (although there are certain individual CIP requirements like CIP-007 R3 that are actually objectives-based. However, until CIP-013 was developed, none of the CIP standards or requirements have been explicitly risk based, meaning that all systems in scope don’t need to be treated exactly the same way, but the controls applicable to them can be based on the risk they pose to the BES).

A few readers may remember that I started saying about a year and a half ago that I was working – with two co-authors – on a book about how the CIP standards can be rewritten to make them much more sustainable than they are now. I confess that we haven’t put many words on paper (or virtual paper) yet. This is mostly because my ideas were still evolving – for example, my realization mentioned above, that the enforcement program itself will need to be rethought, not just the standards themselves.

My ideas are now at a point where I think I have the whole argument in my head, at least in principle (and a lot of the argument can be found in different posts I’ve written. However, it’s in a few sentences or phrases here and there, not in any form where I can copy and paste them into the book); and I have started serious writing. This, coupled with the fact that I will have more free time for the next couple months or so, makes me hope that I will make some serious progress on the book in the near future, although I strongly doubt I’ll be able to finish it that quickly.

[iii] If you read any of my posts from say 2014 through the end of 2016, you may be surprised with this statement. Those posts coincide with the period when the industry (and I) was struggling to figure out what the wording of CIP v5 meant. I frequently suggested that NERC and the regions should go beyond what was strictly allowed by CMEP and the NERC Rules of Procedure and provide true interpretation guidance to entities, simply because there was so much in CIP v5 that was undefined, vague, or – in the case of CIP-002 R1 and Attachment 1 – completely contradictory. Ultimately, both NERC (through the Small Group Advisory Sessions) and the regions (through unofficial advice that is always provided verbally, never written down), ended up doing exactly that, not because I’d advised it but simply because it was the only thing they could do. They couldn’t continually tell the entities “You have to figure it out for yourselves. We can’t tell you anything”, and then expect the entities to meekly submit when they are assessed violations that are due to their misunderstanding what a requirement means (or at least what the region thinks it means).

So my statement that I agree that auditors shouldn’t be providing compliance advice to the entities they audit, in a compliance regime of prescriptive, non-risk-based requirements, only applies in the case where the requirements are very clear and unambiguous. This unfortunately is not the case with CIP v5, although I need to add that I don’t blame this on the drafting team. I blame it on the fact that cyber security is a discipline in which unambiguous prescriptive requirements simply cannot be drafted. A full discussion of why this is the case will probably have to wait for my book.

[iv] And I’m not taking any position on whether or not that pressure is warranted, just stating that it exists and it’s inevitably going to grow.

Moving On….

I was recently surprised to be caught in what appears to be an unannounced RIF at my former employer. While losing a job isn’t exactly number one on my list of favorite things to have happen to me, I accept it and am ready to move on.

I’ll be honest: If any of you know of particular job opportunities I ought to pursue, I’d appreciate your letting me know. Email me at tom@tomalrich.com. And I’ll also be honest in saying that I’d like to know soon about those opportunities, since from early indications it looks like I won’t be on the market for too long. I’m not trying to be coy about this; I’m just stating the truth.

Of course, no matter where I end up, I intend to keep writing this blog. You won’t be rid of me that easily!

Tom

The views and opinions expressed here are my own, and do not reflect those of any organization I work with. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. 

My CIP-013 Presentation at GridSecCon

As usual, NERC’s annual GridSecCon conference, which was held last week in beautiful St. Paul, Minnesota, was a great success. Every year I say this was the best one ever (I’ve only missed two since it began seven years ago), and this was no exception, both for the great programming and for the very competent organization. This year, attendance passed 500 for the first time. Congratulations to Bill Lawrence and the great team that organized the conference.

I had been asked to be part of a panel on supply chain security. Of course, this being a NERC conference (and the panel being led by Howard Gugel, who is in charge of standards development for NERC), most of the focus of the panel was CIP-013. I was certainly no exception to that rule! In starting my presentation (which I tried to hold to seven minutes – I think I succeeded), I admitted that I would be covering a lot of ground in a short time, and rather than forcing people to take a lot of notes I said I’d write up my presentation (and perhaps embellish it) in a post this week. So here it is!

I didn’t have the time or the inclination to do some sort of general disquisition on CIP-013: how it came about, what’s in it, etc. – I assumed the audience would have some idea of this. I also didn’t go into why I like the standard so much and why it’s so important that it is risk-based, since I’ve already discussed these topics. Rather, I just titled the presentation “Important Observations on CIP-013”, and I discussed ten observations that I thought would be of interest to the audience.  Here are the observations (in boldface), and my discussion of them (which definitely goes far beyond what I had time to say at GridSecCon).

CIP-013 is very simple. R1: You develop a plan. R2: You implement it. R3: You review it every 15 months. So the big question is: What should be in your plan? I believe all cyber security requirements should be objectives-based, and the requirements in CIP-013 are certainly that. So the whole question becomes, what is a good plan? Of course, this is where the real work comes in. Because there’s nothing in the requirements themselves – except for the six mandatory items listed in R1.2 – that tells you what needs to be in the plan, or even how to put the plan together in the first place. There’s the excellent – but far too short – Implementation Guidance developed by the drafting team, and I’m sure there will be a lot of other documents written that provide some form of guidance (including my posts, of course), but in the end your organization is going to have to decide for itself how you will put together your plan.

You need a supply chain cyber security risk management plan. It must address risks (threats) from a) procuring hardware and software; b) installing hardware and software; and c) transitions between vendors. Note there are three areas of risk that must be addressed in CIP-013, not just one; FERC specified that the standard needs to address all three. Just about everything I’ve read about the standard (including most of what I’ve written) focuses exclusively on the first area, vendor risk. But the other two areas are just as important, and they are focused much more on what the asset owner does, not the vendor. Since the requirement says all three areas must be addressed, you probably could receive a PNC if you simply omit either b) or c).

Even though R1.2 lists six particular items that need to be in your plan, you will make a big mistake if you confine your plan to those items. I wrote in a post recently that it’s very possible the only way you can get a Potential Non-Compliance finding from a CIP-013 auditor – other than simply blowing the whole standard off as unimportant – is if you omit one of these six items from your plan.

But this doesn’t mean the auditors won’t look at the rest of the plan at all. They will take a comprehensive look at your plan, and if they believe you’ve missed something, they’ll issue an Area of Concern. You would be well advised to address that AoC, even though strictly speaking you can’t be found in violation if you don’t. There are two reasons for this: 1) It’s The Right Thing to Do; and 2) You don’t want to get on the bad side of your auditor. Remember, even though your auditor’s baby is ugly, it’s not a wonderful idea to say that to him or her. The same goes for ignoring an AoC.

CIP-013 is risk-based: You should classify vendors and systems by risk, and apply controls based on the risk level. This is very different from the other CIP standards. The emphasis here is of course on risk management. If you define risk as threat times impact, you can see the difference between the CIP-013 requirements and most (although thankfully not all) of the existing requirements in CIP-002 through CIP-011, which I’ll call prescriptive requirements.

Like non-prescriptive requirements, prescriptive requirements are designed to address a particular threat (or set of threats). However, prescriptive requirements assume the impact of the threat is equal to 1 – the maximum. They also assume that the impact doesn’t vary according to the importance of the cyber asset(s) being protected. With the impact always being one, this means the risk corresponding to each threat is always the same. It also means that no BES Cyber System can be treated any differently from any other BCS in the facility, since each one is assumed to carry the same level of risk (and please don’t tell me that this is what the High/Medium/Low impact rating does in the other CIP standards. Even though CIP-002 R1 and Attachment 1 are written - although not consistently - to make you believe that the impact rating is an attribute of the BES Cyber System, in fact the rating is treated literally 100% of the time as an attribute of the asset itself. Once the asset is designated High, Medium or Low impact, every BES Cyber System within it is treated exactly the same as every other BCS within it, both by NERC entities and by auditors).

Unlike NIST and other cyber standards, “risk” in CIP-013 doesn’t mean risk to the organization. It means risk to the Bulk Electric System. But you also shouldn’t use the H/M/L classifications from CIP-002. You won’t be helping yourself if you do. Some organizations – including all owned by the federal government – are used to ranking risks posed by computer systems, for compliance with FISMA, NIST CSF, and other frameworks and standards. But the risk considered in these other frameworks is almost always risk to the organization itself.

NERC CIP – indeed, all the NERC standards – treats of risks to the Bulk Electric System; that is what NERC standards are designed to protect. So a payroll server for a utility could be hacked and they couldn’t pay their employees for months, causing many of them to quit. This would of course be a terrible thing for the organization, but as long as there isn’t any direct BES impact, it doesn’t mean anything to NERC CIP. This is why CIP only applies to OT systems, and not all of those, either.

And you shouldn’t classify your BCS using the impact level from CIP-002 R1. There’s a good reason for this. Since CIP-013 only applies to High or Medium BCS (although your plan can cover Lows as well. In fact, I can think of reasons why it would require very little additional work to include them in your plan now. I’ll have a post on this sooner or later), you will then have all high or medium risk BCS for CIP-013! So if your plan says you need to do X for high risk BCS, Y for medium risk, and Z for low risk, you will have to follow that and treat every Medium impact BCS from CIP-002 as a medium risk CIP-013 BCS. You won’t be able to put the set of controls Z on any BCS, since you’re not allowing for there to be any lows!

Of course, you don’t have to rank your BCS for CIP-013 purposes as high, medium and low risk. You can rank them as 1,2,3,4 or high/low risk, for example. It’s probably better if you adopt a different nomenclature for risk in CIP-013 than High/Medium/Low, to avoid the inevitable confusion with CIP-002 R1.

CIP-013 is also threat-based: You should make sure that your plan in principle addresses every threat, based on its level of impact (i.e. the risk). You should rate each threat according to its risk, then determine the appropriate level of controls – if any – based on the risk.

You might well ask, “How can I possibly address every threat to a BES Cyber System? There is literally an infinite number of them. There’s probably a small risk that a truck carrying a BCS to our facility will be overrun by a herd of elephants and the BCS will be crushed. How can I even identify all the threats, let alone mitigate them?” And this is where risk comes in; remember, risk is your friend in CIP-013 compliance! You need to consider all the threats (in each of the three risk areas discussed in the previous slide) and determine both the likelihood and impact of each threat; this yields the risk, of course.

Then you need to rank the threats by the risk that corresponds to each one (of course, you don’t have to rank them highest to lowest; you just need to group them in whatever number of risk levels you’ve decided is appropriate). This gives you your prioritized list of threats you need to address. And you can not only prioritize the threats, you can draw a line somewhere in the list and say “Below this line, the risk is so small that I don’t have to do anything more to mitigate it.” You then don’t have to even consider those small risks, although you have to document you are doing this consistently (not arbitrarily saying “Hey, I really don’t think this threat poses much of a risk. I won’t do anything about it). For example, I think it’s safe to say that no NERC auditor will every find you in violation of CIP-013 because you didn’t even consider the threat that a hurricane would pose in Nebraska!

And not only does the risk level help you prioritize the threats you face, it helps you determine the level of controls required. The riskiest BCS and vendors need to have the most stringent controls; the least risky should have the least stringent controls. Of course, this is very different from the other CIP standards!

The above has covered the six points in my first two slides. My final slide listed four points, and was titled “Some Lessons from Healthcare…” This was because I had been talking with a group of cyber security consultants at Deloitte who had been working for four years in the area of medical device security – working both with the vendors and with the hospitals (which are of course the equivalent of electric utilities in “our” space).

The medical device industry has been complying for four years with mandatory cyber security regulations from the Food and Drug Administration. These apply to the vendors, not to the hospitals. However, the hospitals are just as concerned about the vendors’ compliance as are the vendors themselves, so you could say that the hospitals are also subject to the regulations. Since this is a lot like the CIP-013 situation, it is informative to hear about this industry’s experience with mandatory cyber standards.

Utilities and vendors need to partner to secure purchased BCS. Dealing in a hands-off way doesn’t do much to advance security. One way to “cooperate” with a vendor is to throw some contract language over the wall to them and say “Here, Mr. Vendor. Put this language in your contract or else!” The vendor’s lawyers will then review the language, edit it to their taste and throw it back over the wall to you. Your lawyers will edit that document to their taste and throw it back…etc, etc. Of course, this is all great fun and ensures full employment for the lawyers. But it has nothing to do with making the BES more secure.

How about this approach: Your cyber and supply chain people sit down with the vendor’s cyber and sales people and say “How are we going to solve this problem – i.e. the product security and CIP-013 compliance problem – together? Let’s explore the different options….” You look at how the vendor could change their processes, how they could get you better information about their security and the security of their products, how they handle post-implementation support, etc. Sure, you can also discuss contract language and agree on language you both can live with. But that shouldn’t be the main focus of the discussion. It should be cyber security, and there are much better ways to achieve that besides contract language.

Of these two approaches, which do you think will actually increase cyber security? I’m going to go out on a limb and say number two. And by the way, I’m not saying you need to do this with every vendor. Remember the thing about risk? You will want to do this with the riskiest vendors – i.e. those who sell the systems whose loss would have the biggest impact on the BES; for other vendors, maybe contract language and a questionnaire would be enough. Or maybe just contract language, for the least-risky vendors. But if you’re dealing with your EMS vendor in the paper-over-the-wall manner I just described, I think you should seriously rethink how you’re treating supply chain security.

Legacy devices also need to be secured, even though CIP-013 is silent about this. Again, utilities and vendors need to work together to secure legacy devices. CIP-013 only covers BCS purchased after the standard comes into effect. But there are lots of legacy devices in the field, and it’s almost axiomatic that they don’t have as good a level of security as new systems. What should you do about those?

Fortunately, your discussions with vendors on how you’ll cooperate for CIP-013 compliance are the perfect opportunity to discuss legacy systems. What are ways that both of you can cooperate to improve security in those systems? Don’t wait for the next serious vulnerability to make the headlines.

Incident response is the biggest test of cooperation. There’s nothing like a good ol’ hair-on-fire cyber incident to test how well you and your vendor can work together. But guess what? It’s much better not to wait for an incident to find out how well you cooperate, or whether you can cooperate at all. When you have the meeting with your vendor to discuss CIP-013 compliance, bring this up as well.

Procurement needs to lead the conversation; vendors are much more likely to listen to the people who write the PO’s! I realize this may shock you, but vendors tend to pay the most attention to the people who actually issue the PO’s. If you (Mr/Ms cyber/CIP person) want your vendor to do something important, don’t just bring this up to them yourself. Get your procurement people to make the actual call.

The views and opinions expressed here are my own, and do not reflect those of any organization I work with. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. 

Two Lessons from CIP-014: First Lesson

I have recently been wondering how CIP-013 will be enforced, since this is a non-prescriptive, objectives-based standard. I recently concluded that it in effect wouldn’t be “audited” at all, since there will be no way to find an entity in non-compliance with the requirements[i]. Of course, the auditor will still review the plan, and its implementation, from a general supply chain security perspective. If he or she feels that there are parts that need to be improved, they will issue an Area of Concern – which the entity would be well-advised to take to heart. However, I don’t believe there will be any Potential Non-Compliance (PNC) findings issued for R1 or R2 as a whole, unless the entity has simply done nothing or very little to comply with these requirements – and I find it impossible to believe that a NERC entity with Medium or High impact assets would do that.

However, CIP-013 isn’t the first CIP standard that is non-prescriptive and objectives-based. CIP-014 (the physical security standard that applies to certain important substations) is in principle the same (although I would say that CIP-013 goes further in that direction, but not by much). While CIP-014 certainly isn’t being audited yet, there has been a lot more opportunity for entities to talk with their regions about auditing and other compliance questions. What have they found?

I haven’t done any sort of scientific survey, but I did have a long conversation with a NERC physical security compliance person at one of the largest utilities in the US, about his experience so far with CIP-014. He had two stories to tell, which illustrate the challenges ahead for both CIP-014 and CIP-013 compliance enforcement. They also relate very directly to the larger question of how, if all of the CIP standards were re-cast in a non-prescriptive, objectives-based format, they would be complied with and enforced.

The First Lesson

This utility is putting a lot of money into CIP-014 compliance. There was one particular investment of $80 million that was being strongly considered. However, before the powers that be would commit to this investment, they asked my friend to find out whether this investment would enhance their chances of being found compliant with the requirements of CIP-014.

Since some of you may not be familiar with CIP-014, the standard requires the entity to (among other things):

1. Conduct a risk evaluation[ii] to identify which of its facilities (control centers and transmission substations) meet the criteria for inclusion in this standard;
2. Have a qualified third party validate that evaluation;
3. For the substations and control centers that are in scope, conduct an assessment of the facilities’ “potential threats and vulnerabilities” to physical attack;
4. For each facility in scope, develop and implement a physical security plan that will, among other things, address the threats and vulnerabilities identified in the assessment; and
5. Have a qualified third party validate both the assessment in step 3 and the plan developed in step 4. The third party may recommend changes in either document; the entity must change the plan to reflect those recommendations, or document why it did not. And since the plan has to be implemented, these changes will also need to be implemented.

Now that you know how CIP-014 works, you’ll be able to understand my friend’s problem. The plan in step 4 has identified the $80 million investment in question as being required to address one or more threats and vulnerabilities identified in the assessment in step 3. However, since no NERC entity has unlimited funds to address each threat and fix each vulnerability, there need to be trade-offs. This $80 million investment undoubtedly came at the expense of spending an equivalent amount of money to address some threats and vulnerabilities that were not considered to have such a high impact. But the entity – and the third party that reviewed both the assessment and the plan – determined that the impact of the threats addressed by the $80 million investment was sufficiently greater than that of the other threats, that this was the proper way to spend the money.

But management’s concern is this: NERC will give the final “assessment” of the plan when they come for an audit. What if they make the investment, then in a later audit NERC decides that they had their priorities wrong? In other words, that they should have spent the $80 million addressing some of what they thought were lower-impact threats, meaning NERC disagrees with them on their assessment of the impact of the threats in question. Will NERC then order them to spend an additional $80 million addressing these other threats?

It’s certainly a reasonable question, and my friend was tasked with asking it of their Regional Entity; in effect, he was going to ask the region whether they could review their assessment and plan, at least as they pertained to this particular issue. What do you think was their answer? There really was only one thing the region could say: For us to review your plan before you implement it would be a compromise of the time-honored principle of auditor independence. If we tell you how to comply up front, then when we come back to audit we will simply be auditing ourselves.

I don’t think any final decision has been made on the $80 million investment, but my friend thought it very possible he wouldn’t be allowed to proceed with it without some sort of nod from NERC or the region. So the threats and vulnerabilities addressed by that investment will likely remain unaddressed, until NERC audits them and decides they need to make the investment; hopefully, this finding won’t come with a Potential Non-Compliance finding, but just an Area of Concern.

I hope you understand that I’m not in any way saying the region had a choice in how they responded to this entity. Under the NERC Rules of Procedure and Compliance Monitoring and Enforcement Plan, the auditors must maintain strict independence. But that independence comes with a cost. In this case, the cost is a set of substations that are probably not going to be as physically secure as they might be, if the entity had gone ahead with the investment.

How does this relate to CIP-013? CIP-013 asks the entity to develop and implement a supply chain cyber security risk management plan (SCCSRMP). There is a 10-page Implementation Guidance that addresses what should be in the plan, which will – according to NERC’s current views on guidance - carry weight with the auditors (unlike any other guidance that may come out). It’s a very good document, but it could be 1,000 pages and still not cover everything needed to develop and implement a good plan.  

As with CIP-014, the entity will have to develop the plan and implement it, without any official guidance from NERC on whether it’s a good plan or not. As with CIP-014, the entity could go for years believing their plan is good, only to have all of this contradicted years later by a NERC auditor. Since the CIP-013 plan also must address security threats (although this time it’s a question of cyber threats to the supply chain, vs. physical security threats in CIP-014), it’s very possible that a putative future CIP-013 auditor will also disagree with the entity’s assessment of the relative impact of the threats they face. Finally, it’s possible that the CIP-013 auditor will issue a PNC due to this disagreement.[iii]

So it’s very possible that the same thing will happen to you, if you’re involved with CIP-013 compliance, as happened to my friend who was involved with CIP-014 compliance: an important project (or section of a project) will be cancelled or greatly delayed because there is no way that NERC auditors can provide the sort of pre-implementation assurance of compliance that would allow management to feel completely assured in making their investments.

How could management feel assured? What if, before an entity starts implementing a plan (either a physical security plan in CIP-014 or a SCCSRMP in CIP-013), they had to submit it to their NERC region? The region would review it thoroughly, identify any problems they find with it, then point these out to the entity; the entity would then need to change their plan. The entity would then have the comfort of knowing, when they approve a large investment, that it will improve their chances of being compliant.

As you can guess, this wouldn’t be possible in the current NERC environment. First, as I’ve said before, NERC’s CMEP (and probably the NERC Rules of Procedure) needs to be revised to allow for this. But there is another, even more fundamental, change that would need to happen: The NERC auditors would have to turn into something like cyber security consultants. They would review the entity’s assessment and plan, then come back later to review how the entity implemented the plan. They simply couldn’t be called auditors any more (maybe “assessors”, as in PCI).

This might seem like a radical change, but really it’s not. It would simply be a recognition of reality. Prescriptive standards like the NERC O&P (or “693”) standards require traditional auditors, and they need to maintain their independence by not providing compliance advice before the audit. This is necessary because the O&P standards address threats that are absolutely certain, since they’re based on the laws of physics: If you don’t do X, Y will happen.[iv]

But cyber security is very different. There, everything is based on probabilities. The various cyber threats are always changing and they are only probabilities, not certainties. For that reason, prescriptive requirements don’t work in cyber – or more correctly, they do work but at a big cost. Cyber security standards should be non-prescriptive and objective-based (indeed, almost all other cyber standards in the world are such, although the nuclear power industry’s cyber standards are even more prescriptive than the CIP standards). And non-prescriptive standards require a collaborative approach – not an audit in the traditional sense - in order to avoid exactly the sort of snafu that my friend described to me.

In other words, this snafu is almost inevitable when an organization like NERC tries to enforce non-prescriptive standards using a prescriptive compliance regime. The second post in this series will discuss another example of this problem.

I expect to come back to the second lesson in a post a week or so from now. But before I do that, I need to keep a promise I made when I spoke on CIP-013 as part of a panel at NERC GridSecCon last week; the post I promised should be out later this week.

The views and opinions expressed here are my own, and do not reflect those of any organization I work with. If you would like to comment on what you have read here or ask any questions, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] Except in the case of the six items listed in R1.2. Those six items must be incorporated in the entity’s supply chain cyber security risk management plan in R1. If those aren’t so incorporated, then the auditor could issue a Potential Non-Compliance finding. Of course, since R2 requires the entity to implement the plan from R1, the entity could receive a PNC if they haven’t made an effort to implement those six items (which all involve the vendor making a commitment of some sort). However, whether this can actually be enforced is questionable.

[ii] The standard actually uses the word “assessment”; it uses “evaluation” to describe the activity of identifying threats and vulnerabilities in step 3. Since I and most readers are used to thinking of the process of identifying vulnerabilities as an assessment, I have reversed the two words here.

[iii] As I did in the post previously mentioned as well as the first paragraph of this post, I want to point out that I don’t think it’s likely there will be anything more than an Area of Concern issued as a result of a disagreement between the entity and the auditor on an issue like the relative impacts of particular threats. However, I’m sure there will be some in management – and especially in the legal department – who will require more certainty on this point than I’m able to provide.

[iv] This might seem like a big exaggeration – after all, if control room operators don’t communicate properly as required by the NERC COM standards, a cascading outage of the BES won’t necessarily be the result.  If the right set of physical conditions is already in place (hot day in summer, large plant already on outage, etc), there would exist some scenario where if operator 1 does X instead of Y, and operator 2 does A instead of B, then there would be a cascading outage. There is no such certainty anywhere in cyber security. 

Lew Folkerth discusses the "Dark Auditor"

Once again, Lew Folkerth of RF has written a very good article about CIP. This time, it's about the meaning of the phrase (which I hadn't heard before) "Defending against the dark auditor". You can find the article by downloading the latest RF Newsletter and finding "The  Lighthouse".

Why do we need Mandatory Cyber Regulations?

I have more than a few times talked with someone from a large utility who assures me “We really believe in cyber security. We don’t need NERC CIP because we would do all of that anyway.” Both of these sentences have always struck me as strange.

To take the first sentence, when I hear it I always think (but never say!) “I’m really glad you believe in cyber security; that’s good news. Let me go out on a limb and guess that you also really believe in motherhood, apple pie and the Fourth of July.” In other words, talk is cheap. I’ve never heard anyone say “We don’t believe in cyber security at all.” Although some organizations have said so with their actions.

Now let’s look at the second sentence. When someone says that, I usually think (and sometimes say) “Oh, really? You mean that, if you didn’t have to comply with NERC CIP, you would still take great pains to document that – as required by CIP-007 R2.2 - every 35 days you have checked with the patch source for every piece of software installed on a component of a Medium or High impact BES Cyber System or a Protected Cyber Asset to see if there is a new security patch available – even if that vendor has never released a security patch and probably never will? Would you really do this if you weren’t obligated to by CIP?”

Of course they wouldn’t. While some documentation is obviously required as a good security practice, this – and a number of other documentation requirements of CIP – does very little to advance security. Indeed it detracts from it, since if you’re spending your time documenting something like this, you’re taking away time you could spend actually improving security in a way that isn’t required by CIP, such as combating phishing or ransomware (although I’m sure all NERC entities are putting resources into these two threats as well. But since these are IMHO the two biggest cyber threats today, it’s not an exaggeration to say that almost any amount spent combating them isn’t enough).

But let’s leave the question of compliance documentation aside; I’ve already discussed it in another post. Is the second sentence really true? Are there electric utilities or IPPs that would spend as much on cyber security in the absence of NERC CIP as they do in its presence? I’m sure there are a few that would, but for the majority of NERC entities I’m also sure the answer to this question is no. Mandatory requirements (especially if accompanied by potentially huge penalties for non-compliance) are always going to be funded first, even if other non-required areas of cyber security might in some cases deserve more funding, ahead of at least some CIP requirements. And strict logic indicates that it is inevitable that the level of funding for such non-required threats has to be less than it would be in the absence of mandatory cyber requirements.

At the September NERC CIPC meeting in Quebec City, a good discussion broke out (sparked by a presentation by Tobias Whitney of NERC) about getting money for cyber security projects. A couple participants said that they have more than once advocated for a request for funding for security projects not strictly required by CIP by saying that the expenditure would “help CIP compliance”. And lo and behold, the doors to the bank vault were flung open. However, if those three magic words hadn’t been uttered (like sprinkling pixie dust), those projects probably wouldn’t have been approved. So this is the main reason why the electric power industry needs mandatory cyber regulations: Without them, there would be a much lower overall level of funding for cyber security.

You may now ask “So why are you always complaining about NERC CIP? If it’s getting the industry much more funding for cyber projects, it must be making it more secure.” I agree that CIP has made the industry much more secure than it would be otherwise (see this post for more on this topic). However, I think the cost of compliance with the current CIP standards regime outweighs the benefits by a good margin – and that cost is increasing as new standards like CIP-012 and CIP-013 come online. So the question is how to write a standard that a) is mandatory, but b) doesn’t force the entity to invest a lot of time and effort in activities that don’t benefit cyber security very much (and indeed, what is needed is a standard that “forces” the entity to do what they would otherwise do with adequate funding for cyber security, but no mandatory standards).

I believe that these two criteria could be met by a new set of CIP standards - and a compliance regime to go with them - that would 1) be objectives-based, 2) be threat-based, 3) be risk-based and 4) provide a list of threats to address that would be subject to frequent updates by some central group of representatives of NERC entities. CIP-013 actually comes close to this (actually just the first three, but 3 of 4 ain’t bad!), although since I don’t think it can be audited under the current NERC CMEP and RoP, it can’t really be called a “mandatory” standard – except for the six items listed in R1.2. But even not being mandatory, I think there will be lots of pressure on NERC entities to do the right thing and do their best to comply with CIP-013.

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte.

“Associated With”

In early 2014, soon after FERC had approved CIP version 5, a lot of people in the industry (including me) started taking a serious look at the v5 standards, since they were now on their way to becoming the law of the land. One thing we noticed was this important discrepancy in CIP-002 Attachment 1: In Section 1, which discusses classification of High impact BES Cyber Systems, the criterion for deciding whether a system is High impact is if it is “used by and located at” one of the four types of control centers listed in that section. This makes it clear that no system that isn’t physically located at one of those four types of control centers can be High impact. So the BES Cyber Systems located at Medium or Low impact substations and generating stations that are controlled by the High Control Center will be Medium or Low impact respectively, not High impact.

However, in Section 2, which discusses classification of Medium impact BCS, the criterion for deciding whether a system is Medium impact is whether it is “associated with” one of the 13 criteria for assets listed in that section; two of these criteria (2.11 and 2.12) are for Control Centers. This means that, in the case of Medium Control Centers, a system doesn’t have to be physically located at the Control Center in order to be designated Medium impact.

Why is this a problem? Because Control Centers control lots of Cyber Assets (e.g. relays) “in the field” – i.e. located at transmission substations and generating stations. It would be very hard to argue that these field assets aren’t “associated with” the Control Center that controls them, meaning that any system that is located at one of these assets, that meets the definition of BES Cyber System[1], will be Medium impact. And as anyone who has had to comply with CIP v5/v6 for High or Medium assets knows, even if there is only one Medium Cyber Asset located at the asset, there are a number of CIP requirements that automatically become applicable to the asset itself, or to other Cyber Assets located there.

Of course, if the auditors actually interpreted the wording this way (which clearly seems to me to be the right interpretation), there would have been a huge hue and cry from NERC entities with Low impact assets and a Medium impact Control Center, since  a large number (perhaps all) of those Low assets would now be effectively Medium assets. However, they haven’t been interpreting it that way, and the main reason they haven’t has probably been the fact that in the Guidance and Technical Basis for CIP-002-5.1 (page 16), there is the following wording: “Criterion 2.12 categorizes as medium impact those BES Cyber Systems used by and at Control Centers and associated data centers performing the functional obligations of a Transmission Operator and that have not already been categorized as high impact. (my emphasis)”

Of course, this sentence seems to indicate that remote BCS controlled by a Control Center that meets Criterion 2.12 (these will most often be relays in substations) will not take the Medium impact rating of the Control Center. This certainly seems to contradict the language of Attachment 1, but I haven’t heard of any entity being dinged for not declaring those BCS to be Mediums (see this post from early 2014, which provides a different set of reasoning for why Criterion 2.12 should be interpreted to mean that the remote BCS aren’t Mediums. It is a pretty subtle argument, and I haven’t heard it put forth anywhere else).

However, things are changing. As you may know, NERC has decided that the Guidance and Technical Basis in each of the CIP v5 and v6 standards goes beyond what is permitted for NERC guidance. Some of it becomes an “interpretation” of the CIP requirements (and the passage I just quoted seems to be a good example of that. It not only interprets the wording of Attachment 1, it seems to contradict it). Therefore, NERC will remove the G&TB from the standards.

In theory, this shouldn’t make a difference. It has always been said that the G&TB’s aren’t auditable. However, in practice the auditors have paid a lot of attention to them. What will happen when the G&TB for CIP-002 is removed?

My guess is nothing will happen, although I know some people in NERC-land are very fearful of this. Whatever the strict wording of the standard is, this is a case where the consequences of following that wording would be too harmful. I can’t imagine any of the regions would want to do this, especially since they’ve so far all allowed the entities to follow the “interpretation” in the Guidance and Technical Basis; there would be an uprising if they tried to take that away.

So why am I bringing this up? Because this is just one example of the fact that there is a lot of “interpretation” of the CIP standards going on, both by the entities who have to comply and the auditors who have to audit. It’s the grease that allows the wheels to turn, in the creaky engine of NERC CIP.

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte.

---

[1] Of course, a BCS is just a set of BCAs that are grouped together, so BES Cyber Asset is really the operational definition.