A Great Supply Chain Security Forum

In this post from early October, I mentioned (toward the end of the post) a conference I’d just attended outside of Washington DC, called the Software and Supply Chain Assurance Forum (SSCA for short). It’s a free quarterly two-day conference sponsored by NIST, the Department of Defense, the Department of Homeland Security and the Government Services Agency (GSA). It is organized by MITRE Corporation and is always held at MITRE’s offices in McLean, Virginia. I intended to write a post soon thereafter providing all the details you need to get on their mailing list, both in order to hear about future conferences and for general information about supply chain security.

Well, “soon” has proved to be a month and a half, as I got diverted by other things, especially FERC’s approval of CIP-013. But yesterday I received an email announcing their next forum December 18-19, and I decided I really need to write this post now, since some of you may want to attend it (which I can’t do this time, although I will try to attend as often as possible in the future. BTW, there’s no webcast or recording – you have to attend in person).

I asked Bob Martin of MITRE, one of the organizers, to provide a short description of the forum and its history. To be honest, I thought this was probably some effort that had just started a few years ago, when I first heard a lot of talk about supply chain security. As you’ll see, I was quite wrong on that. Face it: the Feds were doing supply chain security long before a lot of us even heard of it. Here’s what Bob said:

“The Software and Supply Chain Assurance started in 2004 as the Software Assurance Forum and Working Groups, a free, DoD-focused public engagement effort to bring together the community on the myriad issues surrounding Software Assurance. The following year it became a joint effort of DHS and DoD, with a continued focus on engagement with industry, academia, and government, and was re-established as part of the Cross-Sector Cyber Security Working Group (CSCSWG) under auspices of the Critical Infrastructure Partnership Advisory Council (CIPAC).

During the first decade, the Forum had an emphasis on working groups to accumulate and promulgate best practices on the Software Assurance aspects of Technology, Processes, Education, and Acquisition. In 2008, the co-sponsoring partnership expanded to include NIST. Then in 2014, GSA joined as the fourth co-sponsor and the Forum refocused to more directly include Supply Chain issues, including renaming itself the Software and Supply Chain Assurance (SSCA) Forum.”

I do want to point out that, although the SSCA clearly started out with almost all Federal government employees and contractors as members, it now includes a significant industry contingent. And the discussions aren’t usually on specific government topics, but supply chain security in general. Here are the topics for the December meeting: the DHS Task Force on supply chain security, software security, software testing, software identification, software certification, international supply chain risk management efforts, updates to US Federal Government policies, and more.

You can sign up for the December meeting here. If you’re a non-US citizen or green card holder, you need to sign up by 12/4. If you’re a US citizen or green card holder, you have until 12/11. And if you want to get on the mailing list, drop an email to Bob Martin at ramartin@mitre.org.

Here are some interesting things I learned from the September meeting. They’re just in the order of the presentations where they were stated. Note that, even when I use quotation marks, I can’t vouch that this is a verbatim transcription of what was said.  

NIST:

CSF R1.1 has a supply chain focus.

800-53 R5 has a map to the CSF, as well as to 800-161 (the supply chain security framework).

800-37 addresses Enterprise Risk Management.

The Open Group:

The O-TTPS cyber security standard is for suppliers. It’s available as a certification.

DoE:

The CITRICS program is testing ICS components for security.

FERC:

Rhonda Dunfee of FERC pointed out that CIP-010 R1.6 just deals with software security patches, not firmware (even though CIP-013 R1.2.5 can be interpreted as applying to firmware as well as software patches).

Unknown:

“Many organizations fear the auditor more than the attacker.” (how true!)

“Software quality defects are really security defects, although the converse is not necessarily true. Quality and security defects are usually dealt with in separate efforts, although it would be better if that weren’t true.”

“If you talk to a CEO about risk, they’ll assure you they have that covered. What they usually mean is financial risk. If you talk to them specifically about operational risk (including cyber), they’ll give you a blank stare.”

Edna Conway of Cisco (a well-known supply chain security expert, and fun to listen to):

“Quality and security aren’t achieved through contract language….If you are dealing with your vendor in a courtroom, you’ve failed.” (how true that is! For my first post on this idea – there will be further posts, I can assure you – go here. There is a follow-up to that post here)

Office of the Director of National Intelligence (this was the best presentation, IMHO):

·         “The hard part in supply chain security is getting all the right people together, not the actual measures.”

·         A good tabletop exercise: What are factors that would make you say no or yes to a new business relationship?

·         “When it comes to SCS, there is no ‘easy’ button: ‘Just tell me what I need to do.’”

·         The Kaspersky threat has been known for a long time. The ODNI talked with supply chain people and warned them of this. But at the time they didn’t have evidence of malfeasance, so many government organizations (and private ones) signed up with Kaspersky anyway.

·         You can’t trade off advantages in cost or schedule for security performance.

·         SCS can’t be a compliance checklist approach; there must be risk mgmt. (how true this is! This is the point of the multiple posts – such as this one - that I’ve written on why CIP-013 R1.1 is the heart of the standard, while R1.2 is just a sideshow, although a mandatory one)

·         4 pillars of supply chain: cost, schedule, performance and security

·         “Security should be like quality. It should be expected all of the time.”

·         “It is not at all inevitable that added security increases cost.”

Howard Gugel, NERC (who spoke on CIP-013):

(in response to a question of whether CIP-013 addresses security risks from custom-developed software) “Those risks are covered in other CIP requirements, notably CIP-010 R3, the Vulnerability Assessment requirement.” I disagree with this, since having an every-15-month VA isn’t the same as having regular patches from a vendor, as new vulnerabilities are identified. This is one of the many risks that should be addressed – for those entities to which it’s applicable – in R1.1. But I haven’t heard NERC say much at all about R1.1 – the focus in everything I’ve heard has been R1.2, and on using contract language to comply with R1.2. IMHO, both of these emphases miss the whole point of the standard.

NCCIC (DHS):

Someone from NCCIC discussed the Russia attacks (about which I wrote a whole bunch of posts this summer, starting with this one), and pointed out that most of the entities compromised were small. This is all well and good, but that certainly wasn’t the tenor of the presentations they gave, even though perhaps they never explicitly said otherwise. It sounded like major utilities had been compromised, and it was just going to take one nod from Vladimir Putin to send the whole US into darkness. That was certainly the tenor of the major news articles, and I haven’t heard of DHS issuing any public (or private) apology for misleading the press on this matter.

This person also seemed to make yet another attempt to try to make sense of those contradictory briefings –as well as what was put out on the subject early this year – by saying that DHS just didn’t understand the difference between “vendors” and “utilities”. I believe we’re now on at least Version 5 of the ongoing saga “What DHS really meant in those briefings”. Unfortunately, this latest explanation is no more consistent with the actual statements than are any of the previous ones. I hope to do a post, summarizing what I understand of this sorry tale, in December.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. We also work with security product or service vendors that need help articulating their message to the power industry. To discuss this, you can email me at the same address.

Stop Making Sense: the Remix

I have been working this year with two organizations that are getting ready for CIP compliance at the Medium impact level, as well as a Canadian organization with Highs, Mediums and Lows, that only had to comply with CIP starting in 2017. These discussions have brought to my mind a number of issues that I addressed in posts and discussed with various entities, as the whole industry was getting ready for CIP v5 compliance in 2014-2016.

A very small number of these issues were actually resolved by revising the standards (although I really can’t think of any issue that was completely resolved in this way. Since changing one of the CIP standards is easily a 3-5 year process from initial idea to having an enforceable standard, it’s not surprising that this happens very rarely). A number of these issues (especially the various problems with CIP-002 R1 and Attachment 1, about which I wrote at least 50 posts, starting with this one) are no longer burning issues, since the auditors and the NERC entities came to a rough consensus on how to resolve them (even though it’s not supported in the requirements as written).

However, there are some issues (like the cloud) on which there hasn’t been any resolution of any sort. The only reason why you don’t hear complaints every day about these issues (from me as well as from others involved with CIP compliance) is that it’s quite clear that nothing can be done about them in the near term, except to do your best to work around them. In fact, I’m sure a lot of CIP compliance people have probably forgotten that these even were issues in the first place. They’re simply part of the landscape, like a mountain. If you’re going from point A to point B and there’s a mountain in between, you’re not going to spend a lot of time complaining about the fact that the mountain is there – you’ll just allow for that and make a big detour when you come near it. But the cost of having to make that detour is very real.

In my discussions with these three entities, they of course have questions on what particular requirements or definitions mean, and naturally try their best to make sense of them. For example, they might say “Surely CIP-007 R2 doesn’t require that we have to check every 35 days, for every piece of software installed within the ESP, to find out if the vendor has released a new security patch! What if the vendor has never released a security patch in ten years? Or what if the software involved is fairly insignificant and has minimal impact on the BES? It only makes sense that we should be able to devote our resources and time to what poses the most risk to the BES.”

To which I will (sagely) reply “I totally agree it makes sense that you should be able to look at risk in how you comply with this requirement (and all of the other CIP requirements as well). But what makes sense has nothing to do with CIP compliance. The only thing that matters is the strict wording of the requirement. If something is allowed by the wording, that’s what you do. If something isn’t allowed, you don’t do it. And if the wording is ambiguous (which it is very often, of course), well…that’s for another discussion.”

I wrote a post about exactly this issue in 2016. I just reread it, and it’s just as relevant now as it was then (in fact, the question I discussed in the post was about the cloud. I agreed then that it would absolutely make sense if NERC entities were free to entrust their BCSI, and their BCS, too, to cloud providers that had demonstrably good cyber security practices – but of course this just wasn’t allowed by CIP. If I’m not mistaken, this is as much an open issue today as it was in 2016, in fact much more so).

However, I’m pleased to say that there’s been some slow progress on addressing this issue. I pointed out in the post that the reason why NERC entities can’t apply reason very often in CIP compliance is that most of the CIP requirements are prescriptive. This is still the case, but it’s also a fact that every significant new CIP requirement developed since CIP v5 has been what I call plan-based, and what others call objective-based: That is, the requirement mandates that the entity develop a plan to achieve a particular objective. The means of achieving it are up to the entity, as long as what you propose in your plan is – are you ready for this? – reasonable. Isn’t that amazing? Not only can you use reason in developing your plan – that is the criterion by which the plan will be judged!

These requirements take different forms, but fortunately they all come down to developing a plan to achieve an objective. The current plan-based requirements are (in order of their development):

CIP-007 R3, Anti-Malware: This was developed with CIP v5. It doesn’t specifically require a plan, but it does definitely state an objective: mitigating the risk posed by malware. You have to go through a planning process to achieve that, since you need to do this for each BES Cyber System, EACMS, PACS and PCA, even though the appropriate means of risk mitigation may be different for different devices.

CIP-011 R1, Information Protection: This was also developed with v5. It requires you to develop an information protection program (same as a plan, as far as I’m concerned) for protecting BES Cyber System Information in storage, transit, and use.

CIP-010 R4, Transient Cyber Assets and Removable Media: This was developed with v6.  The entity must develop a plan to mitigate the risks caused by use of TCAs and RM within the ESP.

CIP-003 R2, Assets containing Low impact BCS: In the v7 version due for compliance on 1/1/20, the entity needs to develop a plan for protecting assets containing Low impact BCS in five different areas.

In addition, there’s CIP-013, which is due for compliance on 7/1/20.  It is the first CIP standard (in fact, probably the first NERC standard, period) that explicitly allows consideration of risk. It does this because FERC knew, when they wrote Order 829 in 2016, that there is no way for a utility to cost-effectively mitigate supply chain security risks if they don’t develop a plan that is based on risk.[i] CIP-012, currently being balloted, is also plan-based.

And this isn’t the end. The CIP Modifications Standards Drafting Team is working on (and currently has out for comment) revised CIP standards and definitions to officially permit CIP compliance for virtualized systems (as opposed to NERC’s current “Don’t ask, don’t tell” policy for virtualization, as well as any other area where the current standards are either ambiguous or just don’t reflect reality).

One key component of this effort is rewriting the most prescriptive requirements, like CIP-007 R2 and CIP-010 R1, as objectives-based. While this is being done specifically to address virtualization, I am sure all NERC entities will welcome this development, regardless of whether they have virtualized systems in their OT environment or not. That's the good news. What's the bad news? The changes the SDT is talking about now are years away from coming into effect. In the mean time, everyone still has the prescriptive requirements they love so much.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post, including compliance with CIP-013; we also work with security product or service vendors that need help articulating their message to the power industry. To discuss my services, please email me at the same address so we can set up a time to talk.

---

[i] And I would take this one step further: There’s no way that a utility can cost-effectively mitigate any security risks except by developing a plan that is based on risk. So what about all the risks addressed by the current prescriptive CIP requirements? They are being effectively mitigated now, but at a cost that is much higher than is needed, simply because the prescriptive requirements don’t allow for any consideration of risk. More importantly, there are a number of major risks – phishing, ransomware, APTs, etc. – that aren’t addressed by CIP at all now, and there’s no appetite on FERC or NERC’s part to go through the long and painful standards development process to incorporate these risks into CIP. If all of the current prescriptive CIP requirements were replaced by a single requirement that mandated that the utility develop a plan to achieve the objectives of mitigating each of the risks currently addressed by the prescriptive requirements, as well as other risks not addressed at all now, then the utility could decide how to allocate its resources, based on risk, so that the maximum amount of total cyber risk was mitigated, given the dollars available to spend. In my opinion, this would be the ultimate solution to CIP’s problems.

Hoover Dam

I must admit that Las Vegas isn’t my favorite city. In fact, I don’t think it makes my top 100 or even top 500. That’s why my heart sank when I heard last year that this year’s GridSecCon would be there. I was especially disappointed because all the other GridSecCon’s that I’ve attended have been in (or near) old historic cities with nice architecture to look at: St. Paul, MN last year, Quebec City in 2016, Philadelphia in 2015, San Antonio in 2014 and Jacksonville, FL in 2013 (OK, Jacksonville itself doesn’t count as old and historic, but it’s not far from St. Augustine, which is definitely both!).

However, there was one really good thing about Las Vegas: It’s very near Hoover Dam, which I’d never seen. And since there was a conference tour scheduled for Friday (there are always tours on Friday morning of the conference, to power plants, substations, etc.), I signed up for that early.

The great thing about this tour was that it was for power industry insiders and was set up by WECC, who co-ran the conference and of course audits the US Bureau of Reclamation, which runs the dam. This proved very fortunate in three ways:

1.  We got a good briefing on the dam before our tour, and during the tour we had two longtime workers who encouraged us to ask “any dam question you want”. We heard a lot of interesting stories about construction, etc., including the poignant fact that, of the 112 deaths associated with construction, the first and the last were a father and his son
2. You have to take an elevator down into the dam for the tour. The first elevators take you to the level where all the generators are visible and we could walk among them – quite neat, of course. But there’s a second, smaller set of elevators that go down to the waterline level, where you can walk out along the base of the dam. And that was very cool, indeed. This second level isn’t available for public tours.
3. There are two elevators from the top of the dam. One is for the public tours, and has an hour wait (it looked like that when we were there). The other is for the workers, which we went on – thus saving a whole hour!

It’s obviously quite an engineering marvel (I love the figure we were quoted regarding throughput: 92,000 gallons per second go through each half of the dam – there are identical halves, one in Nevada the other in Arizona, since the state line runs down the middle of the river – meaning the total is about 184,000 gallons per second!).

But what I found most impressive was how the people we talked to really believed in what the dam was meant to do. Of course, irrigation and flood control were its biggest purpose, but the second was bringing electricity to millions of people, many of whom hadn’t had it before. And a third purpose is seen in its construction period: 1931-1936, the depths of the Depression. It gave jobs to many thousands of people (almost all men, I’m sure. And virtually no minorities. There were limits to the idea of progressivism back then). In an era when government is often seen as at best ineffective and at worst destructive, this is a government project that can truly be said to have done a huge amount of good. And I suspect there are one or two more like it…

I want to thank Brandy Daniels of WECC, who organized the tour and provided the group picture.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013; we also work with security product or service vendors that need help articulating their message to the power industry. To discuss this, you can email me at the same address.

A good news article on CIP-013

I have pointed out many times in this blog that by far the best coverage of cyber security issues in the energy industry is that provided by the web-based Energy and Environment News. Unfortunately, the subscription cost for that is out of the range of us non-one-percenters, but you should look into having your organization subscribe (not just for cyber, but all energy news).

So it’s not surprising that E&E News provided the only coverage that I have seen so far of FERC’s approval of CIP-013 last week. I recommend you read this article (if you are asked to do a free trial subscription, I highly recommend it, since they have good articles every day, with cyber articles about the power industry usually at least a couple times a week. At least you’ll be able to read all the articles while your trial lasts).

In my last post (on Thursday, the day FERC approved CIP-013) I provided a brief summary of FERC’s Order 850 and said I’d have more to say after I had time to read it carefully over the weekend. Well, guess what? What I have to say now isn’t terribly different from what I said in my brief summary:

1. FERC left the implementation period at 18 months, rather than order it be shortened to 12 months, as they had suggested in their NOPR in January.
2. As they also suggested in January, they ordered that NERC add Medium and High impact Electronic Access Control and Monitoring Systems to the applicability of the standard, beyond the current Medium and High impact BES Cyber Systems. They are giving NERC 24 months to do this, which is twice as long as they gave NERC to develop the entire standard in the first place (however, when I say “They”, I need to point out that there is only one Commissioner still in office from when FERC issued Order 829 in July 2016. And that Commissioner – Cheryl LaFleur – dissented from the Order, since she very rightly didn’t believe that one year was enough time for NERC to do a good job of drafting the standard and going through the long and politically fraught approval process).
3. Regarding the other items that FERC suggested in their NOPR should be considered for applicability in CIP-013 – Physical Access Control Systems, Protected Cyber Assets and Low impact BES Cyber Systems – FERC said on Thursday that they will hold any decision until they see the final version of NERC’s supply chain security study, which is due early next year.
4. To be honest, the most interesting part of Order 850 was at the end, in paragraphs 78 and 79. Paragraph 78 discussed comments FERC received about the meaning of “vendor”; FERC’s answer mistakenly said that NERC had defined the term. Actually, the story behind that is a lot more nuanced, and points to a larger problem with the NERC CIP standards in general. Since I specialize in Larger Problems in this blog, I will dig into this later, although I have a number of other posts in the queue before that can happen.
5. Paragraph 79 was sparked by comments suggesting that NERC entities would be on the hook for deficiencies in cyber security practices by their vendors. FERC correctly pointed out that the standard specifically states that entities won’t be on the hook, although they are still responsible for mitigating the risk that the vendor presumably didn’t mitigate. For example, if your vendor agrees (in contract language or just in a letter) that they will help you mitigate new vulnerabilities that affect their products and then doesn’t do that in one case, you still have to take other steps to mitigate the risk caused by that new vulnerability.

But, through work I am currently doing with a vendor to the power industry, I’ve come to see that there’s a very neat, elegant solution to the problem of obligating vendors to take certain steps - and penalizing them if they don’t. What is really cool, though, is that this is also a solution to the vendors’ problem (which I mentioned in the E&E News article) that some NERC entities are downloading contract language from various sources on the internet and sending it to the vendors, demanding that they include that in their contracts. Of course, there’s no way the vendor can deal with this big variety of requests. My solution will also solve that problem. Not bad – two big problems nailed with one solution! Next up: a neat single solution for the two big problems of global poverty and people who talk loudly on their cell phone on trains.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013; we also work with security product or service vendors that need help articulating their message to the power industry. To discuss this, you can email me at the same address.

FERC approves CIP-013 – a quick overview and a wonderful offer

FERC approved CIP-013 this morning. You can find the press release here and the Order here. The only change they ordered was that EACMS be included in the scope for CIP-013. However, rather than ordering NERC to do this very quickly - as their January NOPR seemed to suggest they would - they gave NERC two years to do it (i.e. twice as long as they gave NERC to develop the standard in the first place!). They left some other big questions on the table pending the final version of the NERC/EPRI report I mentioned in my post on Tuesday. I’ll have a full post on Order 850 early next week, after I’ve been able to spend some quality time with it.

I haven’t had time to read the Order (and won’t until the weekend), because I’m still at NERC’s GridSecCon, which – once again – has proved to be a great event. Bill Lawrence and his E-ISAC team have again done a wonderful job of programming the sessions and bringing interesting vendors to the exhibition. You should try to attend GridSecCon 2019, which will be this same week next October, at a location TBD (they will make the decision soon, since some people make their travel plans a year in advance of GridSecCon – it’s so good).

However, while you’re anxiously awaiting my post on the full Order, you should consider the great offer below:

Because FERC didn’t order any change in the Implementation plan (as they also intimated they would do in the NOPR), the compliance date for CIP-013 will be July 1, 2020. However, as I said in this post in January, you really need to aim to have your supply chain cyber security risk management plan (which is the main “deliverable” of CIP-013, of course) finished six months before the compliance date, to give you time to have it reviewed by your Region – and then to make whatever changes they suggest.[i] 

So I recommend you consider January 1, 2020 to be your “plan completion date” (although I’ll give you ‘til Jan. 2. I don’t want to spoil anybody’s New Year’s Day!). Once your Region has given you their comments on your plan and you’ve adjusted the plan to address those comments, you should then put it into place, with hopefully at least a little time before the July 1 compliance date. And if you’re one of the entities that likes to come into complete compliance at least 90 days before the compliance date (as did a number of entities for CIP version 5), then you need to aim for October 1, 2019 – which of course is less than a year away.

Fortunately, Tom Alrich LLC can help you get a good start on developing your program. We are offering a free two-hour webinar workshop for your organization on CIP-013 and what you will need to do to comply with it.

The purpose of the workshop is to get the different groups that will be involved in complying with CIP-013 – supply chain, legal, cyber security and NERC compliance - thinking about the issues that are involved; and in case you haven’t been reading my posts on this subject, complying with CIP-013 will be very different from complying with any of the previous CIP standards. Some of the topics to be addressed in the workshop include:

1. CIP-013 is really the first risk-based NERC standard. While it’s not mandatory, it is highly advised to classify both BES Cyber Systems and vendors by the degree of risk they pose, with different plan strategies corresponding to different degrees of risk. How can you do this?
2. The standard doesn’t list the particular risks that you need to address in your supply chain cyber security risk management plan. How can you compile a credible yet manageable list of risks[ii] for your plan?
3. CIP-013 is the first CIP standard that doesn’t prescribe any particular actions - it simply requires that you develop and implement a plan[iii]. How will the plan and its implementation be audited?
4. While attention has mostly focused on the requirement to mitigate vendor risk, the entity also needs to mitigate implementation risks and risks of transition between vendors, as well as risks posed by services vendors. What are possible strategies for these?
5. While much of the discussion of CIP-013 has focused on the question of getting vendors to agree to contract language, it is a fact that contract language isn’t the only way – or even the preferred way – to get vendor agreement to take actions required by CIP-013. What are good strategies for obtaining vendor commitment, so that the high-cost (both cost in money and cost in ill-feelings between you and the vendor) option of demanding contract language can be avoided, except in cases where it is really needed?
6. How do you document that vendors followed through on their promises? And what do you do if a vendor doesn’t keep its promise, or won’t make any promise to you in the first place?

If you would like to discuss the workshop with me, please drop me an email at tom@tomalrich.com or call me at 312-515-8996. Thanks!

---

[i] As the post points out, it will be a great help to you – Mr/Ms NERC CIP compliance professional – to have your CIP-013 plan reviewed by your NERC Region before the compliance date – since that’s when it needs to be fully implemented. Another reason to review the plan before that date is that most of the Regions won’t want to review any plans after the date, since at that point it becomes auditable and they’ll be worried about auditor independence issues. And I say you should aim for six months before the date, because if you wait until say three months before (i.e. April 1, 2020), there might be a big queue of plans to be reviewed – and the Regions won’t feel compelled to get them all reviewed before the compliance date.

[ii] It would be nice if the drafting team (or somebody else like NERC or the Regions or NATF) had put together a set of risks that should be considered in the plan (but not necessarily mitigated, if the entity thinks they pose low risk to the BES), but that didn’t happen. However, there are certainly other ways to develop a fairly comprehensive list of the important risks; we’ll discuss that in the free workshop.

[iii] CIP-013 R1.2 lists six general risk mitigation goals that must be addressed in your plan, but doesn’t require you to take specific steps to achieve any of these six goals. So the risks behind these goals need to be included in your R1.1 plan – but this doesn’t mean the six goals are all that needs to be included. FERC’s Order 829 made clear they are looking for the entity to at least consider all important supply chain cyber risks, then use its available resources to mitigate those that pose the highest level of risk, in order of their importance.

FERC will approve CIP-013 on Thursday

The first item on the just-released agenda for FERC’s monthly Sunshine Meeting this Thursday reads “RM17-13-000 Supply Chain Rise Management Reliability Standards”. You probably are surprised that FERC seems to be moving in a direction nobody had predicted: Rise Management. And I have to admit that I am fairly surprised at this, although I’m not sure exactly what “rise” refers to and why it needs to be managed. However, the more likely explanation for this neologism is that it’s actually a misspelling of “risk management”. And then it makes perfect sense: FERC has CIP-013 on their agenda for Thursday.

And now I’m going to go way out on a limb: I predict that FERC will approve CIP-013 (along with the new versions of CIP-005 and CIP-010, which are part of the CIP-013 “package”), not remand it (of course, they’ve never remanded a CIP standard that was presented to them, no matter how many problems they found with it – so saying this isn’t much of a stretch). But the big question is what else they will include in the order. I’m not going to predict exactly what they’ll include (a fool’s errand, if there ever was one!), but I will tell you some of the things I’m looking at:

1. EACMS

First, what will they say about Electronic Access Control or Monitoring Systems (EACMS)? In FERC’s Notice of Proposed Rulemaking (NOPR) on CIP-013 this January, they evinced a lot of concern that these aren’t included in CIP-013 now (as approved by NERC, CIP-013-1 only applies to Medium and High impact BES Cyber Systems). I would guess they will order that EACMS be included, but the question is when they will need to be included.

FERC’s normal procedure is to state that the standard they’re approving needs to be changed in a certain way, but not to provide a particular deadline for doing this. Because NERC’s Rules of Procedure don’t allow a standard that has been approved by the NERC Board of Trustees to be changed, this in fact means that this item will go on the agenda for the next version of the standard being approved. But FERC does have the ability to order a “compliance filing” (I may not have the terminology right), providing a short deadline for something to be done. If they do that for EACMS, there would have to be a) a quick drafting/balloting/approval process for version CIP-013-2, which includes EACMS in scope, followed by b) the start of development of CIP-013-3, to include everything else that FERC orders.[I]

2. Low impact BCS

What will they do about Low impact BES Cyber Systems? Should they be included in CIP-013? When FERC ordered NERC to develop a standard for supply chain cyber security risk management, they stated that it should apply to BCS, but they didn’t mention anything about impact ratings. The first draft of CIP-013 included a separate requirement for Low impact BCS, which was perhaps one of the reasons why that draft received 9% approval. Not too surprisingly, Lows were removed from the subsequent drafts.

In their NOPR, FERC didn’t say whether or not they wanted Lows included in CIP-013, but they did order NERC to look at the issue. NERC’s report (really written by EPRI) came out in September. Not too surprisingly, NERC/EPRI didn’t directly weigh in on whether Lows should be included in CIP-013, but it seemed to me that they think it wouldn’t be a terrible idea (of course, nobody thinks that Lows should have to comply at the same level as Mediums and Highs do). So I think it’s likely that FERC will order that Lows be included, but since they won’t order this as a compliance filing, it will be done at the normal NERC standards development and approval pace – meaning the earliest that Lows would have to comply with CIP-013 would be 2 ½ - 3 years from now.

Implementation period

In their NOPR, FERC also suggested that the proposed implementation period for CIP-013, eighteen months, was too long; they suggested that twelve months would be more appropriate. For those keeping score at home, if FERC approves CIP-013 this week, the implementation date will be July 1, 2020. So if the implementation period were shortened to twelve months, the date would be January 1, 2020.

However, I think it’s unlikely that FERC will do this, since they can’t just wave their magic wand and change the CIP-013 implementation plan. In order for the plan to be changed, there will need to be a number of steps, including drafting a Standards Authorization Request, impaneling a Standards Drafting Team (although it’s very likely the team that drafted CIP-013-1 will be assigned this task), drafting the changes (and if FERC orders a compliance filing for EACMS as well, that would also probably be included in the SAR), balloting the first draft, revising the draft after the first ballot doesn’t get the required 67% approval…all the way up to FERC review and approval. As long as this whole process takes less than 90 days (which is like saying “As long as Congress can pass a new budget within a week of introduction…”), the implementation date will still be April 1, 2020. And if it takes more than 90 days, the date will be July 1, 2020 – so FERC will have advanced the implementation date by exactly 0 months, 0 weeks and 0 days, and put through a huge amount of grief in the process. Seems like a lot to go through to achieve nothing.

The big one

The biggest thing I’m looking for in FERC’s Order approving CIP-013 – but I’ll admit I’m not at all expecting to find it – is a discussion of what I see as the fundamental problem with CIP-013:

1. R1.1 is really the guts of the standard. It requires the entity to look over all of the supply chain cyber security risks they face and develop a plan to mitigate the most important of those risks.
2. R1.2 lists six particular mitigations that must be included in the plan (each of which corresponds to a particular risk, which isn’t stated but is definitely implied). These are listed because FERC ordered that those six mitigations be included in the standard.
3. While these six mitigations all address important risks, they are far from being the six most important supply chain cyber security risks facing the participants in the Bulk Electric System. It’s easy to find others.
4. One risk that jumped into the news a couple weeks ago (but has always been in the background), was that of compromise of hardware by a nation-state. Of course, I’m referring to the Chinese motherboard compromise reported by Bloomberg.
5. Another important risk not addressed in R1.2 is the risk that software developed by the NERC entity itself will contain vulnerabilities that aren’t patched. CIP-013 R1.2.4 and R1.2.5, along with the much-loved CIP-007 R2, do a very good job of addressing the risk that software developed by a vendor will contain unpatched vulnerabilities – but obviously they are no help when it comes to software developed by the entity itself.
6. A third serious risk, again not addressed in R1.2, was outlined in a sidebar to the October 8 Bloomberg Business Week article on the Chinese attack. The sidebar recounts how “at least two…customers downloaded firmware…meant to update..network cards. The code had been altered, allowing the attackers to secretly take over a server’s communications…” R1.2.5 wouldn’t have addressed this problem, since it just requires that the end user verify that the patch they’ve downloaded actually came from the vendor. If the vendor itself has been compromised, the user is just SOL (which I believe refers to a new designation in the NERC Functional Model).
7. And when you think about it, if the vendor’s network has been compromised (and DHS recently indicated that the Russians compromised lots and lots of vendors to the power industry, although they seem to have some problems distinguishing vendors from utilities and control centers from generating plants. Another post on this problem is coming to a blog near you soon), this leads to an almost infinite number of risks. For example, what if you get a correctly-formatted technical bulletin from your relay vendor, telling you that they need remote access to five relays immediately, to check to see if they have a vulnerability they’ve just discovered – and when you give them that access, they use all of them to open circuit breakers on key transmission lines? After one such incident, you would realize what was going on, but the damage would have been done (especially if they did that to five other large utilities for say 25 other lines).

The moral of this story is that, if the entity just focuses their CIP-013 compliance effort on R1.2, they will be just as susceptible to the risks described above as if they hadn’t complied at all. And yet I have yet to hear anything from NERC, including the Implementation Guidance for CIP-013 itself, that focuses on anything other than R1.2 – or that even mentions the idea of developing an R1.1 plan that carefully considers all important supply chain cyber security risks.[ii]

Of course, the big problem here is that NERC only officially knows how to audit prescriptive requirements. A non-prescriptive, plan-based requirement like R1.1 just falls off the radar screen – when all you have is a hammer, everything looks like a nail.

So I’m hoping that FERC will make some mention in their Order that they want NERC to work with the entities to develop good supply chain cyber security risk management plans that consider all important supply chain threats (and maybe give them a list of threats they should consider). But I ain’t holding my breath. In other words, I think true supply chain cyber security risk management will likely remain an elusive goal for the power industry even after CIP-013 is fully implemented, even though without doubt (at least in my mind) one of the two biggest sets of cyber risks to all industries now are supply chain risks (think Target, NotPetya, etc).[iii]

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013; we also work with security product or service vendors that need help articulating their message to the power industry. To discuss this, you can email me at the same address.

---

[i] FERC’s NOPR not only questioned why EACMS weren’t covered by CIP-013, but also Physical Access Control Systems (PACS) and Protected Cyber Assets (PCAs). However, they obviously think the lack of EACMS in CIP-013 poses a much bigger BES risk than does the lack of PACS and PCAs, which is why they raised the idea of ordering they be added immediately.

[ii] I readily admit that the big problem with R1.1 is it doesn’t provide any list of risks that need to be considered for inclusion in the plan, so it’s a lot to expect an overstressed NERC CIP staff at any but one of the largest utilities to do a good job of that for themselves. At one point, I hoped that some group like NATF or the trade associations would produce a list of threats that should be considered (not an enforceable requirement of course, but at least a starting point for most utilities). However, the very disappointing guidance that NATF recently put out led me to honestly wonder whether the author(s) had even read R1.1.

[iii] The other biggest risk is that posed by phishing. Which by the way is another risk not addressed at all by NERC CIP.

What does CIP-013 R1.2 tell us?

In a recent post, I pointed out that CIP-013 R1.1 is close to un-auditable. This is because it requires the entity to develop (and implement) a supply chain cyber security risk management plan, but it gives very little information about what should be in the plan. Specifically, it should include a list of risks that should be addressed in the plan, but it doesn’t do that – meaning the entity is on their own to draw up that list[i]. However, R1.1 does provide some high-level information on what the plan should cover; I discussed that in the post.

Now I want to turn to R1.2. What does this requirement tell us? As you know if you’ve read CIP-013, R1.2 provides a list of six “things” that the entity needs to do. What are these things, and how do they relate to the supply chain cyber security risk management plan that’s mandated in R1.1?

Let’s look at the first thing, numbered R1.2.1: “Notification by the vendor of vendor-identified incidents related to the products or services provided to the Responsible Entity that pose cyber security risk to the Responsible Entity”. Let’s pull this apart. It’s essentially saying that a vendor needs to notify you (i.e. the NERC entity responsible for compliance with CIP-013) when they identify a security incident related to products or services you buy from them.

How does this relate to the R1.1 plan? R1.1 says the Responsible Entity needs to identify supply chain cyber security risks. It doesn’t also say you also have to mitigate those risks, but that is clearly implied. What would be the purpose of CIP-013 if you were just required to identify a bunch of big, hairy risks you face, then you could simply declare your job done? It would obviously make no sense to do that, and I predict that anyone who omits risk mitigation in their actual CIP-013 R1.1 plan will not fare well at the hands of their auditor.[ii]

So the six things in R1.2 must be either risks or mitigations. They clearly aren’t risks, so they must be mitigations. In other words, each one lists a high-level mitigation for a particular supply chain cyber security risk. The entity is required to implement all six of these mitigations. While the risks that are being mitigated aren’t explicitly stated, it isn’t too hard to state them:

1. R1.2.1 (quoted above) describes mitigation of the risk that a security breach at a vendor won’t be reported in a timely fashion to users of the vendor’s products, leading to cyber security incidents involving the vendor’s systems installed on user networks.
2. R1.2.2 reads “Coordination of responses to vendor-identified incidents related to the products or services provided to the Responsible Entity that pose cyber security risk to the Responsible Entity”. This is mitigation of the risk that a vendor whose products are found to have a serious vulnerability won’t provide adequate assistance to their users in mitigating that vulnerability, leading again to security incidents at users, caused by the vendor’s products.
3. R1.2.3 reads “Notification by vendors when remote or onsite access should no longer be granted to vendor representatives”. This is a mitigation of the risk that a vendor will terminate an employee who had electronic and/or physical access to users’ systems, and that ex-employee will take out their resentment for this action on those systems.
4. The mitigation described by R1.2.4 is “Disclosure by vendors of known vulnerabilities related to the products or services provided to the Responsible Entity”.  This mitigates the risk that users will experience cyber security incidents that could have been avoided had they been notified of the vulnerabilities.
5. R1.2.5 reads “Verification of software integrity and authenticity of all software and patches provided by the vendor for use in the BES Cyber System”. Of course, this is really two mitigations, addressing two risks. Verification of software integrity mitigates the risk that a malicious actor will tamper with a vendor’s software or firmware during the process of delivery (electronic or on physical media) to the user. Verification of software authenticity mitigates the risk that a malicious actor will set up a fake web site or otherwise impersonate the vendor, so that the user obtains software or firmware that is damaging to their systems.
6. Finally, R1.2.6 describes this mitigation: “Coordination of controls for (i) vendor-initiated Interactive Remote Access, and (ii) system-to-system remote access with a vendor(s)”. The risk here is the one behind the recent (but ongoing) Russian attempts to compromise grid control systems by first compromising vendors and then hijacking their ability to access their products remotely for maintenance and troubleshooting purposes.

So CIP-013 R1.1 calls for a supply chain cyber security risk management plan, but doesn’t identify the risks that need to be addressed in that plan. On the other hand, R1.2 simply lists six supply chain risks (and their mitigations) and tells the entity to “address” them. So how are the two parts of R1 related to each other? Is this an either/or requirement, where you have to comply with R1.1 or R1.2, but not both? My interpretation is that the six risks in R1.2 must without question be included in the R1.1 plan, but they in no way are intended to be the only risks that are identified and mitigated in the plan.

It would be very hard to argue that the six risks are all that need to be included in the plan, since R1.1 calls on the entity to “identify and assess cyber security risk(s) to the Bulk Electric System from vendor products or services resulting from: (i) procuring and installing vendor equipment and software; and (ii) transitions from one vendor(s) to another vendor(s)”. It certainly doesn’t just say “Develop a plan to implement the six risk mitigations in R1.2.”

Yet if you look at almost any of the guidance on, and discussion of, CIP-013 that has come out from NERC or the Regions as well as other organizations like NATF, you’ll see close to no discussion of anything but these six things^[iii]. I haven’t seen any discussion from NERC or anyone else (other than myself) about what should go into a CIP-013 R1.1 supply chain cyber security risk management plan.  Is this because the six risks (and their mitigations) that I’ve just enumerated are far more important than all other supply chain cyber security risks? Is that why the drafting team specifically called them out in R1.2?

I certainly won’t deny that these are important risks. However, they are called out in R1.2 because FERC specifically required that these six mitigations be included in the new standard, when they ordered that it be developed in July 2016[iv].

But there are certainly lots of other supply chain cyber security risks that should be addressed as well. Last week, I was fortunate to be able to attend the Software and Supply Chain Assurance Fall Forum, held at the offices of the MITRE Corporation in McLean, Virginia. This is a quarterly event that has been going on for a couple of years, although I just recently heard about it. It is a gathering of cyber security professionals working at various U.S. Departments (including DoD) and the “Three-letter Agencies”, who are very concerned about supply chain security. I will write a post on some things I learned at this meeting in the near future.

The group now includes attendees from private industry, and this meeting’s theme was the energy industry. There was some discussion of CIP-013, and Howard Gugel of NERC gave a very good presentation on the standard and all the activities going on around it. However, Howard unsurprisingly didn’t dwell on the idea of the R1.1 plan (frankly, I’m not sure he even mentioned it) and spent more time discussing the six mitigations in R1.2, and especially the question of the role of contract language in obtaining vendor participation for addressing those six mitigations (since they are all vendor risks).[v]

In the Q&A period, two audience members brought up risks they thought were very important, that they were surprised weren’t mentioned by Howard. One questioner brought up[vi] the risk of malicious code and back doors being embedded in chips purchased by a vendor in Asian markets because they are low cost – which has been discussed a lot in the past few years.

The other was the risk of unpatched vulnerabilities in software that was developed by the NERC entity itself, or was custom-developed for it. Both CIP-007 R2 and CIP-013 R1.2.5 only deal with patches released by vendors for their software. The gentleman who asked the question (who was the main organizer of the conference) was incredulous that this risk was nowhere addressed in CIP. I pointed out that this was the whole idea of the R1.1 plan – that the entity needed to consider all of the important risks in their plan, not just the six from R1.2. Of course, there are lots of other supply chain risks we could discuss as well – some of which you’ll find in the NERC/EPRI Supply Chain Risk Assessment report from September.

And folks, here’s the point of this post: Supply chain is by far the biggest cyber threat area for most industries now, and without doubt it’s the most serious for the power industry – just look at the Russian attacks. If the industry treats CIP-013 as just being about the six risks in R1.2 and nothing else, CIP-013 won’t mark a watershed in grid security (as well as supply chain security in general) – which it could be if implemented properly. Even more importantly, you will be leaving your OT network completely open to a lot of serious supply chain security risks, just because they didn’t happen to be one of the six listed in R1.2. I certainly hope NERC will wake up and realize they have to make an effort to help the entities understand the need for developing a comprehensive supply chain cyber security risk management plan. And maybe FERC can nudge them a little in that direction when they approve CIP-013.

Before I go, I want to bring up one concern that a lot of people may have – especially when they hear me bringing up the need to “comprehensively” treat all supply chain security risks. Does this mean that, instead of just worrying about six risks, you’ll have to worry about say five times that number? The six risks are already looking like a good amount of work. Do you really have the bandwidth and the resources to address all important supply chain security risks in your plan?

I want to remind you that this isn’t a supply chain cyber security plan; it’s a supply chain cyber security risk management plan. The essence of risk management is that you consider the degree of risk posed by each threat you face, and then allocate your resources toward mitigating the most serious threats. For threats that pose less risk, you allocate fewer resources or perhaps none at all.

Whatever the amount of dollars or hours you have available to spend on risk mitigation, you will always mitigate the most risk – i.e. get the most bang for your buck – if you allocate those toward the greatest risks. With prescriptive requirements – like most of the “legacy” CIP requirements, and like CIP-013 R1.2 – you don’t get any choice on how you allocate your resources. You have to spend whatever it takes to comply with the prescriptive requirements, and if you have a few nickels left over you might try to mitigate a couple other risks to a small degree – even though you may think these other risks are much greater than the ones addressed by the prescriptive requirements. When it is up to you to identify threats, rank them, and mitigate them according to the risk each one poses, your entity will be more secure – and the BES will be, too. What’s to lose by doing this?

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. And if you’re a security vendor to the power industry, TALLC can help you by developing marketing materials, delivering webinars, etc. To discuss any of this, you can email me at the same address.                   

---

[i] To speak more accurately, the problem isn’t exactly that R1.1 doesn’t provide information about what should be in the plan. The real problem is that NERC’s very prescriptive enforcement format (based on auditing, of course), simply isn’t suited for plan-based requirements like CIP-013 R1. If there were a collaborative format in which the Regions could freely work with an entity to make sure they developed a good plan, then worked with them to make sure the plan was implemented well, the fact that R1.1 doesn’t provide much guidance on what should be in the plan wouldn’t matter so much.  

But because NERC only knows how to do strict auditing-based enforcement, it simply isn’t possible for them to officially work with the entity to understand what they need to do to develop a good plan – because of auditor independence concerns, of course. To some degree, all of the Regions actually do work with their member entities as they come into compliance – and some more than others. But it is always a kind of don’t ask/don’t tell sort of enterprise. I hope someday NERC can have a different enforcement model for the CIP standards, since cybersecurity is completely different from electrical engineering – which is ultimately what the 693 standards are based on. Prescriptive requirements and audits just don’t work for cyber. 

[ii] However, this is a pretty amazing omission. It would be like an ordinance for obeying stop signs spending a lot of time describing the different types of stop signs, but never actually requiring that you stop for one! As I said, I don’t think there’s any possibility that this omission will invalidate CIP-013-1, but I certainly hope FERC adds this to their to-do list for NERC for CIP-013-2, as they develop their Order approving CIP-013-1. I can think of a few other things I would like to see on that to-do list as well, so if one of the Commissioners wants to call me or invite me over for beers (I would come, even though I live in Chicago! – although I wish they’d invited me when I was in DC and McLean, VA last week), I’ll be glad to describe my ideas to them. These have all been stated in my posts at various times, but who has time to read through all of those things? I certainly don’t. 

[iii] The one exception to this rule that I know of is the recent NERC Supply Chain Risk Assessment report, which was prepared by EPRI. It is a good document, and does list some different supply chain risks that have nothing to do with the six things. However, it also doesn’t even pretend to provide guidance for CIP-013 compliance. 

[iv] The six mitigations appear at different points in the order, but they all were directly ordered by FERC. The drafting team had the good idea to gather them all in one place in the new standard, rather than have six separate prescriptive requirements for them. 

[v] As much as I think it’s a big mistake just to focus on the “six things” in developing your R1.1 plan, I think it’s an even bigger mistake to believe that the only real way to mitigate the six risks is to use contract language, as I discussed in this post. 

[vi] I admit this person brought up his question during a different discussion of CIP-013 on the first day of the conference (Howard spoke on the second day). My response discussed below was also given on the first day.