Tuesday, February 20, 2018

What about Resiliency?

Yesterday I was interviewed by the always astute Blake Sobczak of Energy and Environment News[i] about a cyber security report issued by the White House last week, and specifically about what it had to say about the cyber security of the power grid. The article appeared today.

The discussion of the grid in the report was certainly good and not terribly inflammatory. My feelings about this report are very similar to my feelings about Ted Koppel’s book Lights Out, which I discussed in this post in early 2016: What Koppel was writing about had very little to do with cyber security. It had everything to do with the amount of devastation that any widespread and prolonged grid event could wreak (and we’re talking about a much more serious event than even the 2003 Northeast blackout), whether caused by a huge weather event (even bigger than Superstorm Sandy), solar storm, EMP event, physical attack, or yes a cyber attack. Even more importantly, the book documented in horrifying detail the country’s almost total lack of preparedness for this. Unfortunately, whoever wrote the book jacket decided the book would sell more copies if it were made to appear as a book about a big cyber attack on the grid, without doubt to leverage the popular movies that had depicted such an attack. So that is how the book is known, but it isn’t what’s actually between the covers.

In the same way, the section of the White House report that I commented on quotes the 2015 study by Lloyd’s and the University of Cambridge that estimated the total cost of a worst-case cyber event at $1 trillion. I totally agree that a worst-case cyber event could cost that much. But there are two considerations: 1) The cost would be the same whether the cause of the event were weather, solar storm, or anything else; and 2) I’d say a worst-case cyber event is probably the least-likely cause of such a huge grid outage – with a probability somewhere around 1 in 10 to the 10th or the 20th power. I think a solar storm is far more likely to cause such an event (and the 1859 Carrington Event probably would have done it, had it occurred today. We’d better hope these aren’t every-200-year events!).

Although the report didn’t advocate any particular policy actions (it was actually quite good as a broad overview of the risks posed by cyber security weaknesses across the US economy), in my comments I anticipated what I thought might be the typical response regarding security of the power grid: “Oh my God, we have to do something about this! We need to really tighten up the cyber security standards so that the power industry (that would be you, Dear Reader) doesn’t let his happen to us!”

My response to this response, whether triggered by the White House report, Ted Koppel’s book or any number of alarmist statements, is that the situation won’t be improved by requiring say a 10-fold increase in the severity of the NERC CIP requirements. There has never been an outage caused by a cyber attack in North America; nor has there ever been even a documented penetration of a control network in a grid asset of any significance (I make this qualification because there was a penetration of a 5MW dam in New York state in 2013. I know of no other such penetration, although if anyone knows of another I’d appreciate your letting me know about that in an email). And of course, any amount of increased cyber security spending by the power industry will do nothing to mitigate the danger of a different type of event like a solar storm (and NERC is currently in the process of approving a draft standard to address solar storm risks by “hardening” grid assets).

I have long believed that the best protection against widespread outages, no matter what the cause, is microgrids. If the great majority of populated areas in North America were protected by microgrids, there could still be widespread grid events which would destroy huge fixed generation and the high-voltage transmission network - but these events wouldn’t cause widespread outages. Each microgrid would automatically activate its local generation resources – wind, solar, gas turbines – and keep on chugging. Of course, the problem at the moment is that microgrids are very expensive to implement and they’ve only been implemented in a small number of high-value locations (like the New Jersey transit system, which was knocked out by Hurricane Sandy, greatly complicating the recovery from the storm). But if we’re talking about throwing a lot more money at grid security, wouldn’t it be a lot better to throw it at a solution to all potential causes of a huge grid outage, rather than just at reducing the already-infinitesimal probability of just one of those causes – a massive cyber attack?[ii]

Before I go, I do want to point one thing out: I don’t believe for a moment that electric utilities are paying too much for OT cyber security controls now. In fact, given the ever-tightening threat environment, I think it’s inevitable they will need to spend more every year for the foreseeable future. My issue is that a large portion of what NERC entities spend on NERC CIP compliance now goes simply to pure compliance activities, not to increasing the level of cyber security. I think that virtually all NERC entities understand that this is a tough world, and they will have to increase their cyber and CIP spending every year just to stay in the same position. But they will be increasingly reluctant to do this without changes in the NERC CIP compliance regime (and in the standards themselves) that will allow a much higher percentage of every dollar they spend on CIP to go towards cyber security.

And what are those needed changes? Funny you should ask. That is the topic of a book I am working on with two co-authors. You may roll your eyes and point out that I’ve been talking about this book for close to two years. That’s true, but we now have a lot of momentum and I’m sure we’ll have something out before the end of the year. And what is the solution we’re advocating? Well, you can find most of it in my posts, although I’ll admit you’ll have to look hard (and be able to tie a lot of bits and pieces together). I hope to have everything in a one-stop-shop this year!

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post. To discuss this, you can email me at the same address or call me at 312-515-8996.

[i] This online publication has the best articles about security of the energy sector of any publication I’ve seen, all written by either Blake or his colleague Pete Behr. They’re all very in-depth (a few even as long as my average post, heaven forbid!) and very well-researched – and this applies to all of the articles, not just those on cyber. Most online news feeds confine themselves to news feeds or reproductions of articles from other publications, so E&E News really stands out. This is a subscription-only publication, but I strongly recommend you try to get your own organization to subscribe to it. I recommended to my boss at Tom Alrich LLC that we purchase a subscription, but he hasn’t replied to my email yet.

[ii] And if you’re tempted to think that the big toughening of the NERC CIP standards would be paid for by “private industry” while the cost of implementing microgrids everywhere would have to be borne by the taxpayers, let me point out something to you: Every taxpayer is a ratepayer to their local electric utility. Where will the utilities get the money to comply with this huge increase in the cost of CIP compliance?

Friday, February 16, 2018

Lew on Patch Management Mitigation Plans

Five days ago, I wrote a post calling attention to Lew Folkerth’s December article on CIP-010 R1 (configuration management) compliance. In that post, I mentioned that you can now sign up to receive notices of new RF newsletters (where Lew’s column appears). I signed up and I’m glad I did, because yesterday I got notice of a new newsletter. This time, Lew’s article (which as usual goes under the name “The Lighthouse”) is about CIP-007 R2 patch management mitigation plans. Lew’s articles are always excellent, but this is still one of his best yet, in my opinion.

I must admit I hadn’t thought very much about patch management mitigation plans. R2.3 says “Mitigation plans shall include the Responsible Entity’s planned actions to mitigate the vulnerabilities addressed by each security patch and a timeframe to complete these mitigations.” I had always just assumed this was all you need to know about these plans.

It turns out I was wrong. Lew does an excellent job of pulling out what really needs to be in a mitigation plan and how it needs to be handled. None of this is in any way an add-on to the requirement, by the way. Lew is simply following the logical implications of what the mitigation plan needs to accomplish, both for security and compliance. Lew has always been most concerned about what makes the most sense from a security point of view. I won’t say everything he says in this article is needed strictly for compliance with the requirement, but doing it will certainly allow you to tell a good story when you get audited.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post. To discuss this, you can email me at the same address or call me at 312-515-8996.

Thursday, February 15, 2018

How Should Vendors “Comply” with CIP-013? Part 2

This is the second in a series of posts of how vendors can “comply” with CIP-013. Of course, vendors don’t have to comply at all, and they don’t have to help their power industry clients comply either. On the other hand, if they wish to remain a vendor to the power industry, a vendor is going to have to do everything they can to help their customers comply. It is inevitable that the vendor’s larger clients will have to comply with CIP-013, and they will need help in doing so.

I do want to emphasize that, if you’re part of a NERC entity who has to comply with CIP-013, you shouldn’t stop reading this post (or the subsequent ones in this series), on the idea that it doesn’t really have anything to do with you. This has everything to do with you, since – as I emphasized in the first post in the series – the only way I see a CIP-013 compliance program succeeding is if the NERC entity at least tries to partner with their vendors for compliance. You need to understand their position as much as they need to understand yours.[i]

In the first post in this series, I suggested that power industry vendors would be well advised to reach out to their customers before the customers reach out to them. I made this suggestion because I think it’s very likely that if the customer reaches out to the vendor first, it will probably be the lawyers and/or purchasing people who do this. This is because most of the discussion so far on CIP-013 compliance (with this blog hopefully being an exception!) has revolved around the question of contract language. But contract language is just one of the ways in which a utility can fulfill its obligations under CIP-013, and it’s probably the bluntest instrument available to the NERC entity to fulfill those obligations. I think that any NERC entity that focuses on contract language first, before even looking at all the options available to it to comply with R1.2, has already done both itself and its vendors a big disservice.

So the vendor should absolutely try to reach out first to their customers. How can they do that? There are of course lots of media available for this: notice on web site, email, USPS, phone call, webinar, onsite meeting, etc. While these are all good, I always favor the more personal ones above all. A webinar might be the ideal first step, since it’s delivered by a person(s) but it still has a structured content. Then the vendor could follow up with each customers by phone or in-person meeting to discuss next steps.

But before you can do a webinar, Mr./Ms. Vendor, you need to know what you’re going to say! And that means you need to have figured out your CIP-013 strategy[ii] – so that’s really the first step. How can you figure out your strategy?

This is of course rough and preliminary, but it seems to me that a vendor’s strategy should be to try to make the CIP-013 compliance process as easy as possible for the customer, period. I can assure you that your customers will be feeling every bit as uncertain and fearful about CIP-013 as you do. If you can convince them that you know the right way to cooperate for compliance, and you follow through on what you say you’ll do, what could possibly go wrong – except, perhaps, if you don’t in fact know the right way to cooperate and both of you end up at some sort of regulatory dead end?

How do you stay away from dead ends? By having a good methodology for designing your CIP-013 strategy. And what should you do first? Well, it seems to me that, if you’re going to design a strategy to help your customers do something, you need to figure out what they have to do. There’s a quick answer to that: They have to comply with CIP-013. Now that we have that out of the way, how do they comply with CIP-013?

As I discussed (at length) in this post, CIP-013 requires the NERC entity to develop and implement a supply chain cyber security risk management plan. Specifically, the plan has to address the three areas of risk listed in R1.1: (a) procuring vendor equipment and software; (b) installing vendor equipment and software; and (c) transitions from one vendor(s) to another vendor(s).

I suggest you start your CIP-013 strategy effort by brainstorming on how you can help your customers address all three of these areas of risk. To be honest, you might decide you can help customers in some ways that are either too expensive to be practical or not likely to yield a lot of benefit compared to the effort required; you can drop these ideas from your strategy.

For procuring vendor equipment and software, the NERC entity needs to address risk in six specific areas; these are listed in R1.2. The reason they are specifically listed in CIP-013 is because FERC specifically required – in Order 829 - that these areas all be addressed in the new standard. The NERC entity needs to do a lot more than simply address these six areas of risk, of course, but because they’re specifically called out you can be sure the auditors will all make sure they’ve been properly addressed in the entity’s plan, probably before they even get around to looking at anything else.

Let’s choose one of the areas in R1.2 as an example:

R1.2.4. Disclosure by vendors of known vulnerabilities related to the products or services provided to the Responsible Entity; 

How can you help your customers address this area of risk? This is of course a particularly difficult one for software vendors, since what might seem like the obvious way to help – disclosing to your customers all of the vulnerabilities in your software that you know about – would be the height of irresponsibility. If a vulnerability hasn’t been patched yet, the last thing you want to do is broadcast the existence of that vulnerability to all of your customers.

On the other hand, you might decide that you do need to provide information on all vulnerabilities (not just the ones that have already been patched, which of course can be advertised to the whole world) to at least your largest and most trusted customers. You need to decide internally in what cases you will do this, what safeguards you will require on the customer end, what alternatives to full disclosure you might suggest to customers for whom you don’t want to take the “Full Monty” approach, etc.

I hope you understand what I’m getting at here. For each of the six items in R1.2, you need to figure out a complete strategy for dealing with all of your customers (and as you can see, you will probably want to deal with particular types of customers in different ways, although the way you break down your customers is likely to vary according to the item in question). This will be an important component of your strategy, Mr./Ms. Vendor.

But this isn’t the only component of your strategy. My next post in this series will go over what else needs to be in your strategy.

If you are with a vendor to the electric power industry, and your company is trying to figure out what you will have to do to comply with CIP-013, Tom Alrich LLC would be pleased to offer you a free one-day (2-6 hours) workshop to review a) what CIP-013 requires, b) what you are likely to get asked to do by your clients, and c) what they should be asking you to do (since b and c won’t usually be the same thing). There will be no charge for my time, but I will require you to pay travel expenses at cost. Please email or call me if you would like to discuss this.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post. To discuss this, you can email me at the same address or call me at 312-515-8996.       

[i] And of course, I’m also doing a series of posts on how NERC entities can comply with CIP-013. Here is the most recent post in that series. Vendors should read this series of posts as well!

[ii] And by the way, NERC entities also need a CIP-013 compliance strategy, which I am gradually laying out in my other series of posts. A lot of elements in both strategies should be the same – just told from different points of view. But a NERC entity’s strategy will inherently include a lot of elements that have nothing to do with vendors, since vendor risk is just one of three areas of supply chain risk that need to be addressed in the supply chain cyber security risk management plan. See this post for a short summary of those three areas, and see this post for a long, painful discussion of that topic – but which ultimately might be more enlightening.

Wednesday, February 14, 2018

I Forgot My Anniversary!

This might seem like an odd topic for a post on Valentine’s Day, but I’m actually referring to the anniversary of my blog. A friend asked yesterday how long I’d been writing the blog, and I at first was going to say three years - because for some reason I seem to have latched onto that number a couple years ago, and never bothered to do the advanced math since then.[i] Then I realized it’s been five years. In fact, just over two weeks ago I missed the fifth anniversary of my first post, written in a hotel room in San Diego on January 28, 2013.[ii]

So anyway, I’m proud that this blog has lasted five years, and I hope it lasts another five! And for those of you who’ve stuck with me for some or even all of the five years, thanks!

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post. To discuss this, you can email me at the same address or call me at 312-515-8996.

Any opinions expressed in this blog post are quite definitely those of my employer, Tom Alrich LLC! If you disagree with what I’ve said, I suggest you take it up with them.

[i] This may also explain why for many years I’ve had the idea that I’m 42. I finally realized that had to be wrong when my son turned 30 last year – and I was sure he hadn’t been born while I was in junior high!

[ii] I had previously written posts for a Honeywell blog that no longer exists, and before that I’d written some “open letters” that Honeywell distributed. The open letters started in 2010, when I attended my first CIP drafting team meeting and found it so interesting – and consequential – that I wrote my first open letter about it.

The topic of my first post wasn't exactly timeless. I was trying to convince people that they had to take CIP version 4 seriously, since after all FERC had approved that version in April of 2012, and the compliance due date was April 1, 2014. I - and many others - felt that CIP version 5 was still a pipe dream that might never be approved by FERC. So it was much better to go with the bird in the hand (v4) rather than the two in the bush (v5). This campaign of mine culminated in what I call my "Dewey Beats Truman" post, when I confidently predicted that v4 would come into effect in 2014, with v5 following a few years later. Of course, just two months later FERC issued their NOPR saying they intended to approve v5, and v4 would go to sleep with the fishes.

Sunday, February 11, 2018

Lew Folkerth on Configuration Management

I apologize, but it seems I’ve fallen behind my minimum quarterly requirement of posts that quote from Lew Folkerth of RF. I just discovered Lew wrote a great article on configuration baselines and CIP-010 R1 for the RF Newsletter dated November/December 2017. You can find it by clicking on The Lighthouse in the table of contents on the left side of the page. I was also pleased to note that RF will now send out emails when new newsletters come out (which is bi-monthly), so neither you nor I will miss any future articles from Lew.

The article speaks for itself, but here are the points I found most interesting[i]:

  • Installed software and firmware listed in CIP-010 R1 should match software and firmware listed in CIP-007 R2 (Patch Management). Auditors check for this now, so you should definitely make sure they match on a regular basis, and even sync the two lists up if possible (page 16, last column).
  • A good tip for simplifying the job of CIP-007 R1 (Ports and Services) documentation by leveraging information from the baseline (p. 17, first column).
  • Benefits of a good baseline for incident response (p. 17, first column).
  • Lew’s recommended list of software and firmware to include in the baseline (p. 17, third column).
  • Lew recommends that firewall rules be under change management, whether or not they’re included in the baseline for the firewall.
  • The box about scripts on page 18 is worth the price of admission by itself! And that certainly doesn’t mean it’s worthless, even though admission to the article is free.
I recommend you all read the article, as well as subscribe to the newsletter.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post. To discuss this, you can email me at the same address or call me at 312-515-8996.

Any opinions expressed in this blog post are quite definitely those of my employer, Tom Alrich LLC! If you disagree with what I’ve said, I suggest you take the matter up with them.           

[i] A few of these aren’t new – in fact, I’ve written about them in previous posts) – but they’re worth repeating.

Thursday, February 8, 2018

How should Vendors “Comply” with CIP-013? Part I

The title of this post should give you pause. Vendors don’t have to comply with CIP-013; only NERC entities do. But this is a distinction without a difference. It has always been the intention of both FERC (in ordering NERC to develop a supply chain security standard) and the NERC standard drafting team (in developing the standard) to require vendors to be involved with compliance – although there will be no penalty to the NERC entity if they try to get a vendor to cooperate in compliance and are unsuccessful.

So it’s very legitimate to ask how a vendor can comply with CIP-013. I have had discussions with several vendors (and many more NERC entities) about this, and here is what we have come up with so far. I do want to point out that this is quite embryonic.

The most important consideration for a vendor is that it will be a big mistake if you look at CIP-013 compliance as some sort of adversary exercise, although you should be forgiven if you have thought of it that way so far. After all, a lot of the guidance so far (and that’s just one 13-page document) focuses on contract language.

My idea of a contract language “negotiation” is one in which the NERC entity’s lawyers and the vendor’s lawyers are sitting on opposite sides of an 8-foot-high brick wall. The entity’s lawyers throw some contract language to the other side and snarl “Here, put this in our contract.” The vendor’s lawyers look it over and throw something else back, saying (perhaps snarling less) “We can accept some parts without change and some with changes, but we can’t accept these other parts at all.” The entity’s lawyers look this over, and reply with their own counter-proposal. This goes back and forth, with the length of time determined by whether or not the lawyers are on salary (in which case they can do this for years) or they are paid by the hour (in which case the company will have to call an end to the fun and tell them to wrap up the “negotiation”).

Of course, this is perhaps an overly bleak view of the contract negotiation process, but I think it illustrates what I’m saying: Both the vendor and the entity should do everything they can to keep their relationship from degenerating to this point. In fact, IMHO if they first resort to contract language in their CIP-013 compliance process, both sides have already lost the game. Contract language is a nice thing to have, but it should never be the first concern.

So what’s the best way for a NERC entity and a vendor to approach the CIP-013 compliance process? I see two possible ways to get this going:

  1. The entity reaches out to the vendor and says “Let’s have a discussion on how we (note the plural pronoun!) can address CIP-013 compliance.”
  2. The vendor reaches out to all of their power industry customers (at least all who have any High or Medium impact assets where their software or hardware product is used), perhaps through an e-mailing or even a – gasp! – USPS mailing, and actually suggests how both sides could work together to achieve the goal of CIP-013 compliance.
So here’s a little quiz: Which of these two actions is more likely to result in a fruitful partnership to achieve CIP-013 compliance? I think they are both good options, but I would strongly recommend that the vendor be the first to reach out. Because if the entity reaches out first, I can almost bet that most of them will do that in the form of a demand for particular contract language. This isn’t because they’re sociopaths, but because NERC compliance up until now has primarily been seen as a legal exercise, whether for NERC CIP or the other NERC standards.

If the vendor reaches out first, and if they don’t immediately focus on contract language (if they do, they’ve obviously decided that it’s important that CIP-013 compliance be made as painful as possible, probably for reasons of job security of the people writing the document), then they can set the terms of engagement. The vendor can suggest actions that favor its strengths – i.e. developing, supporting and securing great hardware and/or software products – and don’t favor the utility’s strength (which is the fact that they are, after all, the buyer, and that money is flowing from them to the vendor, not vice versa. Plus the vendor doesn’t have a monopoly, at least in any electric power product market that I know of).

What should the vendor suggest to their power industry clients? I’ll leave that for the next post in this series. 

If you are with a vendor to the electric power industry, and your company is trying to figure out what you will have to do to comply with CIP-013, Tom Alrich LLC would be pleased to offer you a free one-day (2-6 hours) workshop to review a) what CIP-013 requires, b) what you are likely to get asked to do by your clients, and c) what they should be asking you to do (since b and c won’t usually be the same thing). There will be no charge for my time, but I will require you to pay travel expenses at cost. Please email or call me if you would like to discuss this.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post. To discuss this, you can email me at the same address or call me at 312-515-8996.

Any opinions expressed in this blog post are quite definitely those of my employer, Tom Alrich LLC! If you disagree with what I’ve said, I suggest you take it up with them.

Tuesday, February 6, 2018

I’m an ICS “Influencer”!

I was honored to be asked by Indegy – a company you should really learn about, especially if you own or operate power plants – to join nine other industrial cyber security “influencers” in giving our ideas on what you, i.e. a staff member in an organization operating ICS, can expect in 2018. While I can’t tell you whether your taxes will be lower or you’ll receive a bigger bonus this year, I can talk about what I think is an important development – which anyone who follows this blog might have already heard about once or twice J. You can read the post, on Indegy’s blog, here.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post. To discuss this, you can email me at the same address or call me at 312-515-8996.

Friday, February 2, 2018

Complying with CIP-013, Part 3: What is a good Implementation?

If you’ve read the first two posts in this series (here and here), you realize that for these posts I’ve tried to forget everything I know or have written about CIP-013, and just work through the words of the requirements (and the Purpose statement at the beginning of the standard). My goal has been to mine what is actually written in the requirements and get it all out on the table. The reason I’m doing this is because nothing is binding on the auditors (and therefore on the NERC entity) except what is in the requirements. So it’s important to put in a lot of effort to pull apart the requirements to find everything that’s possible to find in them – before we start going beyond what the wording says and start to think about the best way to actually comply with the requirements.

In our last episode, we wallowed in R1 and found – to my genuine surprise – that there is something important missing from R1.1: While this requirement mandates that the NERC entity “identify and assess” cyber security risks in the supply chain, it doesn’t require them to do anything about them! Of course, this is clearly just a mistake, and I highly recommended that everyone assume the word “mitigate” is also in there, whether or not this problem ever gets fixed in a formal manner.

Now we’re ready to go on to R2, which on the surface seems quite simple:

R2. Each Responsible Entity shall implement its supply chain cyber security risk management plan(s) specified in Requirement R1.

Is this really that simple? Yes and no. It isn’t simple because of what I just pointed out about R1.1: Your supply chain cyber security plan must include mitigation of the risks you’ve identified and assessed, even though that step isn’t included in the requirement part; so you need to mitigate risks when you implement the plan as well. But it is simple because it doesn’t mandate anything about how the plan needs to be implemented.

It seems you’re on your own in this requirement. You have to implement the plan, but you might take one year or 50 years. You might complete one part before starting on the next one, or you might address all parts simultaneously and try to complete them all at the same time. You might prototype your plan in one part of the company before moving it to others, etc. It seems the details are all up to you.

Indeed, this idea is reinforced in spades by the Implementation Guidance document put out by the CIP-013 drafting team last year. The document says literally nothing about what must be done to implement the plan, beyond repeating the substance of the note that appears below R2, emphasizing that the entity isn’t required to renegotiate or abrogate existing vendor contracts. It seems the drafting team couldn’t think of anything important to say about implementation.

However, let’s be realistic: The details of implementing a NERC CIP standard are never up to you. Last fall, I looked at the industry’s early experience with audits of CIP-014; like CIP-013, this standard requires the entity to develop a plan (in this case, a physical security plan for key substations) and then implement it. In two posts (here and here), I told several stories that point to the same conclusion: Even though R2 lacks any details about what constitutes a good vs. a bad implementation, the auditors are going to have their own ideas, and they won’t hesitate to make these known to the entity if they think they haven’t done a good job of implementation[i]. They hopefully won’t actually issue a Potential Non-Compliance (PNC) finding, as they did to one of the entities discussed in the second post. But they will very likely identify an Area of Concern and require the entity to fix the problem identified.

So what are the auditors going to look to when they decide whether or not an entity has properly implemented their supply chain cyber security risk management plan from CIP-013 R1? Darned if I know, but there is one thing of which I’m sure: I’m sure that auditors will look at the actual timeline for completing the implementation, and how that compares with what is in the plan.

Say your plan lays out a two-year implementation schedule, and your next audit is scheduled for about 1 ½ years after you start the implementation. If the auditors show up on that date and decide there’s no way your implementation can be completed by the two-year mark, they will very likely issue an Area of Concern, meaning you will need to develop a new plan with a realistic implementation date. And you’d better make that new date, or you’ll be in trouble three years later at your next audit.

So what does this mean for your plan? Should you always sandbag the finish date in the plan, making it at least a year or two after the date you think you’ll really finish? I don’t really think this is a great idea. I think you should include in your plan what you honestly believe is the date you will finish your implementation. However, I also think you should keep very close tabs on the implementation. Whenever it begins to look like you will not be able to meet your original finish date, you should immediately revise the plan to reflect the new date (of course, you should have change control to document that you made this change. You certainly don’t want to be in the position of needing to convince the auditor that the revised plan was actually the original one!).

Of course, if the auditor doesn’t agree with your excuse for why the implementation date had to be moved back (e.g. “My dog ate our supply chain cyber security risk management plan and we didn’t have any backup. This forced a six-month implementation delay as we re-drafted the plan from scratch.”), they may disagree with you. But IMHO there is no way they can issue you a PNC because of this, since R2 is completely silent on what characterizes a good vs. a bad implementation.[ii]

What’s the moral of this story? Even though R2 looks like it’s wide open to whatever interpretation you might want to give it, this doesn’t mean you should simply do whatever you want when you implement your supply chain cyber security risk management plan.  The auditors will always have their idea of how a plan should be implemented. It is almost certain that they will make their ideas clear to you in your audit, even if – hopefully – they don’t resort to a PNC to do that.

But what if it didn’t need to be this way? What if there were a way you could get your auditors (or at least your Regional Entity) to weigh in on whether and how your plan should be changed, before you started to implement it? That is the question I discussed in this recent post. There needs to be some way a NERC entity can get their CIP-013 R1 plan reviewed by their Region before they embark on implementing it. Moreover, there needs to be some way the Region can check in on the implementation of that plan as it is in progress, to make sure the entity hasn’t made some big mistake that might force it to re-do some or all of what it has already done.

The post described, with the assistance of an auditor, the Entity Development teams that are already implemented in one NERC Region and being implemented in another. The members of these teams aren’t auditors, but they help entities comply with the NERC CIP standards (and possibly other NERC standards). It seems to me that these teams would be the ideal organization to review an entity’s CIP-013 R1 plan as well as its implementation, before the entity has invested a sizable sum in implementation of what could turn out to be a flawed plan

I believe that some mechanism like this needs to be in place, in order for CIP-013 (as well as other plan-based standards like CIP-014 and the upcoming CIP-012, along with plan-based requirements like CIP-003 R2 and CIP-010 R4) to be successful. I hope you agree with me on this (and if not, that you’ll let me know). I hope NERC does, too!

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post. You can email me at the same address or call me at 312-515-8996.

[i] To be honest, the auditor issues in the second of the two posts cited (i.e. Part 3 of the series on lessons from CIP-014) had to do mostly with the plan itself, not with its implementation. But that is just because of timing: The two audits discussed in that post both occurred before the entity had even been able to start to implement their plan. Of course, this was very beneficial, since – had they implemented the plan, which was later determined to be insufficient, they might have been forced to re-do some or all of their implementation, using a new plan that was deemed sufficient.

[ii] They might conceivably issue you a PNC for violating CIP-011 R1.2, since the plan presumably included BES Cyber System Information and you obviously didn’t have a very good plan for protecting BCSI from your dog.

Wednesday, January 31, 2018

What about the Implementation Date for Lows?

I think a lot of CIP compliance professionals have September 1, 2018 circled on their calendars in heavy red ink, given the amount of work they think will need to be done by that date. I’m referring here to the date that compliance with CIP-003-6 R2 Attachment 1, Sections 2 and 3, comes due (the other two sections having already come due). These two sections cover physical and electronic access control for Low impact assets.

However, as I pointed out in a post last November, FERC called that date into question in their October NOPR, in which they proposed to approve CIP-003-7, the revised version of CIP-003 that was approved by NERC early in 2017. The implementation plan for CIP-003-7 says it will come into effect 18 months after FERC approves it, and that CIP-003-6 will be superseded when that happens. In other words, when FERC approves CIP-003-7 (and I believe it is highly likely they will do this, since they said they would!), the clock will start running on the 18 months. And the 9/1/18 date will end up being just another Saturday on a Labor Day weekend.

So what will be the implementation date for CIP-003-7? In the post, I pointed to July 1, 2019; however, I’m not sure what I was smoking when I said that. Since the plan says the implementation date is the first day of the calendar quarter after the 18-month anniversary of the approval by FERC, this means that, for my guess to have been right, FERC would have had to approve CIP-003-7 not much more than a month after I wrote the post (i.e. in December, since I wrote the post on November 12).

Of course, FERC didn’t approve CIP-003-7 in December. If they approve it before March 31, the effective date will be October 1, 2019. And if they approve it in the second quarter, the date will be January 1, 2020.

So, unless FERC decides not to approve CIP-003-7 after all (hardly a likely scenario), CIP-003-6 will never come into effect, and CIP-003-7 will come into effect more than a year after the 9/1/18 date that everyone has circled on their calendars. You can take this as a Valentine’s Day present.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post. I would also love to talk about this; please email me at the same address.

Tuesday, January 30, 2018

On my Own!

As many of you know, I have been looking for a new position since last November, after falling victim to layoffs at my previous employer. I am pleased to announce that I have accepted a position with a new employer: Tom Alrich LLC (you might even say this position was made for me!).

This is literally the first time in my life that I’ve been self-employed, and it looks now like this is definitely the best move for me. I will of course be offering consulting services related to NERC CIP (both related to the current standards as well as CIP-013), but also more generally to power industry cyber security. I will work with NERC entities (utilities and IPPs) as well as with vendors to the industry. For the latter, I can provide assistance in strategy and marketing, as well as in preparing for CIP-013, which will of course affect the vendors almost as much as (in some cases more than) the NERC entities themselves.

I’m pleased to say that I already have a decent amount of work in the hopper for the rest of this year. However, I still have availability! If you would like to discuss a project – or have any other questions, want to complain about something I wrote, etc. – please call me at 312-515-8996 or email me at tom@tomalrich.com.

Sunday, January 28, 2018

Complying with CIP-013, Part 2: Lost in R1.1

This is the second in a series of posts on NERC CIP-013, the upcoming supply chain security management standard. In the first post, I tried to empty my head of everything I already knew (or had opinions on) about CIP-013, and tried to focus on just what was written in the standard itself. I started with the statement of the standard’s purpose, and tried to tease out of that some good pointers on what will be required to comply with the standard. Just analyzing this single sentence showed that complying with CIP-013 will be very different from complying with any previous CIP standard.

In this post, I will look at the requirements of CIP-013 (and they’re all pretty short!). Again, I’ll try to forget anything I know about the requirements or what has been said about them, since in the end all that matters is what is written in the requirements themselves.

Let’s start with R1, which reads:

R1. Each Responsible Entity shall develop one or more documented supply chain cyber security risk management plan(s) for high and medium impact BES Cyber Systems. The plan(s) shall include:

Of course, this is very different from almost all of the other CIP requirements, which require the entity to put in place specific security controls – configuration management, patch management, personnel risk assessments, etc. R1 requires the entity to develop a plan, period. Sure, the plan will need to include certain items to be listed later, but those items aren’t themselves the point of the requirement; the plan is. This isn’t new; there are two currently-enforced CIP requirements that are written in exactly the same way: CIP-003-6 R2 and CIP-010-2 R4. These are also plan-based requirements.[i]

And what is the purpose of the plan required in CIP 13 R1? It is “supply chain cyber security risk management”. In the first post in this series, I already emphasized the fact that the purpose of CIP 13 is risk management, not implementation of particular controls – so the fact that R1.2 lists six particular controls that need to be in place shouldn’t be taken to mean that your plan just needs to address these controls. Your plan needs to address the mitigation of supply chain cyber security risk in general, which is a much broader topic than these six controls.

Let’s continue with the two parts of R1. R1.1 reads:

1.1. One or more process(es) used in planning for the procurement of BES Cyber Systems to identify and assess cyber security risk(s) to the Bulk Electric System from vendor products or services resulting from: (i) procuring and installing vendor equipment and software; and (ii) transitions from one vendor(s) to another vendor(s).

Note that this is pretty curious wording. R1 left off with the words “The plans shall include:” This means that 1.1 should lay out something that must be included in the plan. Instead, 1.1 requires “one or more processes used in planning for the procurement of BES Cyber Systems to…” What does this mean? Concatenating these words with the end of R1 leads to:

“The plan shall include one or more processes used in planning for the procurement…”

Why should a plan include “processes used in planning”? This means that the plan should be for developing further plans. A plan should be for taking particular steps to achieve a particular goal, not for simply developing more plans. Wouldn’t it be simpler just to say the plan should include identification and assessment of cyber security risks to the BES from the two areas of risk listed? Why does there need to be this intermediate step of having “One or more process(es) used in planning…”?

To be honest, I’m not sure at the moment why these words are there; maybe a drafting team member (or someone else) can enlighten me on this.[ii] I recommend you ignore those words. In other words, I suggest you interpret this requirement part as if it were written:

1.1. Identification and assessment of cyber security risk(s) to the Bulk Electric System from vendor products or services resulting from: (i) procuring and installing vendor equipment and software; and (ii) transitions from one vendor(s) to another vendor(s).

If R1.1 were worded this way, what would an entity need to include in its plan in order to be compliant with this requirement? The plan would need to include steps to a) identify and b) assess risks to the BES from vendor products or services resulting from risk areas (i) and (ii).

What does this mean? First of all, I want to point out that item (i) really includes two risk areas, not one. Risks from procuring vendor equipment and software are very different from risks from installing vendor equipment and software. This means I would rewrite the part of the sentence after the colon as “(i) procuring vendor equipment and software; (ii) installing vendor equipment and software; and (iii) transitions from one vendor(s) to another vendor(s).”

Once we’ve made that change, I interpret R1.1 – if written as it is directly above – to require the following steps:

  1. For each of the three risk areas, identify particular risks included in each one. For example, in the area of “procuring vendor equipment and software”, an entity might identify risks such as “the risk that a piece of third-party software included with the product will be infected with malware”; “the risk that the software will include known vulnerabilities that haven’t been mitigated”; etc. This is, of course, potentially a long list. However, IMHO the entity doesn’t have any choice but to list all of the risks[iii] that might be part of procuring vendor equipment and software. As we’ll see in one of the upcoming parts of this series of posts, just the fact that an entity needs to develop a complete list of risks doesn’t mean they will have to spend a lot of time and money mitigating each one; in fact, the entity might determine (and document) that some of those risks are so small that no mitigation at all is required. This is permitted, of course, due to the fact that CIP 13 is the first NERC risk-based requirement.
  2. “Assess” each of these risks. What does this mean? Obviously, there are many aspects of each threat that we might assess. For example, we might assess each threat based on whether it targets our particular department or not; those that don’t affect our department can be ignored.[iv] However, what I think needs to be assessed here is the level of risk in each case.[v] I think that the entity needs to classify each of these risks according to some scheme: high/medium/low; high/low; 1 to 5; etc.

Once the entity has included in the plan each of the above steps (and of course each of these has quite a few implied sub-steps), I think they have properly addressed R1.1 as written. But wait! Haven’t we forgotten something here? We’re going to include in our plan steps to a) identify risks and b) assess those risks. Why are we doing this? Are we going to be satisfied if we simply produce a nice list of risks, each one ranked as – for example - high, medium or low? What else could be needed?

Here’s something else: How about mitigating those risks? After all, isn’t that the whole idea of CIP 13? To actually lower the overall risk to the BES by mitigating supply chain security risks? Yet R1.1 doesn’t require the entity to include anything about mitigation in their plan.[vi]

To be honest, I think it is simply a mistake that mitigation isn’t mentioned in R1.1. It doesn’t make any sense just to identify each risk, assess it, then call it a day and go home. You need to interpret R1.1 as if it included mitigation. In other words, there needs to be a third step in the list above, reading something like:

  1. For each identified risk, mitigate the risk according to the risk level assigned to it in step 2.

Of course, there is a lot more that could be said about this step, and I’ll hope to do that in some future post.

I must admit that, when I started to write this post a few hours ago, I thought I was just going to go over the words of R1 and R1.1 and tease out all of their meaning. Instead, I’ve found that R1.1 doesn’t make much sense unless two changes are made to it (the change incorporated in my revised R1.1 above, as well as adding mitigation to R1.1, as just discussed). So, with greatest respect, I suggest that NERC entities interpret R1.1 to read this way:

1.1. Identification, assessment, and mitigation of cyber security risk(s) to the Bulk Electric System from vendor products or services resulting from: (i) procuring vendor equipment and software; (ii) installing vendor equipment and software; and (iii) transitions from one vendor(s) to another vendor(s).

I’m having a déjà vu moment here. A little less than five years ago, in late April 2013, I set out to write the first of what was going to be a series of posts on how to comply with the CIP version 5 standards; I did this because FERC had about two weeks previously issued a NOPR saying they intended to approve v5 (just as FERC did with CIP-013 less than two weeks ago). Since CIP-002-5.1 was the first standard in order, I started with that.

In that post, as in this one, I tried to work through the steps that would be required to comply with CIP-002-5.1 R1 (including Attachment 1), and I came to a “you can’t get there from here” point – where the logic in the requirement completely broke down. That is a serious flaw in CIP 2 R1 that has never been fixed. At the time, I thought the only solution was to rewrite the requirement (and Attachment 1); I wrote perhaps ten to fifteen posts on this particular topic over the next couple of years (plus many more on other problems in CIP 2 R1), of course without success.

However, it turns out that this flaw didn’t prove to be very harmful as CIP v5 was implemented and then enforced. This is because NERC entities and auditors all came to a common interpretation of CIP 2 R1. The interpretation wasn’t based on the actual wording of the requirement and Attachment 1, but since there was unanimity among entities and auditors, everyone came to believe that this interpretation was actually the correct one.[vii]

Now I believe I’ve found a serious problem with CIP 13 R1.1 (namely the lack of mention of mitigation. The first problem I discussed is more of an annoyance than anything else), but this time I’m not even going to suggest that R1.1 be rewritten. It would easily set back implementation of CIP 13 by more than a year, and I think the solution will be the same as it was with the problem I found in CIP-002-5.1: There will just need to be a common interpretation[viii] of R1.1 that includes mitigation. In fact, I humbly[ix] suggest that my version shown above could be used as that common interpretation – I’ll even make it available royalty-free!

The views and opinions expressed here are my own, and do not reflect those of any organization I work with or for. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

[i] There is another current requirement that is, in my opinion, functionally plan-based. CIP-011-2 R1 requires an information protection program, not a plan. However, the rest of the requirement closely matches CIP-003-6 and CIP-010-2 in form. This is interesting because CIP-011 R1 was developed as part of the “CIP version 5” effort, while the other two requirements were developed several years later, as part of “CIP version 6”.

You may wonder about two other requirements that were originally part of “CIP version 5” that specifically require “plans”. CIP-008-5 R1 requires the entity to develop a cyber security incident response plan, while CIP-009-6 R1 requires recovery plans. However, I distinguish between these two requirements and the three requirements cited above: The plans referred to in CIP-008-5 R1 and CIP-009-6 R1 are actually controls that need to be implemented to protect against threats that might be realized in the future. In CIP 8 R1, the threat is that of a cyber security incident (or more precisely, the threat is that the entity will not respond in an intelligent fashion to a cyber incident and will end up making things much worse). In CIP 9 R1, the threat is that there will be outages of systems that could impact BES reliability unless quickly restored.

The only way you can deal with these two threats is to put a plan in place to address them when they are realized. So CIP 8 R1 actually requires a control – the CSIRP – to address the threat of improper response to cyber incidents, while CIP 9 R1 requires a control – the recovery plan(s) – to address the threat of inability to restore one or more systems when lost. To compare this with a requirement that doesn’t require a plan, CIP 7 R2 requires a control – a patch management program – that mitigates the threat of malicious code. All three of these are prescriptive requirements requiring one or more controls to be implemented; it’s just that in the first two requirements, the control happens to be a plan. That doesn’t make these plan-based requirements using my definition of the term: a requirement to develop a plan for addressing a particular threat or set of threats.

[ii] And I’m not in any way saying the drafting team did anything wrong here; after all, I attended at least two of their face-to-face meetings and some of the call-in meetings, and I never raised this issue at the time. In fact, I didn’t notice this issue, or other issues noted in this post, at all until I started to write the post.

[iii] I would much prefer if R1.1 referred to threats, not risks. I will hopefully address this issue in an upcoming post (and if not, it will definitely be addressed in the book that I and two co-authors are working on, regarding how the CIP standards could/should be rewritten). Briefly, I think the steps should actually be: 1) identify threats; 2) using estimated impact of each threat and the fact that risk = threat X impact, classify each threat by the relative degree of risk it poses; and 3) develop a mitigation strategy for each risk, based on its classification. This would be much clearer if CIP-013 distinguished between threats and risks.

[iv] Obviously, I’m being facetious here.

[v] And as I said in end note ii above, I think this requirement part would be much clearer if it distinguished between threats and risks.

[vi] And, while it might be tempting to think that R1.2 is the part where mitigation is required, that would be a mistake. As I will discuss in the next post in this series, R1.2 lists a set of six specific risks that FERC required NERC to address in this standard when they ordered its development in Order 829; it was never meant to be any sort of comprehensive list of the risks to be mitigated. R1.2 does require mitigation of these six risks, but not of any other risks.

And don’t think that R2 might save the day here. R2 just requires implementation of whatever is in the plan, nothing more. If the plan only includes identification and assessment of risks, then that’s all that has to be implemented.

[vii] Briefly, the issue is that the wording of CIP 2 R1 and Attachment 1 is contradictory. Some of the wording – especially in Attachment 1 Section 3 – leads one to believe that assets should be classified as high, medium or low impact, while most of the wording says that only BES Cyber Systems are classified, not the assets themselves. However, I’d say there is close to 100% unanimity in the NERC community that it is assets, not BCS, that are being classified in Attachment 1. And since this is working, why mess with it? I think I last raised this issue in late 2015 in this post. I don’t intend to raise it again. There are enough real problems with the current CIP standards to be discussed, without dragging out what has turned out to be a fairly harmless wording problem.

[viii] Of course, since it doesn’t make any sense just to identify and assess risks and not mitigate them, it’s very hard to see how R1.1 could be interpreted any other way.

[ix] OK, not so humbly.