Complying with CIP-013, Part 4: The Contract Language Bogeyman

Regular readers of this blog (all three of you!) may have figured out that I’m writing two series of posts on CIP-013: one for NERC entities and one for vendors. The important thing to remember is that both series should be of interest to everybody. The two sets of concerns are almost completely the same, because a) vendors have all sorts of incentive to make sure their customers don’t get into trouble due to something they did or didn’t do; and b) most vendors have their own supply chain, which has to be secured in almost the same way as NERC entities will be securing theirs.

If you ask any NERC CIP compliance professional (or say a supply chain or legal professional at a NERC entity subject to CIP-013) what is the main focus of compliance with CIP-013, the upcoming supply chain security risk management standard, they will most likely tell you it’s contract language. This isn’t terribly surprising, since there was endless debate during the drafting/balloting periods on the role contract language would play in compliance. And I’ve heard that some utilities are already requesting security contract language from their vendors.

I must admit that I was very focused on contract language in the first few posts I wrote about CIP-013 last fall. But as I started my current posts on the standard this year, and focused more on the standard and the Implementation Guidance, I realized that it’s a mistake to think that contract language is in some way the ultimate goal of CIP-013. Far from it.

First, here’s a little quiz: Why was CIP-013 written? The possible answers are: a) to secure NERC entities’ supply chains; or b) to beef up language in vendor contracts. You’re right! The answer is a). Security is the object of CIP-013, not contract language. To see this, you first need to look at the requirements of CIP-013. Do the actual requirements ever refer to contract language? No, they don’t. The only time contract language is mentioned is in this note found after R2:

Implementation of the plan does not require the Responsible Entity to renegotiate or abrogate existing contracts (including amendments to master agreements and purchase orders). Additionally, the following issues are beyond the scope of Requirement R2: (1) the actual terms and conditions of a procurement contract; and (2) vendor performance and adherence to a contract. 

Does either of these sentences say anything about contract language being an objective of CIP-013? The first sentence says that renegotiation of existing contracts isn’t required. I’ll admit this does somewhat imply that, when a new or existing contract is negotiated, there needs to be a discussion of contract language. But that isn’t the same thing as requiring that contract language be the primary means for compliance with the standard, even though so many people apparently believe that is the case.

The first part of the second sentence is even more interesting, since it says that the terms and conditions of a procurement contract are “beyond the scope” of R2. What does this phrase mean? It specifically means that contract language is not the subject of R2 – and if an auditor asks to see an entity’s vendor contract, they can be turned down because it isn’t in scope.

I’ve covered this ground before, but in a different context. In a series of three posts (starting with this one, following up with this one and concluding with this one), I addressed the idea that CIP-013 was unenforceable because of the above provision about contract language being out of scope. A staff member (and former auditor) from one region had made this assertion to me. He argued that NERC entities are required[i] to get their more-risky (i.e. important) vendors to agree to contract language for at least the six items listed in R1.2. However, since the NERC entity isn’t required to show the actual contract to the auditors, this means the auditors don’t have any way of verifying the entity’s assertion that they obtained the contract language.

An auditor (from another region) wrote in after that post to say that NERC entities aren’t required to get the vendor to agree to particular contract language in order to obtain the six items in R1.2 (I wrote about this in the second of the above posts). To quote him, “The requirement is essentially that the Registered Entity ask for the elements of the required process(es), not that the vendor agree to them.  Therefore, the Standard is correct in that the language of the resulting contract is not in scope.  (The procurement steps include) the Request for Quotation or Bid, Request for Contract Amendment, and/or any other Registered Entity-initiated correspondence that sets forth the expectations to be placed upon the vendor.  As long as I see the expectations in the procurement documents issued by the Registered Entity, they have satisfied the requirement.”

I agree with the auditor that succeeding in getting language placed in a contract isn’t required, and that it should be sufficient that the Registered Entity “initiated correspondence that sets forth the expectations to be placed upon the vendor”. But I would go even further. I don’t think the entity needs to show any procurement documents at all, because I don’t think the entity needs to prove that they “initiated correspondence” with the vendor to implement certain controls, if they can show other evidence of compliance. And what other evidence can the entity show? The best evidence will demonstrate that the vendor is actually doing what they were supposed to do![ii] After all, isn’t the goal of CIP-013 to protect the BES against supply chain risks, not simply document that you’ve asked your vendors to do certain things?

Suppose you want to show your auditor that your vendor is providing “Notification…when remote or onsite access should no longer be granted to vendor representatives..” as required by R1.2.3. Let’s say the vendor has had several people leave their company in the last year who had electronic or physical access to the vendor’s systems at say one of your substations, and each time this has happened they’ve sent you an email asking you to remove the person’s access. All you should have to do is show those emails to the auditor. What possible further benefit would there be from also showing them procurement documents, let alone a contract? Of course, if you also have these, that’s fine. But if the vendor verbally agreed to do something, or just sent you a letter outside of the procurement process, but you can show they’re actually doing what they said they’d do, that should be sufficient evidence.

Let’s turn from the requirements of CIP-013 to the Implementation Guidance. This document has language that implies that contract language is a goal of the standard (although certainly not the only one), while at the same time it also says just the opposite. However, I can see how people who have read the document have come to believe that contract language is the primary control they should be focusing on.

First off, let me point out that there is nothing mandatory about the Implementation Guidance. The auditors aren’t bound to follow it in their audits, and the entities aren’t bound to follow it in their implementation of CIP-013. That being said, you can be sure most entities and all auditors will have read the document carefully and will usually try to follow it as often as they can. So it is important to at least understand what it says. On page 1, you’ll find this paragraph:

First, in developing their supply chain cyber security risk management plan(s), Responsible entities should consider how to leverage the various components and phases of their processes (e.g. defined requirements, request for proposal, bid evaluation, external vendor assessment tools and data, third party certifications and audit reports, etc.) to help them meet the objective of Requirement R1 and give them flexibility to negotiate contracts with vendors to efficiently mitigate risks. Focusing solely on the negotiation of specific contract terms could have unintended consequences, including significant and unexpected cost increases for the product or service or vendors refusing to enter into contracts.

Here, the Implementation Guidance is specifically warning the entity against relying solely on contract language. However, the message is very different when you get to pages 5-8, where the six specific items required under R1.2 are addressed. For each of these items, there are one or more processes listed. These are described as “Examples of ways that a Responsible Entity could, through process(es) for procuring BES Cyber Systems required by Part 1.2, comply with Parts 1.2.1 through 1.2.6.” (my emphasis) It seems that, at least for the six risks addressed in R1.2, the members of the Standards Drafting Team that wrote the Implementation Guidance were primarily considering controls that occur in the procurement phase of the vendor relationship – including contract language.[iii]

When you look at the 20 actual processes suggested under the parts of R1.2, you will notice that all but three of these begin with one of two phrases: “In an RFP or during contract negotiations, request that the vendor include in the contract provisions…” or “During procurement”. Of course, both of these phrases simply confirm the fact that whoever wrote this section of the Implementation Guidance was primarily looking at the process of obtaining contract language – or other processes focused entirely on the procurement phase of the vendor relationship – to achieve the mitigation of risk required in each of these six cases.

But in my opinion, the big emphasis on the procurement process in this section of the Implementation Guidance is a mistake. Essentially, what is being implied in this section is that the only possible way to coordinate security controls with vendors is during the procurement process, when obviously the customer has the most leverage over the vendor. This might be a good model for a purely commodity market, where you have a bunch of vendors competing viciously with each other and they all receive a very low margin (on presumably a high volume of sales). In this case, the customer clearly has to obtain everything they want from the vendor up front, and they need to negotiate simultaneously with multiple vendors and play them off against each other for concessions.

I don’t know about you, but I don’t think this is how it works in the market for control systems that run the grid. I am betting that almost all vendors in this market will do almost anything they can to make their customers happy, since they are counting on having a long-lasting relationship with them. And different vendors’ products are very different, so it’s impossible to play vendors off against each other, as in a commodity-type market.

Clearly, a customer’s leverage isn’t limited to the procurement process. I am sure that, in most cases, a field technician (backed up by their VP) with a serious complaint about a vendor, midway through the contract lifetime, will usually have as much leverage as a purchasing agent on the eve of contract renewal.[iv] Even though it is notoriously difficult to change vendors of control systems in the power industry, I really don’t believe that any vendor would last very long if they didn’t do everything they could to provide what their customers need.[v]

Again, I’m saying that I think the emphasis on the procurement process in this section of the CIP-013 Implementation Guidance is a mistake. I really doubt there are auditors who will insist that NERC entities need to wait for the procurement (or contract renewal) process in order to request any meaningful changes to vendor security controls. And it follows almost certainly that auditors won’t require that contract language be the primary vehicle for obtaining agreement on controls – especially since that’s the one type of evidence that the auditors won’t be able to see!

If I’m so sure that the emphasis of this whole section of the Implementation Guidance is wrong, how did it end up there in the first place? To answer that, you need to know that at least half of the members of the drafting team for this standard were supply chain people – and I’m speculating that this particular section was drafted by some of them.

For supply chain people, the procurement process is kind of the whole show, since that is when they have by far the most interaction with vendors, as well as when what they do and say will have the most impact. But they don’t get involved much with the day-to-day interaction with vendors that goes on in the field. So supply chain people are naturally going to consider the procurement process – and especially contract language – as the primary vehicle for influencing what vendors do. They don’t see how vendors often easily agree to requests that come through completely different channels.

The moral of this story? It would be a big mistake for NERC entities to focus on the procurement process – and especially on contract language – as the sole, or perhaps even the predominant vehicle for getting vendors to agree to implement security controls. There are lots of paths to achieve this goal, including simple phone calls from a field technician to their friend Joe at the vendor. However, what will need to change with CIP-013 compliance is that those calls or emails – when related to vendor controls the entity has decided need to be in place in order for it to comply with CIP-013 – will need to be logged and archived so they can be evidence of compliance.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post. To discuss this, you can email me at the same address or call me at 312-515-8996.

---

[i] The staff member based this on the fact that FERC seemed to have made this a requirement in their Order 829, in which they ordered NERC to draft this standard. The auditor replied that FERC’s statements have no bearing on how a standard should be interpreted, once they have approved it. You can read their back-and-forth in the second of the three posts I linked above.

[ii] And keep in mind that, if the vendor absolutely refuses to do something that the entity requested, the entity isn’t in violation for that reason. However – and this is important – they are still under obligation to if possible mitigate the risk in question, presumably using a different method that doesn’t require the vendor to do something for them. Of course, there are certainly some risks that it won’t be possible to mitigate without the cooperation of the vendor, such as the risk of someone who had access to your systems leaving the vendor’s employment, and your organization not being notified that this has happened (this is, of course, the risk addressed in R1.2.3). In these cases, if the vendor simply won’t cooperate, the entity isn’t obligated to change vendors; but the entity should document how they tried to obtain cooperation.

[iii] Of course, you need to keep in mind that the six items listed in R1.2 are there because FERC ordered these six items be specifically included in the standard. They aren’t in any way intended to be the only items that need to be included in the entity’s supply chain cyber security risk management plan. The plan needs to address each of the three risk areas listed in R1.1: a) risks from procuring vendor equipment and software, b) risks from installing vendor equipment and software, and c) risks from transitions from one vendor to another. The six items in R1.2 all correspond to risks that fall into category a). However, there are certainly many other risks that fall into that category, which aren’t included in these six. And of course, the six items don’t address any risks that fall into categories b) and c).

[iv] Of course, I’m not talking here about huge vendors like Microsoft and Adobe, for whom the electric power industry accounts for a very small part of their revenue. They will of course provide product support, but if XYZ Power calls up and demands that Microsoft insert a new feature into the current Windows version, I rate the likelihood they will succeed in this demand as very low.

[v] Of course, if anyone has a different view on this, I’d love to hear it. Send me an email or comment on this post.

What about Resiliency?

Yesterday I was interviewed by the always astute Blake Sobczak of Energy and Environment News[i] about a cyber security report issued by the White House last week, and specifically about what it had to say about the cyber security of the power grid. The article appeared today.

The discussion of the grid in the report was certainly good and not terribly inflammatory. My feelings about this report are very similar to my feelings about Ted Koppel’s book Lights Out, which I discussed in this post in early 2016: What Koppel was writing about had very little to do with cyber security. It had everything to do with the amount of devastation that any widespread and prolonged grid event could wreak (and we’re talking about a much more serious event than even the 2003 Northeast blackout), whether caused by a huge weather event (even bigger than Superstorm Sandy), solar storm, EMP event, physical attack, or yes a cyber attack. Even more importantly, the book documented in horrifying detail the country’s almost total lack of preparedness for this. Unfortunately, whoever wrote the book jacket decided the book would sell more copies if it were made to appear as a book about a big cyber attack on the grid, without doubt to leverage the popular movies that had depicted such an attack. So that is how the book is known, but it isn’t what’s actually between the covers.

In the same way, the section of the White House report that I commented on quotes the 2015 study by Lloyd’s and the University of Cambridge that estimated the total cost of a worst-case cyber event at $1 trillion. I totally agree that a worst-case cyber event could cost that much. But there are two considerations: 1) The cost would be the same whether the cause of the event were weather, solar storm, or anything else; and 2) I’d say a worst-case cyber event is probably the least-likely cause of such a huge grid outage – with a probability somewhere around 1 in 10 to the 10^th or the 20^th power. I think a solar storm is far more likely to cause such an event (and the 1859 Carrington Event probably would have done it, had it occurred today. We’d better hope these aren’t every-200-year events!).

Although the report didn’t advocate any particular policy actions (it was actually quite good as a broad overview of the risks posed by cyber security weaknesses across the US economy), in my comments I anticipated what I thought might be the typical response regarding security of the power grid: “Oh my God, we have to do something about this! We need to really tighten up the cyber security standards so that the power industry (that would be you, Dear Reader) doesn’t let his happen to us!”

My response to this response, whether triggered by the White House report, Ted Koppel’s book or any number of alarmist statements, is that the situation won’t be improved by requiring say a 10-fold increase in the severity of the NERC CIP requirements. There has never been an outage caused by a cyber attack in North America; nor has there ever been even a documented penetration of a control network in a grid asset of any significance (I make this qualification because there was a penetration of a 5MW dam in New York state in 2013. I know of no other such penetration, although if anyone knows of another I’d appreciate your letting me know about that in an email). And of course, any amount of increased cyber security spending by the power industry will do nothing to mitigate the danger of a different type of event like a solar storm (and NERC is currently in the process of approving a draft standard to address solar storm risks by “hardening” grid assets).

I have long believed that the best protection against widespread outages, no matter what the cause, is microgrids. If the great majority of populated areas in North America were protected by microgrids, there could still be widespread grid events which would destroy huge fixed generation and the high-voltage transmission network - but these events wouldn’t cause widespread outages. Each microgrid would automatically activate its local generation resources – wind, solar, gas turbines – and keep on chugging. Of course, the problem at the moment is that microgrids are very expensive to implement and they’ve only been implemented in a small number of high-value locations (like the New Jersey transit system, which was knocked out by Hurricane Sandy, greatly complicating the recovery from the storm). But if we’re talking about throwing a lot more money at grid security, wouldn’t it be a lot better to throw it at a solution to all potential causes of a huge grid outage, rather than just at reducing the already-infinitesimal probability of just one of those causes – a massive cyber attack?[ii]

Before I go, I do want to point one thing out: I don’t believe for a moment that electric utilities are paying too much for OT cyber security controls now. In fact, given the ever-tightening threat environment, I think it’s inevitable they will need to spend more every year for the foreseeable future. My issue is that a large portion of what NERC entities spend on NERC CIP compliance now goes simply to pure compliance activities, not to increasing the level of cyber security. I think that virtually all NERC entities understand that this is a tough world, and they will have to increase their cyber and CIP spending every year just to stay in the same position. But they will be increasingly reluctant to do this without changes in the NERC CIP compliance regime (and in the standards themselves) that will allow a much higher percentage of every dollar they spend on CIP to go towards cyber security.

And what are those needed changes? Funny you should ask. That is the topic of a book I am working on with two co-authors. You may roll your eyes and point out that I’ve been talking about this book for close to two years. That’s true, but we now have a lot of momentum and I’m sure we’ll have something out before the end of the year. And what is the solution we’re advocating? Well, you can find most of it in my posts, although I’ll admit you’ll have to look hard (and be able to tie a lot of bits and pieces together). I hope to have everything in a one-stop-shop this year!

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post. To discuss this, you can email me at the same address or call me at 312-515-8996.

---

[i] This online publication has the best articles about security of the energy sector of any publication I’ve seen, all written by either Blake or his colleague Pete Behr. They’re all very in-depth (a few even as long as my average post, heaven forbid!) and very well-researched – and this applies to all of the articles, not just those on cyber. Most online news feeds confine themselves to news feeds or reproductions of articles from other publications, so E&E News really stands out. This is a subscription-only publication, but I strongly recommend you try to get your own organization to subscribe to it. I recommended to my boss at Tom Alrich LLC that we purchase a subscription, but he hasn’t replied to my email yet.

[ii] And if you’re tempted to think that the big toughening of the NERC CIP standards would be paid for by “private industry” while the cost of implementing microgrids everywhere would have to be borne by the taxpayers, let me point out something to you: Every taxpayer is a ratepayer to their local electric utility. Where will the utilities get the money to comply with this huge increase in the cost of CIP compliance?

Lew on Patch Management Mitigation Plans

Five days ago, I wrote a post calling attention to Lew Folkerth’s December article on CIP-010 R1 (configuration management) compliance. In that post, I mentioned that you can now sign up to receive notices of new RF newsletters (where Lew’s column appears). I signed up and I’m glad I did, because yesterday I got notice of a new newsletter. This time, Lew’s article (which as usual goes under the name “The Lighthouse”) is about CIP-007 R2 patch management mitigation plans. Lew’s articles are always excellent, but this is still one of his best yet, in my opinion.

I must admit I hadn’t thought very much about patch management mitigation plans. R2.3 says “Mitigation plans shall include the Responsible Entity’s planned actions to mitigate the vulnerabilities addressed by each security patch and a timeframe to complete these mitigations.” I had always just assumed this was all you need to know about these plans.

It turns out I was wrong. Lew does an excellent job of pulling out what really needs to be in a mitigation plan and how it needs to be handled. None of this is in any way an add-on to the requirement, by the way. Lew is simply following the logical implications of what the mitigation plan needs to accomplish, both for security and compliance. Lew has always been most concerned about what makes the most sense from a security point of view. I won’t say everything he says in this article is needed strictly for compliance with the requirement, but doing it will certainly allow you to tell a good story when you get audited.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post. To discuss this, you can email me at the same address or call me at 312-515-8996.

How Should Vendors “Comply” with CIP-013? Part 2

This is the second in a series of posts of how vendors can “comply” with CIP-013. Of course, vendors don’t have to comply at all, and they don’t have to help their power industry clients comply either. On the other hand, if they wish to remain a vendor to the power industry, a vendor is going to have to do everything they can to help their customers comply. It is inevitable that the vendor’s larger clients will have to comply with CIP-013, and they will need help in doing so.

I do want to emphasize that, if you’re part of a NERC entity who has to comply with CIP-013, you shouldn’t stop reading this post (or the subsequent ones in this series), on the idea that it doesn’t really have anything to do with you. This has everything to do with you, since – as I emphasized in the first post in the series – the only way I see a CIP-013 compliance program succeeding is if the NERC entity at least tries to partner with their vendors for compliance. You need to understand their position as much as they need to understand yours.[i]

In the first post in this series, I suggested that power industry vendors would be well advised to reach out to their customers before the customers reach out to them. I made this suggestion because I think it’s very likely that if the customer reaches out to the vendor first, it will probably be the lawyers and/or purchasing people who do this. This is because most of the discussion so far on CIP-013 compliance (with this blog hopefully being an exception!) has revolved around the question of contract language. But contract language is just one of the ways in which a utility can fulfill its obligations under CIP-013, and it’s probably the bluntest instrument available to the NERC entity to fulfill those obligations. I think that any NERC entity that focuses on contract language first, before even looking at all the options available to it to comply with R1.2, has already done both itself and its vendors a big disservice.

So the vendor should absolutely try to reach out first to their customers. How can they do that? There are of course lots of media available for this: notice on web site, email, USPS, phone call, webinar, onsite meeting, etc. While these are all good, I always favor the more personal ones above all. A webinar might be the ideal first step, since it’s delivered by a person(s) but it still has a structured content. Then the vendor could follow up with each customers by phone or in-person meeting to discuss next steps.

But before you can do a webinar, Mr./Ms. Vendor, you need to know what you’re going to say! And that means you need to have figured out your CIP-013 strategy[ii] – so that’s really the first step. How can you figure out your strategy?

This is of course rough and preliminary, but it seems to me that a vendor’s strategy should be to try to make the CIP-013 compliance process as easy as possible for the customer, period. I can assure you that your customers will be feeling every bit as uncertain and fearful about CIP-013 as you do. If you can convince them that you know the right way to cooperate for compliance, and you follow through on what you say you’ll do, what could possibly go wrong – except, perhaps, if you don’t in fact know the right way to cooperate and both of you end up at some sort of regulatory dead end?

How do you stay away from dead ends? By having a good methodology for designing your CIP-013 strategy. And what should you do first? Well, it seems to me that, if you’re going to design a strategy to help your customers do something, you need to figure out what they have to do. There’s a quick answer to that: They have to comply with CIP-013. Now that we have that out of the way, how do they comply with CIP-013?

As I discussed (at length) in this post, CIP-013 requires the NERC entity to develop and implement a supply chain cyber security risk management plan. Specifically, the plan has to address the three areas of risk listed in R1.1: (a) procuring vendor equipment and software; (b) installing vendor equipment and software; and (c) transitions from one vendor(s) to another vendor(s).

I suggest you start your CIP-013 strategy effort by brainstorming on how you can help your customers address all three of these areas of risk. To be honest, you might decide you can help customers in some ways that are either too expensive to be practical or not likely to yield a lot of benefit compared to the effort required; you can drop these ideas from your strategy.

For procuring vendor equipment and software, the NERC entity needs to address risk in six specific areas; these are listed in R1.2. The reason they are specifically listed in CIP-013 is because FERC specifically required – in Order 829 - that these areas all be addressed in the new standard. The NERC entity needs to do a lot more than simply address these six areas of risk, of course, but because they’re specifically called out you can be sure the auditors will all make sure they’ve been properly addressed in the entity’s plan, probably before they even get around to looking at anything else.

Let’s choose one of the areas in R1.2 as an example:

R1.2.4. Disclosure by vendors of known vulnerabilities related to the products or services provided to the Responsible Entity; 

How can you help your customers address this area of risk? This is of course a particularly difficult one for software vendors, since what might seem like the obvious way to help – disclosing to your customers all of the vulnerabilities in your software that you know about – would be the height of irresponsibility. If a vulnerability hasn’t been patched yet, the last thing you want to do is broadcast the existence of that vulnerability to all of your customers.

On the other hand, you might decide that you do need to provide information on all vulnerabilities (not just the ones that have already been patched, which of course can be advertised to the whole world) to at least your largest and most trusted customers. You need to decide internally in what cases you will do this, what safeguards you will require on the customer end, what alternatives to full disclosure you might suggest to customers for whom you don’t want to take the “Full Monty” approach, etc.

I hope you understand what I’m getting at here. For each of the six items in R1.2, you need to figure out a complete strategy for dealing with all of your customers (and as you can see, you will probably want to deal with particular types of customers in different ways, although the way you break down your customers is likely to vary according to the item in question). This will be an important component of your strategy, Mr./Ms. Vendor.

But this isn’t the only component of your strategy. My next post in this series will go over what else needs to be in your strategy.

If you are with a vendor to the electric power industry, and your company is trying to figure out what you will have to do to comply with CIP-013, Tom Alrich LLC would be pleased to offer you a free one-day (2-6 hours) workshop to review a) what CIP-013 requires, b) what you are likely to get asked to do by your clients, and c) what they should be asking you to do (since b and c won’t usually be the same thing). There will be no charge for my time, but I will require you to pay travel expenses at cost. Please email or call me if you would like to discuss this.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post. To discuss this, you can email me at the same address or call me at 312-515-8996.       

---

[i] And of course, I’m also doing a series of posts on how NERC entities can comply with CIP-013. Here is the most recent post in that series. Vendors should read this series of posts as well!

[ii] And by the way, NERC entities also need a CIP-013 compliance strategy, which I am gradually laying out in my other series of posts. A lot of elements in both strategies should be the same – just told from different points of view. But a NERC entity’s strategy will inherently include a lot of elements that have nothing to do with vendors, since vendor risk is just one of three areas of supply chain risk that need to be addressed in the supply chain cyber security risk management plan. See this post for a short summary of those three areas, and see this post for a long, painful discussion of that topic – but which ultimately might be more enlightening.

I Forgot My Anniversary!

This might seem like an odd topic for a post on Valentine’s Day, but I’m actually referring to the anniversary of my blog. A friend asked yesterday how long I’d been writing the blog, and I at first was going to say three years - because for some reason I seem to have latched onto that number a couple years ago, and never bothered to do the advanced math since then.[i] Then I realized it’s been five years. In fact, just over two weeks ago I missed the fifth anniversary of my first post, written in a hotel room in San Diego on January 28, 2013.[ii]

So anyway, I’m proud that this blog has lasted five years, and I hope it lasts another five! And for those of you who’ve stuck with me for some or even all of the five years, thanks!

  

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post. To discuss this, you can email me at the same address or call me at 312-515-8996.

Any opinions expressed in this blog post are quite definitely those of my employer, Tom Alrich LLC! If you disagree with what I’ve said, I suggest you take it up with them.

---

[i] This may also explain why for many years I’ve had the idea that I’m 42. I finally realized that had to be wrong when my son turned 30 last year – and I was sure he hadn’t been born while I was in junior high!

[ii] I had previously written posts for a Honeywell blog that no longer exists, and before that I’d written some “open letters” that Honeywell distributed. The open letters started in 2010, when I attended my first CIP drafting team meeting and found it so interesting – and consequential – that I wrote my first open letter about it.

The topic of my first post wasn't exactly timeless. I was trying to convince people that they had to take CIP version 4 seriously, since after all FERC had approved that version in April of 2012, and the compliance due date was April 1, 2014. I - and many others - felt that CIP version 5 was still a pipe dream that might never be approved by FERC. So it was much better to go with the bird in the hand (v4) rather than the two in the bush (v5). This campaign of mine culminated in what I call my "Dewey Beats Truman" post, when I confidently predicted that v4 would come into effect in 2014, with v5 following a few years later. Of course, just two months later FERC issued their NOPR saying they intended to approve v5, and v4 would go to sleep with the fishes.

Lew Folkerth on Configuration Management

I apologize, but it seems I’ve fallen behind my minimum quarterly requirement of posts that quote from Lew Folkerth of RF. I just discovered Lew wrote a great article on configuration baselines and CIP-010 R1 for the RF Newsletter dated November/December 2017. You can find it by clicking on The Lighthouse in the table of contents on the left side of the page. I was also pleased to note that RF will now send out emails when new newsletters come out (which is bi-monthly), so neither you nor I will miss any future articles from Lew.

The article speaks for itself, but here are the points I found most interesting[i]:

• Installed software and firmware listed in CIP-010 R1 should match software and firmware listed in CIP-007 R2 (Patch Management). Auditors check for this now, so you should definitely make sure they match on a regular basis, and even sync the two lists up if possible (page 16, last column).
• A good tip for simplifying the job of CIP-007 R1 (Ports and Services) documentation by leveraging information from the baseline (p. 17, first column).
• Benefits of a good baseline for incident response (p. 17, first column).
• Lew’s recommended list of software and firmware to include in the baseline (p. 17, third column).
• Lew recommends that firewall rules be under change management, whether or not they’re included in the baseline for the firewall.
• The box about scripts on page 18 is worth the price of admission by itself! And that certainly doesn’t mean it’s worthless, even though admission to the article is free.

I recommend you all read the article, as well as subscribe to the newsletter.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post. To discuss this, you can email me at the same address or call me at 312-515-8996.

Any opinions expressed in this blog post are quite definitely those of my employer, Tom Alrich LLC! If you disagree with what I’ve said, I suggest you take the matter up with them.           

---

[i] A few of these aren’t new – in fact, I’ve written about them in previous posts) – but they’re worth repeating.

How should Vendors “Comply” with CIP-013? Part I

The title of this post should give you pause. Vendors don’t have to comply with CIP-013; only NERC entities do. But this is a distinction without a difference. It has always been the intention of both FERC (in ordering NERC to develop a supply chain security standard) and the NERC standard drafting team (in developing the standard) to require vendors to be involved with compliance – although there will be no penalty to the NERC entity if they try to get a vendor to cooperate in compliance and are unsuccessful.

So it’s very legitimate to ask how a vendor can comply with CIP-013. I have had discussions with several vendors (and many more NERC entities) about this, and here is what we have come up with so far. I do want to point out that this is quite embryonic.

The most important consideration for a vendor is that it will be a big mistake if you look at CIP-013 compliance as some sort of adversary exercise, although you should be forgiven if you have thought of it that way so far. After all, a lot of the guidance so far (and that’s just one 13-page document) focuses on contract language.

My idea of a contract language “negotiation” is one in which the NERC entity’s lawyers and the vendor’s lawyers are sitting on opposite sides of an 8-foot-high brick wall. The entity’s lawyers throw some contract language to the other side and snarl “Here, put this in our contract.” The vendor’s lawyers look it over and throw something else back, saying (perhaps snarling less) “We can accept some parts without change and some with changes, but we can’t accept these other parts at all.” The entity’s lawyers look this over, and reply with their own counter-proposal. This goes back and forth, with the length of time determined by whether or not the lawyers are on salary (in which case they can do this for years) or they are paid by the hour (in which case the company will have to call an end to the fun and tell them to wrap up the “negotiation”).

Of course, this is perhaps an overly bleak view of the contract negotiation process, but I think it illustrates what I’m saying: Both the vendor and the entity should do everything they can to keep their relationship from degenerating to this point. In fact, IMHO if they first resort to contract language in their CIP-013 compliance process, both sides have already lost the game. Contract language is a nice thing to have, but it should never be the first concern.

So what’s the best way for a NERC entity and a vendor to approach the CIP-013 compliance process? I see two possible ways to get this going:

1. The entity reaches out to the vendor and says “Let’s have a discussion on how we (note the plural pronoun!) can address CIP-013 compliance.”
2. The vendor reaches out to all of their power industry customers (at least all who have any High or Medium impact assets where their software or hardware product is used), perhaps through an e-mailing or even a – gasp! – USPS mailing, and actually suggests how both sides could work together to achieve the goal of CIP-013 compliance.

So here’s a little quiz: Which of these two actions is more likely to result in a fruitful partnership to achieve CIP-013 compliance? I think they are both good options, but I would strongly recommend that the vendor be the first to reach out. Because if the entity reaches out first, I can almost bet that most of them will do that in the form of a demand for particular contract language. This isn’t because they’re sociopaths, but because NERC compliance up until now has primarily been seen as a legal exercise, whether for NERC CIP or the other NERC standards.

If the vendor reaches out first, and if they don’t immediately focus on contract language (if they do, they’ve obviously decided that it’s important that CIP-013 compliance be made as painful as possible, probably for reasons of job security of the people writing the document), then they can set the terms of engagement. The vendor can suggest actions that favor its strengths – i.e. developing, supporting and securing great hardware and/or software products – and don’t favor the utility’s strength (which is the fact that they are, after all, the buyer, and that money is flowing from them to the vendor, not vice versa. Plus the vendor doesn’t have a monopoly, at least in any electric power product market that I know of).

What should the vendor suggest to their power industry clients? I’ll leave that for the next post in this series. 

If you are with a vendor to the electric power industry, and your company is trying to figure out what you will have to do to comply with CIP-013, Tom Alrich LLC would be pleased to offer you a free one-day (2-6 hours) workshop to review a) what CIP-013 requires, b) what you are likely to get asked to do by your clients, and c) what they should be asking you to do (since b and c won’t usually be the same thing). There will be no charge for my time, but I will require you to pay travel expenses at cost. Please email or call me if you would like to discuss this.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post. To discuss this, you can email me at the same address or call me at 312-515-8996.

Any opinions expressed in this blog post are quite definitely those of my employer, Tom Alrich LLC! If you disagree with what I’ve said, I suggest you take it up with them.

I’m an ICS “Influencer”!

I was honored to be asked by Indegy – a company you should really learn about, especially if you own or operate power plants – to join nine other industrial cyber security “influencers” in giving our ideas on what you, i.e. a staff member in an organization operating ICS, can expect in 2018. While I can’t tell you whether your taxes will be lower or you’ll receive a bigger bonus this year, I can talk about what I think is an important development – which anyone who follows this blog might have already heard about once or twice J. You can read the post, on Indegy’s blog, here.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post. To discuss this, you can email me at the same address or call me at 312-515-8996.

Complying with CIP-013, Part 3: What is a good Implementation?

If you’ve read the first two posts in this series (here and here), you realize that for these posts I’ve tried to forget everything I know or have written about CIP-013, and just work through the words of the requirements (and the Purpose statement at the beginning of the standard). My goal has been to mine what is actually written in the requirements and get it all out on the table. The reason I’m doing this is because nothing is binding on the auditors (and therefore on the NERC entity) except what is in the requirements. So it’s important to put in a lot of effort to pull apart the requirements to find everything that’s possible to find in them – before we start going beyond what the wording says and start to think about the best way to actually comply with the requirements.

In our last episode, we wallowed in R1 and found – to my genuine surprise – that there is something important missing from R1.1: While this requirement mandates that the NERC entity “identify and assess” cyber security risks in the supply chain, it doesn’t require them to do anything about them! Of course, this is clearly just a mistake, and I highly recommended that everyone assume the word “mitigate” is also in there, whether or not this problem ever gets fixed in a formal manner.

Now we’re ready to go on to R2, which on the surface seems quite simple:

R2. Each Responsible Entity shall implement its supply chain cyber security risk management plan(s) specified in Requirement R1.

Is this really that simple? Yes and no. It isn’t simple because of what I just pointed out about R1.1: Your supply chain cyber security plan must include mitigation of the risks you’ve identified and assessed, even though that step isn’t included in the requirement part; so you need to mitigate risks when you implement the plan as well. But it is simple because it doesn’t mandate anything about how the plan needs to be implemented.

It seems you’re on your own in this requirement. You have to implement the plan, but you might take one year or 50 years. You might complete one part before starting on the next one, or you might address all parts simultaneously and try to complete them all at the same time. You might prototype your plan in one part of the company before moving it to others, etc. It seems the details are all up to you.

Indeed, this idea is reinforced in spades by the Implementation Guidance document put out by the CIP-013 drafting team last year. The document says literally nothing about what must be done to implement the plan, beyond repeating the substance of the note that appears below R2, emphasizing that the entity isn’t required to renegotiate or abrogate existing vendor contracts. It seems the drafting team couldn’t think of anything important to say about implementation.

However, let’s be realistic: The details of implementing a NERC CIP standard are never up to you. Last fall, I looked at the industry’s early experience with audits of CIP-014; like CIP-013, this standard requires the entity to develop a plan (in this case, a physical security plan for key substations) and then implement it. In two posts (here and here), I told several stories that point to the same conclusion: Even though R2 lacks any details about what constitutes a good vs. a bad implementation, the auditors are going to have their own ideas, and they won’t hesitate to make these known to the entity if they think they haven’t done a good job of implementation[i]. They hopefully won’t actually issue a Potential Non-Compliance (PNC) finding, as they did to one of the entities discussed in the second post. But they will very likely identify an Area of Concern and require the entity to fix the problem identified.

So what are the auditors going to look to when they decide whether or not an entity has properly implemented their supply chain cyber security risk management plan from CIP-013 R1? Darned if I know, but there is one thing of which I’m sure: I’m sure that auditors will look at the actual timeline for completing the implementation, and how that compares with what is in the plan.

Say your plan lays out a two-year implementation schedule, and your next audit is scheduled for about 1 ½ years after you start the implementation. If the auditors show up on that date and decide there’s no way your implementation can be completed by the two-year mark, they will very likely issue an Area of Concern, meaning you will need to develop a new plan with a realistic implementation date. And you’d better make that new date, or you’ll be in trouble three years later at your next audit.

So what does this mean for your plan? Should you always sandbag the finish date in the plan, making it at least a year or two after the date you think you’ll really finish? I don’t really think this is a great idea. I think you should include in your plan what you honestly believe is the date you will finish your implementation. However, I also think you should keep very close tabs on the implementation. Whenever it begins to look like you will not be able to meet your original finish date, you should immediately revise the plan to reflect the new date (of course, you should have change control to document that you made this change. You certainly don’t want to be in the position of needing to convince the auditor that the revised plan was actually the original one!).

Of course, if the auditor doesn’t agree with your excuse for why the implementation date had to be moved back (e.g. “My dog ate our supply chain cyber security risk management plan and we didn’t have any backup. This forced a six-month implementation delay as we re-drafted the plan from scratch.”), they may disagree with you. But IMHO there is no way they can issue you a PNC because of this, since R2 is completely silent on what characterizes a good vs. a bad implementation.[ii]

What’s the moral of this story? Even though R2 looks like it’s wide open to whatever interpretation you might want to give it, this doesn’t mean you should simply do whatever you want when you implement your supply chain cyber security risk management plan.  The auditors will always have their idea of how a plan should be implemented. It is almost certain that they will make their ideas clear to you in your audit, even if – hopefully – they don’t resort to a PNC to do that.

But what if it didn’t need to be this way? What if there were a way you could get your auditors (or at least your Regional Entity) to weigh in on whether and how your plan should be changed, before you started to implement it? That is the question I discussed in this recent post. There needs to be some way a NERC entity can get their CIP-013 R1 plan reviewed by their Region before they embark on implementing it. Moreover, there needs to be some way the Region can check in on the implementation of that plan as it is in progress, to make sure the entity hasn’t made some big mistake that might force it to re-do some or all of what it has already done.

The post described, with the assistance of an auditor, the Entity Development teams that are already implemented in one NERC Region and being implemented in another. The members of these teams aren’t auditors, but they help entities comply with the NERC CIP standards (and possibly other NERC standards). It seems to me that these teams would be the ideal organization to review an entity’s CIP-013 R1 plan as well as its implementation, before the entity has invested a sizable sum in implementation of what could turn out to be a flawed plan

I believe that some mechanism like this needs to be in place, in order for CIP-013 (as well as other plan-based standards like CIP-014 and the upcoming CIP-012, along with plan-based requirements like CIP-003 R2 and CIP-010 R4) to be successful. I hope you agree with me on this (and if not, that you’ll let me know). I hope NERC does, too!

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post. You can email me at the same address or call me at 312-515-8996.

---

[i] To be honest, the auditor issues in the second of the two posts cited (i.e. Part 3 of the series on lessons from CIP-014) had to do mostly with the plan itself, not with its implementation. But that is just because of timing: The two audits discussed in that post both occurred before the entity had even been able to start to implement their plan. Of course, this was very beneficial, since – had they implemented the plan, which was later determined to be insufficient, they might have been forced to re-do some or all of their implementation, using a new plan that was deemed sufficient.

[ii] They might conceivably issue you a PNC for violating CIP-011 R1.2, since the plan presumably included BES Cyber System Information and you obviously didn’t have a very good plan for protecting BCSI from your dog.