UDP ports in NERC CIP

A client asked me a question last week about UDP ports and compliance with CIP-007 R1.1. Since I didn’t know the answer, I posted the question on the Western Interconnect Compliance Forum board, where there are often very good discussions of CIP issues. This time didn’t disappoint; there were some very good responses (including several I haven’t included here). Below is that discussion, followed by my short discussion of the lesson I’ve learned from the discussion.

My original post on WICF:

A client is asking the following question. Does anyone have ideas on this?

“We have numerous hardened devices such as HP iLOs (Integrated Lights Out) which don't have any sort of NetStat command to show ports. These devices get monthly scans from our Nessus scanning tool. But each month, the monthly scans produce a different list of UDP ports for the devices. Since UDP ports are, by nature, stateless, they cannot be opened or closed. What is the best way to document these ever-changing UDP ports in our CIP-010 R1 baseline?”

Response from WECC entity 1:

Ports on any hardware device are tough. A NETSTAT or scan at any moment doesn't tell the whole story, especially as many listening ports may actually be ephemeral responses to outgoing traffic.

Usually, a good bet when you can't get clean port information from the device itself is to go to manufacturer's documentation. But there's nothing easy or direct to basic web searches. I get this:

http://danielcheah.blogspot.com/2015/06/hp-ilo-integrated-lights-out-port.html

But it just gives TCP ports. What's going on with UDP?

It's possible that the UDP ports could all be DNS responses. The iLO wants to resolve a domain name like HPE.com, so it shoots out a request to port 53 and listens on a dynamic port for the response. So that port will always change, and you won't know what it's for unless you catch the port 53 UDP packet when it goes out.

The tricky thing with UDP is that it could be "fire-and-forget." It doesn't form a connection. There could be tons of UDP traffic for which you never see an open port. There could be leftover open ports that don't actually do anything.

Usually, NETSTAT and NMAP will tell you open TCP ports, but they won't tell you all the ports being used for UDP. Instead of capturing ports, the only reliable way is to capture traffic in transit.

You can wrap it by putting it behind a firewall and review the firewall logs to see what went through. You can restrict rules until you see what breaks. Or you can span the port on the switch and pass the span port into Wireshark. Maybe you can even find a useful command in the SSH shell. Or maybe you come from a company with enough clout that you can get HP themselves to tell you what the ports are for.

There's almost always a way to get ports with some degree of justification.

Response from Kevin Perry, retired former Chief CIP Auditor of SPP Regional Entity (note: I contacted Kevin by email, not through WICF):

The above person's response is spot on. The difference between TCP and UDP is that UDP is stateless. There is no negotiated, bi-directional communication session like there is with TCP. The sender transmits the UDP packet with an IP address and port, and hopes it is picked up by the destination. There is no acknowledgment from the destination. On the destination device, the UDP port has to be raised to catch the packet. Because there is no "state", there is no distinct declaration that the port is "listening" as opposed to another state such as "established", like there is with TCP ports. Essentially, all raised UDP ports not tied to the local loopback address are network accessible and are considered to be listening. And, yes, there are exploits that make use of UDP ports. Network Time Protocol (ntp - UDP port 123) had an active exploit a couple of years ago.

In the case of ntp, both the client and the time server have to raise UDP port 123. The client sends a time update request out over UDP port 123 to the time server. The time server has UDP port 123 raised in order to receive the request. It then sends the reply time packet back to the requester (the client) using UDP port 123. That means the client must also have UDP port 123 raised in order to get the time hack. In some environments, the time server automatically broadcasts the time on a defined interval and again, the devices needing the time hack have UDP port 123 raised.

Nmap struggles with UDP ports. It may report falsely that a UDP port is open (or not). And, as Joseph pointed out, it may simply be an instant communication like dns that tries to mimic TCP bi-directional communication without creating and then tearing down a session. So, like Joseph said, vendor documentation is the best source if you cannot interrogate the device using something like netstat.

As far as documentation goes, you can document a port range and associate it with the service or application, whether TCP or UDP. In the case of the ILO card, you cannot turn the logical ports off, so they are deemed necessary. And, the ports need to be identified, documented, and restricted as appropriate at the perimeter firewall, not just because CIP-005 says so, but because you need to minimize your attack surface as a good utility and good cyber security practice.

(Kevin’s response to follow-up email from me)

A number of entities have been disabused of the idea that UDP ports were excluded because they were not in a “listening” state.

Entity approaches I have seen vary, but they are usually some combination of vendor documentation, internal native commands like netstat, host-run programs that do the equivalent of internal native commands, and external scanners like nmap.  I have not seen anyone do network packet captures and analysis, but it would certainly show what was passing back and forth.

Response from Brent Sessions, VP Risk and Reliability Compliance, Western Area Power Administration (WAPA):

In the case of UDP, tools that can help you demonstrate what UDP traffic is going to and from a device could be your network IDS, a host-based IDS, or a passive vulnerability scanner.  I've used these in the past to demonstrate the random port issue - you can set rules around those for monitoring.  Cumbersome, maybe, but it can capture the UDP traffic if the sensors are in the right place. 

Also, even with UDP, there is a service that has to have a handle to bring up a port and listen.  Knowing what service that is and how it behaves is a good way to demonstrate you understand your network port baseline.  Like others have said, vendor docs are invaluable here.  But if the vendor is not responsive, looking at traffic patterns over time using the methods described here, you might be able to build a profile.  Better than nothing.

And now here’s Tom’s response to this discussion:

It seems clear to me that there are lots of things a NERC entity can do to identify UDP ports, although no technique will ever provide the definitive list. So the question really becomes “What is enough for compliance?”

If CIP-007 R1 were a risk-based requirement, this would be a very easy question to answer. The requirement would say something like “Take appropriate steps to mitigate the risk posed by open logical ports on BES Cyber Systems.” When deciding how it will respond for UDP ports vs. TCP ports, the entity will need to decide the relative risks posed by the two types of ports. Since there have been very few successful UDP attacks and since they don’t seem to have caused much damage when they happened (vs. – as we all know – huge amounts of damage caused by successful TCP attacks), it would be very easy to document that the relative levels of risk posed by UDP vs. TCP ports justified focusing the entity’s resources (which aren’t unlimited, of course) on the latter.

But as we all know, CIP-007 R1 isn’t a risk-based requirement, which is why the only correct answer to the question what is enough for compliance is “Whatever your auditor will accept.” Of course, this is the case with other CIP requirements as well, but it seems it’s especially appropriate here, since there simply is no such animal as a definitive list of open UDP ports. Of course, this isn't a very satisfying answer, since it shows that here - as with many other questions about the prescriptive CIP requirements - you have to rely on the "Don't ask, don't tell" NERC CIP compliance structure, in which entities are forced either to ask an auditor from their Region the answer to a compliance question (and then only receive a verbal response, which won't at all be binding on any future auditor) or take their best guess and hope future auditors will consider it close enough to what they want to see that they will give them a pass (although perhaps with an Area of Concern to address before the next audit). 

So what’s the root of the problem? Is it that the CIP v5 drafting team (who drafted this requirement) got lazy and didn’t bother to define clear criteria for identifying open UDP ports? No. As with a lot of questions having to do with cybersecurity, there simply is no correct way to do this, and there doesn’t even seem to be any real meaning to the phrase “the correct way to identify open UDP ports”.

The problem is that NERC standards are prescriptive. If you think of the Operations and Planning standards (most of which long predate CIP), they are prescriptive because they have to be. They’re ultimately based on the laws of physics, which – the last time I checked – don’t change from time to time or situation to situation. If you do something once and the rest of the environment hasn’t changed, you’ll get the same results if you do it again.

However, since there are no immutable “laws” of cybersecurity, prescriptive requirements simply don’t make sense there. If you’ve been dealing with CIP for a long time, you know this is hardly the only instance – or even the twentieth or thirtieth - where there simply is no good answer to a question of how to comply with a prescriptive requirement. The only approach that works for cybersecurity regulation is a risk-based (also called “plan-based” or “objectives-based”. They amount to the same thing) one. Currently, CIP-003 R2, CIP-007 R3, CIP-010 R4, CIP-011 R1, CIP-013 and CIP-014 are risk based (and the drafts of CIP-012 indicate that will be risk-based as well). Ultimately, I think all of the CIP standards will be risk-based (and it’s interesting that literally all of the new standards and requirements drafted since CIP v5 have been risk-based. I don’t think anyone would even propose that a new NERC CIP requirement be anything but risk-based). There’s simply no other way to write successful security standards.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. My offer of a free webinar on CIP-013, specifically for your organization, remains open to NERC entities and vendors of hardware or software components for BES Cyber Systems. To discuss this, you can email me at the same address.

Reminder: IEEE Smart Grid Cybersecurity Workshop

Just a reminder that IEEE is having their 2019 Smart Grid Cybersecurity Workshop in Atlanta on December 12^th and 13^th. It will be at the same hotel as the NERC CIPC meeting the two previous days, and CIPC participants will receive a discount for this workshop.

The agenda looks very good. And oh…I almost forgot. I’ll be the keynote speaker, discussing “Developing Your Supply Chain Cyber Risk Management Plan”. I hope to see you there!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. My offer of a free webinar on CIP-013, specifically for your organization, remains open to NERC entities and vendors of hardware or software components for BES Cyber Systems. To discuss this, you can email me at the same address.

The dreaded D-word

My last post, which discussed the most recent Wall Street Journal article on grid security, is a good example of something that’s happened to me a number of times during my storied career as a blogger: I started the post by thinking the topic was fairly well-defined and narrow, but I ended it by realizing that there was a larger issue to be addressed (you can get away with this when you’re a blogger. When you write newspaper articles like Rob Barry and Rebecca Smith do, you have to lead with the large issue, not discover it unintentionally at the end).

I indicated this larger issue when I wrote “..NERC and the industry may have been barking up the wrong tree all along, in considering that the biggest goal of cyber attackers would be causing a cascading outage in the Bulk Electric System; the right way to counter that is through the NERC CIP standards, which apply exclusively to BES assets. But if we’re talking about specific facilities like dams and military bases, the substations and generating plants that serve them are often purely distribution ones, to which the CIP standards don’t apply.”

This was also a takeaway from Rob and Rebecca’s January article (this link is to my post on the article, but if you want to see the article itself, see this funny link that I included in a footnote to my post). That article pointed out how the Russians are targeting utilities that serve military bases, which of course are also served primarily at the Distribution level, not the BES level. I wanted to write a post last January saying that everybody should be paying more attention to the risks posed by purely Distribution assets, but I started to build up a big backlog of posts to write, so I haven’t gotten around to it until now – and I only did because of the new WSJ article, which makes very clear that the primary focus of attacks on the US grid nowadays is on particular assets on the Distribution network, not the BES.

Why hasn’t there been a lot of focus on Distribution cyber regulation before this? It’s simple – because right now, it’s nobody’s job. NERC and FERC have jurisdiction solely over the Bulk Electric System (although FERC always uses the term Bulk Power System, and NERC sometimes uses that term. Nobody has been able to explain to me why one term is used rather than the other, so I will continue to believe it’s simply the preference of the person writing the document); it would take an act of Congress to change that. And if someone in Congress had the appetite to change that now, it would require a lot of cajoling and horse-trading with the state PUCs, who currently have jurisdiction over some Distribution assets in their states, and aren’t likely to give up anything to FERC and NERC without a fight.

And to be honest, the PUCs – while they talk earnestly about cybersecurity at the NARUC meetings all the time – aren’t doing a lot about grid security in their states, because they don’t have the authority for that, either. For one thing, the PUCs by definition don’t regulate anything but IOUs (for example, the state of Nebraska doesn’t have any IOUs at all, which is why the Nebraska Public Services Commission doesn’t even mention electric power on its website). And even with IOUs, the PUCs are officially only allowed to regulate pricing and safety – which is why all cybersecurity initiatives are ultimately cast as safety measures. Currently, I know of only one state – New Jersey – that has a full cybersecurity standard (and a well-written one at that); but it’s not audited, and of course it’s completely voluntary for the coop and municipal utilities in the state.

So if anybody’s thinking about cybersecurity regulation of Distribution-level assets, it’s very hard to see where it will come from barring an act of Congress, as I said earlier.

So why should we want regulation of Distribution-level assets? Haven’t I been complaining since 2014 about the CIP standards and how many problems they have? That’s true, and I certainly don’t recommend that the CIP standards as currently written should be applied to Distribution – in fact, I don’t think they should be applied to Transmission either. As I discussed in my recent webinar, I think the CIP standards need to be entirely rewritten, so that they’re much more like CIP-013 than CIP-007 R2 and CIP-010 R1. Pending that development, I wouldn’t wish the CIP standards on any other industry or country.

But, as I recently pointed out, the only good way I know for cyber professionals in any industry to get the resources they need to effectively mitigate cyber risks is to have mandatory cyber regulations. If we really want to mitigate the Distribution-level threats – which seem to be much larger than the Transmission-level ones – we need to figure out a way to develop regulations that don’t impose the big regulatory burden that the NERC CIP standards do on Transmission and large Generation entities. So extending the current CIP standards to Distribution – even if this could be pushed through Congress – isn’t a good option, and isn’t likely to be realized.

Should NERC develop a “CIP lite” that would apply just to Distribution entities? I think there should be a CIP lite (completely risk-based, as described in my webinar), but it should apply to all NERC entities, and replace the current CIP standards. While I think this will happen in the next few years (mainly because of pressure on NERC to do something to allow utilities to put Medium and High impact BES Cyber Systems in the cloud – not just BCSI), it’s currently not even on the floor, let alone on the table.

But the question is who should promulgate and enforce this new Distribution CIP standard? I vote for DoE. But there needs to be support in Congress to do this, and I know of none at the moment. So, despite the fact – well documented in the WSJ – that Distribution is probably the big problem area for cybersecurity, not Transmission, we’re a very long way from doing anything about it.

One of my favorite jokes is about a man who goes outside at night and finds his neighbor on hands and knees, looking for something under a streetlight. He asks his neighbor what he's looking for and the neighbor replies "My keys". He then asks him "Where did you lose them?", and the neighbor points to his dark lawn. So the man asks "Well, why are you looking for them here?". The neighbor replies "Because the light's better here."

This strikes me as an excellent metaphor for the current situation with cyber regulation of the US power grid. We know the focus of attacks nowadays is Distribution, but we can't do anything about it due to jurisdictional issues. So we put all of our efforts into regulating Transmission and large Generation, not Distribution. After all, the light's better there.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. My offer of a free webinar on CIP-013, specifically for your organization, remains open to NERC entities and vendors of hardware or software components for BES Cyber Systems. To discuss this, you can email me at the same address.

A new WSJ story on utility cyberattacks, with a difference

I’ve written a lot previously about Wall Street Journal articles about cyberattacks on the US power grid, including this post and this one. Now, WSJ reporters Rebecca Smith and Rob Barry are back with another article on this topic, but this one quite different from the previous two. While WSJ online articles are normally behind a paywall, Rob sent me this free link.

The previous two articles dealt with Russian attacks that came almost entirely through the vendor community – in the case of the 2018 article, via attacks on the vendors’ remote access systems, and in the January 2019 article, via phishing attacks on vendors.

In this new article, they discuss direct attacks on utilities (in case you though those weren’t happening any more – ha ha), and the provenance of those attacks isn’t at all clear. It seems the attacks are coming partly through malware-laden emails and partly through direct assault on firewalls; but the article doesn’t describe any attacks coming through vendors or other third parties.

There are two very interesting aspects to the new attacks (and this is a campaign that is fairly recent, unlike the Russian supply chain attacks, which have been going on literally for years): First, the utilities targeted are on the smallish side. The largest of the utilities named in the article is a large Generation and Transmission cooperative, and none are investor owned utilities (IOUs). Of course, none of these attacks is said to have succeeded, even in penetrating the IT network.

Second, the utilities don’t seem to be targeted simply in order to cause economic disruption through outages, but because they are located near particular strategic facilities, such as the important Sault Ste. Marie Locks between Lakes Superior and Huron, and Federal dams (even though the facilities may not actually be served by the utility target – which the attackers, presumably from overseas, wouldn’t usually know).

This was a theme of Rebecca and Rob’s January article: the Russians, at least in some cases, seem to be targeting specific facilities, including especially military bases. No military targets are mentioned in the article, which might indicate the attackers in this case are a different group than the one discussed in January, and they might not be Russian at all.

Would you like to know my big takeaway from this article?...I didn’t think so, but I’ll tell you anyway. It’s that NERC and the industry may have been barking up the wrong tree all along, in considering that the biggest goal of cyber attackers would be causing a cascading outage in the Bulk Electric System; the right way to counter that is through the NERC CIP standards, which apply exclusively to BES assets. But if we’re talking about specific facilities like dams and military bases, the substations and generating plants that serve them are often purely distribution ones, to which the CIP standards don’t apply.

In theory, distribution assets are regulated by the PUCs of the states in which they’re located. However, a little-understood fact about PUCs is they usually have direct authority only over IOUs, and as I said, none of the utilities mentioned in the article are IOUs. So probably a majority of the substations and generating plants that serve the strategic assets under attack aren’t subject to any cyber regulations (and only a few state PUCs have true cyber regulations, even for their IOUs).

Of course, this doesn’t at all mean these utilities have bad security – and it certainly doesn’t mean that the NERC CIP standards should be extended to them in some way. But it does show that any faith in NERC CIP as the country’s primary bulwark against the encircling cyber hordes is increasingly misplaced, even if it might have been justified five or ten years ago.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. My offer of a free webinar on CIP-013, specifically for your organization, remains open to NERC entities and vendors of hardware or software components for BES Cyber Systems. To discuss this, you can email me at the same address.

"(You) don’t know how lucky you are, boys…"

I noted a number of interesting statements at GridSecCon this year – as well as the Friday trip to SecureWorks’ headquarters in Atlanta, which proved to be very interesting - which I’d like to tell you about. I’ll gradually work through them as I get time; I hope I’m finished by the next GridSecCon!

One comment I found especially interesting was during a panel on natural gas security. Robert Mims of Southern Companies is in charge of cyber security for their natural gas division. He lamented the fact that his team consists of exactly four people, while his peer on the electric side has “hundreds” of people working for him.

What’s the difference? He doesn’t face a mandatory cyber standard like NERC CIP. He had no doubt that if he did, he would have a much bigger head count than he does (of course, even if the electric side didn’t have any mandatory cyber standards to worry about, I’m sure their team would still be multiple times the size of the gas team. There are just a lot more moving parts on the electric side, and in general I believe that the dangers of a cyber attack causing serious physical damage in gas are much lower than in electric).

And this goes to the real reason why mandatory standards are needed in some cases: the flow of money from management increases substantially when there are penalties to worry about (and I don’t think the monetary penalties are the biggest incentive for compliance. I’ve always said that power companies would do almost everything they could to avoid violations even if the “penalty” were a trip to Disney World. The reputational, etc. damage is much more painful than the monetary damage, in the long run).

So as much as I complain about problems with the CIP standards, I don’t want to see mandatory standards go away. However, I do think all security standards should follow one Golden Rule: As much as possible, they should simply require the entity’s cyber staff to do on their own what they would do if they received the same level of funding as they now do with the current NERC CIP standards, yet they didn’t have to comply with any standard. I contend they would “identify and assess” their cyber risks (to use the words found in CIP-013 R1.1, which is the best example so far of this approach), and mitigate the most important ones. And they would mitigate them using the most efficient approach possible – since they wouldn’t have to follow prescriptive requirements that inherently aren’t the most efficient approach, sometimes not by a long shot.

In other words, I would rewrite all of the CIP standards like CIP-013, although I’d make improvements to that, and there are other considerations as well[i]. But if you take away mandatory standards, you turn off the money spigot. Thus, a number of NERC entities have freely admitted to me that they would never get the same level of cyber funding as they do now, were it not for NERC CIP. Some even admit to justifying purchases as being “required by NERC CIP”, when in reality that’s not the case. But you didn’t hear that here, of course…

So tonight, you should thank your lucky stars that NERC CIP is mandatory, not a voluntary framework. As the Beatles said (in “Back in the USSR” from the White Album), “…(you) don’t know how lucky you are, boys…”

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. My offer of a free webinar on CIP-013, specifically for your organization, remains open to NERC entities and vendors of hardware or software components for BES Cyber Systems. To discuss this, you can email me at the same address.

---

[i] If you’d like to hear more about this, listen to my recent webinar, or drop me an email and I’ll send you the slides.

How NOT to comply with CIP-013, part I

I will admit that I’ve emphasized a little too much that there are many possible ways to comply with CIP-013 R1, and I need to swing the pendulum back in the other direction a little. While it’s true that the standard gives you very little to go on regarding what should be in your R1 plan, it’s also true that there are some things that definitely are required. This is especially true since NERC provided good information in their most recent Evidence Request Tool about what audits will focus on. If something will be audited on, you need to do it – at least that’s what my parents raised me to believe.

This post (and part II to follow very shortly) describes mistakes you can make, that can lead to your being non-compliant with CIP-013.

I. You don’t address both R1.1 and R1.2 in your plan

I’m sure that, even a year after FERC approved CIP-013, many long-time NERC compliance professionals look at R1 and have the following reactions:

1. Their eyes glaze over when they read 1.1. All this talk about identifying and assessing risks, without telling you any concrete steps that you need to undertake, sounds like some sort of mystic writing from the Upanishads, not a NERC requirement. Every other NERC requirement they’ve seen tells you exactly what you need to do and when you need to do it, period.
2. When their eyes fall on R1.2, they grab onto it like an old friend, even though it’s a friend that’s changed since the last time they saw him. The six items in R1.2 don’t tell you exactly what you need to do and when, but they at least do tell you particular objectives you need to meet, such as “Disclosure by vendors of known vulnerabilities related to the products or services provided to the Responsible Entity”.

It’s inevitable that many of these people will decide that R1.2 is the whole of R1, meaning that as long as their plan covers these six risks (there are actually eight, since R1.2.5 and R1.2.6 address two risks each), they’re fine. In fact, when a vendor of a software tool to aid in CIP-013 compliance spoke at GridSecCon, it was clear that he thought R1.2 was all that was required to be in the entity’s supply chain cyber security risk management plan.

Well, it’s not. R1.1 isn’t there just for the fun of it, but because the CIP-013 drafting team was taking FERC seriously when they said in Order 829 that NERC entities should identify supply chain risks to mitigate. True, FERC did require all of the specific items in R1.2 at various points in their Order – and the SDT helpfully gathered them all into one place in the standard – but these are in no way intended to be the only supply chain risks that NERC entities face, or even the most important ones. It’s still up to the entity to identify (in complying with R1.1) other risks that are important and mitigate those, along with the risks in R1.2.

And if you don’t believe me, take a look at NERC’s Evidence Request Spreadsheet 3.0, discussed in the last item below.

II. You don’t address the five risk categories in R1.1

Although it’s not too obvious, R1.1 identifies five categories of supply chain risk that need to be “identified and assessed” in your plan. They are[i]:

1. Product risks from procurement of equipment and software;
2. Risks from procurement of services applying to BCS equipment and software;
3. Product risks from installation of equipment and software;
4. Risks from use of services for BCS (Note: This applies to services on all of your Medium and High impact BCS, not just those you procure after 7/1/20); and
5. Transitions between vendors.

Of course, each of these risk categories can include many different risks. I’ll provide one example for each category:

1. The risk that a supplier’s software development environment will be compromised, and someone will plant a backdoor in software destined for one of your BES Cyber Systems, which they’ll later exploit (as happened with Delta Airlines last year, although in their case it was 800,000 credit card numbers that were stolen).
2. The risk that a utility will contract with a vendor of services for BCS who doesn’t vet their new hires properly. A person with malicious intent will inadvertently be hired and allowed to access – remotely or onsite – some of your BCS.
3. The risk that a software product you install on one of your BCS won’t have the most recent patches applied to it and will be compromised after you install it.
4. The risk that a vendor will fire an employee who has onsite access to your BCS and won’t immediately tell you about it. The fired employee will come to your site and take revenge on the vendor by disabling one of your BCS (of course, this is the risk behind R1.2.3. The risk behind R1.2.6 – actually two risks – also falls in this category. The other four items in R1.2 fall into category a).
5. The risk that, when you leave one vendor and start purchasing from a different one, the first vendor won’t destroy all of the data they have stored about your BES Cyber Systems. Someone will hack into their network and steal the data.

I freely admit that, if you omit one of these categories in your plan (e.g. transitions between vendors), you might not be cited for a potential violation. When most people talk about CIP-013 (including people at NERC and the Regions), they usually discuss risks that fall into categories a and d. However, all five of these categories are called out in R1.1. You need to at least consider all of them. If you decide there aren’t any risks that are likely to apply to you in one of the categories, you should document why you made this decision, but you don’t need to try to dream up risks just for the sake of having at least one in each category.

III. Your procedures don’t match you plan

Whatever is in your R1 plan, there is one thing that is very certain: In R2, you need to do what you said you would do in R1. But what if your procedures don’t match what’s in your plan? CIP-013 isn’t that different from the other CIP standards, in that what really counts is the procedures you write to comply with it. The big difference is that, with say CIP-007 R2, you have to write patch management procedures that match what the different parts of that requirement mandate. With CIP-013 R2, you need to write supply chain security risk management procedures that match the plan that you developed in R1. So in R1.1, you’re literally writing the “requirements” that you have to have procedures for in R1.2!

So how do you avoid being non-compliant in this way? Simple (at least in theory): when you write your plan, you need to make sure there’s nothing in it that you aren’t positive you can do in practice. And if there’s something you’re not sure about, take it out of the plan. You can always add it back later.

For example, suppose you commit in your plan to conducting full security audits of say your top five suppliers. However, when it comes time to do the supplier audits (say, within a year of CIP-013 becoming enforceable), you realize how much staff time and money these will require, and you begin to think of all the other things you could do to mitigate supply chain security risks if you hadn’t committed the time and money to audits. You should certainly be able to change your mind at this point, but you will need to revise your plan so that it will match your new course of action. When you’re audited, you will need to provide both plans, as well as explain why you made the change between them (and emphasize how it will result in greater supply chain security risk mitigation, since you'll use your resources more efficiently).

This isn’t a new development in CIP. Since CIP v1, it has always been clear that you must show that you complied in full with any plan or program that you develop as part of the compliance process. For example, if in your CIP-008 R1 cyber security incident response plan you indicate that you will take a certain step in your incident response, yet your record of a CSIRP tabletop exercise doesn’t indicate you took that step, you can be (and entities have been) cited for this omission.

IV. You don’t have evidence that you fulfilled your plan

CIP-013 R2 is also like most of the other CIP requirements, in that it requires that you have documentation to show you did what you said you would do in your R1 plan. But it differs from most of the other CIP requirements, in that you’re not required – with one exception, discussed in the next section - to have documentation of every instance when you did something that was called out in the plan.

The Measures section of R2 says you must have “documentation to demonstrate implementation of the supply chain cyber security risk management plan(s), which could include, but is not limited to, correspondence, policy documents, or working documents that demonstrate use of the supply chain cyber security risk management plan.” In other words, you must not only show the auditors that you developed a good R1 plan, but that you implemented the plan – yet at the same time, you don’t have to show that you followed it without fail in every instance.

For example, suppose your plan says you will send out a security questionnaire to vendors and suppliers every year. If the answers reveal that a vendor’s controls are weak in a particular area, you will talk with them to point out that you really need them to implement these controls, and to ask if they need your help or advice in implementing those controls (e.g., maybe a vendor really needs a SIEM, but has no experience or understanding of how to go about implementing and maintaining that).

Do you need to show documentation of every questionnaire you sent out and received, as well as of every phone call or email you sent to your vendors to follow up on deficiencies? No, but you do need to provide enough evidence to convince the auditor that you followed up on a deficiency pointed out in a vendor's questionnaire response. The evidence can be emails, records of phone calls, notes of an in-person meeting, etc.

However, there is one part of your plan that does require evidence of every instance in which it was applied. I’ll talk about that in the next section.

V. You don’t execute and document procurement risk assessments (PRAs)

NERC’s most recent Evidence Request Spreadsheet v3.0 shows what evidence will be required in audit Data Requests:

1. In the Level 1 DR, you will be required to “Provide each documented plan(s) that addresses the applicable requirement parts in CIP-013 R1.” In other words, you have to provide your plan and show it addressed both R1.1 and R1.2.
2. In the Level 1 DR, you will also have to list all of the “procurements” that were planned or in effect for high/medium impact BCS during the audit period. If you’re not sure exactly what a procurement is and when it starts and ends, see this recent post.
3. In the Level 2 DR, the auditors will take a random sample of procurements from the list you provided, and ask you to provide the following for each selected procurement:
1. “..evidence of the identification and assessment of cyber security risk(s) to the Bulk Electric System from vendor products or services resulting from: (i) procuring and installing vendor equipment and software; and (ii) transitions from one vendor(s) to another vendor(s).” In other words, evidence that you complied with R1.1 in your PRA.
2. “..the following evidence:

                                                               i.      Notification by the vendor of vendor-identified incidents;

                                                             ii.      Coordination of responses to vendor-identified incidents;

                                                            iii.      Notification by vendors when remote or onsite access should no longer be granted to vendor representatives;

                                                           iv.      Disclosure by vendors of known vulnerabilities;

                                                             v.      Verification of software integrity and authenticity of all software and patches provided by the vendor for use in the BES Cyber System; and

                                                           vi.      Coordination of controls for (i) vendor-initiated Interactive Remote Access, and (ii) system-to-system remote access with a vendor(s).

In other words, evidence that you complied with R1.2 in your PRA.

I think items 1 and 2 above are fairly self-explanatory at this point. However, what does item 3 mean? In other words, what should your PRA evidence contain, and what documentation will you need for it? At a high level, I think it should contain the following:

1. The list of risks (although I use the term threats here) that you considered in the PRA. Do you remember the set of threats that you identified for mitigation in R1.1? You should consider a subset of these, namely those that apply to the vendor or supplier in question. And how do you determine what these are? Well, that’s magic, but if you’re interested in more information on it, see the last paragraph below.
2. How you assessed those threats, to determine which were already mitigated (either by the supplier or vendor or by your organization itself) and which still need to be mitigated (at this point, that usually means your organization has to mitigate them, but there is still time to ask the supplier or vendor to help you – e.g. require that the vendor provide a security vulnerability assessment before shipping the product(s) to you). You do this using the vendor or supplier risk score (which should be calculated for each threat, not just an overall score for the vendor or supplier), combined with the product or service risk score (which would be a single overall score for that product or service) to get a “procurement risk score”. If that score is low, then this risk is mitigated for this procurement and no further action is needed on it. If it’s high or medium (or perhaps just high, if you’re being conservative), then you do need to determine what additional mitigations are required during the procurement to bring it to low. In my way of seeing things, a low risk score means the risk is mitigated - in fact, that's my definition of mitigation.
3. How you determined the mitigations required in this procurement, based on the risks you considered in the first step above (and that’s magic, too! Again, see the last paragraph below).

Even though you don’t need to provide instance-by-instance evidence for all other items in your R1.1 plan, for procurements you do, since the auditors will randomly sample the procurements and require evidence for certain ones - so you obviously need evidence for all of them before the audit. The spreadsheets, etc. that you create and utilitize as part of the PRA should be retained, and if they're properly designed, will provide compliance evidence without any further changes.

Would you like a little more explanation of some of the things I’ve just said? I’d be glad to do that – for your organization alone. I’m still offering a free two-hour (or less, if you prefer) webinar on CIP-013 compliance, for anybody in your organization who will be involved in the program. This includes people in procurement, legal, cybersecurity, and – oh, yes! – NERC compliance. The agenda is very flexible, so we’ll develop it in a discussion before the webinar. If you’re interested in this offer, please drop me an email at tom@tomalrich.com, and I’ll send you more information.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. My offer of a free webinar on CIP-013, specifically for your organization, remains open to NERC entities and vendors of hardware or software components for BES Cyber Systems. To discuss this, you can email me at the same address.

---

[i] I’m not going to go through the reasoning I used to arrive at this list, but I hope it won’t be too hard to discern. If you question how I got anything on this list, drop me an email and I’ll tell you.

My webinar recording is up!

The recording of the webinar I did last week on “How can we regulate cyber-security for critical energy infrastructure?” is now available. I thought the webinar went very well, with some very good questions. But I’d certainly appreciate hearing your comments as well, positive or negative. Just email them to me at tom@tomalrich.com.

BTW, if you'd like to see the slides, I'll be glad to send them to you. Just email me.

Any opinions expressed in the webinar are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

Lew covers Lows, and Tom rants

The September/October edition of RF’s newsletter contains another great article by Lew Folkerth[i]. He has been concentrating on CIP-013 (as I have) since late last year, but in this article he switched to the other big current concern of the NERC CIP community: Low impact assets, and specifically the two new versions of CIP-003 R2 that are coming out early next year. And per what we’ve come to expect, he covers Lows with typical Folkerthian (I’ve just coined a word. Remember, you heard it here first!) thoroughness.

Specifically, Lew discusses every CIP requirement that applies to Lows, both those in effect now and those on the way. And he provides good insights into everything he discusses. So where are the flaws in this article? Nowhere that I can see.

I’ve had a few people ask me lately what’s the deal with the two new CIP-003 versions: CIP-003-7, which becomes effective on January 1, 2020, and CIP-003-8, which comes into effect on April 1, 2020. Most people can’t tell the difference between them, which is quite subtle, I’ll admit. Here’s the story (and here I’m speaking more directly than Lew, being a NERC employee after all, is permitted to): You should really just consider CIP-003-7 as the next version, and ignore CIP-003-8[ii]. Literally the only difference between the two is that CIP-003-8 incorporates a small change that FERC ordered when they approved CIP-003-7, but it’s almost inconceivable that you won’t already have incorporated that change when you develop your CIP-003-7 program.

That change has to do with the Transient Cyber Asset (i.e. laptop) controls in Section 5 of attachment 1 of CIP-003. When FERC approved CIP-003-7, they pointed out that the suggested controls relating to TCAs owned by a third party (almost always a vendor, of course) in Section 5 of Attachment 1 just suggest that the entity review what the vendor has done to mitigate the risk of malware being on the TCA, not actually mitigate it. They ordered that NERC revise the requirement so that it also requires mitigation of the risk.

Therefore, Section 5 of CIP-003-8 Attachment 1 contains the sentence

5.2.2 For any method used pursuant to 5.2.1, Responsible Entities shall determine whether any additional mitigation actions are necessary and implement such actions prior to connecting the Transient Cyber Asset.

Let me reword this for you: CIP-003-7 stated that you should review the vendor’s controls on TCAs, but it didn’t explicitly require you to do anything if you didn’t like those controls. This means, for example, that you would

1. Discover that one vendor wasn’t doing anything at all to prevent malware on their laptops used at customers (which of course is highly unlikely in itself, since all vendors have lawyers who have told them in no uncertain terms the liability they’d face if one of their laptops infected – say – a critical substation and caused an outage).
2. However, FERC assumed you would do nothing more about this, without being prodded by the CIP standards. So you wouldn’t.
3. Then, when someone from the vendor came to your substation bearing one of those suspect laptops, you would just wave them through saying “Go ahead and connect to any device you want, even though there’s a good chance you have malware on your laptop. What could possibly go wrong?”

Of course, this is a ridiculous idea. Nobody who works in IT or OT, who isn’t literally insane, would do this. Yet FERC seemed to think this was a big problem, so they ordered NERC to change the requirement to make it clear the entity needs to do something when they have reason to believe a vendor isn’t doing what it should to protect its laptops from malware. This is roughly the equivalent of writing a law requiring parents to take action if they’re told their house is on fire and their children are trapped inside, based on the assumption that without such a law, they’ll just go about their business as usual, while saying “Really? Now that’s interesting….”

So FERC just ordered that NERC make this one change. But as you know, NERC can’t just edit a standard to put in a small change, then publish it and go out to lunch. To make the change that FERC ordered, NERC’s Rules of Procedure (RoP) require it to

1. Write a Standards Authorization Request for this change, and ballot that SAR;
2. Have a drafting team make the change (in this case, it was the CIP Modifications SDT, which drafted CIP-003-7 originally and which is still working today); then
3. Submit it for a first ballot by the NERC ballot body. There were 205 votes for and 30 against in that first ballot in the fall of 2018, yet the RoP doesn’t consider that final (not because this isn’t a sufficient majority, but because I believe there always has to be another ballot after any standard meets the required majority, for some reason). Thus, there had to be a “Final Ballot”, which was submitted and approved more than six months after the first one.
4. Of course, after that the NERC Board of Trustees had to approve CIP-003-8 at their next quarterly meeting, and finally FERC had to approve it, which took a few months in itself.

So now we have a new version of CIP-003 that differs by one sentence from CIP-003-7, and in practice literally requires entities to do something they’re all already doing for CIP-003-7 compliance. Yet NERC still has to go through the above process, which stretched out more than a year. Granted, I doubt anyone had to devote a huge amount of time to this, since it was all fairly routine. And since the SDT that had developed CIO-003-7 was still operating, NERC didn’t have to put together a new SDT, which would have been a big job in its own right. But the fact that everyone had to go through all these steps for what could easily have been a one-hour fix (which shouldn’t have even been needed in the first place) is pretty disappointing.

So who’s to blame for this? FERC deserves some blame, because they ordered the change. But the funny thing is that FERC always orders NERC to make changes to CIP standards that they approve (and there are few if any CIP approvals where they haven’t ordered at least one change), yet never admits that they know that NERC’s Rules of Procedure require it to go through the full standards development process for even the slightest change; instead, they just state that NERC should change the standard. Of course, I’m sure FERC isn’t allowed by law to consider NERC’s internal procedures when they decide a change is needed to a standard they’ve just approved.

So the real blame falls on the NERC Rules of Procedure – although not on any particular NERC employees. Since the above is the only way to make a change in a NERC standard that’s already approved, it’s a real inhibitor to improving existing standards (e.g. coming up with some “definition” of “programmable”, which NERC put in the SAR for the CIP Modifications SDT, but which they’ve never addressed and never will, perhaps because there really isn’t a workable definition of that word), as well as developing new standards to address new threats like phishing and ransomware.

Phishing and ransomware have both been around for many years (especially phishing) and are perhaps the two most destructive cyber threats today, yet there is now not even any discussion about developing new CIP requirements to address them. I think this is a problem, but I’m not at all inclined to suggest developing these two requirements now. In my webinar last week – for which the recording should be available soon – I described a mechanism for removing changes to CIP from the standards development process. Once that is done, then any important new threat could be easily incorporated into CIP – but this would also require that all of CIP be rewritten as non-prescriptive, risk-based requirements. But until that happens – and I think various pressures will drives this to happen, within a few years – trying to develop any more CIP standards not explicitly required by FERC will literally end up causing more harm than good.

Have a nice day. 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. My offer of a free webinar on CIP-013, specifically for your organization, remains open to NERC entities and vendors of hardware or software components for BES Cyber Systems. To discuss this, you can email me at the same address.

---

[i] There’s also a PDF that includes all 34 of the CIP/cybersecurity articles that Lew has written for the newsletter, starting in late 2015 (including this one). You can find that here.

[ii] Of course, you can’t really ignore it, since your compliance documentation will need to reflect that you were complying with CIP-003-7 for three months and then you became compliant with CIP-003-8. And don’t be snarky – like this post – and point out to the auditors that you were compliant with CIP-003-8 on January 1 anyway, since there was no way you could develop a CIP-003-7 program without also complying with CIP-003-8. Leave that to me.

Upcoming speaking engagement

I was quite honored to be asked recently to be the keynote speaker at the second annual IEEE Smart Grid Cybersecurity Workshop, which will be held on Thursday and Friday December 12 and 13 at the same hotel in Atlanta where the NERC CIP will meet on Tuesday and Wednesday (which I’ll also attend); the agenda is here. My topic will be – what else? – “Developing your Supply Chain Cyber Risk Management Plan”.

Of course, if you are a cynical person like me, you might point out that the smart grid has to do primarily with distribution, while CIP-013 (which of course requires the entity to develop a supply chain cybersecurity risk management plan) is a standard for Bulk Electric System assets. Why talk about CIP-013 at this workshop?

Fortunately, I already have my answer for you: I’m not talking just, or even primarily, about CIP-013. Any utility, in fact any organization that runs using computing hardware and software that they purchase, is subject to supply chain cybersecurity risks, and should have a risk mitigation plan. Exactly the same considerations go into developing a plan for cyber assets to be deployed on the distribution grid as for BES cyber assets. So there, smarty pants. And I won’t be alone in discussing supply chain security. There’s a panel on that topic right before me.

I’ll hope to see you there! 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. My offer of a free webinar on CIP-013, specifically for your organization, remains open to NERC entities and vendors of hardware or software components for BES Cyber Systems. To discuss this, you can email me at the same address.