UDP ports in NERC CIP

A client asked me a question last week about UDP ports and compliance with CIP-007 R1.1. Since I didn’t know the answer, I posted the question on the Western Interconnect Compliance Forum board, where there are often very good discussions of CIP issues. This time didn’t disappoint; there were some very good responses (including several I haven’t included here). Below is that discussion, followed by my short discussion of the lesson I’ve learned from the discussion.

My original post on WICF:

A client is asking the following question. Does anyone have ideas on this?

“We have numerous hardened devices such as HP iLOs (Integrated Lights Out) which don't have any sort of NetStat command to show ports. These devices get monthly scans from our Nessus scanning tool. But each month, the monthly scans produce a different list of UDP ports for the devices. Since UDP ports are, by nature, stateless, they cannot be opened or closed. What is the best way to document these ever-changing UDP ports in our CIP-010 R1 baseline?”

Response from WECC entity 1:

Ports on any hardware device are tough. A NETSTAT or scan at any moment doesn't tell the whole story, especially as many listening ports may actually be ephemeral responses to outgoing traffic.

Usually, a good bet when you can't get clean port information from the device itself is to go to manufacturer's documentation. But there's nothing easy or direct to basic web searches. I get this:

http://danielcheah.blogspot.com/2015/06/hp-ilo-integrated-lights-out-port.html

But it just gives TCP ports. What's going on with UDP?

It's possible that the UDP ports could all be DNS responses. The iLO wants to resolve a domain name like HPE.com, so it shoots out a request to port 53 and listens on a dynamic port for the response. So that port will always change, and you won't know what it's for unless you catch the port 53 UDP packet when it goes out.

The tricky thing with UDP is that it could be "fire-and-forget." It doesn't form a connection. There could be tons of UDP traffic for which you never see an open port. There could be leftover open ports that don't actually do anything.

Usually, NETSTAT and NMAP will tell you open TCP ports, but they won't tell you all the ports being used for UDP. Instead of capturing ports, the only reliable way is to capture traffic in transit.

You can wrap it by putting it behind a firewall and review the firewall logs to see what went through. You can restrict rules until you see what breaks. Or you can span the port on the switch and pass the span port into Wireshark. Maybe you can even find a useful command in the SSH shell. Or maybe you come from a company with enough clout that you can get HP themselves to tell you what the ports are for.

There's almost always a way to get ports with some degree of justification.

Response from Kevin Perry, retired former Chief CIP Auditor of SPP Regional Entity (note: I contacted Kevin by email, not through WICF):

The above person's response is spot on. The difference between TCP and UDP is that UDP is stateless. There is no negotiated, bi-directional communication session like there is with TCP. The sender transmits the UDP packet with an IP address and port, and hopes it is picked up by the destination. There is no acknowledgment from the destination. On the destination device, the UDP port has to be raised to catch the packet. Because there is no "state", there is no distinct declaration that the port is "listening" as opposed to another state such as "established", like there is with TCP ports. Essentially, all raised UDP ports not tied to the local loopback address are network accessible and are considered to be listening. And, yes, there are exploits that make use of UDP ports. Network Time Protocol (ntp - UDP port 123) had an active exploit a couple of years ago.

In the case of ntp, both the client and the time server have to raise UDP port 123. The client sends a time update request out over UDP port 123 to the time server. The time server has UDP port 123 raised in order to receive the request. It then sends the reply time packet back to the requester (the client) using UDP port 123. That means the client must also have UDP port 123 raised in order to get the time hack. In some environments, the time server automatically broadcasts the time on a defined interval and again, the devices needing the time hack have UDP port 123 raised.

Nmap struggles with UDP ports. It may report falsely that a UDP port is open (or not). And, as Joseph pointed out, it may simply be an instant communication like dns that tries to mimic TCP bi-directional communication without creating and then tearing down a session. So, like Joseph said, vendor documentation is the best source if you cannot interrogate the device using something like netstat.

As far as documentation goes, you can document a port range and associate it with the service or application, whether TCP or UDP. In the case of the ILO card, you cannot turn the logical ports off, so they are deemed necessary. And, the ports need to be identified, documented, and restricted as appropriate at the perimeter firewall, not just because CIP-005 says so, but because you need to minimize your attack surface as a good utility and good cyber security practice.

(Kevin’s response to follow-up email from me)

A number of entities have been disabused of the idea that UDP ports were excluded because they were not in a “listening” state.

Entity approaches I have seen vary, but they are usually some combination of vendor documentation, internal native commands like netstat, host-run programs that do the equivalent of internal native commands, and external scanners like nmap.  I have not seen anyone do network packet captures and analysis, but it would certainly show what was passing back and forth.

Response from Brent Sessions, VP Risk and Reliability Compliance, Western Area Power Administration (WAPA):

In the case of UDP, tools that can help you demonstrate what UDP traffic is going to and from a device could be your network IDS, a host-based IDS, or a passive vulnerability scanner.  I've used these in the past to demonstrate the random port issue - you can set rules around those for monitoring.  Cumbersome, maybe, but it can capture the UDP traffic if the sensors are in the right place. 

Also, even with UDP, there is a service that has to have a handle to bring up a port and listen.  Knowing what service that is and how it behaves is a good way to demonstrate you understand your network port baseline.  Like others have said, vendor docs are invaluable here.  But if the vendor is not responsive, looking at traffic patterns over time using the methods described here, you might be able to build a profile.  Better than nothing.

And now here’s Tom’s response to this discussion:

It seems clear to me that there are lots of things a NERC entity can do to identify UDP ports, although no technique will ever provide the definitive list. So the question really becomes “What is enough for compliance?”

If CIP-007 R1 were a risk-based requirement, this would be a very easy question to answer. The requirement would say something like “Take appropriate steps to mitigate the risk posed by open logical ports on BES Cyber Systems.” When deciding how it will respond for UDP ports vs. TCP ports, the entity will need to decide the relative risks posed by the two types of ports. Since there have been very few successful UDP attacks and since they don’t seem to have caused much damage when they happened (vs. – as we all know – huge amounts of damage caused by successful TCP attacks), it would be very easy to document that the relative levels of risk posed by UDP vs. TCP ports justified focusing the entity’s resources (which aren’t unlimited, of course) on the latter.

But as we all know, CIP-007 R1 isn’t a risk-based requirement, which is why the only correct answer to the question what is enough for compliance is “Whatever your auditor will accept.” Of course, this is the case with other CIP requirements as well, but it seems it’s especially appropriate here, since there simply is no such animal as a definitive list of open UDP ports. Of course, this isn't a very satisfying answer, since it shows that here - as with many other questions about the prescriptive CIP requirements - you have to rely on the "Don't ask, don't tell" NERC CIP compliance structure, in which entities are forced either to ask an auditor from their Region the answer to a compliance question (and then only receive a verbal response, which won't at all be binding on any future auditor) or take their best guess and hope future auditors will consider it close enough to what they want to see that they will give them a pass (although perhaps with an Area of Concern to address before the next audit). 

So what’s the root of the problem? Is it that the CIP v5 drafting team (who drafted this requirement) got lazy and didn’t bother to define clear criteria for identifying open UDP ports? No. As with a lot of questions having to do with cybersecurity, there simply is no correct way to do this, and there doesn’t even seem to be any real meaning to the phrase “the correct way to identify open UDP ports”.

The problem is that NERC standards are prescriptive. If you think of the Operations and Planning standards (most of which long predate CIP), they are prescriptive because they have to be. They’re ultimately based on the laws of physics, which – the last time I checked – don’t change from time to time or situation to situation. If you do something once and the rest of the environment hasn’t changed, you’ll get the same results if you do it again.

However, since there are no immutable “laws” of cybersecurity, prescriptive requirements simply don’t make sense there. If you’ve been dealing with CIP for a long time, you know this is hardly the only instance – or even the twentieth or thirtieth - where there simply is no good answer to a question of how to comply with a prescriptive requirement. The only approach that works for cybersecurity regulation is a risk-based (also called “plan-based” or “objectives-based”. They amount to the same thing) one. Currently, CIP-003 R2, CIP-007 R3, CIP-010 R4, CIP-011 R1, CIP-013 and CIP-014 are risk based (and the drafts of CIP-012 indicate that will be risk-based as well). Ultimately, I think all of the CIP standards will be risk-based (and it’s interesting that literally all of the new standards and requirements drafted since CIP v5 have been risk-based. I don’t think anyone would even propose that a new NERC CIP requirement be anything but risk-based). There’s simply no other way to write successful security standards.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. My offer of a free webinar on CIP-013, specifically for your organization, remains open to NERC entities and vendors of hardware or software components for BES Cyber Systems. To discuss this, you can email me at the same address.