Wednesday, April 19, 2017

Lew Folkerth’s “Zones of Authority”

For what seems the 50th time, I’m starting a post by referring to an article by Lew Folkerth of Reliability First, found in the March-April issue  of RF’s newsletter (as always, under the heading “The Lighthouse”). As has been the case on every previous occasion, Lew has put his finger on an important issue in the current NERC CIP compliance regime.

The article[i] specifically addresses the question whether the use of virtualization is permitted by the current CIP standards, and if so what is permitted and how it should be implemented in a compliant manner. Lew answers the first question by saying virtualization is neither permitted nor prohibited. This is of course not earth-shattering news, but Lew quickly moves on to something that is quite interesting: the concept (which he invented, at least as far as CIP goes) of “zones of authority”.

Lew brings this concept up for the following reason: Even though there are no actual CIP requirements that apply to virtualized environments, this hasn’t stopped NERC and the regions from coming up with a few “guiding principles” for virtualization; probably the most important of these is that there should not be “mixed-trust” virtualized environments. Briefly, this implies that a host server for multiple VMs shouldn’t host both ESP and non-ESP devices, and a switch with multiple VLANs shouldn’t host both ESP and non-ESP networks.

Lew says (or implies) that auditors are having differences of opinion with entities on the security of mixed-trust switches. It seems these entities have switches that implement both ESP and non-ESP VLANs. When the auditors tell them this isn’t secure, the entities point out that the non-ESP VLANs have just as good security as the ESPs do[ii]. So why aren’t they safe?

Lew doesn’t dispute that it is very likely that, from a pure security point of view, these entities are right: mixed-trust switches are just as safe as switches that only implement ESPs. But this is where zones of authority come in. Lew points out that the auditor only can look at CIP assets, like ESPs. Anything else, such as a VLAN that isn’t an ESP, is completely out of his or her purview; the auditor just has to assume these are completely insecure, and thus shouldn’t be found on the same switch as ESP VLANs.

Of course, Lew is completely right about this: The NERC CIP standards only take account of BES Cyber Systems and their associated ESPs, PCAs, EACMS, etc. Anything that isn’t one of these is completely out of scope. This is a double-edged sword, of course. On the one hand, the auditors can’t look at what isn’t in an ESP, which limits the kinds of evidence an entity can show them. On the other hand, the entity has no compliance obligations for something that isn’t in an ESP or that protects ESP devices, like an EACMS or PACS. My guess is most entities, if told they could force CIP auditors to consider security controls they have implemented for non-ESP networks, but only in return for having at least some of the cyber assets on those non-ESP networks fall into scope for CIP, would say “Thanks but no thanks. We’ll leave things as they are.”

And this is exactly the issue: It would make a lot of sense from a security point of view to have all cyber assets and networks that are in the entity’s control be in scope for CIP[iii]. Just look at the Ukraine attacks, which started with phishing attacks on the IT networks. It was only after completely taking over those networks that the attackers were able to find a way into the relays that were their ultimate targets, even though they were on the OT network.[iv] But the idea of having all or even just a few of their IT cyber assets suddenly become BES Cyber Systems would cause a lot of CIP professionals to bash their head against the nearest hard object, or seriously consider making a career change to fast food service.

So are we stuck here? While it would certainly be a good idea to have all networks under the NERC entity’s control be in scope for CIP, I would agree with my hypothetical CIP professional that the burden of making these networks ESPs, and making even a few of their cyber assets be BES Cyber Systems, would be too great to justify the benefit: the enhanced ability to demonstrate their security posture to an auditor.

But we’re not stuck here. Were we to rewrite the CIP requirements so that they weren’t prescriptive (i.e., so they were objectives-based), and enclose them in an appropriate “wrapper” so that the entity was simply required to address a certain set of threats (and this list of threats were regularly updated, outside of the standards development process), I think we could safely expand CIP to include all networks under an entity’s control. But there would still have to be an appropriate classification of cyber assets in scope, depending on how “near” they are to the BES. Those that are directly involved in control of the BES (as BCS are in the current standards) would have to have stronger controls than those whose involvement is only indirect (such as the IT network assets at the Ukrainian utilities that were attacked).

Of course, I’ve made this point about the need to rewrite CIP previously. And I will continue to make it, including in my next two posts.

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte.

[i] I recommend you read the article yourself, of course, even though I will summarize parts of it here. Don’t worry – it’s just a page. If I had written the article, it would have at least been the length of one chapter in a book, if not an entire book.

[ii] I’m speculating here. I’m not sure if auditors are actually running into this problem already, or if Lew just anticipates that they will. Either way, Lew’s discussion provides an excellent window into a real problem.

[iii] I recently was introduced to another CIP blogger, Larry G, who publishes a blog called NERD CIP; while I haven’t had time to spend a lot of quality time with his posts yet, they look quite insightful. In particular, he recently wrote a post that starts off with a consideration of the same Lew Folkerth column that this post started with (although he has a different take on it). In that post, Larry points to the PCI definition of Trusted Network, which reads “Network of an organization that is within the organization’s ability to control or manage.” Since all Trusted Networks are potentially in scope for PCI, they will all need to be considered – either because they themselves hold payment card data or because their compromise could lead to compromise of the networks that do hold that data. In fact, I will venture a guess that CIP is the only set of cyber security standards that limits itself to a subset of the networks actually controlled by the entity.

[iv] And please don’t tell me that it was only because the three Ukrainian utilities had poor separation between their IT and OT networks (lack of two-factor authentication, lack of intermediate server, etc) that the latter were penetrated. When attackers have complete control of the IT network for months, they will one way or the other be able to find their way into the OT network, no matter how good the security may be. How about this: One day all of the relay engineers get an email from their boss (when he is on vacation) that due to some need for emergency maintenance they need to open all of the circuit breakers at a certain time that day. Some might not follow this instruction; others might do it. In any case, there would be a coordinated outage. If this happened at a number of utilities, it could cause a serious BES problem.


  1. "When the auditors tell them this isn’t secure, the entities point out that the non-ESP VLANs have just as good security as the ESPs do[ii]. So why aren’t they safe?"

    The solution is to place all the devices in this mixed-trust inside of the ESP from a compliance standpoint. No need to reduce security, keep the two trust segments segregated from a technical L2/L3 standpoint, just as they are now. The reality is that the switch is a BCA and anything connected to it is inside the an ESP. From a technical standpoint, nothing needs to change. It's all from a compliance standpoint: everything connected to the switch is in one ESP, and therefore must be afforded PCA protection at the high water mark. Now the auditors are happy because they can audit. From a technical standpoint, it is just as secure and the least amount of switches/firewall hardware is required. We are never restricted from having multi-trust levels within an ESP - just the same as we can have multi-trust levels within a PSP. The auditors don't get to audit the ESP rules between the mixed-trust environments, only from/to outside/inside the ESP, the same as they can only audit the Physical APs from the PSP and not the doors within which may have more limited permissions.

    If you follow my logic, this clearly applies the same to a virtual environment. You can virtualize all you want, but as we see time and time again, the entire environment is really one trust level, even if you can apply some nuanced restrictions between guests and vlans. Why is this? Because hypervisors, switches, fabric, etc., are found to be exploitable with vulnerabilities and go beyond the configured permissions. Just look at the most recent Amazon exploit allowing a covert ssh channel between two guest VMs sharing the same CPU. Virtualize all you want, but everything connected is part of the same ESP and will be either BCAs or PCAs at the same high water mark. This must include storage as well, because as we all know, virtualization falls over when the storage goes away.


  2. The main point really isn't that the security of "non-CIP" is as good as "CIP". The most important thing is that network isolation is maintained between them. The configuration of devices in one VLAN are irrelevant to devices in the other VLAN. If it's just a Layer 2 VLAN, there is no "mixed trust" because the traffic from "non-CIP" doesn't mix with that of "CIP".