For what
seems the 50th time, I’m starting a post by referring to an article
by Lew Folkerth of Reliability First, found in the March-April
issue of RF’s newsletter (as always,
under the heading “The Lighthouse”). As has been the case on every previous
occasion, Lew has put his finger on an important issue in the current NERC CIP
compliance regime.
The article[i] specifically
addresses the question whether the use of virtualization is permitted by the
current CIP standards, and if so what is permitted and how it should be
implemented in a compliant manner. Lew answers the first question by saying
virtualization is neither permitted nor prohibited. This is of course not
earth-shattering news, but Lew quickly moves on to something that is quite
interesting: the concept (which he invented, at least as far as CIP goes) of
“zones of authority”.
Lew brings
this concept up for the following reason: Even though there are no actual CIP
requirements that apply to virtualized environments, this hasn’t stopped NERC
and the regions from coming up with a few “guiding principles” for
virtualization; probably the most important of these is that there should not
be “mixed-trust” virtualized environments. Briefly, this implies that a host
server for multiple VMs shouldn’t host both ESP and non-ESP devices, and a
switch with multiple VLANs shouldn’t host both ESP and non-ESP networks.
Lew says (or
implies) that auditors are having differences of opinion with entities on the
security of mixed-trust switches. It seems these entities have switches that
implement both ESP and non-ESP VLANs. When the auditors tell them this isn’t
secure, the entities point out that the non-ESP VLANs have just as good
security as the ESPs do[ii]. So why
aren’t they safe?
Lew doesn’t
dispute that it is very likely that, from a pure security point of view, these
entities are right: mixed-trust switches are just as safe as switches that only
implement ESPs. But this is where zones of authority come in. Lew points out
that the auditor only can look at CIP assets, like ESPs. Anything else, such as
a VLAN that isn’t an ESP, is completely out of his or her purview; the auditor
just has to assume these are completely insecure, and thus shouldn’t be found
on the same switch as ESP VLANs.
Of course,
Lew is completely right about this: The NERC CIP standards only take account of
BES Cyber Systems and their associated ESPs, PCAs, EACMS, etc. Anything that
isn’t one of these is completely out of scope. This is a double-edged sword, of
course. On the one hand, the auditors can’t look at what isn’t in an ESP, which
limits the kinds of evidence an entity can show them. On the other hand, the
entity has no compliance obligations for something that isn’t in an ESP or that
protects ESP devices, like an EACMS or PACS. My guess is most entities, if told
they could force CIP auditors to consider security controls they have
implemented for non-ESP networks, but only in return for having at least some
of the cyber assets on those non-ESP networks fall into scope for CIP, would
say “Thanks but no thanks. We’ll leave things as they are.”
And this is
exactly the issue: It would make a lot of sense from a security point of view
to have all cyber assets and networks that are in the entity’s control be in
scope for CIP[iii].
Just look at the Ukraine attacks, which started with phishing attacks on the IT
networks. It was only after completely taking over those networks that the
attackers were able to find a way into the relays that were their ultimate
targets, even though they were on the OT network.[iv] But the
idea of having all or even just a few of their IT cyber assets suddenly become
BES Cyber Systems would cause a lot of CIP professionals to bash their head
against the nearest hard object, or seriously consider making a career change
to fast food service.
So are we
stuck here? While it would certainly be a good idea to have all networks under
the NERC entity’s control be in scope for CIP, I would agree with my
hypothetical CIP professional that the burden of making these networks ESPs,
and making even a few of their cyber assets be BES Cyber Systems, would be too
great to justify the benefit: the enhanced ability to demonstrate their
security posture to an auditor.
But we’re
not stuck here. Were we to rewrite the CIP requirements so that they weren’t
prescriptive (i.e., so they were objectives-based), and enclose them in an
appropriate “wrapper” so that the entity was simply required to address a
certain set of threats (and this list of threats were regularly updated,
outside of the standards development process), I think we could safely expand
CIP to include all networks under an entity’s control. But there would still
have to be an appropriate classification of cyber assets in scope, depending on
how “near” they are to the BES. Those that are directly involved in control of
the BES (as BCS are in the current standards) would have to have stronger
controls than those whose involvement is only indirect (such as the IT network
assets at the Ukrainian utilities that were attacked).
Of course, I’ve
made this point about the need to rewrite CIP previously. And I will continue
to make it, including in my next two posts.
The views and opinions expressed here are my own and don’t
necessarily represent the views or opinions of Deloitte.
[i]
I recommend you read the article yourself, of course, even though I will
summarize parts of it here. Don’t worry – it’s just a page. If I had written
the article, it would have at least been the length of one chapter in a book,
if not an entire book.
[ii]
I’m speculating here. I’m not sure if auditors are actually running into this
problem already, or if Lew just anticipates that they will. Either way, Lew’s
discussion provides an excellent window into a real problem.
[iii]
I recently was introduced to another CIP blogger, Larry G, who publishes a blog
called NERD CIP; while I haven’t
had time to spend a lot of quality time with his posts yet, they look quite
insightful. In particular, he recently wrote a post that
starts off with a consideration of the same Lew Folkerth column that this post
started with (although he has a different take on it). In that post, Larry
points to the PCI definition of Trusted Network, which reads “Network of an
organization that is within the organization’s ability to control or manage.”
Since all Trusted Networks are potentially in scope for PCI, they will all need
to be considered – either because they themselves hold payment card data or
because their compromise could lead to compromise of the networks that do hold
that data. In fact, I will venture a guess that CIP is the only set of cyber security standards that limits itself to a subset of the networks actually controlled by the entity.
[iv]
And please don’t tell me that it was only because the three Ukrainian utilities
had poor separation between their IT and OT networks (lack of two-factor
authentication, lack of intermediate server, etc) that the latter were
penetrated. When attackers have complete control of the IT network for months,
they will one way or the other be able to find their way into the OT network,
no matter how good the security may be. How about this: One day all of the
relay engineers get an email from their boss (when he is on vacation) that due
to some need for emergency maintenance they need to open all of the circuit
breakers at a certain time that day. Some might not follow this instruction;
others might do it. In any case, there would be a coordinated outage. If this
happened at a number of utilities, it could cause a serious BES problem.
"When the auditors tell them this isn’t secure, the entities point out that the non-ESP VLANs have just as good security as the ESPs do[ii]. So why aren’t they safe?"
ReplyDeleteThe solution is to place all the devices in this mixed-trust inside of the ESP from a compliance standpoint. No need to reduce security, keep the two trust segments segregated from a technical L2/L3 standpoint, just as they are now. The reality is that the switch is a BCA and anything connected to it is inside the an ESP. From a technical standpoint, nothing needs to change. It's all from a compliance standpoint: everything connected to the switch is in one ESP, and therefore must be afforded PCA protection at the high water mark. Now the auditors are happy because they can audit. From a technical standpoint, it is just as secure and the least amount of switches/firewall hardware is required. We are never restricted from having multi-trust levels within an ESP - just the same as we can have multi-trust levels within a PSP. The auditors don't get to audit the ESP rules between the mixed-trust environments, only from/to outside/inside the ESP, the same as they can only audit the Physical APs from the PSP and not the doors within which may have more limited permissions.
If you follow my logic, this clearly applies the same to a virtual environment. You can virtualize all you want, but as we see time and time again, the entire environment is really one trust level, even if you can apply some nuanced restrictions between guests and vlans. Why is this? Because hypervisors, switches, fabric, etc., are found to be exploitable with vulnerabilities and go beyond the configured permissions. Just look at the most recent Amazon exploit allowing a covert ssh channel between two guest VMs sharing the same CPU. Virtualize all you want, but everything connected is part of the same ESP and will be either BCAs or PCAs at the same high water mark. This must include storage as well, because as we all know, virtualization falls over when the storage goes away.
-JJR
The main point really isn't that the security of "non-CIP" is as good as "CIP". The most important thing is that network isolation is maintained between them. The configuration of devices in one VLAN are irrelevant to devices in the other VLAN. If it's just a Layer 2 VLAN, there is no "mixed trust" because the traffic from "non-CIP" doesn't mix with that of "CIP".
ReplyDeletehttp://nerd-cip.blogspot.com/2017/04/a-tale-of-two-viewpoints-mixed-trust-vs.html