Friday, July 12, 2024

NERC CIP and the cloud: The sad story of how we came close to fixing the problem in 2018, but didn’t


It’s good to see that interest is finally picking up in removing the barriers that prevent NERC entities with medium and/or high impact BES environments from utilizing many cloud-based services, especially those whose use would amount to utilizing either BES Cyber Systems (BCS) or Electronic Access Control or Monitoring Systems (EACMS) – like security monitoring services - in the cloud.

One good indicator of this interest is the fact that there seem to be a lot of people interested in serving on the new Standards Drafting Team, which is now being constituted and will presumably start meeting later this year. Unfortunately, I think it will be 5-6 years before their work is finished and the new “CIP cloud” standards are implemented, but I’m glad the process has at least started.

Fortunately, the new SDT doesn’t have to start from scratch. In fact, they can study the lessons to be learned from the story of the CIP Modifications SDT[i], which in 2018 outlined – and started to develop – a set of CIP changes that would have completely solved the cloud problem (although that’s not why they introduced them). Were it not that they had overlooked one crucial step, there would probably be no need for a new SDT today; use of the cloud would already be a non-issue for any NERC entity, just as it is for almost any organization in any other industry today.

You can learn what happened in detail by reading (or at least skimming) these three posts that I wrote on the SDT’s ideas in 2018: Part 1, Part 2, and Part 3. Here’s what led up to the events of 2018:

1.      When CIP version 6 was approved in early 2016, FERC – as they usually do, at least when they approve new or revised CIP standards – ordered NERC to make some further changes, which I won’t discuss here. The SDT that was appointed to develop those changes was called “Modifications to CIP Standards”. That SDT is still in operation today.

2.      Usually, when FERC orders a new standard or changes to existing standards, a new SDT is chosen; this happened in 2016 as well. To guide the new SDT, a Standards Authorization Request (SAR) is normally developed that closely mirrors what FERC ordered. The general idea is that the new SDT should just focus on giving FERC what they asked for; it is risky to ask them to do more than that.

3.      However, it seems one of the members of the new SDT must have had bad karma, since the CIP Mods team’s SAR had one slight modification: They needed to figure out how to incorporate virtualization into the CIP standards. After all, how hard could that be?

4.      As it turns out, it’s quite hard. The team is still working to achieve that goal, even though they achieved their other goals about six years ago and even though I suggested at one point that it wouldn’t be such a bad thing if they just followed Sen. George Aiken’s advice for ending the Vietnam War: “Declare victory and leave”. After all, they had given FERC everything they asked for. Incorporating virtualization into CIP was never intended to be their primary goal.

When the SDT started focusing in earnest on virtualization in 2017 (or early 2018), they realized their job would be made much easier if they focused on two changes that would affect all the CIP standards (at the time, there were only ten standards: CIP-002 through CIP-011).

The first (and more significant) of these was changing CIP asset identification from being focused entirely on physical devices to being focused on systems. Even though CIP-002-5 had introduced the concept of BES Cyber System (BCS) and nominally made that the basis for all the CIP requirements, a BCS was defined as simply an aggregation of BES Cyber Assets – and those are individual physical devices. A NERC entity can’t identify its BCS without first identifying its BCAs. Since the BCA definition references Cyber Assets, the first step in the whole process is identifying devices that meet the Cyber Asset definition: “Programmable electronic devices…”.

The SDT proposed breaking that device connection through two steps. First, they would rewrite the BCS definition to include what was in the BES Cyber Asset definition, including 15-minute impact on the BES. Second, they would eliminate the terms Cyber Asset and BES Cyber Asset altogether, so a NERC entity would never again have to identify devices as the basis of their CIP compliance program; instead, they would simply look at all the systems they operate, both physical and virtual, and decide which of those have a 15-minute impact on the BES. There would be no need to consider how a system is physically implemented.

The result of this process would be that the entity has a list of their BES Cyber Systems, which they will then classify into high, medium and low impact[ii], based on the criteria in CIP-002 R1 Attachment 1.

This alone was a great idea, but the SDT realized it wasn’t enough; further changes would need to be made to the CIP standards to make this change workable. They made it clear in their 2018 webinar (the slides for which are still available here. The recording has been taken down) that some of the existing CIP requirements would need to be rewritten to be more “objective based” and “technology neutral” (see slide 29).

Within the NERC community, these two terms have come to describe a requirement that simply states the objective the NERC entity must achieve, rather than the means they use to achieve it. The webinar gave the example of CIP-007 R3 as an objective based and technology neutral requirement (slide 26), and CIP-005 R1 (slide 27) as the opposite. In fact, the webinar described how (part of) CIP-005 R1 might be rewritten to make it objective based (slide 28).

When I listened to the webinar in 2018, I was very focused on the problems that the need to comply with prescriptive CIP requirements (including CIP-005 R1, CIP-007 R2 and CIP-010 R1) poses for NERC entities, although I usually referred to my proposed solution as “risk based” or “plan based” requirements, not usually as “objective based” requirements. In practice, I believe these are all names for the same thing. I was really excited about the webinar (as shown by the fact that I wrote three posts about it, all linked earlier), since it seemed clear that implementing the changes they suggested would fix both the problem I was concerned with and the problem they were concerned with (virtualization).

However, it turns out the SDT’s proposal would also have fixed another problem: CIP and the cloud. This is because the biggest problem with CIP in the cloud is the fact that the focus on individual devices (whether physical or virtual) in the CIP requirements makes it almost impossible for the cloud service provider to furnish their CIP customers the evidence they need to demonstrate compliance with prescriptive requirements like CIP-007 R2 and CIP-010 R1.

After all, a CSP doesn’t want to track every possible physical or virtual device that a BES Cyber System (or more accurately, any part of a BCS) might have been installed on over a three-year NERC audit period and document how each of those physical or virtual devices has been compliant with a large number of CIP requirements over that period. Yet, that is what a lot of CSPs would have to do if any of their offerings met the definition of EACMS, BCS or PACS.

The two elements of the CIP Modifications SDT’s 2018 proposal would have eliminated this problem, for two reasons. First, the CSP would no longer be obligated to track physical or virtual devices on which any part of a BCS or EACMS was installed during the audit period, and show that every requirement of CIP was complied with for each of those devices.

Second, the CSP would also no longer need to document compliance with prescriptive requirements like CIP-007 R2, which require documentation of compliance for every device in scope for the requirement in every instance in which compliance was required. Instead, since all CIP requirements would be rewritten as risk-based, the CSP would simply have to document they had developed and followed appropriate policies and procedures. In many cases, these p’s and p’s would already be in place, due to the CSP’s obligation to comply with FedRAMP or ISO 27001.

However, it turns out that the SDT’s proposal met an inglorious end: As the SDT started to get to work on drafting the two parts of their proposal (moving to systems as the basis of CIP compliance and making certain prescriptive requirements objectives-based), the larger electric utilities (especially the IOUs) realized that the changes the SDT was proposing would essentially require them to throw away a lot of the training, software, etc. that they had implemented for CIP compliance over the years.

These utilities (including their main trade organization, EEI) decided there was no way they were going to make such a drastic change, especially since it would probably have to be made all at once when the new standards were implemented. They put their (collective) foot down, and that was the end of the SDT’s proposal. Instead, the SDT went back to what had been their original idea: they would keep CIP’s focus on devices, but define virtual devices, not just physical ones. The requirements could remain almost the same, but their applicability would be expanded. This requires making substantive changes to most CIP requirements and getting them approved by the NERC ballot body, in most instances. This effort is ongoing today, and shows no sign of ending soon.

When I heard this had happened (at the end of 2018, at a NERC CIPC meeting in Atlanta), I was quite disappointed, and initially thought the industry had been let down by the big boys. However, I later came to realize it was quite unrealistic to expect so many organizations to make such a drastic change all at once. Since I didn’t see any way that these attitudes would change easily, I started to think there wasn’t much hope that NERC entities with high and/or medium impact BES environments would ever be able to make full use of the cloud.

However, I clearly wasn’t thinking creatively enough; it’s good that some other people were more creative than I was. Last year, a new SAR was approved by the NERC Standards Committee in December; that SAR is the basis for the drafting team that’s now being formed. One feature of the new SAR was one that I hadn’t thought of at all: there should be two “tracks” for NERC CIP compliance, one applicable to on-premises systems and one applicable to cloud-based systems.

Under the SAR, any NERC entity that has no use for the cloud and is happy keeping all their OT systems on-premises will be welcome to do so; the CIP requirements they had to comply with will be almost unchanged from the ones they were already following. Meanwhile, those entities that do want to start using the cloud can start following the new “cloud CIP track” for their cloud-based systems, although they need to continue to follow the “on-premises track” for on-premises systems.

This sounded messy to me until I did a kind of “test” of the idea in preparation for a webinar that I did with Lew Folkerth of RF last October. I was quite surprised to find that implementing the two tracks using the existing CIP-002 wording was much easier than I anticipated. This is because CIP-002 nowhere mentions either Cyber Assets or BES Cyber Assets. Instead, it starts with identification of BES Cyber Systems and lets the NERC entity discover for themselves that they can’t do that without first identifying Cyber Assets and BES Cyber Assets.

Thus, it’s easy to simply divide “BES Cyber Systems” into two types: “On premises BCS” and “Cloud BCS”. To identify the former, the NERC entity needs first to identify Cyber Assets and BES Cyber Assets. However, to identify Cloud BCS, the entity can start with systems implemented in the cloud. My guess is that, as NERC entities gain more experience complying with the “Cloud CIP” standards and realize how much simpler it is to comply for cloud-based systems, they will be more inclined to utilize the cloud for their OT systems.

Are you a vendor of current or future cloud-based services or software that would like to figure out an appropriate strategy for selling to customers subject to NERC CIP compliance? Or are you a NERC entity that is struggling to understand what your current options are regarding cloud-based software and services? Please drop me an email so we can set up a time to discuss this!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.


[i] I initially called this the “CIP v7 SDT”. This was because I thought the changes would all be made as part of a new set of revisions to all the standards, as had always been done previously (i.e., CIP versions 1-6). However, the CIP Modifications drafting team could see there would be different timelines for approval of the changes they were working on; the idea of “revving” all the standards at once no longer made sense. That is why there are at least 7 or 8 version numbers attached to the current CIP standards. There will almost certainly never be a single version number for all CIP standards, nor is that necessary. 

[ii] Of course, identifying low impact BCS is a very complicated topic, which I don’t want to get into here.

No comments:

Post a Comment