NERC CIP: Great! Now we have 4 CIP cloud problems, not 3. Merry Christmas!

It’s well known that the NERC CIP standards, not because of design but because of CIP’s history as a standard for use of important systems on premises, put substantial obstacles in the way of NERC entities wishing to use the cloud in their OT environment, as most of them are now doing (to different degrees) in their IT environment.

As recently as last July, I believed there were three distinct problems:

First, implementation of medium and high impact BES Cyber Systems (BCS) in the cloud was effectively verboten, mainly because the CSP would have to provide the NERC entity with evidence for compliance with requirements like CIP-010 R1 and CIP-005 R1 and maybe 15 others that require tracking the devices on which the systems (or parts thereof) are stored or processed. No CSP can ever furnish such evidence, meaning the NERC entity that stores medium and or high BCS in the cloud will end up in violation of just about every CIP requirement in the book over the entire audit period. At $1 million per day per system and per requirement violated, that would be…a lot more than I earn in a month!

Second, I knew that Electronic Access Control or Monitoring Systems (EACMS) for medium and/or high impact ESPs were being prevented from residing in the cloud by the fact that the entity could never get evidence of compliance with a subset of the problems holding up medium/high impact BCS. However, the fact that only a subset of the CIP requirements apply to EACMS was cold comfort, since EACMS were prevented just as effectively by the remaining requirements, especially those of CIP-006, which would effectively require the CSP to declare physical access to some of their data centers (or perhaps all of them) to be under the control of each NERC entity that had EACMS (or BCS) stored in them. 

The employees of each data center would need to badge in once for every NERC entity with any of their systems stored there; each entity would have to run a Personnel Risk Assessment on each employee before they authorized them to enter the data center…but you get the idea. This is simply impossible.

And I knew something else about the EACMS prohibition (I still know it, BTW): this prohibition has seriously damaged the security of the grid, and the rate of that damage is now increasing. How do I know this? It’s because there are many security services that monitor traffic into and out of (and sometimes within) the NERC entity’s Electronic Security Perimeters (ESPs) and stream the data out to a server in the cloud. That server processes all the information and finds evidence of threats that are affecting other organizations (not just utilities). Then it feeds that information back to the NERC entity as actionable intelligence.

It has always been a core element of security services like this that their servers be in the cloud, since that gives them real-time access to information about attacks and threats worldwide. This makes the information they provide more valuable than the information that can be provided by a server located within the NERC entity’s ESP, which may not receive this worldwide information for days, if at all.

And guess what else security services like this have in common? They’re all completely forbidden by NERC CIP. That’s because the servers in the cloud perform access monitoring (not usually access control), and since EACMS means Electronic Access Control or Monitoring (not Electronic Access Control and Monitoring as it used to), the fact that those cloud-based servers are monitoring access makes them EACMS. An EACMS has to be within a Physical Security Perimeter (PSP, in case you couldn’t guess the acronym), meaning that the horrors about badging, access control, etc. that I described above would apply to any cloud data center that held even a portion of the server.

This means that, at least for the past six or seven years (and maybe longer), NERC entities with medium and/or high impact BES Cyber Systems (and therefore medium or high ESPs, which means medium/high EACMS) have been unable to take advantage of these cloud-based security services and have instead had to rely entirely on services that could be delivered using only on-premises equipment.

In fact, I remember a well-respected NERC auditor telling me a number of years ago that he had to tell a medium-large utility in his Region to rip out the cloud-based security service they were using and find a totally on-premises system. He said it “broke his heart” to tell them that, since he knew their level of security would be lower because of it (although I think more than one or two people in his Region might have questioned the idea that he had a heart in the first place. It’s tough being an auditor!).

Six or seven years ago, the idea of cloud-based security services was fairly new, so this was an unusual situation. However, I’m told that in the past year, more and more such services with on-premises hardware are giving notice to their customers that, even though they may still offer them an on-premises service, all of their enhancements from now on will go to their cloud-based service; some are simply telling their NERC entity customers that they can’t use their service anymore if they can’t use the cloud-based option. “Yes, we’re sorry to lose our NERC CIP customers, but for every one of you, there are hundreds of customers that don’t even know what NERC CIP is. Given the choice, we have to go with the bulk of our customers.” Who can blame them for saying this?

The third problem was storage of BES Cyber System Information (BCSI) for medium and/or high impact BCS in the cloud. There are only a small number of CIP requirements that apply to BCSI, but BCSI in the cloud is – as of today - still technically no more permitted than BCS or EACMS in the cloud. Why was this? It’s complicated, but it had to do with just two words in two sub-requirements (called “Requirement Parts” in CIP): “storage locations”. Those two words prevented BCSI from being officially “legal” in the cloud.

It might seem odd that storage of BCSI (or any information) in the cloud could be so important. After all, there are plenty of very inexpensive on-premises storage options nowadays, so the NERC entity that can’t store BCSI in the cloud can’t be suffering that much harm, can they?

Aha, but here’s the rub: SaaS applications (software as a service) in some cases need access to BCSI to work their magic. Think of a configuration management service that can help with CIP-010 R1 compliance. It requires a lot of configuration information on BCS, EACMS, and PACS. That information will usually be BCSI. And guess what? If the application can’t get the information it needs, it will be useless for a NERC entity with medium or high impact systems to sign up for it.

So that’s what happened. Because of those two words, all use of SaaS was technically off limits for NERC entities with high and/or medium systems. I know there were at least one or two SaaS providers that not only retained their NERC entity customers but expanded their numbers, in full view of the NERC auditors. In 2018, I even attended a user group meeting for NERC entities in one of the Regions, who were customers of one of those SaaS providers. They all reported that their auditors did nothing to discourage them from using that provider, even though they were in theory in violation of at least two Requirement Parts every day of the audit period.

Fortunately (or so it seemed to me until a fateful day a few weeks ago), we were all happy to hear that the BCSI problem was solved when a drafting team developed new versions of CIP-011 and CIP-004, which included what initially seemed to me to be quite an elegant fix for this problem. Those two new versions are set to come into effect on January 1, 2024.

So, does this story have a happy ending? Sure, the BCS and EACMS problems remain, but at least the BCSI problem will be fixed in less than a month. Progress is good, right?

Well, no. It seems multiple CIP auditors now agree that there’s a problem with the wording of CIP-004-7 R6 that literally blocks all SaaS from being usable by NERC entities with medium and/or high impact BCS. What is this problem? Is it some high-minded concern about the security of these SaaS providers?

Nope. Once again, the problem is caused by two words in that requirement. However, the two words aren’t the same as they were previously. The problem with “storage locations” is gone, but now it’s been replaced by a problem with “provisioned access” (maybe that’s progress, but I doubt it). Briefly, it seems that, even if the BCSI used by a SaaS app is encrypted going to the cloud and while in the cloud, it still needs to be decrypted in order for the app to use it. Even though that decryption might not last even one second, the fact that during that second an employee of the CSP (who of course hasn’t been authorized by name by the NERC entity) might be able to see the unencrypted data is enough to possibly make the NERC entity noncompliant with R6.

I want to emphasize that this new problem has nothing to do with security, any more than the three problems described above have anything to do with security. This is all an unintentional consequence of decisions made by the team that drafted CIP-004-7 and CIP-011-3 (the two standards that come into effect on 1/1/24), and I’m sure there were good reasons why they made those decisions. Moreover, I’m sure nobody even thought this might be a problem. But here we are.

How will the problem get fixed? Can NERC just put out an “errata” document saying, “Oops, we didn’t mean to imply that the BCSI could never be decrypted once it reaches the cloud. After all, that wouldn’t make sense, would it? Let’s just interpret the word in a sensible way. We’ll say that the BCSI can be decrypted by the CSP employees while loading it into the SaaS app”.

Unfortunately, a solution like that is simply not possible in the world of NERC CIP. I agree that prohibiting use of SaaS due to a wording glitch doesn’t make sense, any more than previous problems caused by the words “programmable”, “routable”, “used by”, etc. made sense. But you know what? Making sense and $5.25 can get you a tall no-foam latte that you can drink while you mull over this problem.

I thought the problem could be fixed if a drafting team (either one that is already in existence or one that is created for this purpose) could just spend a few weeks on it, but I was assured by someone who knows a lot more about NERC compliance than I do that it’s a much more difficult problem – not because there’s any real question about what a secure solution is. Instead, it’s about what a compliant solution is, that can be implemented without having to change the NERC Rules of Procedure, etc.

The irony of this is that many NERC entities with high or medium impact BCS have been using a small number of SaaS apps for years. In fact, in 2018 I attended a meeting of NERC entities in one of the Regions, that were all using a particular SaaS application for configuration management of systems within their ESP (including BCS, of course). None of those entities reported that their auditors had any problem with this, even though they knew exactly what the entity was doing. On the other hand, I doubt any of those entities sent an email to their audit team asking for permission to do this; they would without much doubt have been turned down had they done that.

I’m sure those entities (and probably many more) are still using that SaaS application without any problems, and at last report the Bulk Electric System in that Region has not collapsed because of this gross violation of the NERC CIP regulations. But all other NERC entities need to be warned: If you ask for permission to use SaaS in a high or medium impact environment after January 1, you’ll probably be turned down. I’ll let the astute reader figure out what they should do about this, but I’ve seen nothing in writing warning NERC entities that, contrary to previous expectations, they won’t be allowed to use SaaS after January 1. I’ll leave it at that.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

I lead the OWASP SBOM Forum. If you would like to learn more about what that group does or contribute to our group, please go here.