NERC CIP in the Cloud: Is it time to start using SaaS?

I’ve written at least a few times about the difference between SaaS (software-as-a-service, although “software that runs in the cloud” is a more accurate description) and BCS (BES Cyber Systems) that are deployed in the cloud (for those unfamiliar with NERC CIP, BCS are the systems that the 13 current NERC CIP standards, CIP-002 through CIP-014, are there to protect. These are the systems whose loss r compromise would impact the Bulk Electric System (BES) within 15 minutes, although the impact wilusually be instantaneo.

Note from Tom 6/19: Kevin Perry, former Chief CIP Auditor for the SPP Regional Entity and co-chair of the team that drafted CIP versions 2-4, pointed out a couple of problems with this post as I wrote it yesterday. I've discussed them in italics below, although I decided not to change what I wrote yesterday - just point out where I was wrong.

Both SaaS and cloud BCS consist of software running in the cloud. They both provide advice and/or monitoring data to their operators. However, since the loss of a BCS, whether deployed in the cloud or not, will by definition impact the BES within 15 minutes, this means a BCS can provide more than advice; it can provide control or real-time monitoring. In other words, a BCS always has some sort of connection with a device (like an electronic relay that controls a circuit breaker) that impacts or monitors the power grid. Most importantly, the BCS needs to directly control, or monitor the output of, the grid-connected device. If a system’s impact on the grid is dependent on a human being taking some action first, then it’s not a BCS. 

Note: Kevin Perry pointed out that the converse of this last statement isn't true. That is, if a system's (negative) impact on the grid is dependent on a human being not taking a particular action (and that negative impact occurs within 15 minutes), then it's a BCS. He used the example of a SCADA system that notifies an operator when there's a problem that needs attention, so the operator can take actions to fix the problem. If that system is compromised and doesn't notify the operator of a problem - and that lack of notification leads to a negative impact on the BES within 15 minutes - then clearly the system can have a 15-minute BES impact and is therefore a BES Cyber Asset and also part of a BES Cyber System. 

This means that the same software product could be deployed in the cloud in two quite different ways. Let’s use the example of software that monitors power flows and can detect a dangerous anomaly in real time. If the software simply sets off an alarm and warns the operator of the anomaly – on the assumption that the operator will perform the steps necessary to protect the BES - then the software is SaaS. However, if the software is directly connected to an electronic relay in a substation, which itself directly controls the circuit breaker, then it may be a BCS. This is because it affects the BES within 15 minutes, without requiring human intervention.

When I mentioned earlier that the purpose of the CIP standards is to protect BES Cyber Systems, I omitted the fact that the CIP standards also protect BES Cyber System Information (BCSI)[i]. Since there are far more than ten times as many requirements (and “requirement parts”, which is NERC’s word for what are usually known as subrequirements) that apply to BCS than there are for BCSI, the main emphasis is almost always on protecting BCS. A BCS consists of one or more hardware devices and the software that runs on those devices.

However, when it comes to the cloud, hardware essentially disappears. Of course, everyone knows that the software in the cloud runs on hardware, but the big advantage of using the cloud is that the end user doesn’t need to ensure protection of the hardware – just the software. There are two basic ways to utilize software installed in the cloud.

1. One way is to install and manage the software yourself (although in many cases the OS and other supporting software products are managed by the CSP). This is how BES Cyber Systems can be utilized. Currently, only a small number of low impact BCS are installed in the cloud, since the CIP standards don’t pose any impediment to installing low BCS in the cloud.

However, it is close to impossible to “legally” install medium and high impact BCS in the cloud. This isn’t because the current CIP requirements directly forbid this happening, but because it would be literally impossible for the CSP (by which I mean a “platform” CSP) to provide the required evidence of compliance with requirements like CIP-007 R2 patch management and CIP-010 R1 configuration management. Any NERC entity that utilizes cloud-based BCS but can’t provide the required evidence for compliance with all current CIP requirements that apply to BCS is likely to be hit with a lot of violations. This is almost certainly why I have never heard of a NERC entity that has knowingly installed medium or high impact BCS in the cloud.

2. On the other hand, since SaaS consists of just software and is completely abstracted from the hardware it runs on, the only CIP requirements that apply to a NERC entity’s use of SaaS are the three requirements (along with a total of seven requirement parts) that apply to BCSI: CIP-004-7 R6, CIP-011-3 R1 and CIP-011-3 R2. NERC entities with high or medium impact BCS are free to use SaaS all they want, but if the information they store and/or utilize meets the BCSI definition, they must comply with those requirements.

In the previous versions of CIP-004, several requirement parts were worded so that they effectively prohibited usage in the cloud. This wasn’t intentional, of course. However, until recently the NERC community and FERC considered the cloud to be too untrustworthy to be considered as a home for anything having to do with the power grid. Therefore, it was never even considered to be an option for systems subject to CIP compliance. CIP-004-6 (the version of CIP-004 that was in effect until January 1, 2024) didn’t make any provision for encrypting BCSI at rest, since all storage locations for physical or electronic BCSI were assumed to be under the direct control of the NERC entity. For this reason, anyone with physical or logical access to the server(s) where BCSI was stored was considered to have access to the BCSI itself, whether encrypted or not.

Fortunately, this attitude was changing by 2018, when a new NERC Standards Drafting Team was constituted to fix the wording problems in CIP-004. The team was tasked with making it clear that encryption of BCSI makes it inaccessible to anyone without access to the decryption key, even if they have physical and/or electronic access to the server(s) where the BCSI is stored.

Besides making changes to CIP-004, the drafting team also changed CIP-011 R1, which requires that NERC entities with medium and/or high impact BCS develop an Information Protection Program for all BCSI, no matter where it resides or is used. The changes (to the Methods section of R1.2) made it clear that encryption – along with some other methods of data obfuscation – renders BCSI unusable to anyone that does not have access to the decryption key. Encryption wasn’t considered to be a protection in previous CIP versions; in fact, encryption was never even mentioned in the CIP standards until CIP-011-3 came into effect (along with CIP-004-7) on January 1, 2024.

In 2018, I thought that the changes needed to fix the BCSI-in-the-cloud problem would be extremely complicated. However, the SDT came up with an ingenious solution that involved minimal changes, including:

1.      Modifying the existing CIP-004 requirements to remove the language that effectively prohibited cloud storage of BCSI;

2.      Adding a new requirement CIP-004-7 R6 that introduced a concept called “provisioned access”. R6 states that provisioned access occurs when “…an individual has both the ability to obtain and use BCSI. Provisioned access is to be considered the result of the specific actions taken to provide an individual(s) the means to access BCSI (e.g., may include physical keys or access cards, user accounts and associated rights and privileges, encryption keys).” (my emphasis added)

3.      In other words, someone with physical or electronic access to a server that contains BCSI, who does not also have access to the decryption key, does not have provisioned access to the BCSI. By the same token, someone with access to both the server that stores the BCSI and the key does have provisioned access even if the BCSI is still encrypted. This is because the person could decrypt the data if they wanted to.

4.      Modifying the wording of CIP-011-3 R1.1 and R1.2 to separate identification of BCSI (in R1.1) from protection of BCSI (in R1.2).

5.      Modifying the Methods section of R1.2 to add a new category of BCSI called “off-premise BCSI”, and to make it clear that encryption is an option for protecting that new category (this was also the first time that encryption was mentioned in the CIP standards, as well as the first time that use of the cloud for any purpose was mentioned).  

The two revised standards, CIP-004-7 and CIP-011-3, including the changes described above, came into effect on January 1, 2024. Since the reason for making these changes was to enable storage and use of BCSI in the cloud, and since use by SaaS applications was the primary intended use for BCSI in the cloud, I and some others in the NERC community thought that NERC entities would be happy that they could finally move computationally intensive data analysis tasks, that required some use of BCSI, to the cloud.

Even more importantly, I thought that vendors of the software that enables those tasks would be even happier. After all, instead of having to individually support each of their customers using their software with on-premises hardware, they could become a SaaS provider. Thus, they would just need to maintain one big instance of the software[ii] for all their customers.

However, what happened was quite different from what I expected: After being told for years that storing or using BCSI in the cloud was verboten, few NERC entities were ready to take the leap to using BCSI in a SaaS application – absent strong encouragement from NERC and/or their own Region(s). But that encouragement, strong or otherwise, was as far as I know totally absent. For example, even though some previous changes to the CIP standards have been accompanied by multiple NERC webinars and presentations at Regional Entity meetings, I have yet to hear of one of these happening that dealt with the two revised standards that came into effect on 1/1/2024.

But there is another explanation for why NERC entities have been reluctant to start using SaaS that requires use of BCSI: Late in 2023, some NERC Regional Entity staff members realized that there was a potential “showstopper” problem regarding the wording of CIP-004-7 R6. I described that problem in detail in this post in January 2024, but here is a quick summary:

1.      BCSI must be encrypted from the moment it is transmitted to the cloud. It needs to remain encrypted through when it is stored in the cloud and utilized in a SaaS application.

2.      Few SaaS applications can make use of encrypted data. Therefore, some person who is an employee or contractor of the SaaS provider will need to decrypt the BCSI and “feed it in” to the application. That person will need to have access to both the BCSI and the decryption key. At first glance, that appears to meet the “definition” of provisioned access included in the first section of R6: “..an individual has both the ability to obtain and use BCSI.”

3.      Requirement Part CIP-004-7 R6.1 makes it clear that provisioned access must be authorized by the NERC entity. Therefore, if Staff Member Y of the SaaS provider needs to feed BCSI from Electric Utility ABC into the application, ABC will have to authorize the provider to provision Y with access to their BCSI. Similarly, if Utility 123 needs to have their BCSI fed into the same SaaS application, they will also need to authorize provisioning of the staff member that does this, even if that staff member is also Y.

4.      Since SaaS is used day and night and since staff members get sick, take vacations and are transferred, there will always need to be multiple people with provisioned access to the BCSI of each NERC entity customer. If the provider has 100 NERC entity customers and at any time there are six staff members who need access to each customer’s BCSI (corresponding to three 8-hour weekday shifts and three 8-hour weekend shifts), that means there will need to be up to 600 individuals with provisioned access to BCSI at any one time.

5.      For each of these individuals, the supplier will need to provide evidence to each NERC entity customer that they are in continual compliance with CIP-004-7 Requirement R6 Parts R6.1, R6.2 and R6.3. Needless to say, this will require a huge amount of paperwork on the part of the SaaS provider; many (if not most) SaaS providers will be unwilling to undertake this responsibility. Therefore, it isn’t surprising that no NERC entity I know of decided to take the leap to using SaaS with BCSI after 1/1/2024.[iii]

Fortunately, help is on the way with this problem. A group I am (a small) part of, the informal NERC Cloud Technology Advisory Group (CTAG), discussed the above problem at length and realized that the BCSI access required for a SaaS provider staff member does not need to be explicitly provisioned; therefore, Requirement CIP-004-7 R6 does not apply in the use case described (which is fundamental to almost all SaaS use, of course).

However, knowing this might not in itself be helpful. This might be simply filed away under the “useless, but  still nice to know” category, if the only way it could be used to change the situation would be by changing one of the NERC CIP requirements (presumably CIP-004-7 R6). I say this because changing an existing CIP requirement could very easily require three years or even longer, starting with the day the change is first proposed to the NERC Standards Committee (in a Standards Authorization Request or SAR).

However, no change to a NERC CIP requirement (or definition) is needed. The CIP auditors (who are all staff members of one of the six NERC Regional Entities) occasionally produce “CMEP[iv] Practice Guides” that provide direction to audit staff on “approaches to carry out compliance monitoring and enforcement activities.” Our group turned over our findings – which are in part based on collateral NERC documents – to the committee that is in charge of drafting new CMEP Practice Guides. While it is not guaranteed that will be the outcome, we are optimistic they will develop a new Guide for BCSI that will make a recommendation on this issue (the last Guide for BCSI was published in 2019; it is based on the previous version of CIP-004, so it is now obsolete).  

Of course, it may be six months (or even longer) before a new CMEP Practice Guide is published. Does this mean that software vendors with current CIP customers should wait six months before they start offering SaaS services to those customers? Or that current SaaS providers need to wait six months before they start approaching NERC entities that are subject to CIP compliance about using their applications? Or that NERC entities that would like to move certain data intensive applications to the cloud should wait six months before they even start talking to SaaS providers about doing that?

In all these cases, the answer is no. The important thing to remember is that the concerns that were raised in late 2023 about provisioned access being necessary for SaaS application provider staff members were in retrospect overblown. The “default position” should always have been that provisioned access was not necessary.

Tom 6/19: Kevin also pointed out a recent development that I didn't know about. Without going into a lot of detail, it seems the NERC Regional auditors, who would need to prepare a CMEP Practice Guide, don't think a new one is needed in this case. In effect, they say that statements in a document that NERC endorsed as Implementation Guidance for CIP-004-7 Requirement R6 and CIP-011-3 Requirements R1 in December 2023 sufficiently undercut the concerns that were being raised about the wording regarding "provisioned access" in the first part of CIP-004-7 Requirement R6. 

Moreover, they said the question is more properly dealt with by the NERC Responsible Entity in the Information Protection Program that is mandated by CIP-011. In other words, they agree that the problem I described above shouldn't be a concern in most cases, but they also don't think a CMEP Practice Guide is required to explain this. Auditors don't like to waste time, especially when it's their own!

Thus, the new CMEP Practice Guide will do nothing more than restore the status quo before the concerns arose. It does not in any way amount to a change in how CIP requirements are normally interpreted. After all, had the concerns been correct, it would effectively have meant that use of BCSI with SaaS was still completely off limits for NERC entities with high and/or medium impact BES Cyber Systems, and the work of the drafting team from 2019 to 2023 was all for naught. 

My blog is more popular than ever, but I need more than popularity to keep it going. I’ve often been told that I should either accept advertising or charge a subscription fee or both. However, neither of those options appeals to me. It would be great if everyone who appreciates my posts could donate a $20-$25 “subscription fee” once a year (of course, I welcome larger amounts as well!). Will you do that today?

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] The NERC definition of BCSI is “Information about the BES Cyber System that could be used to gain unauthorized access or pose a security threat to the BES Cyber System. BES Cyber System Information does not include individual pieces of information that by themselves do not pose a threat or could not be used to allow unauthorized access to BES Cyber Systems, such as, but not limited to, device names, individual IP addresses without context, ESP names, or policy statements. Examples of BES Cyber System Information may include, but are not limited to, security procedures or security information about BES Cyber Systems, Physical Access Control Systems, and Electronic Access Control or Monitoring Systems that is not publicly available and could be used to allow unauthorized access or unauthorized distribution; collections of network addresses; and network topology of the BES Cyber System.”

[ii] Of course, in practice I’m sure that SaaS providers maintain multiple instances of their software in the cloud, for various reasons. But that is still a big improvement over maintaining a single instance for each customer, as they did before they moved to SaaS delivery of their product.

However, I’m deliberately overlooking the “multi-tenant problem”. This arises when a standalone enterprise software product that includes its own database is moved without modification to the cloud – with the result that users from different organizations and even different countries might end up sharing a single database. Even though there are protections between different users in the database, they are not likely to be equivalent to the protections that exist when each organization in each country operates its own database instance. I hope to address this topic soon.

[iii] While this sentence is accurate, it’s misleading. The fact is, there are at least one or two SaaS applications that NERC entities have been using to document CIP-010 R1 (configuration management) compliance since 2016 or 2017; of course, configuration data on BES Cyber Systems will almost certainly include BCSI. It is likely those NERC entities are still using those SaaS applications today.

[iv] CMEP stands for Compliance Monitoring and Enforcement Program.