I’ve written at least a few times about the difference between
SaaS (software-as-a-service, although “software that runs in the cloud” is a
more accurate description) and BCS (BES Cyber Systems) that are deployed in the
cloud (for those unfamiliar with NERC CIP, BCS are the systems that the 13 current
NERC CIP standards, CIP-002 through CIP-014, are there to protect. These are
the systems whose loss r compromise would impact the Bulk Electric System (BES)
within 15 minutes, although the impact wilusually be instantaneo.
Note from Tom 6/19: Kevin Perry, former Chief CIP Auditor for the SPP Regional Entity and co-chair of the team that drafted CIP versions 2-4, pointed out a couple of problems with this post as I wrote it yesterday. I've discussed them in italics below, although I decided not to change what I wrote yesterday - just point out where I was wrong.
Both SaaS and cloud BCS consist of software running in the
cloud. They both provide advice and/or monitoring data to their operators. However,
since the loss of a BCS, whether deployed in the cloud or not, will by
definition impact the BES within 15 minutes, this means a BCS can provide more
than advice; it can provide control or real-time monitoring. In
other words, a BCS always has some sort of connection with a device (like an
electronic relay that controls a circuit breaker) that impacts or monitors the power
grid. Most importantly, the BCS needs to directly control, or monitor
the output of, the grid-connected device. If a system’s impact on the grid is dependent
on a human being taking some action first, then it’s not a BCS.
Note: Kevin Perry pointed out that the converse of this last statement isn't true. That is, if a system's (negative) impact on the grid is dependent on a human being not taking a particular action (and that negative impact occurs within 15 minutes), then it's a BCS. He used the example of a SCADA system that notifies an operator when there's a problem that needs attention, so the operator can take actions to fix the problem. If that system is compromised and doesn't notify the operator of a problem - and that lack of notification leads to a negative impact on the BES within 15 minutes - then clearly the system can have a 15-minute BES impact and is therefore a BES Cyber Asset and also part of a BES Cyber System.
This means that the same software product could be deployed
in the cloud in two quite different ways. Let’s use the example of software
that monitors power flows and can detect a dangerous anomaly in real time. If the
software simply sets off an alarm and warns the operator of the anomaly – on the
assumption that the operator will perform the steps necessary to protect the BES
- then the software is SaaS. However, if the software is directly connected to
an electronic relay in a substation, which itself directly controls the circuit
breaker, then it may be a BCS. This is because it affects the BES within 15
minutes, without requiring human intervention.
When I mentioned earlier that the purpose of the CIP
standards is to protect BES Cyber Systems, I omitted the fact that the CIP standards
also protect BES Cyber System Information (BCSI)[i].
Since there are far more than ten times as many requirements (and “requirement
parts”, which is NERC’s word for what are usually known as subrequirements) that
apply to BCS than there are for BCSI, the main emphasis is almost always on
protecting BCS. A BCS consists of one or more hardware devices and the software
that runs on those devices.
However, when it comes to the cloud, hardware essentially
disappears. Of course, everyone knows that the software in the cloud runs on
hardware, but the big advantage of using the cloud is that the end user doesn’t
need to ensure protection of the hardware – just the software. There are two
basic ways to utilize software installed in the cloud.
1. One way is to install and manage the software
yourself (although in many cases the OS and other supporting software products
are managed by the CSP). This is how BES Cyber Systems can be utilized.
Currently, only a small number of low impact BCS are installed in the cloud,
since the CIP standards don’t
pose any impediment to installing low BCS in the cloud.
However, it is close to impossible to “legally” install medium
and high impact BCS in the cloud. This isn’t because the current CIP
requirements directly forbid this happening, but because it would be literally
impossible for the CSP (by which I mean a “platform” CSP) to provide the required
evidence of compliance with requirements like CIP-007 R2 patch management and
CIP-010 R1 configuration management. Any NERC entity that utilizes cloud-based BCS
but can’t provide the required evidence for compliance with all current CIP
requirements that apply to BCS is likely to be hit with a lot of violations.
This is almost certainly why I have never heard of a NERC entity that has knowingly
installed medium or high impact BCS in the cloud.
2. On the other hand, since SaaS consists of just
software and is completely abstracted from the hardware it runs on, the only
CIP requirements that apply to a NERC entity’s use of SaaS are the three
requirements (along with a total of seven requirement parts) that apply to BCSI:
CIP-004-7 R6, CIP-011-3 R1 and CIP-011-3 R2. NERC entities with high or medium
impact BCS are free to use SaaS all they want, but if the information they
store and/or utilize meets the BCSI definition, they must comply with those
requirements.
In the previous versions of CIP-004, several requirement
parts were worded so that they effectively prohibited usage in the cloud. This
wasn’t intentional, of course. However, until recently the NERC community and
FERC considered the cloud to be too untrustworthy to be considered as a home
for anything having to do with the power grid. Therefore, it was never even
considered to be an option for systems subject to CIP compliance. CIP-004-6
(the version of CIP-004 that was in effect until January 1, 2024) didn’t make
any provision for encrypting BCSI at rest, since all storage locations for physical
or electronic BCSI were assumed to be under the direct control of the NERC
entity. For this reason, anyone with physical or logical access to the
server(s) where BCSI was stored was considered to have access to the BCSI
itself, whether encrypted or not.
Fortunately, this attitude was changing by 2018, when a new NERC
Standards Drafting Team was constituted to fix the wording problems in CIP-004.
The team was tasked with making it clear that encryption of BCSI makes it
inaccessible to anyone without access to the decryption key, even if they have
physical and/or electronic access to the server(s) where the BCSI is stored.
Besides making changes to CIP-004, the drafting team also changed
CIP-011 R1, which requires that NERC entities with medium and/or high impact
BCS develop an Information Protection Program for all BCSI, no matter where it
resides or is used. The changes (to the Methods section of R1.2) made it clear
that encryption – along with some other methods of data obfuscation – renders BCSI
unusable to anyone that does not have access to the decryption key. Encryption wasn’t
considered to be a protection in previous CIP versions; in fact, encryption was
never even mentioned in the CIP standards until CIP-011-3 came into effect (along
with CIP-004-7) on January 1, 2024.
In 2018, I thought that the changes needed to fix the
BCSI-in-the-cloud problem would be extremely complicated. However, the SDT came
up with an ingenious solution that involved minimal changes, including:
1.
Modifying the existing CIP-004 requirements to
remove the language that effectively prohibited cloud storage of BCSI;
2.
Adding a new requirement CIP-004-7 R6 that introduced
a concept called “provisioned access”. R6 states that provisioned access occurs
when “…an individual has both the ability to obtain and use BCSI.
Provisioned access is to be considered the result of the specific actions taken
to provide an individual(s) the means to access BCSI (e.g., may include
physical keys or access cards, user accounts and associated rights and
privileges, encryption keys).” (my emphasis added)
3.
In other words, someone with physical or
electronic access to a server that contains BCSI, who does not also have access
to the decryption key, does not have provisioned access to the BCSI. By the
same token, someone with access to both the server that stores the BCSI and the
key does have provisioned access even if the BCSI is still encrypted. This is
because the person could decrypt the data if they wanted to.
4.
Modifying the wording of CIP-011-3 R1.1 and R1.2
to separate identification of BCSI (in R1.1) from protection of BCSI (in R1.2).
5.
Modifying the Methods section of R1.2 to add a
new category of BCSI called “off-premise BCSI”, and to make it clear that encryption
is an option for protecting that new category (this was also the first time
that encryption was mentioned in the CIP standards, as well as the first time
that use of the cloud for any purpose was mentioned).
The two revised standards, CIP-004-7 and CIP-011-3, including
the changes described above, came into effect on January 1, 2024. Since the
reason for making these changes was to enable storage and use of BCSI in the
cloud, and since use by SaaS applications was the primary intended use for BCSI
in the cloud, I and some others in the NERC community thought that NERC
entities would be happy that they could finally move computationally intensive
data analysis tasks, that required some use of BCSI, to the cloud.
Even more importantly, I thought that vendors of the
software that enables those tasks would be even happier. After all, instead of
having to individually support each of their customers using their software with
on-premises hardware, they could become a SaaS provider. Thus, they would just
need to maintain one big instance of the software[ii]
for all their customers.
However, what happened was quite different from what I
expected: After being told for years that storing or using BCSI in the cloud
was verboten, few NERC entities were ready to take the leap to using BCSI
in a SaaS application – absent strong encouragement from NERC and/or their own
Region(s). But that encouragement, strong or otherwise, was as far as I know totally
absent. For example, even though some previous changes to the CIP standards
have been accompanied by multiple NERC webinars and presentations at Regional
Entity meetings, I have yet to hear of one of these happening that dealt with the
two revised standards that came into effect on 1/1/2024.
But there is another explanation for why NERC entities have
been reluctant to start using SaaS that requires use of BCSI: Late in 2023, some
NERC Regional Entity staff members realized that there was a potential “showstopper”
problem regarding the wording of CIP-004-7 R6. I described that problem in
detail in this
post in January 2024, but here is a quick summary:
1.
BCSI must be encrypted from the moment it is
transmitted to the cloud. It needs to remain encrypted through when it is
stored in the cloud and utilized in a SaaS application.
2.
Few SaaS applications can make use of encrypted
data. Therefore, some person who is an employee or contractor of the SaaS
provider will need to decrypt the BCSI and “feed it in” to the application.
That person will need to have access to both the BCSI and the decryption key. At
first glance, that appears to meet the “definition” of provisioned access
included in the first section of R6: “..an individual has both the ability to
obtain and use BCSI.”
3.
Requirement Part CIP-004-7 R6.1 makes it clear
that provisioned access must be authorized by the NERC entity. Therefore, if Staff
Member Y of the SaaS provider needs to feed BCSI from Electric Utility ABC into
the application, ABC will have to authorize the provider to provision Y with access
to their BCSI. Similarly, if Utility 123 needs to have their BCSI fed into the same
SaaS application, they will also need to authorize provisioning of the staff
member that does this, even if that staff member is also Y.
4.
Since SaaS is used day and night and since staff
members get sick, take vacations and are transferred, there will always need to
be multiple people with provisioned access to the BCSI of each NERC entity
customer. If the provider has 100 NERC entity customers and at any time there
are six staff members who need access to each customer’s BCSI (corresponding to
three 8-hour weekday shifts and three 8-hour weekend shifts), that means there
will need to be up to 600 individuals with provisioned access to BCSI at any
one time.
5.
For each of these individuals, the supplier will
need to provide evidence to each NERC entity customer that they are in
continual compliance with CIP-004-7 Requirement R6 Parts R6.1, R6.2 and R6.3. Needless
to say, this will require a huge amount of paperwork on the part of the SaaS
provider; many (if not most) SaaS providers will be unwilling to undertake this
responsibility. Therefore, it isn’t surprising that no NERC entity I know of decided
to take the leap to using SaaS with BCSI after 1/1/2024.[iii]
Fortunately,
help is on the way with this problem. A group I am (a small) part of, the informal
NERC Cloud Technology Advisory Group (CTAG), discussed the above problem at
length and realized that the BCSI access required for a SaaS provider staff
member does not need to be explicitly provisioned; therefore, Requirement
CIP-004-7 R6 does not apply in the use case described (which is fundamental to almost
all SaaS use, of course).
However,
knowing this might not in itself be helpful. This might be simply filed away under
the “useless, but still nice to know” category, if the only way it could be used to
change the situation would be by changing one of the NERC CIP requirements
(presumably CIP-004-7 R6). I say this because changing an existing CIP requirement
could very easily require three years or even longer, starting with the day the
change is first proposed to the NERC Standards Committee (in a Standards
Authorization Request or SAR).
However,
no change to a NERC CIP requirement (or definition) is needed. The CIP auditors
(who are all staff members of one of the six NERC Regional Entities) occasionally
produce “CMEP[iv]
Practice Guides” that provide direction to audit staff on “approaches to carry
out compliance monitoring and enforcement activities.” Our group turned over
our findings – which are in part based on collateral NERC documents – to the
committee that is in charge of drafting new CMEP Practice Guides. While it is
not guaranteed that will be the outcome, we are optimistic they will develop a
new Guide for BCSI that will make a recommendation on this issue (the last Guide
for BCSI was published in 2019; it is based on the previous version of CIP-004,
so it is now obsolete).
Of course,
it may be six months (or even longer) before a new CMEP Practice Guide is
published. Does this mean that software vendors with current CIP customers should
wait six months before they start offering SaaS services to those customers? Or
that current SaaS providers need to wait six months before they start
approaching NERC entities that are subject to CIP compliance about using their
applications? Or that NERC entities that would like to move certain data
intensive applications to the cloud should wait six months before they even
start talking to SaaS providers about doing that?
In all these cases, the answer is no. The important thing to remember is that the concerns that were raised in late 2023 about provisioned access being necessary for SaaS application provider staff members were in retrospect overblown. The “default position” should always have been that provisioned access was not necessary.
Tom 6/19: Kevin also pointed out a recent development that I didn't know about. Without going into a lot of detail, it seems the NERC Regional auditors, who would need to prepare a CMEP Practice Guide, don't think a new one is needed in this case. In effect, they say that statements in a document that NERC endorsed as Implementation Guidance for CIP-004-7 Requirement R6 and CIP-011-3 Requirements R1 in December 2023 sufficiently undercut the concerns that were being raised about the wording regarding "provisioned access" in the first part of CIP-004-7 Requirement R6.
Moreover, they said the question is more properly dealt with by the NERC Responsible Entity in the Information Protection Program that is mandated by CIP-011. In other words, they agree that the problem I described above shouldn't be a concern in most cases, but they also don't think a CMEP Practice Guide is required to explain this. Auditors don't like to waste time, especially when it's their own!
Thus,
the new CMEP Practice Guide will do nothing more than restore the status quo
before the concerns arose. It does not in any way amount to a change in how CIP
requirements are normally interpreted. After all, had the concerns been
correct, it would effectively have meant that use of BCSI with SaaS was still completely
off limits for NERC entities with high and/or medium impact BES Cyber Systems,
and the work of the drafting team from 2019 to 2023 was all for naught.
My blog is more popular than
ever, but I need more than popularity to keep it going. I’ve often been told that
I should either accept advertising or charge a subscription fee or both.
However, neither of those options appeals to me. It would be great if everyone
who appreciates my posts could donate a $20-$25 “subscription fee” once a year (of course, I
welcome larger amounts as well!). Will you do that today?
If you would like to comment on
what you have read here, I would love to hear from you. Please email me
at tom@tomalrich.com.
[i] The
NERC definition of BCSI is “Information about the BES Cyber System that could
be used to gain unauthorized access or pose a security threat to the BES Cyber
System. BES Cyber System Information does not include individual pieces of
information that by themselves do not pose a threat or could not be used to
allow unauthorized access to BES Cyber Systems, such as, but not limited to,
device names, individual IP addresses without context, ESP names, or policy
statements. Examples of BES Cyber System Information may include, but are not limited
to, security procedures or security information about BES Cyber Systems,
Physical Access Control Systems, and Electronic Access Control or Monitoring
Systems that is not publicly available and could be used to allow unauthorized
access or unauthorized distribution; collections of network addresses; and
network topology of the BES Cyber System.”
[ii] Of
course, in practice I’m sure that SaaS providers maintain multiple instances of
their software in the cloud, for various reasons. But that is still a big
improvement over maintaining a single instance for each customer, as they did
before they moved to SaaS delivery of their product.
However, I’m deliberately overlooking the “multi-tenant
problem”. This arises when a standalone enterprise software product that
includes its own database is moved without modification to the cloud – with the
result that users from different organizations and even different countries
might end up sharing a single database. Even though there are protections between
different users in the database, they are not likely to be equivalent to the
protections that exist when each organization in each country operates its own
database instance. I hope to address this topic soon.
[iii] While
this sentence is accurate, it’s misleading. The fact is, there are at least one
or two SaaS applications that NERC entities have been using to document CIP-010
R1 (configuration management) compliance since 2016 or 2017; of course, configuration
data on BES Cyber Systems will almost certainly include BCSI. It is likely
those NERC entities are still using those SaaS applications today.
[iv]
CMEP stands for Compliance Monitoring and Enforcement Program.
