How will NERC entities assess their CSPs?

I participated in the third of the four panels in NERC’s successful Cloud Technical Conference on November 1. Two of the three questions that all panel members were asked were:

·        How does the shared responsibility model in cloud computing reshape the way utilities manage accountability for security and compliance, and what best practices can help clearly define these responsibilities between utilities and cloud service providers?

·        How can utilities effectively manage and verify that cloud providers are fulfilling their security responsibilities, and what role do audits and third-party assessments play in this process?

Of course, these are both variations of the question, “How will NERC entities assess their CSPs, once they are able to fully utilize their services?” While I am satisfied with the separate answers I provided to these two questions during the conference, I now realize it is much better to answer the unified question.

First, I want to point out that in this post I’m treating the term “cloud service provider” (CSP) to mean two types of organizations: “Platform” CSPs like AWS, Azure and Google Cloud, and SaaS (“software-as-a-service”) providers, meaning software providers that offer subscriptions for access to their software in the cloud. Usually, I distinguish between the two, as I did in this post.

In the new or revised CIP standards that the NERC Project 2023-09 Risk Management for Third-Party Cloud Services Standards Drafting Team will start drafting in 2025, I think the CSPs should be assessed in two ways:

1.      While the CSPs are not subject to the jurisdiction of either NERC or FERC directly, there needs to be an annual “audit” of the CSPs. It should be conducted by the NERC ERO; the CSPs will never agree to be audited by every NERC entity that is a customer. Kevin Perry, former Chief CIP Auditor for SPP Regional Entity, suggested the Regional auditors could conduct a joint audit (they perform these all the time).

a.      The audit will have two parts. First, there should be an assessment of the audit report from either the CSP’s ISO 27001 certification or their FedRAMP authorization. This assessment does not need to cover the entire report, but only certain topics that the current “cloud” Standards Drafting Team (i.e., the team that is meeting now) has decided should be a focus of the assessment. These might include topics such as background checks for personnel, incident response plan, internal network security monitoring (INSM), etc. The NERC assessors will look for adverse findings in any of these areas and note them.

b.      For the second part of the audit, the current SDT should identify cloud risks that are not addressed by the CSP’s authorizations or certifications. The NERC assessors will need to interview CSP personnel regarding the degree to which the CSP has mitigated each of these risks. They might include:

                                                    i.     Multitenant databases in SaaS products. This isn’t itself a risk, since a SaaS provider can never provide each customer with their own instance of the product without completely breaking their business model. On the other hand, NERC entities shouldn’t be sharing a database with organizations from Russia and Iran. The SDT will need to debate this issue and come up with reasonable measures that mitigate risk without putting the SaaS provider out of business.[i]

                                                   ii.     Whether the CSP is properly training their customers in how to manage the security controls for their own cloud environment.

                                                  iii.     How well the platform CSP vets third parties that broker access to services in their cloud.

c.      The ERO auditors will prepare a report on their assessment of each platform CSP and SaaS provider and make these available on request by NERC entities that are customers of those services, as well as to the CSP itself.

d.      NERC will not “certify” the CSPs. Their job is only to assess particular risks to which the CSP is subject, whether these risks are addressed in a certification or whether they are subject to the separate risk review described in item b above.

I want to point out that there is currently no provision in the NERC Rules of Procedure for NERC to conduct assessments of third parties that are not subject to NERC’s jurisdiction – which is the case with CSPs, of course. If what I have just described is to come to pass, there will probably need to be RoP changes; however, no Standards Drafting Team is currently empowered to make those.

This is one of the many unknowns that will impact the likely implementation date for the revised CIP Reliability Standards. In a recent post, I stated that I think the most likely date is Q2 of 2031; I also pointed out that if a change to the Rules of Procedure is required, even that date might be too optimistic. Guess what? I now believe an RoP change (or at least some sort of change to NERC rules, which the SDT has no authority to change on their own) is required. Ergo, Q2 2031 is an optimistic estimate; it would be safer to use a later one, although I have no idea what that would be.

This gets me back to the conclusion of the post I just linked: Asking NERC entities to wait until new or revised CIP standards are in place to make full (and secure) use of the cloud isn’t workable. There are partial measures that can be taken on an interim basis to enable at least some cloud use by NERC entities with high or medium impact BES environments. I believe it’s time to make some decisions on what needs to be done in say the next two years, and how to do it.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] I can see this debate alone taking six months; I’m sure there are a few other topics that could be equally contentious. That is why I am now anticipating that new and/or revised CIP standards that address cloud issues won’t be in place until 2031.

Reply to Boris Polishuk, Cybellum – November 5, 2024

 

From Tom Alrich: Boris is Chief Architect of Cybellum, the Israeli company that is one of the leading service providers in the SBOM world; he is a member of the OWASP SBOM Forum. He wrote me in response to the white paper that Tony Turner and I, who are co-leaders (with Jeff Williams) of the SBOM Forum, recently developed, titled “purl needs to identify proprietary software”. The paper describes the project we are proposing to expand the scope of the “purl” software identifier to address proprietary software as well as closed source software. We hope to be able to start that project soon.

To summarize why this project is important, the software security “industry” is plagued by the problem that software products all have different names in different contexts. In order to learn about vulnerabilities (usually designated with a “CVE number” like CVE-2021-44228) that affect a software product they rely on, an end user organization needs to locate the product in a vulnerability database. To do that, they need to learn (or be able to construct from information they already have) the identifier for the product in the database.

The most widely used vulnerability database in the world, the US National Vulnerability Database (NVD), utilizes exclusively the “CPE” identifier, which stands for “Common Platform Enumeration”. CPE has been in use, in multiple versions, for around twenty years. Unfortunately it is an unreliable identifier, as the SBOM Forum described on pages 4-6 of this 2022 white paper. Even more unfortunately, serious problems in the NVD since February 2024 have resulted in over two thirds of the vulnerabilities reported this year being totally invisible to a search using a CPE name. Clearly, an alternative software identifier is needed.

Below, I have posted each paragraph from Boris’ letter in bold roman type, with my response in italics.

Hi Tom,
Thanks for sharing your thoughts on using SWID to generate PURLs for proprietary software. We've considered this approach, but we have some reservations about its feasibility at this time. Here's our reasoning:

• Decentralized Ecosystem: Proprietary software exists in a decentralized ecosystem with no central authority to enforce naming standards or manage a unified repository. This increases the risk of duplicate or conflicting PURLs, even when generated through SWID.

I agree that proprietary software exists in a decentralized ecosystem. However, I think you’ll agree that the same can be said for open source software. Despite that fact, not only is purl by far the leading software identifier used in open source vulnerability databases today, it is almost the only one. I am sure there have been duplicate purls for OSS, mainly due to somebody making a mistake. But I don’t know of any case where a purl was correctly generated, yet still exactly duplicated a purl for a different product. I also don’t know of any case where two different purls were created for the same product (this happens often with CPEs). Do you know of any such cases?

·        Limited SWID Adoption: While SWID offers a potential solution, its adoption has been limited. In our experience, many organizations are unfamiliar with SWID or its implementation. Relying on SWID for proprietary software identification might face similar obstacles as CPE, hindering widespread adoption and effectiveness.

I agree that SWID has had limited adoption. You probably know that NIST developed the SWID specification (and got it approved as ISO/IEC 19770-2) to be a replacement for the CPE identifier in the NVD. For a few years, it enjoyed modest success; for example, Microsoft included a SWID tag in the binaries for all their new products or versions for at least a couple of years. However, SWID never reached the point where NIST felt comfortable dropping CPE altogether and only using SWID in the NVD. They have more recently acknowledged they will never make that change.

However, you need to understand the reason why we’re proposing that SWID tags be used to create purls for proprietary software that isn’t distributed through app stores: The supplier needs to decide the name and version string for every product they release. This information needs to be made publicly available in a single document.

We could have defined our own format for the document, but since SWID includes the fields needed for the purl (as well as many more that aren’t needed and don’t need to be filled in) and since it is already an ISO standard, we decided to use SWID. In fact, Steve Springett has created a SWID tag generator, which a software supplier (or a third party, if the supplier has not done this already) can use to create a SWID tag for one of their current or legacy products (note that the majority of the fields in Steve’s tool are optional in the purl). The suppliers won’t need to know anything about SWID to create a SWID tag.

·        PURL and CVE.org: The lack of CVE.org's full embrace of PURL as a primary identifier raises concerns about its long-term viability as the sole solution for vulnerability management.

The CVE 5.1 specification (which allows purls in a CVE report but doesn’t say anything about how they will be created or entered in the report) was only adopted by CVE.org this past spring (the SBOM Forum submitted the pull request to add purl to the CVE spec in 2022); very few CNAs are using v5.1 yet. This is mostly because including a purl in a CVE report won’t do any good now, because the NVD is at least several years away from being able to handle purl (Sonatype’s OSS Index vulnerability database supports both CVE and purl today, so it might be able to utilize purls included in CVE v5.1 records with little or no change required. We would like to test this in our proof of concept).

Given these challenges, we believe that a more comprehensive approach is needed to address the complexities of identifying proprietary software. This might involve:

• Collaboration with CVE.org: Working closely with CVE.org to establish clear guidelines and standards for both PURL and CPE, ensuring they complement each other and address the limitations of each system.

I agree this is important. In fact, it is already one of the “deliverables” of our project (which are described on pages 6-9 of the preliminary project plan that I made available last week). We plan to work with CVE.org and the CNAs (which report to CVE.org) to develop whatever rules are required for them to correctly create a purl for a product described in a CVE report, and to include the purl in the report.

Regarding CPE, the reason we’re doing this project is that, while purl is clearly a much better identifier than CPE for open source software and components – and should be adopted by the NVD for OSS as soon as possible – it doesn’t identify proprietary software products, unless they are delivered through a package manager like PyPI or Maven Central (which rarely happens). Currently, CPE is the only identifier for proprietary software, but it’s subject to the problems listed on pages 4-6 of the OWASP SBOM Forum’s 2022 white paper linked earlier.

When our project is finished and after our recommendations are implemented, we believe that purl will prove to be superior to CPE as an identifier for proprietary software, as it is now for open source software (see our 2022 white paper for why purl is in general superior to CPE). However, the 240-250,000 CVE records that now include CPE can’t simply be thrown away. That means the NVD, and the other vulnerability databases based on the NVD, will need to support both identifiers, perhaps for a very long time.

However, given that the NVD currently has a backlog of more than 19,000 CVE records that don’t include a CPE or purl – and that backlog is growing all the time – integrating purl into the NVD isn’t likely to happen anytime soon. Fortunately, there are multiple open source vulnerability databases that support purl, although I don’t think there are any that also support CPE.

·        Hybrid Model: Exploring a hybrid approach that leverages PURL's strengths for open-source components while incorporating alternative mechanisms, potentially drawing inspiration from SWID or other identification schemes, for proprietary software.

Essentially, this is what we’re proposing. Purl is currently used in almost all vulnerability databases for open source software (except the NVD and other databases based on it), but it doesn’t address proprietary software currently. SWID by itself was intended originally to be an identifier for both OSS and proprietary software products, but I don’t know any vulnerability database that supports SWID as a software identifier now, nor do I know of any other identifiers for proprietary software, other than the very flawed CPE.

We’re not saying that anybody who is happy with using CPE to identify software products needs to give it up. But we are saying that, once our recommendations are implemented, we believe purl will prove to be a superior identifier for both open source and proprietary software.

·        Industry-Wide Standards: Promoting the development of industry-wide standards for software identification to ensure interoperability and consistency across different ecosystems.

That’s what we want to do! Once purl and CPE are both able to handle open source and proprietary software, the software community will “decide” (through their choices in the marketplace) whether they want to use one or the other identifier, or both. As mentioned earlier, both will continue to be in use for many years.

We appreciate your insights and the work you're doing in the OWASP SBOM Forum. While we don't see SWID -> PURL as a viable solution for proprietary software at this point, we're open to further discussions and exploring alternative approaches.

SWID -> PURL is certainly not a viable solution for proprietary software at this point, but we want to make it one as soon as possible! However, I want to point out that the idea of using SWID tags to generate purls for proprietary software isn’t set in stone; if anyone has a better idea for expanding purl to cover proprietary software not distributed through app stores (purls for software in app stores can be easily created without using SWID tags, as described in our new white paper), please bring it up to us. We hope to start the project soon, at which point the participants will be able to determine its direction in both synchronous and asynchronous online discussions.

We welcome anyone who wants to participate and/or to contribute to the effort through directed donations to the SBOM Forum through OWASP (a 501(c)(3) nonprofit organization).

When will the “NERC CIP in the cloud” problem be solved for good? You won’t like the answer…

NERC’s 6-hour virtual Cloud Technical Conference on November 1 was quite successful. The conference included three panels of industry types (including me) discussing questions mostly posed to them in advance, followed by a discussion by members of the team that will draft changes to the CIP standards to address what I call the “CIP in the cloud” problem.

The discussions were very productive and produced some great insights. I took a lot of notes and will produce multiple posts on those insights in the coming month or so. However, I’m going to start off with a question that wasn’t discussed in the conference, but was very much hanging over it: When will new or revised CIP standards be in place, so that full, but secure, use of the cloud by NERC entities with BEES assets will finally be possible?

This isn’t an academic question at all. As multiple panelists pointed out, previously on-premises software products and security services are moving to the cloud all the time. In some cases, they retain an on-premises option, with the caveat that the most innovative changes will only go into the cloud. In other cases, the vendor is making a clean break with on-premises systems, leaving NERC entities with high- or medium-impact BES environments with no other choice than to find a totally on-premises replacement. And as Peter Brown of Invenergy pointed out (in the conference as well as an earlier webinar sponsored by the informal NERC Cloud Technology Advisory Group and SANS, which I wrote about in this post), those replacements are inevitably more expensive and offer less functionality.

In January, I wrote a post that examined this question. I concluded by saying:

So, if we get lucky and there are no major glitches along this path, you can expect to be “allowed” to deploy medium and high impact BCS, EACMS and PACS in the cloud by the end of 2029. Mark your calendar!

Of course, the end of 2029 sounds like a long time to have to wait, especially with security services and software already abandoning their on-premises options. Do I still think the industry will have to wait five years for the cloud to be completely “legal”? I have good news and bad news, but finally some good news, for you:

·        The first good news is I no longer think the end of 2029 is the likely date by which cloud-based systems and services for systems to which the CIP standards apply will be fully “legalized”.

·        The bad news is I think it will probably be later than 2029.

·        However, the second good news is that, given how this problem is affecting more and more NERC entities all the time, it’s unlikely there won’t be at least a partial solution to this problem before 2029 – although don’t ask me what form that solution will take. This is very much uncharted ground.

Here's a short summary of my timeline and the reason for my changes:

1.      I had thought the new Standards Drafting Team (SDT) would start their drafting work when they convened in July. However, it turns out they are now working on revising their Standards Authorization Request (SAR). They will finish that by the end of this year and will submit it to the NERC Standards Committee for approval. That approval is likely to be quickly granted, so the team will probably start drafting in January 2025, not last July as I had anticipated.

2.      There are some huge issues that will need to be discussed when the SDT starts drafting. I attended a lot of the meetings of the CSO706 SDT that drafted CIP version 5. V5 completely rewrote the CIP standards and definitions that had been put in place with CIP version 1. Even though there were a lot of fundamental questions discussed in those meetings, I also know the SDT had a good idea of what they needed to do when they started drafting v5 in early 2011. Even then, developing the first draft took a year and a half (see the January post linked above). The “cloud” SDT might take that long or even longer to develop their first draft.

3.      Once the SDT has their first draft, they will submit it to the NERC Ballot Body for approval. It’s 100% certain it won’t be fully approved on the first ballot. With each ballot, NERC entities can submit comments – which, of course, mainly discuss why the commenter didn’t vote for the standard in question (each new or revised standard will be voted on separately). The drafting team needs to respond to every comment, although in practice they group similar comments and respond to them at one time. For just one of the CIP v5 ballots, 2,000 pages of comments were submitted.

4.      It’s close to certain that the new or revised standards will go through at least four ballots before they’re approved, with three comment periods in between them. The balloting process alone took the CIP v5 SDT a year, and I assume the new SDT’s experience will be roughly the same. Adding that to the estimate of 18 months to draft the first version of the new standads, we’re at 2 ½ years, starting in January.

5.      When the new or revised standards have been approved by the ballot body, they will go to the NERC Board of Trustees for approval at their next quarterly meeting; it’s close to certain the BoT will approve it in one meeting. So, BoT approval requires 3 months, bringing us to two years and nine months for the process so far.

6.      At that point, the standards go to FERC for approval. Even though individual FERC staff members have been quite supportive of the need for changes to accommodate cloud use (and two staff members spoke in the technical conference), the staff might very well not be in line with some of the actual changes that are proposed. Of course, the five FERC Commissioners are the ones who must approve those changes; they always take a lot of time to come to general agreement. I’ll stick with my earlier estimate of one year for FERC to approve the new or revised standards, but it could well be longer than that. We’re now at three years and nine months from next January, which is the third quarter of 2029.

7.      However, FERC approval doesn’t mean that NERC entities can rush off and start using the cloud. There will without doubt be an implementation period of more than one year; I’ll say it will be 18 months[i], but even that may be a low estimae. This puts us at the first or second quarter of 2031, before the new or revised CIP standards are enforced.[ii]

Thus, instead of saying that the cloud will be completely “legal” for NERC entities by the end of 2029, I’m now saying this will happen by the second quarter of 2031, which is 6 1/2 years from now. But that isn’t all: In my January 2024 post, I pointed out that I thought it was possible that the changes required for the cloud will also require changes to the NERC Rules of Procedure; I now believe it’s likely this step will be needed.

The SDT has no power to make RoP changes, and my guess is there might need to be a separate drafting team for those changes. Of course, this alone could add a couple more years to the whole process. Since I don’t know what’s involved, I won’t change my estimate of Q2 2031 as the date when systems subject to NERC CIP compliance can be freely used in the cloud, subject to the controls in the CIP standards. But there’s now a big asterisk beside that date.

If you’re like some NERC entities, as well as some members of the NERC ERO, you’ll probably look at my Q2 2031 date and say something like, “This is unacceptable! The NERC community can’t wait this long.” You would be right; this is unacceptable. This is why I’m sure that some measures will be taken long before that date to allow at least some cloud use cases for BES Cyber Systems, EACMS and PACS. However, I have no clear idea of what those measures will be, beyond my own wishful thinking. 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] The CIP v5 standards were approved by FERC in November 2013, but were enforced on July 1, 2016. That was 2 ½ years after approval.

[ii] Since many NERC entities are eager to start using the cloud for OT systems, there will probably be accommodations for entities that wish to follow the new standards before the implementation period is finished. However, only a small number of NERC entities will be allowed to take advantage of those accommodations, and they will be closely monitored. This was done when CIP v5 had been approved by FERC in 2013. At that time, NERC established the Version 5 Technical Advisory Group (V5TAG), a small group of NERC entities that implemented the v5 standards before the enforcement date. They were closely monitored by NERC and documented their experiences.