Thursday, June 13, 2024

Cloud CIP, Part 4: What can we learn from the CIP-013 experience?

Lew Folkerth of the RF Region and Chris Holmquest of SERC wrote a great article recently titled “The emerging risk of not using cloud services” (my emphasis); I linked to it in this post. It eloquently made the point that, far from the cloud being too risky for NERC entities to use, the risk of not using the cloud is currently greater – and growing all the time. This is because more and more software and service providers (especially security service providers) are announcing that in the coming years they will only offer a cloud-only version of their product, or at least they will no longer implement new features in their on-premises version. Both reliability and security will suffer as a result.

One sentence in the article stood out for me: “New Reliability Standards will be required, and those standards will need to be risk-based.” The article goes on to say, “There are too many variables in cloud environments to be able to write prescriptive standards for these cases.”

Of course, I completely agree with this statement, but how will this work in practice? Fortunately, these won’t be the first risk-based standards. The honor of being the first entirely risk-based standard goes to CIP-013, which came into effect in 2020. If any of you were reading this blog regularly at that time, you may remember I was a huge fan of the fact that CIP-013 is a risk based standard; I was sure it would be very successful and make everyone in NERC (both NERC entities and NERC auditors) love the idea of risk-based standards. But it didn’t quite work out like I’d hoped. Here’s my story:

I had been writing about CIP-013 from the beginning, when FERC issued Order 829 in July 2016. I participated in, and wrote about, the drafting team’s efforts to develop the standard. And I celebrated in 2018 when FERC approved CIP-013. Plus, I was the first person (that I know of) that advocated for pushing back the compliance date for CIP-013 when it became clear in March 2020 that Covid-19 was going to require the electric power industry - as well as almost every other industry - to drastically alter how they conducted their business.

Not coincidentally, I worked with some NERC entities to understand what CIP-013 compliance means. In the process of doing that, I developed what seemed to be a clear interpretation of what CIP-013 required (I wrote about this interpretation in a number of posts in 2017-2020). Here is my summary of the requirements of CIP-013 (it doesn’t matter whether you’re looking at version 1 or 2 of the standard; the only important difference between the two is greater applicability in v2):

1.      CIP-013-2 R1 requires the NERC entity to “…develop one or more documented supply chain cyber security risk management plan(s) for high and medium impact BES Cyber Systems…” R1 goes on to list, in R1.1 and 1.2, items that must be included in the plan.

2.      R1.1 requires that the plan “…identify and assess cyber security risk(s) to the Bulk Electric System from vendor products or services resulting from: (i) procuring and installing vendor equipment and software; and (ii) transitions from one vendor(s) to another vendor(s).”

3.      R1.2 lists six risks (or more specifically, mitigations for risks) that must be included in the plan. These six risks had been cited by FERC at different places within Order 829, as items that needed to be included in the plan. The Standards Drafting Team that developed CIP-013 gathered these all into one Requirement Part: R1.2.

4.      R2 requires the NERC entity to implement the plan they’ve developed in R1.

5.      R3 requires the NERC entity to review and update the plan every 15 months.

That’s it. The entire Standard (minus the Measures) fits on a single page. When it was developed, I marveled at the sheer simplicity of CIP-013.

Of course, the heart of CIP-013 is R1, since that describes the plan that’s required. I interpreted R1 (and still do) to mean that the NERC entity must do the following:

1. Identify supply chain cybersecurity risks to the Bulk Electric System resulting from the BES Cyber Systems the entity may procure. Where should the NERC entity look to find these risks? Of course, there are lots of lists, the NATF Criteria being one list that is especially relevant to the BES. But the entity doesn’t have to confine itself to a pre-conceived list. One way to identify risks is to read the news.

Here’s an example of that: Right after CIP-013 came into effect in 2020, the SolarWinds attack was discovered (when it was discovered, the attackers had been present in SolarWinds’ development network for 15 months. During that time, they carefully prepared and tested their malware before infecting seven releases of the Orion platform. They even started by introducing a proof of concept for their malware design, to see if it could infect the platform with a benign piece of code; that succeeded. I fully expect the attackers to publish a case study of this engineering triumph someday).

Surely, the risk that a software supplier has an insecure development network is a big risk. One mitigation for that risk would be requiring your software suppliers to fill out the Attestation Form that CISA recently released for compliance with Executive Order 14028.

2. Rate each risk as high or low, based on its likelihood and impact. In this case, estimating impact is easy: BES Cyber Systems are classified as such precisely because the impact of their loss, compromise, etc. is high. This means the impact of any supply chain attack on BCS will always be high. Therefore, the only real variable is likelihood. To rate each risk, the entity must simply ask, “Is the likelihood high or low that this risk will be realized?” If likelihood is high, risk is high; if it’s low, risk is low as well.

Fortunately, if you just divide likelihood into high and low levels, estimating it is easy. For example, someone may point out that, if even a small meteorite crashed into your relay supplier’s factory, the factory might be incinerated; that would of course mean your organization would need to find another supplier of relays. That’s a huge impact, but what’s the likelihood? Probably less than the likelihood of being struck by lightning on a sunny day. This is a low risk.

Once you’ve rated your risks as high or low, you then need to focus on the high ones; obviously, those are the only ones that need to be mitigated. But are you obligated to mitigate every high risk? No. An important principle of risk management is that no organization has unlimited resources available for risk mitigation. Your organization needs to decide which of the high risks you can afford to mitigate, and just focus on those. This means you should assess your vendors just based on the risks you’re trying to mitigate. In your questionnaires, you shouldn’t ask a vendor about a risk if you don’t care what their answer is. You’re just wasting your and the vendor’s time.

3. Once you have developed a list of risks you wish to mitigate, you need to add to that list the six risks in R1.2 (if you haven’t already identified them as important risks independently). You need to do this, not because these are the most significant supply chain cybersecurity risks to the BES (although they are all important risks), and certainly not because they’re the only supply chain risks to the BES. The SDT included those risks in R1.2 because FERC had mandated them at various disconnected places in Order 829. In other words, the SDT was saying, “We want you to identify risks you think are important and mitigate them. But, since FERC wants these six risks to be in your plan, you need to make sure you include them as well.”

Given my interpretation of CIP-013, how did I think it would be audited? It seemed quite logical to me:

a)      R1.1 would be audited based on how good a job the entity did of “identifying and assessing” risks. If they had made an honest effort to determine at least some of the most important supply chain cyber risks to the BES, that would be fine.

b)     R1.2 would be audited based on whether the entity included the six risks in R1.2.1 – R1.2.6 in its plan.

c)      R2 would be audited based on how well the entity implemented its plan – i.e., whether it took steps to mitigate all the risks it had said it would mitigate in the plan.

d)     R3 would be audited based on whether the entity had reviewed its plan every 15 months, and whether they had honestly taken steps to fix any problems or deficiencies they found in the plan.

During the runup to CIP-013 implementation in 2017-2020, I wrote a number of posts on what CIP-013 means, in which I elaborated on the above logic. Frankly, I thought that logic was so compelling that it would be widely adopted by NERC entities. After all, why would the CIP-013 drafting team tell NERC entities to develop a plan to “identify and assess” risks to the BES if they didn’t mean it?

But I was wrong. From what I’ve heard, there are few NERC entities that have interpreted CIP-013 to be about anything more than R1.2.1 – R1.2.6. And now, I wonder why I ever thought otherwise. After all, if NERC entities have learned anything from their 15 or so years of experience with NERC CIP compliance, it’s that they need to keep their “compliance footprint” as small as possible. That is, they need to keep their nose close to the grindstone and never stray beyond the strictest possible interpretation of the standards. To do anything more than what’s strictly required doesn’t win you any Brownie points; in fact, it might possibly leave you with a completely avoidable violation – an “own goal”, if you will.

However, I’m not blaming NERC entities for this situation. I’m also not blaming NERC, and certainly not the auditors. I’m blaming these two facts:

First, the standard, which I had admired for its pristine simplicity, was in retrospect too simple. Instead of simply telling NERC entities to “identify and assess” risks, R1 should have given them suggestions on how to do that within R1 itself. For example, R1.1 might have included a set of ten or so “areas of risk” that must be identified in the plan, e.g. “vendor remote access”, “software development process”, “secure shipment”, etc. The entity would be required to scrutinize each of these areas for risks that they should add to their plan. In some cases, they would be justified in ignoring one of those areas entirely; for example, if they don’t allow vendor remote access at all, they obviously don’t need to worry about securing their vendor remote access system.

Doing this would also have given the auditors something to hang their hat on when they audited the entity for CIP-013 compliance, other than simply determining whether the entity had done a good job of developing their plan. Instead, they could have verified that the entity examined each of the ten areas and made a conscious effort to determine whether there were important risks for them in each area. Since they didn’t do that, NERC entities focused entirely on the six items that were clearly required by CIP-013 R1: the six risks in R1.2.

Thus, the first lesson to be learned from the CIP-013 experience is that, in the world of prescriptive requirements like CIP-010 R1 (configuration management) and CIP-007 R2 (patch management), handing a blank slate to both the entity and the auditor and saying “You can figure this out for yourself” – which is unfortunately what the SDT did[i] - is asking for trouble.

Second and more importantly, NERC auditors in general (i.e. for all the standards, not just the CIP standards) aren’t trained to judge how well an entity has assessed and mitigated risks; they’re trained to determine if they did or didn’t do X. While I’m sure some of them, especially CIP auditors, understand risk very well (if for no other reason than that almost every other mandatory cybersecurity standard is based on risk), for many of them it’s a foreign concept. NERC needs to develop methods for auditing risk-based requirements, not just prescriptive ones, and then train the auditors on those methods.

Of course, fixing these two problems won’t be easy. But if NERC CIP is going to make a successful transition to the cloud, these two problems will need to be addressed.

Are you a vendor of current or future cloud-based services or software that would like to figure out an appropriate strategy for the next few years, as well as beyond that? Or are you a NERC entity that is struggling to understand what your current options are regarding cloud-based software and services? Please drop me an email so we can set up a time to discuss this!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.


[i] Having participated in many of that SDT’s discussions, I know why they made this mistake: FERC had given NERC a strict deadline to develop and approve the new supply chain security standard. The SDT couldn’t afford to add any provisions to CIP-013 that might stir up controversy and result in extra ballots being necessary (although there were controversies anyway). In other words, FERC’s deadline backfired spectacularly. This points to a big problem with NERC’s standards development process, at least when it comes to cybersecurity: you can have a comprehensive standard that takes a long time to approve, or you can have a minimal standard that gets approved relatively quickly. But you can’t have both.

Tuesday, June 4, 2024

The road to cloud CIP, part 3: the EACMS problem


There are several key components of the problem of complying with the NERC CIP Reliability Standards by NERC entities with high and/or medium impact Bulk Electric System (BES) environments. By far the most important of those (and the most important for the upcoming NERC “Risk Management for Third-Party Cloud Services” Standards Drafting Team to address) is what I call the EACMS problem. This is so important because a) it inhibits many NERC entities from using existing cloud-based security monitoring services, and b) because more and more on-premises security services are moving exclusively to the cloud, thus becoming off-limits to many NERC entities.

Why is this the case? This is the problem in a nutshell:

1.      EACMS stands for Electronic Access Control or Monitoring System. That is, any system that monitors or controls access to a high or medium impact Electronic Security Perimeter (ESP) or high or medium impact BES Cyber Systems (BCS) is ipso facto an EACMS. I italicized “or”, because that’s very significant. The previous definition included “and”, meaning a system had to both monitor and control access, to be an EACMS. That definition applied to a much more limited set of systems.

2.      While there are lots of security monitoring systems that don’t “control” access to an ESP or BCS, there probably aren’t many that don’t monitor access. After all, knowing who’s knocking on your door to the internet, and especially knowing who made it through, is probably the most important consideration for security monitoring.

3.      If a cloud-based system monitors access to a medium or high impact ESP, it’s an EACMS. Therefore, it needs to be compliant with all the CIP Requirements and Requirement Parts that apply to medium or high impact EACMS (of course, the terms ESP and EACMS are only defined within medium or high impact environments).

4.      While some of the CIP Requirements that apply to EACMS (e.g. the requirements of CIP-008 and CIP-009) aren’t impossible to comply with in a cloud environment, there are others, especially the requirements in CIP-006, for which strict compliance is almost impossible in the cloud. For example, since the security service’s EACMS must be within a PSP controlled by the NERC entity, some auditors might interpret this to require that each NERC entity (with high or medium impact BCS) that utilizes the service must authorize entry for any CSP employee who has access to the systems on which the service is implemented. Unless the systems are somehow segregated with access controlled by the entity, that will probably mean every employee that is allowed to enter a data center that contains one or more systems that implement the security service will need to first be approved by the NERC entity. Of course, no CSP will ever allow this to happen.

As I mentioned in the previous post in this series, one way that a cloud-based security service provider could get around this problem would be to create a separate instance of their service just for NERC entities. They would then lock the servers on which that instance is implemented in a separate room, with access controlled by the NERC entity. Of course, this solution technically breaks the cloud model and raises the CSP’s costs considerably. However, to retain their NERC CIP customers, the CSP may decide to “eat” this cost temporarily, pending full resolution (through the standards drafting process) of the problems with CIP compliance in the cloud.

So, how can a security service provider who wants to move to an entirely cloud-based model remain “CIP compliant” after their move – i.e., how can they avoid making their CIP customers non-compliant with most of the CIP requirements? Other than the locked-room solution just described, I know of no way to do that today, other than to change the nature of the service they offer, so that it doesn’t monitor access to the ESP (or so that ESP access is still monitored locally). That is why I say this is the biggest problem facing NERC entities with medium or high impact BES environments that want to utilize cloud services today.

Are you a vendor of current or future cloud-based services or software that would like to figure out an appropriate strategy for the next few years, as well as beyond that? Or are you a NERC entity that is struggling to understand what your current options are regarding cloud-based software and services? Please drop me an email so we can set up a time to discuss this!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

Saturday, June 1, 2024

We’re making progress on vulnerability database issues

Vulnerability database expert Brian Martin and I have been having a good back-and-forth discussion on LinkedIn about vulnerability database issues in general, including discussion of my proposal for a Global Vulnerability Database.

Today, Brian put up a new post that moves the discussion forward. His post includes 6 or 7 passages that point to what I think are common misconceptions that haven’t been well articulated previously. Because Brian has articulated them so clearly, I want to comment on each one of them. I’ll quote each of Brian's passages in red and then comment in black italics: 

While a Persistent Uniform Resource Locators (PURL) is one solution, it isn’t the only one used by vulnerability databases. So not only do you need to have an intelligent mapping from PURL to PURL, you also need it from CPE to PURL, and possibly other identifiers. It’s easy to have multiple valid PURLs all for the same piece of software.

BTW, purl in this context stands for “package URL”. Here is a good description of purl, posted by Philippe Ombredanne, the creator of purl.

Brian, when you say “mapping from purl to purl”, I think you’re talking about my earlier comment about comparing a CVE-purl connection in OSS Index with the same connection in CVE.org (once the CNAs start creating those). That’s a very special case, which I’d prefer to discuss with you offline.

However, “mapping CPE to purl” is literally impossible if there is more than one package manager for a particular OSS project. This is because most CPEs for open source software don’t refer to the package manager (except sometimes as part of the product name), meaning the user has no way of knowing which PM the vulnerability is found in.

Regarding the last sentence, “It’s easy to have multiple valid PURLs all for the same piece of software”, the problem is there’s no way to be certain that the code for a product named “log4core” in one package manager is bit-for-bit identical to the code for the “same” product in another package manager. Given that, the fact that CVE-12345 is found in one PM doesn’t allow you to conclude that it will be found in another PM.

This in one way is a limitation of purl, since you can’t make a statement that for example, CVE-12345 applies to all package managers that contain a product called “log4core”. You can only make that statement if you have tested log4core in all package managers. Purl will keep the CNA honest, meaning they will only list a purl in a CVE report if they have tested the product in that package manager – and a user should never assume a CVE in one PM will apply to another. In other words, CPE gives the user a false sense of comprehensiveness. 

Somewhere there are / were CPE specifications, likely before NVD took control of it. Early in the VulnDB days, we used them so we could generate our own CPE for products that didn’t appear in NVD. The fact that a seasoned vulnerability practitioner isn’t sure standards exist speaks volumes to how poorly NVD has managed CPE.

As unaccustomed as I am to defending NVD, I need to do so now. There’s simply no way there can be a unique CPE for any product – i.e., one that any user will always be able to create accurately. Pages 7-9 of the OWASP SBOM Forum’s 2022 document on the naming problem differentiate extrinsic identifiers like CPE from intrinsic identifiers like purl.

Briefly, an extrinsic identifier requires the user to do a lookup to at least one external database, before they can be sure they have the correct identifier. In the case of CPE, that database is the CPE Dictionary. On the other hand, an intrinsic identifier like purl just requires the user to enter information they already know with certainty: the package manager from which they downloaded the software, the product name in that package manager, and the version string in that package manager.

The reason that CPE is ultimately unworkable is the fact that creating a CPE name usually requires making arbitrary choices (e.g., “version 1.2” vs. “v1.2”), rather than only requiring information that can always be exactly verified by a user, Nobody can know for sure what choice was made by the person that created the CPE without doing a search of the CPE dictionary, and perhaps multiple searches using fuzzy logic or something like that.

(quoting Tom) “As long as you know the package manager (or source repository) that you downloaded an open source component from, as well as the name and version string in that package manager, you can create a purl that will always let you locate the exact component in a vulnerability database. This is why purl has literally won the battle to be the number one software identifier in vulnerability databases worldwide, and literally the only alternative to CPE.”

Unless… you end up having half a dozen PURLs for the same package, because it is available on a vendor’s page, GitHub, GitLab, Gitee, and every package manager out there.

And this is exactly the point about using purl in a vulnerability database: It only tells you what the CNA that created the CVE report with purl knows: the package manager, product name and version string of the software in which they found the vulnerability. The user can’t draw any conclusion about a product with the same name and version string in any other PM, unless the CNA that produced the report added purls for them as well (meaning they tested the same product and version in each PM). 

Who will maintain this epic list of PURLs? As of this blog, there are only 379 CNAs with tens of thousands of software companies out there. Not to mention the over one hundred million repositories on GitHub alone. While a PURL may be an open standard where CPE is not, it forces the community to set a PURL for every instance of the location of that software. That sounds like the big database you don’t think is viable?

Again, that’s the point of purl: no list is required. Any user can create the correct purl just from the three pieces of information they already know. As Steve Springett often says, every open source product in a package manager already has a purl – there’s no need to create it. 

(quoting Tom) “However, there is one big fly in the purl ointment: It currently doesn’t support proprietary (or “closed source”) software.”

And the other shoe drops. =) So, this is not a critique by any means, just highlighting the problems the community faces. The problems we faced 10 years have just compounded and here we are. Not that there were realistic solutions to all of these problems back then, and even if there were, we certainly didn’t address them then.

That’s correct. Currently, purl only covers open source software, although Steve Springett (who worked with Philippe to create purl, as mentioned in Philippe’s post that I linked above) points out that any online software “store” (Google Play, the Apple Store, etc.) could easily be made into a purl type, since the store controls the namespace of the proprietary products that are for sale in the store (just like a package manager controls the namespace of the packages in the PM).

In other words, what is needed is a controlled namespace, so one product will always have one name. Steve also suggested that SWID tags could be a more general way to identify proprietary software. He wrote the purl PR for a new identifier called SWID – which was adopted in 2022. See below. 

(quoting Tom) “I think this is a solvable problem, but it will depend – as a lot of worthwhile practices do – on a lot of people taking a little time every day to solve a problem for everybody. In this case, software suppliers will need to create a SWID tag for every product and version that they produce or that they still support. They might put all of these in a file called SWID.txt at a well-known location on their web site. An API in a user tool, when prompted with the name and version number of the product (which the user presumably has), would go to the site and download the SWID tag – then create the purl based on the contents (there are only about four fields needed for the purl, not the 80 or so in the original SWID spec).”

Unfortunately, I think at this point, this is a pipe dream. I am quite literally discovering new, well-known “standards” only by seeing them as requests ending in a 404 response in my web logs. So any such solution based on well-known I think isn’t viable now, and likely won’t be moving forward.

Please read what the OWASP SBOM Forum proposed regarding SWID on pages 11 and 12 of our 2022 paper. The point is that there needs to be some unique user-discoverable source of information on the product. Otherwise, the only alternative is to create (and maintain) a hugely expensive database of all proprietary software, along with the different product names and vendor names it was associated with through its lifetime – and that requires a huge number of very subjective judgments.

For example, if Product A from Vendor X is sold to Vendor Y who renames it Product B, is it the same product or not? If B is very different from A, you would just say it’s different. But if B is literally just A with a different name, you’d say it’s the same. Where do you draw the line between these two cases? There’s simply no way to do so.

There are certainly other ways that information on proprietary software could be made user-discoverable, so that no big secondary database (probably much larger than the vulnerability database itself) is required. One way is Steve Springett’s Common Lifecycle Enumeration project. That will take much longer to put in place than our SWID proposal, but IMO is ultimately the correct thing to do. If you have other ideas, we’d love to hear them. 

(Tom here) Of course, all of the above discussions are examples of the Naming Problem. There’s no question that this problem will be with us for a long time and will never be “solved” in any final way. However, the good news about the Global Vulnerability Database idea is that the naming problem doesn’t need to be solved first, precisely because the GVD won’t require “harmonization” of software names. 

The software will be named what it’s named in the vulnerability databases to which queries are routed; it will be up to the individual databases to continue their (presumably ongoing) efforts to improve their naming. If there's reason to believe there are serious naming problems in one vuln DB, the GVD might suspend routing queries to it. The GVD will be no more accurate than the individual DBs, but it won’t be less accurate, either. 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Also, if you would like to learn more about or join the OWASP SBOM Forum, please email me.

My book "Introduction to SBOM and VEX" is now available in paperback and Kindle versions! For background on the book and the link to order it, see this post.

 

Friday, May 31, 2024

The NVD continues to underwhelm


Yesterday, the NVD put up the latest episode of their ongoing soap opera, “As the NVD declines”, in the form of this announcement on their website:

May 29, 2024: NIST has awarded a contract for additional processing support for incoming Common Vulnerabilities and Exposures (CVEs) for inclusion in the National Vulnerability Database. We are confident that this additional support will allow us to return to the processing rates we maintained prior to February 2024 within the next few months.

In addition, a backlog of unprocessed CVEs has developed since February. NIST is working with the Department of Homeland Security’s Cybersecurity and Infrastructure Security Agency (CISA) to facilitate the addition of these unprocessed CVEs to the NVD. We anticipate that that this backlog will be cleared by the end of the fiscal year. 

As we shared earlier, NIST is also working on ways to address the increasing volume of vulnerabilities through technology and process updates. Our goal is to build a program that is sustainable for the long term and to support the automation of vulnerability management, security measurement and compliance.

With a 25-year history of providing this database of vulnerabilities to users around the world and given that we do not play an enforcement or oversight role, NIST is uniquely suited to manage the NVD. NIST is fully committed to maintaining and modernizing this important national resource that is vital to building and maintaining trust in information technology and fostering innovation. 

Moving forward, we will keep the community informed of our progress toward normal operational levels and our future modernization plans.

This announcement was loudly trumpeted by an article in Cybersecurity Dive today. The headline made me open the article, where I was immediately disappointed by the first sentence: “The National Institute of Standards and Technology expects to clear the towering backlog of unanalyzed vulnerabilities in the National Vulnerability Database by the end of September, the agency said in a Wednesday update.”

Why was this disappointing? To understand why, you need to understand the two most important activities performed by the NIST NVD staff:

1.      Importing CVE reports produced by CVE.org and integrating them into the NVD database.

2.      “Analyzing” the reports, which primarily consists of a) creating and adding a CVSS score (if not already present), b) adding CWEs, and c) adding CPE names. CPE names are by far the most important of those items, since without them, the CVE report is the rough equivalent of a car without a steering wheel: You know there’s a new vulnerability out there, but you have no idea what product(s) is vulnerable to it, unless you read the text of the report. However, text isn’t enough. The CPE name of a vulnerable product needs to be in the report, since without it, nothing will appear in the NVD to link the vulnerability to the product.

However, the NVD didn’t lie when they said in their announcement, “NIST has awarded a contract for additional processing support for incoming Common Vulnerabilities and Exposures (CVEs) for inclusion in the National Vulnerability Database”, right before they said, “In addition, a backlog of unprocessed CVEs has developed since February.” The first quote refers to item 1 above: the “additional processing support” in the new contract is to help the NVD ingest CVEs into the NVD. The second quote refers to the enrichment of those CVEs. That’s the “additional backlog” that they haven’t even thought about addressing yet, let alone found the funds to reduce it (of course, reducing that backlog will require a lot more hours of effort, although it will not be technically difficult). CISA is trying to reduce it themselves, but they’re only doing so for a small percentage of the backlogged CVEs.

This is more than passing strange. After all, the NVD has been processing CVE reports since the early years of this century. Since the processing doesn’t add anything to the report beyond what the CNAs (who work on behalf of CVE.org) have already included, and since by now, parsing the reports and populating the appropriate NVD fields should be performed as soon as the report is received, why does the NVD even have a backlog of CVEs to process, let alone need to pay $865,000 to a contractor to lower the backlog?

I’m sure the reason is the same as the one that probably explains the collapse of the enrichment function during one day (Feb. 12) in February: the fact that the NVD’s hardware and software infrastructure was created two decades ago. Presumably, whoever developed them has long ago departed the NVD (and perhaps this world), perhaps not leaving behind what would be considered top-notch documentation today.

Contrary to what the article says, CISA’s funding cutback doesn’t explain the sudden collapse of the database on February 12. Nor does the fact that they received a larger-than-normal number of new CVE reports then. No modern database should choke and be down for 3 ½ months (and counting) due to a sudden increase in workload. In fact, no modern database should be down for even a day due to any technical problem, let alone 3 ½ months. But we’ve known for a long time that there are multiple single points of failure in the NVD’s infrastructure.

By the way, has anyone heard the NVD’s explanation for what happened in February, or even an apology?...I didn’t think so, since they still haven’t provided one (note they didn’t do that in their announcement yesterday, either). This must mean one of two things, neither of which is good:

1.      They still haven’t figured out what happened; or

2.      They know what happened, but don’t think their worldwide users deserve to hear that.

I’m not sure which is worse.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Also, if you would like to learn more about or join the OWASP SBOM Forum, please email me.

My book "Introduction to SBOM and VEX" is now available in paperback and Kindle versions! For background on the book and the link to order it, see this post.

 

Wednesday, May 29, 2024

The Road to Cloud CIP, part 2: the CIP-013/ISO 27001 solution


In my first post in this series, I pointed out what I think is an essential element of new CIP standards that will accommodate cloud use for NERC entities with medium and/or high impact environments: having separate “tracks” for cloud vs. on-premises systems (with the requirements for the latter being almost exactly the same as the current CIP requirements).

However, as I pointed out in this post, the full NERC process of developing the new standards and getting them approved by NERC and FERC – which started three weeks ago - will likely take at least five years. This isn’t good, since more and more software and services vendors (and especially security services vendors) are moving to a cloud-only model, even though they know some of their electric utility customers won’t be able to follow them there. It is likely that the security and reliability of the North American Bulk Electric System (BES) will soon be negatively impacted because of this situation.

In this more recent post, I pointed out three things. First, I noted that the big problem inhibiting cloud use for CIP systems is the fact that the CIP requirements are based on the idea of protecting the physical device (e.g. a server) on which a system in scope is housed – yet, the software that comprises those systems moves rapidly from server to server all the time. Strict application of the CIP requirements (which were mostly developed before cloud use became widespread, and which never mention the cloud once) mandates first that every cloud server that holds even a small piece of a system in scope for CIP - say, an EACMS - comply with all the CIP requirements that apply to that asset type, no matter how inapplicable they may be to the cloud.

Second, those requirements also implicitly mandate that every server that might in the future hold part of one of those systems be continually protected during an entire three-year audit cycle. Given that there are over 40 applicable CIP requirements and over 100 applicable requirement parts (sub-requirements), that would require an unbelievably huge amount of work for the CSP (applicable systems are medium or high impact BES Cyber Systems or BCS, Electronic Access Control or Monitoring Systems or EACMS, and Physical Access Control Systems or PACS). Moreover, the CSP would need to provide literal mountains (well, hills) of documentation, individualized for each NERC entity that is a customer.

Finally, I noted that CIP-013-2, the supply chain security standard, may furnish a way out of this mess in the near term, while still allowing the full 5+-year process of drafting new CIP standards (and probably changes to the NERC Rules of Procedure) to proceed in the background.

However, it turns out I made a mistake in both statements. In the first statement (really, a group of statements), I made the implicit assumption that a cloud service provider (CSP) would never “break the cloud model” by installing systems in scope for NERC CIP on dedicated, unchanging servers, even though the requirements were all written for such servers. But I’ve since learned that this might indeed be an option for CSPs. If BCS, EACMS and PACS never jump from physical device to physical device, CIP compliance suddenly becomes much easier.

In the second statement (about CIP-013), I followed the same implicit assumption – that systems in scope for CIP compliance would continually jump from server to server in the cloud, meaning there would be an unbelievably large number of servers in scope for CIP-013 compliance. However, if the servers are unchanging (and locked within a single Physical Security Perimeter subject to the requirements of CIP-006), this suddenly makes what I proposed in the post – that the NERC entity declare the CSP to be a vendor under CIP-013-2, and that the method for assessing their cybersecurity be examining their ISO 27001 certification evidence – much more realistic.

In other words, I now like my suggestion about CIP-013 even more than I did in April. It seems there could be a fairly easy way for usage of the cloud by medium and high impact BCS, EACMS and PACS to be “legal” relatively soon - maybe in two years, vs. the 5+ years that the full standards development process will take. Of course, the full process still needs to take its course, since it is very likely that the new Standards Drafting Team will want to see more evidence from the CSP than just their ISO 27001 certification, and since changes will be needed in CIP-002 to enable the “two-track” CIP compliance process I described in the first post in this series.

What will it take for this to happen? I think the NERC Board of Trustees will need to take some action to get this all moving. It would be nice to be able to say there is currently motion in that direction. However, I don't see that now. 

I do want to point out that the NERC Cloud Technical Advisory Group (CTAG) and SANS are going to sponsor a set of 6 or 7 webinars on use of the cloud by NERC entities, starting at the end of June and running biweekly through the end of August. That may raise some awareness.

Are you a vendor of current or future cloud-based services or software that would like to figure out an appropriate strategy for the next few years, as well as beyond that? Or are you a NERC entity that is struggling to understand what your current options are regarding cloud-based software and services? Please drop me an email so we can set up a time to discuss this!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

Saturday, May 25, 2024

NERC CIP: The road to “Cloud CIP”, part 1


Last December, the NERC Standards Committee approved a Standards Authorization Request (SAR) that set in motion the process of making revisions to the NERC CIP Standards (and perhaps the NERC Rules of Procedure as well) that will finally allow NERC entities with high and/or medium impact BES environments to make full use of cloud services for those environments.

However, when I say “set in motion” I’m using that phrase loosely, since the committee assigned the project medium priority - meaning it would not even start until the third quarter of this year. I pointed out in this post that, because of all the cats that need to be herded for this project to succeed, it will probably take 5-6 years (at least) between the day the project starts and the day the barriers to full use of the cloud by NERC entities are finally removed.

I also pointed out there is growing concern among NERC and Regional Entity staff members about the steadily increasing numbers of software and service providers who are telling their NERC entity customers they no longer have the option of providing a totally on-premises solution. Those NERC entities will soon face the choice (or have already faced it) between doing without those software products and services, and being in violation of a slew of CIP requirements if they don’t move away from them.

These staff members fear that within 2-3 years there may be real damage to the reliability - and especially the security - of the Bulk Electric System. This is because some important NERC entities will no longer be able to utilize software and services (especially security services) that they depend upon today to keep the lights on. I speculated there might need to be some sort of “break glass” measure that would allow at least some NERC entities to utilize the cloud for high and medium impact BES environments, while still allowing the standards development process to proceed at its accustomed geologic pace (in fact, I suggested one such measure, which I think is still an option that needs to be discussed. In fact, it will not break much glass at all).

However, it seems the Standards Committee has been hearing about these problems from other sources as well, since last week there was an unexpected announcement that Project 2023-09: Risk Management for Third-Party Cloud Services has been set up and is now soliciting comments on the SAR. Of course, there’s a huge journey ahead, but it’s nice to see that the first step is being taken earlier than originally planned.

In October, I was invited to present for a monthly webinar (called a Tech talk) presented by the RF NERC Region; I chose as my topic the question of how I would rewrite the NERC CIP standards to “pave the road” to full use of the cloud by NERC entities. Lew Folkerth of RF – a good friend who has made regular appearances in this blog for almost ten years – interviewed me for the webinar.

As the basis for the webinar, I put together a lengthy article describing in some detail the changes I would make; I published it in this post (I also published a PDF of the article, which I’ll be glad to provide to anyone who emails me for it).

Of course, now that the standards drafting process is finally starting, it’s now more important than ever to get ideas on the table for what the new standards should look like. The ideas in my article haven’t changed hugely since I wrote it, but I would like to make them more accessible now by discussing them in a set of short posts; this is the first of those posts. Since I’m sure my ideas will evolve as the new Standards Drafting Team (SDT) meets and starts having substantive discussions, this might be a series that goes on for years.

Something like this is needed since, unlike almost every other NERC CIP standards drafting process since the CIP v1 drafting team started meeting in 2006, this process is not driven by a FERC order. Even though FERC staff members understand that the changes I’m hereby naming “Cloud CIP” are sorely needed, and even though they are providing assistance when they can, the drafting team doesn’t have an official FERC “blueprint” to follow. Instead, it is up to the drafting team to figure out what it wants to be when it grows up (the team hasn’t been constituted yet. If you work for a NERC entity, you might consider getting nominated to the team. Having been an active observer of several previous standards drafting efforts, I can promise it will take a lot of your time, but I can also promise it will probably be one of the most interesting efforts you’ve ever participated in).

I certainly can’t say I know exactly what is needed to solve the problem of CIP in the cloud, but at least the posts I write will help clarify people’s ideas. It’s almost impossible to get very far if you start with a completely blank slate, which is essentially what the drafting team has been presented with (the SAR rightfully doesn’t try to prescribe what the team needs to do). It’s better to start from what later proves to be a dead end position, than to start from no position at all.

My first topic in this series is an idea that I definitely didn’t originate, but which I now realize is probably the key to a successful Cloud CIP effort. This is an idea that the CIP Modifications drafting team learned the hard way in 2018. I hope to describe that bit of history in another post soon, but to summarize, that drafting team proposed a thoroughgoing change to CIP that in retrospect was exactly what’s needed to fix the cloud problem (it was actually intended to be a framework for integrating virtualization support into the CIP standards). However, the SDT’s proposal was going to require that every NERC entity throw away most of their existing CIP program (including documentation, training, software, etc.) and start with a brand new one.

The new CIP program that the SDT outlined (which I discussed in this and two subsequent posts) would have rewritten many of the CIP requirements so they were all risk-based. It was certainly the right overall approach, but a lot of big utilities, who had millions of dollars invested in their existing CIP programs and neither the budget nor the inclination to throw all of that away and start over, made it clear they would never do that. The drafting team realized they’d been beaten and dropped the whole idea.

I had been a big supporter of the drafting team’s ideas in 2018, but after they went down in flames, I decided there’s no fighting City Hall; I stopped advocating for those changes. About once a year, I put out a post stating that I saw no prospect for the cloud becoming completely “legal” for NERC entities until the NERC community had a change of heart and decided that the long term benefit of having CIP  requirements that would allow full use of the cloud was worth the short-term hassle of having to throw away their existing processes and start over.

However, early last year a new SAR was developed that was quite short on details but threw in one new concept which turned out to be the key to making Cloud CIP a real possibility. This SAR (which developed into the one that was adopted in December) raised the idea of two CIP “forks” for two different groups.

One group is the set of NERC entities (which might even be the majority, although I have no way to know if that’s the case now) that is perfectly fine with the existing CIP standards, and more importantly doesn’t want to make a radical change to what they’re doing now. They don’t particularly care about making full use of something they don’t think they need anyway: use of the cloud by medium and high impact BES Cyber Systems, Electronic Access Control or Monitoring Systems (EACMS), etc. The other group is NERC entities that are painfully aware of how much not being able to make full use of the cloud is hurting both their organization’s bottom line and increasingly their levels of reliability and security, as their most important vendors start to tell them they are moving to the cloud – and by the way, will they join them there?

For the first group, the solution is simple: They can keep doing exactly what they’re doing now. The CIP requirements they comply with won’t change at all, except for changes already proceeding that have nothing to do with the cloud. For the second group, the CIP changes will be big (including completely risk-based requirements), but only for systems they wish to “outsource” to the cloud – either by use of SaaS offerings or by actually transferring existing on-premises systems to the cloud. For their on-premises systems, there will be no change at all in the CIP requirements.

Does this two-track system sound like a big mess to you? I thought that might be the case, but when I looked at how it could be accomplished, I realized that in principle it’s not that hard. The principal changes required are a) defining new types of assets with “Cloud” in the name (e.g., “Cloud BES Cyber System”) and b) making some surprisingly minor changes to wording in CIP-002 R1 and Attachment 1. Almost no changes are required in the other CIP standards, since they will henceforth just apply to on-premises systems (i.e., what they apply to now). The requirements that apply to cloud systems will be found in new CIP standards that apply only to cloud-based systems.[i]

There’s a reason why the changes to the existing CIP standards to accommodate the two-track Cloud CIP system turn out to be so easy to describe. That’s a subject for one of the next posts in this series. I’m giving you something to look forward to.

Are you a vendor of cloud-based services or software (or services or software you would like to be cloud-based, were it not for problems like those discussed above), that would like to figure out an appropriate strategy for the next few years, as well as beyond that? Or are you a NERC entity that is struggling to understand what your current options are regarding cloud-based software and services? Please drop me an email so we can set up a time to discuss this!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.


[i] NERC entities that choose to put systems in the cloud under Cloud CIP will still need to follow the “classic” CIP standards for their on-premises systems.

Sunday, May 19, 2024

Clarifying the Global Vulnerability Database

Brian Martin recently put up a long, thoughtful post on LinkedIn that critiqued my post from last November about what I called (and still call) the Global Vulnerability Database (GVD). That post was one of many I’ve written that started by being about one thing (in this case, a GVD that would be a single database) but evolved as I was writing the post into something else by the end (something that isn’t a single database at all, but more of an “intelligent switching hub” for evaluating vulnerability database queries and routing them among the most appropriate existing databases).

This evolution didn’t bother me, since one of the advantages of calling something you write a blog post, rather than an essay or a white paper, is that you don’t have to go back and rewrite the whole post when such an evolution occurs – people expect inconsistency (and I deliver it, I’m proud to say). I figured that it wasn’t worth rewriting the post unless it drew a lot of interest, so I let it stand as I’d written it.

It didn’t draw much interest until Brian wrote his post a couple of weeks ago. Brian has a lot of experience with vulnerability databases and has a good following for his thoughts, so his post drew a lot of comments.

When I read Brian’s post, I realized that a number of his objections wouldn’t have been valid if I’d rewritten the post so that it focused on the single idea of a switching hub, not an actual database – although I presume I won’t be thrown in jail if users think of it as a single database. The point is that it should be possible in 2024 to field diverse queries – regarding different types of products (open source software, proprietary or “closed source” software, and intelligent devices), different types of vulnerabilities (CVE, OSV, GitHub Security Advisories, etc.), and different identifiers (CPE and purl) - and have an intelligent engine that decides, for each query, which is the best database or combination of databases to resolve the query.

Once that decision has been made, the appropriate queries will go out to the different individual databases (CVE.org, the NVD if it still exists, OSV, OSS Index, VulnCheck, VulnDB, etc.). Then, the results will be processed and returned to the user as an answer to their question. There would need to be a lot of intelligence behind both of these steps, since they won’t be easy at all (and they will require quite a lot of prior knowledge, such as whether a report in OSS Index that a particular software product – identified with a purl – is affected by a CVE has the same status as a report in CVE.org that the same purl is affected by the same CVE, since they will have been derived very differently).

To rectify my sin of last November in not rewriting my post before I put it up, I put up a new post on May 9. This made a single, coherent statement, but still doesn’t include all the detail (such as what’s in the paragraphs above) that I would include if I had the time to write a white paper. I think this answers many of Brian’s questions (such as whether the GVD would be hugely expensive and require legions of volunteers. That would be the case if we tried to put up a single “harmonized” database that maintains data from all existing vulnerability databases, and that’s the reason why last November I switched in mid-post to the idea of a switching hub).

However, Brian brought up one important issue that I want to address now (he brought up others, which I hope were mostly addressed in my May 9 post. If there are other issues that you still want me to address, Brian, please let me know).

Is CPE a dead end?

Brian repeated a sentence from my November post, “Specifically, a new identifier is needed for proprietary software, since I (and others) regard CPE as a dead end, even though it was pioneering in its time.” Brian ended up basically agreeing with that statement, but his reasoning isn’t mine, and I’d like to describe that.

Pages 4-6 of the OWASP SBOM Forum’s 2022 paper on how to fix the naming problem in the NVD (which is still valid today, even though the NVD now seems to be on its way to extinction or worse: irrelevance) describe some serious problems with CPE. However, they don’t address what I consider to be the most serious problem: the fact that there will never be a way to populate fields like vendor and product name in a way that will be universally agreed on, without resort to more databases – which themselves will need to be constructed, maintained, etc.

For example, a CPE listing “Microsoft” as the vendor will be different from one listing “Microsoft, Inc”, which will be different from one listing “Microsoft, Inc.” with a period, etc. Because a CPE won’t be easily found unless it matches the CPE that’s searched for, trying to search for a particular product or vendor always involves guessing about the choices that were made by the person (usually an NVD staff member, until recently) who created the CPE. The NVD may have some sort of criteria they follow (e.g., “Always put a comma before ‘Inc’ and a period after it.”), but they’re clearly just rough rules of thumb if they exist at all, since CPE names vary for seemingly random reasons.

Because, as Brian points out, the CNAs will probably be creating most CPE names from now on and the CNA is often the developer of the product being named, this in theory is better. Yet, how is an end user that wants to know whether the product I’m using to write this post is called “Microsoft Word”, “Word”, “Word 365”, “Microsoft Office Word”, etc. going to find which one of those Microsoft (which is a CNA) uses? Even worse, the product name might vary by the division within Microsoft that creates the CPE, etc.

You might say something like, “What does Microsoft call the product on their web site?” And I ask, which of the Microsoft web sites are you referring to? Is Microsoft going to enforce standard naming across all web sites worldwide? And what about blog posts on the Microsoft sites? Will they follow some sort of internal Microsoft standard? Etc., etc.

What some people, including some who should know better, have suggested is that there should be a centralized database of product names, company names, version strings (since versions can be identified in many ways), etc. Then “all you have to do” to find the correct CPE is look up the company name, product name, and version string (which also varies a lot) in the directory. The company will hopefully rigorously enforce use of their chosen names, and the CNAs will be severely disciplined if they use any other in naming their products in a CVE report…And while we’re at it, the lion will lie down with the lamb and I will study war no more and people will stop having loud cell phone conversations on trains; that is, all the world’s problems will be solved…

By the way, who will pay for that inordinately expensive database of product and company names? It will cost a huge amount of money, both to put together and to maintain – much more than the cost of the NVD and CVE.org databases combined. Face it: an identifier that requires an expensive auxiliary database to make it work is a dead end. Even if all the other problems with CPE didn’t exist, this alone would ultimately sink it.

This is why the OWASP SBOM Forum recommended purl as the replacement for CPE in our 2022 paper. The paper goes to inordinate lengths to explain why purl is better, but the main reason is that no lookup is required. As long as you know the package manager (or source repository) that you downloaded an open source component from, as well as the name and version string in that package manager, you can create a purl that will always let you locate the exact component in a vulnerability database. This is why purl has literally won the battle to be the number one software identifier in vulnerability databases worldwide, and literally the only alternative to CPE.

Currently, there are no purls in CVE.org. However, the fact that CVE now supports purl in CVE Format 5.1 (formerly “CVE JSON spec v5.1) – a change requested by the SBOM Forum two years ago – means there will be purls when the CNAs start adding them to their CVE reports (which unfortunately will probably not be soon, given the substantial training that will need to be conducted.

However, there is one big fly in the purl ointment: It currently doesn’t support proprietary (or “closed source”) software. Our 2022 paper did suggest a solution for that problem (proposed by Steve Springett, who is a purl maintainer, among many other things): There should be a new purl type called SWID, which will be based on the contents of a SWID tag created by the supplier. Anybody with the SWID tag for the product they want to inquire about (and for at least a few years, some big software suppliers like Microsoft included a SWID tag with the binaries for all of their products) will be able to create exactly the same purl that the supplier used to report the vulnerability. In fact, Steve got the SWID type added to purl.

What’s preventing this from being the solution for naming proprietary software is that there’s no good way for an end user, who might not have access to the binaries of a product they’re using – or who is using a legacy product that doesn’t have a SWID tag – to find the tag, if there is one.

I think this is a solvable problem, but it will depend – as a lot of worthwhile practices do – on a lot of people taking a little time every day to solve a problem for everybody. In this case, software suppliers will need to create a SWID tag for every product and version that they produce or that they still support. They might put all of these in a file called SWID.txt at a well-known location on their web site. An API in a user tool, when prompted with the name and version number of the product (which the user presumably has), would go to the site and download the SWID tag – then create the purl based on the contents (there are only about four fields needed for the purl, not the 80 or so in the original SWID spec).

There can be other solutions like this as well, and they don’t even have to be based on SWID tags (as long as they’re based on purl). The point is that we should no longer have to rely on a software identifier like CPE, that requires a separate database (or databases) to work. Of course, since there are so many CVE reports that have only CPEs on them (in fact, I think they all do today), it will be years (if not decades) before we can finally be done with CPE. But we should try to move to purls as soon as possible, so we can at least stop the bleeding.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Also, if you would like to learn more about or join the OWASP SBOM Forum, please email me.

My book "Introduction to SBOM and VEX" is now available in paperback and Kindle versions! For background on the book and the link to order it, see this post.