Should you be worried or happy when you find no vulnerabilities in your software?

One of the unfortunate byproducts of vulnerability management compliance requirements is that they tend to reward negative findings. That is, if you search a vulnerability database for products that you use and find that some product searches don’t yield any vulnerabilities at all, you might take this as evidence that the products are very secure and you must be compliant.

However, often that’s far from the truth. The negative finding can mean:

1.      The person who created the identifier for the product in the vulnerability database made a mistake or didn’t follow the specification. Because you searched using the correct identifier, the search failed.

2.      The product you’re searching for has been acquired by a different supplier, who enters new vulnerabilities under their name, not that of the previous supplier (from which you acquired the product). That search fails, too.

3.      The supplier name reported was for example “Microsoft, Inc.”, but it is recorded in the identifier as “Microsoft Inc.” Again, the search fails.

4.      A product has been entered in the vulnerability database using multiple names, e.g. “OpenSSL” vs “OpenSSL_Framework”. However, a vulnerability is seldom reported using multiple names, since it’s unlikely that the CVE Numbering Authority (CNA) that reported the vulnerability knew there were multiple names available. Thus, due to pure chance, one name might have no vulnerabilities reported against it, while the other may have a lot of vulnerabilities, even though they’re in fact the same product. If you happen to enter the name for the version with no vulnerabilities reported, your search will again fail.

5.      Even though the identifier in the database is correct, you fat-fingered the identifier in the search bar, so the search failed.

6.      The supplier of the software product has never reported a vulnerability for it. Instead of the negative finding being good news, it means the supplier cares very little for the security of their products or their customers. The great majority of software vulnerabilities are reported by the developer of the software or the manufacturer of the device. However, it’s certain that reported CVEs (about 260,000 have been reported since 1999) are a tiny fraction of all software vulnerabilities. Lack of vulnerability reporting is the biggest obstacle in the way of securing the software supply chain.

In the National Vulnerability Database (NVD), the above problems are compounded by the error message the user receives almost every time a search is unsuccessful: “There are 0 matching records”. Since this message also appears when a product truly doesn’t have any vulnerabilities reported against it, in all the above scenarios you might reasonably assume the product is vulnerability-free. In fact, human nature dictates that most of us will make that assumption. Of course, in most cases it’s probably not true.

What is the solution to these problems? I don’t know of any solution to all of them, but I can point out that the first four problems will not appear if the identifier used in the vulnerability database is purl (the product identifier used in almost all open source software databases) as opposed to CPE (the identifier used by the NVD and databases that are based on the NVD).

This is why the CVE Program (run by cve.org) is now looking at including purl as another identifier in what I call the “CVE ecosystem”. This includes all vulnerability databases that utilize CVE as the vulnerability identifier (there are many other vulnerability identifiers, mostly for open source software products. However, CVE is by far the major vulnerability identifier worldwide).

Of course, the most widely used vulnerability database in the CVE ecosystem is the NVD. When the CVE Program adopts purl as an alternative identifier, will purl suddenly be usable for searches in the NVD? No. To support purl, a vulnerability database that currently just supports CPE will need to make several changes. Given the problems that the NVD has been experiencing over the past 11 months, it isn’t likely they will be able to suddenly devote resources to making them.

However, other databases will be able to display CVEs that apply to a software product when the user enters a purl[i]. This means that at least some of the CVE Records published by CVE.org will be accessible to users and developers of open source software products.

It will be at least a couple of years before purl is fully supported in the CVE ecosystem. That might seem to be a long time, if it weren’t for the fact that six months ago I would have told you it’s unlikely that purl will be supported by CVE in this decade. Things are starting to move in the right direction.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

My book "Introduction to SBOM and VEX" is now available in paperback and Kindle versions! For background on the book and the link to order it, see this post.

---

[i] There are two large vulnerability databases that display CVEs that apply to an open source software product/version when the user enters the purl for that product/version. However, these databases don’t include the CVE Records published by the CVE Program, since those records currently don’t include purls. When that changes, those two databases will be able to accept new CVE Records, thus making those records searchable using purl.

Here’s how we can chop a couple of years off the wait for the “Cloud CIP” standards

 

I have posted recently on the need to rewrite two NERC CIP requirements: CIP-007 Requirement R2 (patch management) and CIP-010 Requirement R1 (configuration management). The primary reason that both requirements need to be rewritten is that they are by far the most prescriptive CIP requirements. In fact, since CIP version 5 (when both these requirements were substantially revised) came into effect in 2016, I have heard that complying with just these two requirements accounts for a substantial percentage of all NERC compliance costs, not just NERC CIP compliance costs.

However, the second reason why these two requirements need to be rewritten is that they are currently the two biggest barriers to use of the cloud by NERC entities with medium or high impact BES environments. The main reason for this is that the two requirements apply on the level of individual BES Cyber Assets, even though they’re written to apply to BES Cyber Systems (BCS). This means that a cloud service provider would have to produce documentation for the NERC entity that showed the CSP had taken every required step in CIP-007 R2 and CIP-010 R1 for every device on which any part of the BCS resided during the audit period.

One of the main reasons why use of the cloud is so inexpensive is that systems (i.e., the software and data in systems) can be moved from server to server and datacenter to datacenter whenever it’s advantageous to do so. It would be hugely expensive if a CSP were required to provide that information, and it’s doubtful that any CSP would even entertain the idea of doing that. None of the other CIP requirements require providing documentation at anywhere near that level of detail.

Fortunately, both the prescriptiveness problem and the cloud documentation problem can be cured with the same medicine: rewriting CIP-007 R2 and CIP-010 R1 to make them “objectives-based” (that is NERC’s term, although mine is “risk-based”. They mean effectively the same thing). When will that happen?

Last summer, a new NERC Standards Drafting Team started working on what will undoubtedly be a huge multi-year project to revise (and/or add to) the existing NERC CIP standards to make them “cloud-friendly”. They haven’t worked out their agenda yet, but I recently estimated that the new and/or revised standards will be fully approved and enforced around 2031. This is based on the experience with CIP version 5, which took almost that long and which in some ways was easier to draft than “cloud CIP” will be.

However, one thing is certain about the SDT’s agenda: it will include rewriting CIP-007 R2 and CIP-010 R1. Given how controversial both requirements are, and the fact that CIP-007 R2 needs to be rewritten as a vulnerability management, not a patch management, requirement, I think just rewriting and balloting those two requirements will take 1 ½ to 2 years. While this work will undoubtedly require some coordination with the “Risk Management for Third-Party Cloud Services” drafting team, this is something that NERC drafting teams do all the time.

So here’s my idea: Why not create a new Standards Authorization Request (SAR) that just requires rewriting the two requirements? This would take CIP-007 R2 and CIP-010 R1 completely off the cloud SDT’s plate, meaning they might be able to finish their work in five years, not seven. And it would allow the two revised requirements to be drafted by a fresh team that’s excited about being able to fix the two biggest “problem children” among the NERC CIP requirements, rather than a team that’s midway through a 7-year slog and wondering if perhaps long-distance truck driving would have been a better career choice.

While I would technically be allowed to draft that SAR, I don’t have the time to do it – and more importantly, a SAR has much better chance of approval if it’s prepared by one or two NERC entities (with perhaps a vendor also participating). However, if a NERC entity wants to take the lead on this, I’d be pleased to help draft it.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

My book "Introduction to SBOM and VEX" is available! For context, see this post. 

Why do we need to replace NERC CIP-010 R1?

 

I have written multiple posts recently (including this one and this one) on why CIP-007 R2, the NERC CIP patch management requirement, needs to be rewritten as a risk-based (or “objectives-based”) vulnerability management requirement. My two main reasons for saying this are:

1.      The inordinately high cost of documenting compliance with CIP-007 R2 causes NERC entities with high and medium impact BES environments to invest a lot of resources in activities that don’t yield much of an increase in security. The same can be said for CIP-010 R1.

2.      CIP-007 R2 and CIP-010 R1 are very likely the two largest obstacles to implementing medium and high impact BES Cyber Systems (BCS), Electronic Access Control or Monitoring Systems (EACMS) and Physical Access Control Systems (PACS) in the cloud today. This is because a cloud service provider may be required to provide compliance evidence for these two requirements on an individual device (both physical and virtual device) level; that will be quite difficult for the CSP to do.[i]

I’ve stated many times since CIP version 5 came into effect in 2016 that CIP-010 R1 and CIP-007 R2 are the two biggest examples of the high cost of complying with prescriptive CIP requirements. One NERC entity with high impact Control Centers told me that, of all the documentation they must generate for compliance with all NERC requirements (not just NERC CIP requirements), half of that is due to CIP-010 R1 and CIP-007 R2. While it’s indisputable that security measures need to be documented, a number of NERC entities have also told me they think half of the CIP compliance documentation they produce doesn’t improve security at all.

Cybersecurity is a risk management exercise. This means the objective of any cybersecurity requirement should be to reduce risk. My main problem with CIP-007 R2 is that its objective, patch management, doesn’t address a risk at all. Instead, it’s a mitigation for an important cyber risk that isn’t directly addressed in CIP today: the risk posed by unpatched software vulnerabilities in BCS, EACMS, Protected Cyber Assets (PCA) and PACS. Since there are other mitigations for this risk besides patch management, and because in many cases the risk posed by an unpatched vulnerability is so low that it’s a waste of time for a user organization to apply the patch for it, CIP-007 R2 needs to be rewritten as a vulnerability management requirement.

How should CIP-010 R1, the CIP configuration management requirement, be rewritten as a risk-based requirement? As with patch management, configuration management is a mitigation for a cybersecurity risk, not a risk in itself. What is the risk that configuration management mitigates?

Think about what happens if you misconfigure a system – for example, you disable authentication while working on the system and forget to re-enable it. This leaves the system vulnerable to exploit in a similar way to how a software vulnerability leaves the system vulnerable to exploit. In other words, the purpose of configuration management is to prevent vulnerabilities that are due to accidental misconfiguration, just as the purpose of patch management is to prevent vulnerabilities that are due to software flaws. Does this mean that CIP-007 R2 and CIP-010 R1 address the same problem: risk due to unmitigated vulnerabilities?

I don’t think it does, and here’s why: There is usually little or nothing that most organizations can do to prevent vulnerabilities from being present in software they didn’t develop. However, there is a lot that an organization can do to prevent configuration vulnerabilities. In fact, that’s the purpose of CIP-010 R1: to prevent a configuration error from leaving the system in a vulnerable state.

Given that’s the case, you might wonder why I’m making such a big point about CIP-010 R1. Since its purpose is to prevent misconfigurations from occurring in the first place, why does the requirement need to be rewritten? Obviously, if there are no misconfigurations, there’s no risk from misconfiguration, right?

Unfortunately, no. No true risk can ever be completely “prevented”. I can prevent the risk of food poisoning by not eating, but I don’t know many people who would think that’s a good idea. Similarly, the way to prevent the risk of computer misconfiguration is not to use computers at all, but that’s neither possible nor desirable for most of us.

Any risk management exercise needs to begin by acknowledging there is no way to reduce the risk to zero. Even if a NERC entity follows every letter of CIP-010 R1, it’s always possible that someone on the staff will have a bad day and forget to do something really basic, leaving the NERC entity exposed to both cybersecurity and compliance risk.

Of course, you could substantially reduce the risk of a bad day making someone forgetful by hiring two staff members for every position and making sure that each person is always watching the other. If that doesn’t work, you could hire three staff members for each position – although even that isn’t guaranteed to work…

You probably get the idea: If you’re willing and able to throw more resources into mitigating a risk, you can probably reduce it to as close to zero as you want, but at an increasingly higher cost per amount of risk mitigated. At a certain point, your boss will tell you the company has better uses for its resources than trying to reduce this particular risk to .00000001%. In fact, your boss might accompany that message with a pink slip.

This is why it’s so important that all the NERC CIP requirements be made risk- (or objectives-) based: If an overzealous auditor wants to require a NERC entity to ignore all other options for use of its time and money and focus the whole organization on mitigating one particular risk (which might be the risk of computer misconfiguration), the entity can argue that the auditor is being unreasonable.

However, there are a small number of prescriptive CIP requirements that mandate certain actions come hell or high water. For example, CIP-007 R2 requires you to apply or develop a mitigation plan for every applicable security patch that’s been released for a software product used in the ESP within the last 70 days. Even if the vulnerability addressed by the patch has never been exploited in the wild since it was discovered ten years ago – which means the vulnerability addressed by the patch poses very little risk to the organization – it still is in scope for this requirement.

Fortunately, besides CIP-007 R2 and CIP-010 R1, I think all other CIP requirements are risk-based, even when they don’t mention risk. For example, the many CIP requirements that point to a plan or process as their objective are risk-based, since developing and following a plan always requires identifying and managing risks.

I’m proposing that CIP-007 R2 and CIP-010 R1 both be rewritten as risk-based requirements. You might ask why I want to change them now, since the Risk-Management-for-Third-Party-Cloud-Services standards drafting team may well decide to change them on their own. I agree the SDT will feel compelled to revise these two standards themselves, but I also believe it will be 2031 (or even later) before all of the new and/or revised CIP standards they develop will come into effect.

CIP-007 R2 and CIP-010 R1 have been a burden on NERC entities for years, even without any consideration of using them in the cloud. Removing them from the Cloud Drafting Team’s agenda and creating a separate drafting team for them could easily cut 2-2 ½ years off the Cloud SDT’s work, as well as make it possible for NERC entities to comply with the two new risk-based standards at least 2-3 years before they would otherwise be able to. Even NERC entities that don’t plan to put BES Cyber Systems in the cloud anytime soon will benefit from this.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

My book "Introduction to SBOM and VEX" is available! For context, see this post.

---

[i] I used to say it’s impossible for CSPs to provide this evidence. I now say it’s difficult, but not necessarily impossible.

Et tu, CISA?

Yesterday, I pointed out that the National Vulnerability Database has not only failed to perform one of its most important jobs – adding CPE names to all new CVE records, a process called “enrichment” – this year, but they put an exclamation point on that record during the last two weeks of the year (ending yesterday 12/30) when they only added 15 of the 1,181 CPEs they should have added during that period. This means that, during those two weeks, they were only operating at a rate of 1%, vs. 45% in the previous 50 weeks (even 45% is abysmal, but it’s 45 times better than 1%, if my math is correct).

Of course, when there’s a shortfall like that, you have to wonder whether the NVD (which is part of NIST, which is itself part of the Department of Commerce) has simply given up on creating CPEs. As I explained in the post, a CVE record – i.e., an announcement of a newly-identified vulnerability, with a textual description of the software product in which it is found – without a CPE name is invisible to automated searches using that product name. And since there have been almost 40,000 new CVEs reported in 2024, manually searching through the text of each of those records is probably not going to be a lot of fun.

I honestly don’t think the NVD is completely giving up on the idea of creating more CPEs, simply because it would be quite hard for anyone there to justify holding onto their job if they really do that. On the other hand, I find it hard to believe the NVD will ever make up for the ground they’ve lost over this year, since they would not only have to go back to creating CPEs at their old pace, but they would have to at least triple that rate, for three reasons:

1.      The volume of new CVEs this year is close to double what it was last year. If that rate continues next year, the NVD will need to more than double their effort just to keep up.

2.      Since they added less than half the CPEs they were supposed to add this year, in order to catch up to where they should be, they need to multiply their doubled rate by a factor of at least two. This means they really have to triple or quadruple their effort from last year. Do you think they can do that?

3.      In 2023, there were about 28,000 new CVEs, and I believe the NVD assigned at least one CPE to each one of those. Yet this year, they assigned fewer than 18,000 CVEs, which is 10,000 fewer than last year. If anything, the trend line of CPE assignments is sharply downward, not upward. And given the current growth rate of CVEs, there could quite easily be over 50,000 new CVEs next year. Even if the NVD were able to get back to the 2023 rate of 28,000 per year, that still leaves their backlog at 22,000 at this time next year – which happens to be about their current backlog. No matter what it is, this certainly isn’t progress.

Meanwhile, there’s been another development that only adds to this problem. Someone pointed out to me today that CISA put up this announcement on GitHub a month ago: “…as of December 10, 2024, CISA will no longer be adding CPE strings to the enriched dataset.” You may know that within a couple of months after the NVD started having its problems in February, CISA gallantly announced that it was stepping up to help and would be adding CPE names and a couple other items (CVSS scores and CWEs, I believe) to CVE records – this was called the “Vulnrichment” program. While the program didn’t produce a huge number of CPE names, the fact that CISA was doing this was a morale booster for people who still had some faith in the NVD (a group that no longer included me by that time).

However, it seems CISA has changed its mind about producing CPEs (although not the other items). Of course, they’re not providing any explanation for why they changed their mind. But I can certainly guess that, given the many broken promises from the NVD this year, CISA is getting a little fed up with them. So, just as CISA’s announcement of the Vulnrichment program last spring was a morale booster, their repudiation of that announcement in December has to be taken as a motion of no confidence, in both the NVD and in CPE itself. I’ll note that just a few days earlier, CVE.org employees had pointed out, in their annual CNA Workshop (which I wrote a post about), that they’re trying to learn more about purl. Indeed!

What’s the solution to this problem? Remember, the NVD was the leading vulnerability database in the world less than a year ago. What is it now? You might be inclined to say it’s just a fraction of its former self, but I want to point out that even that is a wild overestimation. Just because the NVD has managed to add CPEs to about 45% of its new CVEs this year, that doesn’t mean it’s providing 45% of the value that it did before that; in fact, it’s providing much less value.

Consider why you search the NVD, or any other vulnerability database, for vulnerabilities that apply to a software product your organization uses. Are you expecting to learn about only 45% of the vulnerabilities that affect that product? I doubt it. Don’t you really want to learn about every vulnerability that affects the product? Wouldn’t it be much better if someone told you that the 55% of vulnerabilities that you won’t find by searching the NVD will be easy to find if you also search XYZ database? However, nobody can do that, since there are no machine-readable identifiers to tell anyone what’s in the approximately 22,000 CVE records that don’t have CPE names (i.e., the backlog).

There is one thing you can do. If you’re concerned mostly about open source products, you shouldn’t be looking for those in the NVD; in fact, you should never have been doing that in the first place. The OSV and OSS Index open source databases both have a much larger collection of open source vulnerabilities than does the NVD. Moreover, searches in those two vulnerability databases are much more reliable than they are in the NVD.

Why are these two databases more reliable than the NVD? It’s because they use a much more reliable identifier than CPE: purl. You can read why purl is so much more reliable than CPE in this post, but one difference that should be very obvious by now is the fact that nobody needs to “create” a purl. In other words, no single government agency can gum up international vulnerability management activities by stopping work without any explanation (not that this would ever happen, of course 😊).

An open source product that’s made available in a package manager already has a purl, since it can be created by anybody who has the spec and also knows the name of the package manager, as well as the name and version string for the product in that package manager. Of course, all three of those items are readily verifiable.

You may be wondering why, given that purl is so much more reliable than CPE and that CPE is rapidly going down the tubes, we can’t use purl now to search for vulnerabilities in the NVD? There are two reasons:

1. The CVE Numbering Authorities (CNAs), who report new vulnerabilities to CVE.org (from where the records are passed on to the NVD and other databases that are based on CVE identifiers), have until this past May not been allowed to include any software identifier other than CPE in a CVE record.[i] In May, the new CVE v5.1 spec included a field for a purl. However, there has been probably zero use of that field, mainly because currently there’s no vulnerability database that can utilize CVE records that include purls. If the CNAs add purls to CVE records today, the records will be all dressed up with nowhere to go.

2. Purl currently does not provide a way to identify commercial software products, but only open source products distributed in package managers. The only currently available identifier for commercial software products is CPE. It’s not a good identifier for commercial products (or for open source products either, of course), but it’s the only one available today. Faute de mieux, as the French say.

I’m proud to announce that the OWASP SBOM Forum is starting to work on both of these issues now. You can read how we’re working on the second issue in this post that I linked earlier. However, we won’t get very far without more participation, as well as financial support (which can be in the form of donations of $1,000 or more to OWASP – a 501(c)(3) nonprofit organization – that are “restricted” to the SBOM Forum). If you can contribute time or funds, please email me.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

My book "Introduction to SBOM and VEX" is now available in paperback and Kindle versions! For background on the book and the link to order it, see this post.

---

[i] If you aren’t intimately familiar with CVE records, CVE.org, the CNAs and similar exotica, you should go to this post.  

Has the NVD thrown in the towel?

I’ve been relying on three people to follow the ongoing saga with the National Vulnerability Database (NVD) for me: Andrey Lukashenkov of Vulners, Patrick Garrity of VulnCheck, and Bruce Lowenthal of Oracle. Early this morning (Chicago time) I emailed all three of them to ask for an update, since the last post I wrote on this subject (I’ve written many previous ones, which you can find by searching for NVD in the search bar at the top of this post) was on December 9.

Andrey replied first, but I must admit that he had an unfair advantage, since Patrick and Bruce both live in the US and Andrey lives in Spain. Regardless, I was shocked by what he told me: In the last two weeks of the year, the NVD “enriched” (i.e., added one or more CPEs to) only 15 CVE records out of a total 1,188. That’s just 1% (plus, all 15 were in the week beginning December 16; no CVEs at all were enriched during the last week in Andrey’s spreadsheet, which ended today, December 30).

To provide some perspective on this, in the first 50 weeks of this year, the NVD enriched 17,577 of 38,593 CVE records, or 45%. In previous years, the NVD enriched almost 100% of new CVE records, usually within a few days of receiving them from CVE.org. Of course, it was shocking that the NVD only enriched 45% of CVE records through mid-December, but – holiday or no holiday – it’s well beyond shocking that they seem to have literally thrown in the towel the last two weeks (after working during these weeks in previous years, with time off for the federal holidays).[i]

If you want to understand what this means, I’ll refer you back to the December 9 post. In brief, the problem is (and if you don’t understand any of the terms below, please read this post):

1.      New software vulnerabilities are reported to CVE.org by the CVE Numbering Authorities (CNAs). The CVE report includes a description of the vulnerability, a CVE number assigned to it by the CNA, and a textual description of the affected product or products.

2.      CVE.org includes the new record in their database and forwards it on to the NVD.

3.      Up until February 12, 2024, the NVD usually created the machine-readable CPE name, which is required for a user to look up vulnerabilities for a particular product in the NVD.

4.      If there is no CPE name attached to a CVE record in the NVD (as is the case with 55% of the new CVE records created this year), a user of the product that is described as affected in the text of the record will not see that CVE number when they search using the product’s CPE name. In other words, that CVE number will be invisible to searches for that product[ii].

5.      Of course, the user won’t see an error message when this happens. In fact, if that CVE is the only one listed for that product in the NVD, the user will just learn that there are no vulnerabilities that apply to that product. Of course, this isn’t true.

In summary, the fact that 55% of the 2024 CVE records in the NVD are invisible to automated product searches means that any search for vulnerabilities found in a product (including searches initiated by a scanning tool) is going to miss more than 55% of the vulnerabilities reported for the product this year (including the great majority of vulnerabilities reported after February 12).

It remains to be seen whether the NVD has given up on enriching CVE records altogether, but the fact that I even must mention that possibility shows how far the NVD has fallen. I don’t recommend that anybody bet the farm on the NVD coming back for perhaps a year or two, if even that. We need to move as quickly as possible toward introducing purl into the CVE ecosystem. More posts are coming on that subject.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

My book "Introduction to SBOM and VEX" is now available in paperback and Kindle versions! For background on the book and the link to order it, see this post.

---

[i] In case you were wondering if the NVD has deigned to explain to us why this happened, I’ll point out that the last entry on their “news updates” page is from November 15.

[ii] The fourth paragraph of the December 9 post mentions, “..a few firms like Vulners (Andrey’s employer) and VulnCheck have taken it upon themselves to add their own CPEs to some of the unenriched CVE records..” This means you will find more vulnerabilities applicable to a particular product if you search either of those databases.

Why do we need to replace NERC CIP-007 R2?

I have written two recent posts about the need to replace (not just revise) the patch management requirement CIP-007 R2. Since both posts were written for other purposes as well, I’d like to summarize what they say and add some context that I previously left out.

The first post from October pointed out that the two most prescriptive CIP requirements, CIP-007 Requirement R2 and CIP-010 Requirement R1, are without doubt also the two most burdensome; this is because of the huge amount of documentation that the NERC entity must produce to prove compliance with each of them. The evidence must be provided for every physical (and soon, virtual) device in scope.

Moreover, the two requirements pose the biggest impediment to implementation of medium and high impact BES Cyber Systems in the cloud. This is because no CSP could ever produce the required device-level evidence, even if the CSP’s patch and configuration management programs in fact followed all the steps mandated by the two requirements. As every NERC compliance professional knows, “If you didn’t document it, you didn’t do it.”

I went on to point out that both problems – the overwhelming documentation demands for on-premises systems and the fact that it is virtually impossible for a CSP to provide the compliance evidence required by the strict wording of the two requirements – can be solved by replacing the current prescriptive wording of each requirement with a risk-based (or “objective-based”, to use NERC’s preferred term) requirement.

Doing this won’t be hard for CIP-010 R1, since the objective of the current prescriptive requirement won’t change in the new risk-based version: reduction of the risk that an inadvertent configuration change in a system will leave it vulnerable to a cyberattack. The difference is that the objective will be achieved in the revised requirement through risk management, not through providing careful documentation of each instance of compliance, as is the case currently.

However, in the October post I noted that the objective of the current CIP-007 R2 is to ensure that every security patch released for any software or firmware product installed on a system in scope is applied, or that the vulnerability is otherwise mitigated. There is no consideration of the risk posed by the vulnerability that is addressed by the patch; if the supplier (or other patch source) has released a security patch within the previous 35 days, either the patch needs to be applied or the vulnerability it addresses needs to be mitigated in some other way. The NERC entity doesn’t have the option of pointing out that applying the patch will address little or no risk, and therefore it would be better for them to spend their time applying patches hat do address risk, even if they aren’t required for compliance with CIP-007 R2.

In other words, the problem with CIP-007 R2 is that patching isn’t a security objective in itself; it’s a mitigation of the risk posed by one or more unpatched software vulnerabilities. If CIP-007 R2 is going to be rewritten as risk-based, it must be a vulnerability risk management requirement, not a patch application requirement.

2. In this post in December, I asked what a CIP vulnerability management requirement would look like. I pointed out how much more efficient that would be than the current patch management requirement.  This is especially true because at most 6-7 percent of all vulnerabilities (CVEs) are ever actively exploited. This means that a policy of applying every security patch that is released for a software product inevitably leads to a lot of wasted effort. However, this is the policy that is required by the current CIP-007 R2.

As I pointed out in the December post, the annual volume of newly identified software vulnerabilities (CVEs) when the original CIP patch management requirement was developed was a small fraction of what it is currently. This meant that organizations didn’t have a big backlog of unapplied patches; therefore, it was possible to apply every security patch that had been released for a software product.

Today, because of the huge volume of newly identified vulnerabilities – 35,000 this year alone vs. about 29,000 in 2023 - the biggest concern in vulnerability management is prioritizing patch applications, so they are as efficient as possible. In other words, risk needs to be reduced by the greatest amount possible, given the resources available for applying patches. This is why I stated in my December post that the new CIP-007 R2 should require that the vulnerability risk management plan prioritize patch applications “…according to criteria listed in the plan”.

Note that I’m not calling for the CIP-007 R2 vulnerability risk management plans to require use of specific criteria to prioritize patching vulnerabilities based on the degree of risk they pose. However, if a NERC entity just follows the FIFO method (first in, first out) for applying patches, they should be cited for that, since the patch’s time of arrival isn’t an indicator of the risk of the vulnerability being patched.

The most widely used vulnerability prioritization criterion today is probably CVSS score. However, many vulnerability management professionals will tell you that CVSS isn’t a measure of risk, but of severity. Risk is some combination of likelihood and impact. CVSS assumes that likelihood = 100%, so it only estimates the worst case impact. This doesn’t mean that CVSS should be ignored by an organization that is trying to prioritize patches according to the risk posed by the vulnerability or vulnerabilities they mitigate. But it does mean that CVSS should never be the only criterion you look at, or even the first criterion.

The best way to measure the risk posed by a particular vulnerability is by determining whether it is currently being exploited or is likely to be exploited, using these two measures:

1.      Whether the vulnerability (CVE) is in CISA’s Known Exploited Vulnerabilities catalog, and

2.      The vulnerability’s current EPSS score, which provides the likelihood (as a percentage between 0 and 1) that the vulnerability will be exploited within the next 30 days[i].

Using one or both of the above in your vulnerability prioritization methodology is much better than relying solely on CVSS, but you can use other criteria as well. These can include:

A.     CVSS, or even better, one or more components of the CVSS score. For example, one of the components estimates the degree of effort required to exploit the vulnerability. You can just pull out that component and include it as on of you criteria.

B.     A measure of the importance of an asset. Of course, since we’re talking specifically about NERC CIP compliance now, all assets will be important. However, some will be less so (e.g., a system handling I/O to a purely distribution substation vs a primary EMS server).

C.      A measure of the exposure of an asset, such as whether it is directly exposed to the internet, or whether it is in a DMZ vs. an ESP.

Of course, there are many other possible prioritization criteria. An organization should decide the right combination of criteria for their needs (and different areas of the organization might use different criteria).

Replacing CIP-007 R2 with a vulnerability risk management requirement will help accomplish three goals:

1.      It will reduce the amount of software vulnerability risk experienced by NERC entities with high or medium impact CIP environments, since entities will be able to prioritize their patch management efforts according to the amount of risk mitigated by each patch.

2.      It will greatly reduce the amount of compliance documentation required for NERC entities with high or medium impact on-premises CIP environments.

3.      It will go a long way toward eliminating the biggest obstacle to deployment of CIP-protected systems in the cloud today: the fact that CIP-007 R2 and CIP-010 R1 both require the CSP to provide evidence of prescriptive actions taken on a huge number of individual physical and virtual cyber assets over a three-year CIP audit period.

Of course, the Project 2023-09 Risk Management for Third-Party Cloud Services Standards Drafting Team (SDT) is just about to start drafting revisions and/or additions to the CIP standards that will finally make use of the cloud completely “legal” on the OT sides of all NERC entities. Since that team will almost certainly need to rewrite CIP-007 R2 and CIP-010 R1, why propose that these two standards be rewritten separately from that effort? There are two reasons why:

1.      Since CIP version 5 was implemented in 2016, I have continually been told that CIP-007 R2 and CIP-010 R1 are by far the most burdensome of all the requirements that apply in on-premises CIP environments. In other words, we need to fix these two requirements as soon as possible, regardless of any consideration of cloud use in the future.

2.      I recently estimated that it will be 2031 – i.e., seven years from now - before the new or revised CIP standards for the cloud are implemented. Since I think that rewriting CIP-007 R2 and CIP-010 R1 alone will require around 2 to 2 1/2 years, and since this needs to be accomplished as soon as possible, it makes sense to have a separate Standards Authorization Request (SAR) and SDT for this purpose. In fact, taking this item off the current SDT’s agenda will probably trim that agenda by two years, meaning the final CIP “solution” for the cloud could be in place by 2029 instead of 2031.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

My book "Introduction to SBOM and VEX" is available! For context, see this post.

---

[i] For an excellent webinar recording that discusses EPSS, go here.

Was this the first risk-based CIP requirement?

On July 4, 2016, I wrote about a meeting I had just attended: the second meeting of the NERC “Project 2016-02  Modifications to CIP Standards” drafting team. Coincidentally, this team finally finished their work earlier this year, when the revised CIP standards to accommodate virtualization were approved by the NERC ballot body, approved by the NERC Board of Trustees and submitted to FERC for their approval.

This meeting occurred soon after an important turning point in the CIP standards:

1.        The CIP versions 5 and 6 standards had come into effect on July 1, 2016 – i.e., three days before I wrote the post.

2.        The “CIP Mods” team (as the new team came to be known) was created to address issues that had come to light as NERC entities were preparing for implementation of CIP v5 and v6.

In my post, I noted an issue that I had started writing about at the beginning of 2016:

…since the beginning of this year I have put out a few posts - such as this one and this one - that take a different tack. They don’t criticize the v5 wording, but do criticize the whole idea of what I call “prescriptive standards” – which is what all the CIP versions have been. It may be clear from these two posts that I think the prescriptive approach doesn’t work for cyber security standards, even though it might be fine for the other NERC standards. And since the requirements of CIP v5 and v6 are prescriptive ones (with two exceptions, which I’ll discuss below), it seemed certain that v7 will be prescriptive as well.

I cited two other posts in the above paragraph. Near the end of the first of those two posts (from February, 2016), I asked, “Is CIP v5/v6/v7 Sustainable?” I pointed to the new items that were on this new drafting team’s plate (or might be soon). I wondered whether they would result in even more prescriptive requirements (as if the CIP community didn’t have enough of those already). I added:

I have asked a small number of compliance people for NERC entities what percentage of every dollar they spent on CIP compliance actually went to cyber security and what percentage went to the paperwork required to prove compliance. Of course, there is no objective way to measure this, but the estimates I have received range from 35 percent to 70 percent. Maybe the average is 50 percent, but’s let’s assume my small sample is biased downward and the average is closer to 70 percent. Even this figure means that 30% of every dollar spent on CIP is only spent to prove compliance, not to improve cyber security. To phrase it differently, if we could figure out a way to get the percentage of spending that goes to security up to 90%, we would in principle have up to a 20% increase in security (budgets) without having to spend a single additional dollar.

I also quoted a respected utility cyber security manager who asked, when told that virtualization was next up in the new CIP standards lineup, “Why can’t we just be trusted to do the right thing, rather than have to spend a huge amount of time documenting our compliance for virtualized systems?”

That’s really the issue, isn’t it? Why should NERC entities have to spend so much time and money documenting compliance with prescriptive CIP requirements, when most other mandatory cybersecurity compliance regimes (PCI, HIPAA, etc.) have managed to develop requirements that let the entity focus on achieving an objective, not on taking a set of prescriptive steps to prove their compliance in every instance?

Of course, the answer to that question is that NERC entities should not have to spend so much time and money documenting compliance with prescriptive CIP requirements. How can the requirements be revised (or more importantly written differently in the first place) to eliminate this problem?

I didn’t answer that question in the February 2016 post, but I did in the July 4 post that referenced it. The second half of that post concentrated on the most important item on the drafting team’s agenda for that meeting: fixing the definition of “low impact external routable connectivity” (which was at the time affectionately known as LERC) found in the original version of CIP version 6[i].

LERC (a term that was invented for use with CIP v6) turned out to be a mistake and a big source of confusion when NERC entities prepared to comply with V6. Briefly, LERC was intended to be the low impact version of External Routable Connectivity (ERC), a term invented with CIP v5, which only applies in high and medium impact environments.

However, it turned out that the requirement where LERC applied (CIP-003-6 R2, Attachment 1 Section 3) implicitly allowed LERC to be “broken” by a number of factors; this led to long arguments between NERC entities and their auditors about whether LERC was completely broken by a particular measure like a data diode, or whether it was only partially broken by a measure like a firewall. In the first case, the entity would have been found compliant with the requirement; in the second case, it would not have.

It was quite interesting to watch the SDT, during the 2 1/2-day meeting, evolve from trying to make this a workable prescriptive requirement to realizing that couldn’t happen, since doing so would make both the requirement and the definition a hopeless mess. At that point, it seems “another observer” (although I can’t remember who that was) suggested that the SDT should “rewrite the requirement so that it simply stated that an entity with LERC needed to take steps to mitigate the risk posed by LERC and discuss different options for doing this in the Guidelines and Technical Basis section.”

In other words, the other observer realized it was a huge waste of time to torture the requirement until it yielded a fixed set of measures that the entity could take to “break” LERC (and thus comply with the prescriptive basis of the requirement). Instead, the person suggested it was better to “rewrite the requirement so it simply stated that an entity with LERC needed to take steps to mitigate the risk posed by LERC and discuss different options for doing this in the Guidelines and Technical Basis section.”[ii]

I found this to be quite interesting, because the other observer admitted it would never be possible to treat this as a purely prescriptive requirement, for which the Responsible Entity would be given an up-or-down judgment of compliance. Instead, the entity needed to decide what was the best way for them to mitigate the risk in question; it was up to the auditor to decide whether the measures the entity took sufficiently mitigated the risk.

Folks, this was heady stuff! Here was a revised NERC CIP requirement for which the goal of compliance was simply to mitigate the risk addressed by the requirement, without trying to a) prescribe the steps required to mitigate that risk or b) attempt to measure the total risk that was mitigated by the steps the entity took (which would have been a fool’s exercise, in any case).

In other words, this may have been the first NERC CIP requirement (and perhaps the first NERC requirement, period) whose objective is clearly risk mitigation[iii] and nothing more (there already were at least two requirements in CIP version 5 that were in fact risk-based, including CIP-007 R3 anti-malware and CIP-011 R1 information protection. However, neither one of them explicitly mentions risk or risk mitigation).

Since 2016, other risk-based (which NERC calls “objectives-based”) requirements, and even whole standards (CIP-012, CIP-013 and CIP-014) have been developed and put into effect. Moreover, I am reasonably sure that after prescriptive requirements like CIP-007 R2 and CIP-010 R3 were developed as part of CIP version 5 (in 2011 or 2012, probably), not a single new prescriptive CIP requirement has been developed. Moreover, I’m certain that no new prescriptive CIP requirements ever will be developed; there clearly is no more appetite for them in the NERC community.

This leaves open the big questions, How are risk-based CIP requirements being audited today? And how should they be audited? These questions are especially important because any new or revised CIP requirements that will apply to assets in the cloud will have to be risk-based. Will developing procedures for auditing risk-based CIP requirements require changes to the NERC Rules of Procedure? Given that I haven’t found anyone who can tell me exactly what is required to change the RoP, that might not be a lot of fun…

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

My book "Introduction to SBOM and VEX" is available! For context, see this post.

---

[i] That definition was “Direct user-initiated interactive access or a direct device-to-device connection to a low impact BES Cyber System(s) from a Cyber Asset outside the asset containing those low impact BES Cyber System(s) via a bi-directional routable protocol connection.”

[ii] The “Guidelines and Technical Basis” section of CIP-003 has been renamed “Supplemental Materials” and is now found at the end of the CIP-003-8 standard (the current version) starting on page 33. It provides diagrams of ten possible methods which the entity might take to mitigate the risk posed by LERC, without saying which of those is the “best” or the “most compliant”.

[iii] CIP-003-8 R2 Attachment 1 Section 3 doesn’t explicitly mention risk. However, it requires the entity to “Permit only necessary inbound and outbound electronic access as determined by the Responsible Entity.” Since the entity is allowed to determine for itself what constitutes “necessary” access and since any judgment about whether something is necessary must always consider risk, this makes it a risk-based requirement in my book.