Monday, March 25, 2024

“Fixing” the NVD is technically easy. But politically? Not so much.

For our regular meeting of the OWASP SBOM Forum last Friday, we had an open agenda as usual. Not surprisingly, everyone (including me) wanted to discuss the current problems with the National Vulnerability Database. For those who haven’t been keeping score at home, around mid-February the number of CVE reports added to the NVD dropped a lot – initially, to a small fraction of what they had been previously. Even more importantly, most of the reports that were added didn’t include the CPE identifier for the vulnerable product; this is needed to find out what software or devices a new CVE applies to. Having a CVE report without a CPE name (or any other identifier) for the affected product(s) is roughly equivalent to having a car without a steering wheel.

Fortunately, I’ve just heard that starting in March, most or all of the CVE reports that are being added have CPE names, although the volume of reports added is still less than a third of the previous average. This is better, but considering how critical the NVD is to software vulnerability efforts worldwide, some explanation should have been provided by now – but there has been none.

Fortunately, Tanya Brewer of NIST, the head of the NVD team, will be participating in a previously scheduled panel discussion on Wednesday at 2PM Eastern Time at the first (but certainly not the last) annual VulnCon conference; I would think she’ll say something about the problem then. There are still virtual seats available for $100 for the 3-day conference starting today, March 25. The conference is loaded with interesting presentations, at least if software vulnerabilities are important to what you do. You can sign up here. Note that anyone who has signed up for virtual access will be able to see streams of each day’s presentations, starting at the end of that day but continuing for the next 3-4 weeks. However, I was sorry to learn that virtual attendees aren’t allowed to attend the happy hour.

There were some very knowledgeable people in our meeting on Friday, including Pete Allor, Director of Product Security for Red Hat (and a member of the CVE.org board), Bruce Lowenthal, Director of Product Security for a major American software developer, and Steve Springett, leader of the OWASP CycloneDX and Dependency Track projects – whose side job is supervising software security for about 1,000 developers at ServiceNow. Other people contributed as well. Below is what I learned (I can’t guarantee these statements are completely accurate, but I’m sure I have the general picture right).

By the way, Pete is the founder and organizer of VulnCon. When he started musing last fall about having a conference just to discuss vulnerabilities, I thought they’d be lucky to get 100 people to discuss that topic. However, they sold out quickly for the onsite meeting (with at least 200 seats, I’m sure). There will also certainly be a lot of virtual attendees[i]. Pete deserves a lot of credit for putting this together; he thinks there could be at least 1,000 onsite attendees next year.

What caused the NVD’s problem? We won’t know until there’s some announcement to that effect, but around the time the slowdown began in mid-February, a new CVE Naming Authority (CNA) (see the post linked above for a description of what CNAs do) dumped a huge number of CVE reports on the NVD (it was in the thousands. Given that about 20,000 new CVEs are added to the NVD every year, this might have been almost half a year’s worth in one day). The reports were different from the normal ones, because they lacked a CVE number. Could this have caused the problem?

You might think it’s not surprising that a database would suffer serious problems if it suddenly had to process half a year’s worth of defective entries at once. But remember, databases are supposed to be resilient to these kinds of problems. Normally, I would expect the database to be able to reject invalid entries, rather than get bogged down trying to process them. Why didn’t that happen in this case?

But there’s a bigger question: My last post pointed out that every CVE report is submitted by a CNA. There are currently around 330 of these, all over the world. For example, Oracle and ServiceNow are both CNAs. Red Hat is one of five Root CNAs. The CNA reports to the CVE.org organization (popularly known as MITRE, since the staff is entirely composed of contractors from MITRE). The report describes a vulnerability that the CNA identified in one of its own products, or else in a product from another developer that isn’t a CNA, who came to the CNA for assistance in creating the report. The CNA’s primary job is to describe the vulnerability correctly in the report and assign a CVE number to it; they also need to describe the affected product(s) and version(s) accurately.

What the CNA doesn’t do is assign the CPE name (CPE is the software identifier on which the NVD is based) for the affected product and version, even though the CNA is usually the developer of the product. Instead, once the CNA has submitted their CVE report and it’s been entered in the CVE.org database, it goes to the NVD (which is part of NIST). There, a staff member creates the CPE using the supplier, product and version information entered on the report by the CNA. The NVD adds the CPE name(s) to the report, as well as the CVSS score and CWEs.

Here's the problem: It seems the NVD people who create the CPE name often make mistakes (or at least they make questionable decisions), and the CPE name ends up diverging from the CPE specification. Obviously, if the CPE name doesn’t follow the spec, it will be close to impossible to find it by a normal search.

Bruce Lowenthal has been pointing out regularly – in the 3+ years that I’ve known him – that the CNA, who developed the software in question, isn’t allowed to generate the CPE name. This creates a big problem for his company, because when the name created by NIST is incorrect, customers get mad at the developer, thinking they’re responsible for the error. So, the logical question is: Why not let the CNA create the CPE name and add it to the CVE report themselves? Why does the NVD have to be involved? It seems the NVD staff are literally contributing to the problem, not fixing it.

Bruce also makes similar comments about CVSS scores and CWE information. Those items are also added to the CVE report by NVD staff members, not the developer who reported the vulnerability in the first place. Creating those requires detailed understanding of the vulnerability; Bruce says his company could do a much better job of producing both CVSS cores and CWE information for their own products than the NVD staff members do. I’m sure many other developers would say the same thing.

Note that making this change would kill two birds with one stone. One is that it would result in more accurate CPE names. But the other is that it would go a long way toward fixing the immediate problem with the NVD – which is that they seem to be having trouble creating CPE names, period. Whatever is the cause of the problem, there’s no disputing that having the CNA create the CPE name – before the CVE report even goes to the NVD – would prevent the current problem from recurring.

But there’s another aspect of this problem: the fact that the NVD’s infrastructure, which is built on a foundation put in place more than two decades ago, is…how shall I say it?...not state of the art. For one thing, how many heavily used databases today aren’t totally redundant, so that if any piece of software or hardware fails, the database will be up and running very quickly, perhaps without users even knowing there was a problem? The answer? Not many. Yet I’ve heard for a couple years that there are multiple single points of failure in the NVD. Even though we don’t yet know what the NVD’s problem is, it’s clear that if the NVD had true high availability, they wouldn’t still be crippled by the problem, more than 40 days after it first appeared.

What’s ironic about this is there’s already another database that has all the information that the NVD does, yet it’s built on a much more robust, highly available infrastructure: CVE.org. All the information in the NVD is from CVE reports; those are first entered in CVE.org. Why should NIST continue to maintain the NVD and struggle (sometimes unsuccessfully, as in the past two months) to keep the thing running with duct tape and prayer, when they could just let CVE.org handle all the queries? That way, the NVD staff members could just concentrate on their other assigned task: adding CPE names, CVSS scores and CWE information to the CVE reports.

Of course, the answer to the question in the above paragraph should be clear when you read the last sentence. I pointed out earlier that Bruce Lowenthal and other CNAs want to take back responsibility for creating these three items from the NVD, since the developer that identified the vulnerability can create them much better than anybody else. Now I’m saying it would make much more sense for the NVD database to be shut down and replaced by an expanded CVE.org. What’s left for the NVD staff to do if they don’t do either of these things?

Good question. And that’s exactly why the NVD (and NIST) will fight any such proposal. But consider the following:

1.      Even if all their current responsibilities are taken away, the NVD staff members are all part of NIST. In case you didn’t know it, NIST does a huge amount of valuable work in the cybersecurity area. I strongly doubt those staff members would be laid off from NIST. Moreover, NIST will still receive the federal funding for the NVD, at least through the end of the fiscal year and maybe beyond that. Surely, they can figure out some creative way to keep the (now ex-) employees of the NVD gainfully employed for at least another few years.

2.      If CVE.org gets additional funding in future years, due to assuming the responsibility of operating “the database formerly known as NVD”, the NVD’s ex-employees will presumably get first consideration when they apply for a job at CVE.org.

3.      If all else fails and there is no longer any public employment available for ex-NVD staff members, there’s always the private sector. Given the experience and knowledge they’ve gained by working for the NVD, I doubt they’ll have any problem finding work.

If someone decided tomorrow that both above actions should be taken, how long would it take for them to be effective? Moving responsibility for CPE, CVSS and CWE additions from the NVD staff to the CNAs could be done very quickly. However, given there will need to be a lot of training as part of this, I think three months is a good estimate for it.

Moving the NVD to CVE.org will take more time, even though the overall structure of the database won’t change. This is because the querying facilities, data feeds, APIs, etc. that the NVD offers (not always successfully, though) will need to be reproduced in CVE.org. Since CVE.org can build on what is currently included in the NVD, I doubt this will take too long – perhaps six months. I think, based on technical considerations alone, the entire move from the NVD to CVE.org could be accomplished within six months.

Unfortunately, the problems in making this change won’t be technological but political. The NVD is part of the Department of Commerce and CVE.org is part of the Department of Homeland Security. Here’s Tom’s Law of Government: Whenever you have two Cabinet-level departments involved in a change, each with their own budget to protect, there will be resistance to that change from one or both of those departments.

The only governmental officer who can bang the appropriate heads together to effectuate that change is the one officer who outranks both Secretaries. That’s…you guessed it, the president! Even though the number of dollars and the number of staff members involved in this change isn’t even a rounding error in the tenth decimal point of the federal budget, the problem will probably require White House intervention. Only God knows how long that will take, but I’ll guess this will take at least 2-3 years (and 2-3 budget cycles, of course).

However, things aren’t as bad as they look. That’s because there’s nothing that either CVE.org or the NVD does that can’t be reproduced by an organization completely outside of government. For example, the CNAs are obviously a key part of this process, but they’re all volunteers. While the government has “CVE” trademarked, there’s nothing to prevent outside organizations from creating their own vulnerability identifier (perhaps “TOM”?) and having the CNAs create both “TOM Reports” and CVE Reports. They’ll forward the latter on to CVE.org (and through them, to the NVD), and combine the CVE and TOM data in a single vulnerability database. A good example of this is OSV, which was originally created by Google but is now a full-fledged open source vulnerability database. OSV has its own vulnerability identifiers, although the database also includes CVEs.

In fact, following this all-private path would bring an important added benefit: The new database could easily include purl identifiers as well as CPEs, because the OWASP SBOM Forum made a successful “pull request” to CVE.org in 2022. As a result, the most recent version of the “CVE JSON spec” (v 5.1) – which is already being used by some CNAs to create their CVE reports - allows purls to be added to the CVE reports, along with CPEs[ii]. That will solve a good portion of the naming problem in the NVD, which clearly will never be solved if the NVD is left to its own devices (it will be at least 2-3 years before the NVD will be prepared to adopt the 5.1 version of the CVE JSON spec, but a new private database would be able to adopt it from the start).

Of course, even if the entirely private route is taken, that alone will require at least a year, since we will need at least to give an all-government solution a chance to work first. There’s no question that this would be the best outcome. But I strongly doubt it will work.

 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

My book "Introduction to SBOM and VEX" is now available in paperback and Kindle versions! For background on the book and the link to order it, see this post.


[i] I’m finishing this post after I watched four of the sessions in Monday’s conference. It’s definitely worthwhile, and since you’ll be able to see all of the sessions via streaming, you don’t have to feel bad if you missed today’s sessions.

[ii] See this 2022 paper by the SBOM Forum.

No comments:

Post a Comment