The NVD responds to the concerns, although not well

 

Tanya Brewer of NIST, the leader of the National Vulnerability Database, spoke about their current problems at a very well-attended special session of the VulnCon conference on Wednesday[i]. Here is how she started out:

·        She didn’t explain what the problem was that came up in mid-February, although she said the NVD will explain it when the statement is approved (it sounds like that might be a couple of weeks). I think they might take some crisis management lessons from the White House and other government agencies. A cardinal rule is that you need to get as much information as you can in front of the public as soon as possible. The fact that the problem appeared six weeks ago and still hasn’t been explained publicly doesn’t exactly fit that bill.

·        She did say that the cause was some sort of “silly governmental problem”, which everyone would think was ridiculous if they knew about it. Unfortunately, the people in the audience (I attended virtually) didn’t seem to be enjoying a hearty belly laugh with this statement. This incident – which is ongoing, of course – has caused real pain to real people worldwide.  Moreover, by minimizing it like she did, she sent exactly the wrong message. People might have been comforted if they’d learned that what happened in February was an out-of-the-blue disaster that is unlikely to happen again. Instead, they heard it was something simple. That means it could easily happen again. Oops.

·        She emphasized that the NVD isn’t shut down. However, I’ve heard it’s still only operating at about a third of its normal level. Somebody in the audience pointed out that there are still about 4,000 CVE reports that have been filed with CVE.org[ii] but still haven’t appeared in the NVD. I have been told this is a huge increase from the NVD’s normal backlog.

She talked about some of the things she would like to do with the NVD, but emphasized that these are very hard to do. Once again, she sent exactly the wrong message:

·        She said she would like to support the JSON CVE 5.0 specification (the CVE JSON spec is what the CVE reports follow) but pointed out that it’s hard for them to implement this. She said the same thing to the OWASP SBOM Forum last May. It’s unfortunate that the NVD hasn’t been able to make progress on this problem since then, but it’s not at all surprising: The NVD, built on a foundation that was laid almost 25 years ago, will always have a hard time adapting any new or updated technologies, even though they could be implemented much quicker in a modern infrastructure.

·        She said she would like to implement the ability to use purl identifiers (to search for software in the NVD). This is also something she told the SBOM Forum last May. We advocated for this in the paper we published in September 2022. In that paper, we also pointed out that this goal could be achieved in the CVE JSON 5.1 specification. Tony Turner of the SBOM Forum submitted a pull request to make the required changes, before the 5.1 spec was finalized in 2022. The pull request was accepted, and the 5.1 spec includes those changes.

·        She also said they were trying to figure out how to make purl work in the NVD. In theory, they just need to support the 5.1 spec, but since they can’t even get 5.0 to work yet (she said they were most of the way there, but “most of the way” doesn’t help very much when you either support the spec or you don’t), it will clearly be several years before purl can be implemented in the NVD. But who knows whether it will ever be implemented, since the NVD seems to lurch from crisis to crisis (in her talk at VulnCon today, Tanya alluded to some sort of crisis that came up last summer. This was probably why she stopped communicating with the SBOM Forum, after having two very promising meetings with us)?

·        So, it will be years before the NVD can get the 5.1 spec working and incorporate purl searches. If you read the SBOM Forum’s paper, you’ll understand why this is such a problem; the CPE identifiers the NVD uses now are inadequate in many ways (see pages 4-6 of the paper). When we finalized the paper a year and a half ago, I honestly thought it would be 1-2 years at the most before the NVD would have purl working. Now, there’s a serious question whether they’ll ever be able to do that.

She then turned to the Consortium she’s proposing, which is more or less like what she described to the SBOM Forum last May (see my previous post, linked above). She said:

1.      The Consortium will only be open to organizations that are capable of spending months negotiating a Cooperative R&D Agreement (CRADA) with NIST. There will probably be a membership fee.

2.      The NVD will release a “letter of intent”, which will describe what they would like the Consortium to do. I assume that is close to what she said they wanted last May: Warm bodies that can work onsite for six months or so for free. They will first spend a couple of months learning the obscure bits of old technology that the NVD is based on, so they can then be of assistance in fixing problems. It’s clear that the NVD will tell the Consortium members what needs to be done, not the other way around (not that I would have expected anything different from any other government agency, of course).

Another issue: I won’t go through the details, but the audience members expressed a lot of dissatisfaction with the fact that the NVD insists on creating the CPE names, CVSS scores, and CWE lists that are added to the original CVE report. That report is created by the CVE Naming Authority (CNA), which is usually the developer of the software. Why not let the organization that discovered the vulnerability in their software create these three things and include them in the CVE report that they created?

Her short answer to this question was that these tasks are currently assigned to the NVD, but her longer answer implied that the CNAs can’t be trusted to be as objective in these matters – when it comes to their software – as the developers themselves. Of course, that stands to reason, but let’s put this in perspective:

Vulnerability databases depend on software and device suppliers reporting vulnerabilities in their own products. However, in another VulnCon session on Monday, two researchers reported that that only 10 software developers (all large, of course) report the majority of CVEs – in fact, more than the bottom 60% of all developers report. These ten developers are exactly the entities that the NVD doesn’t trust to be objective in evaluating their own products.

I don’t dispute that the developers will likely be biased toward minimizing the severity of vulnerabilities in their own software, but I want to compare these ten developers to a huge group of product suppliers that should also be reporting vulnerabilities to the NVD: manufacturers of intelligent devices.

It’s become clear to me that almost no manufacturers of intelligent devices (Cisco being the shining exception to this rule) ever report a vulnerability in their products. For example, Tom Pace of NetRise told the SBOM Forum in 2022 that he had identified one device – whose manufacturer has never reported a vulnerability for any of their products – that had at least 40,000 firmware vulnerabilities. He was clear that this was just a device he happened to study. He doesn’t have time to do a scientific survey of devices, but he expects he’d find similar results with almost any device.

In January, a fairly simple search in the NVD led me to realize that, of the top ten medical device makers worldwide, eight of them either never report vulnerabilities to CVE.org (aka MITRE) or they have only done so in a small number of cases. The other two device makers aren’t perfect, but they can’t be measured, since they’re part of much larger organizations that make lots of hardware and software products. There’s no way to isolate the numbers for just that organization’s medical devices in the NVD. Of course, given that medical devices are subject to more cyber regulation than any other devices or software, the fact that their manufacturers seldom if ever report vulnerabilities makes it almost certain that other device manufacturers almost never report them, either.

Think about this: The NVD doesn’t trust the ten organizations that are responsible for well over half of all vulnerabilities reported worldwide, while at the same time, device makers seem to have a clean bill of health when you look them up in the NVD. This is because, if a device has never reported a vulnerability, someone who searches the NVD for the device will see the same results as they would see if the device had never had a vulnerability (even though the device might well have 40,000 unreported vulnerabilities). Wouldn’t it be better if the NVD focused their efforts on expanding coverage of devices, rather than insisting that only the NVD should perform four tasks that the software developers could do much better, and which they want to perform?

Clearly the NVD has a lot of problems, but most of these are due to its old, creaking infrastructure. I pointed out in my last post that the CVE.org database has a much more modern infrastructure; moreover, that database already has all the data that the NVD does. This is because the CVE reports come to CVE.org first, before they get passed on to the NVD. Why not just make CVE.org the nation’s (and the world’s) premier vulnerability database? After all, CVE.org already supports the JSON 5.1 CVE spec, meaning it would be easy to add the ability to search on purl identifiers to it.

I certainly can’t say I’m sure that moving the NVD to the CVE.org infrastructure is the right thing to do now. But I am sure that the wrong thing is for the private sector to join in a big project to “Save the NVD!”, when the robust alternative to the NVD is already in place – it’s just in a different Department. Maybe there needs to be some sort of commission to study this issue, including people from both the Departments of Commerce and Homeland Security, plus someone from the White House to make sure a decision comes out of the process, and it’s fairly arrived at.

Meanwhile, the NVD obviously needs to get back on its feet, but let’s be clear: the Consortium will have nothing to do with that, since it won’t be in place for at least six months and probably longer than that. Tanya said her staff knows what needs to be done and is working on doing it. Fine. Let them do it. But we have to decide whether, in the longer term, the NVD should move to the CVE.org infrastructure or stay where it is.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

My book "Introduction to SBOM and VEX" is now available in paperback and Kindle versions! For background on the book and the link to order it, see this post.

---

[i] The session was different from the previously scheduled panel discussion she participated in on Wednesday afternoon, which didn’t address the NVD’s problems at all. The session where she spoke suddenly appeared on the conference calendar yesterday, replacing another one.

[ii] In my previous post (already linked), I tried to explain the complex structure linking the NVD (part of the Department of Commerce) and CVE.org (part of DHS). You might want to review that now, since calling it Byzantine is to slander Byzantium.

“Fixing” the NVD is technically easy. But politically? Not so much.

For our regular meeting of the OWASP SBOM Forum last Friday, we had an open agenda as usual. Not surprisingly, everyone (including me) wanted to discuss the current problems with the National Vulnerability Database. For those who haven’t been keeping score at home, around mid-February the number of CVE reports added to the NVD dropped a lot – initially, to a small fraction of what they had been previously. Even more importantly, most of the reports that were added didn’t include the CPE identifier for the vulnerable product; this is needed to find out what software or devices a new CVE applies to. Having a CVE report without a CPE name (or any other identifier) for the affected product(s) is roughly equivalent to having a car without a steering wheel.

Fortunately, I’ve just heard that starting in March, most or all of the CVE reports that are being added have CPE names, although the volume of reports added is still less than a third of the previous average. This is better, but considering how critical the NVD is to software vulnerability efforts worldwide, some explanation should have been provided by now – but there has been none.

Fortunately, Tanya Brewer of NIST, the head of the NVD team, will be participating in a previously scheduled panel discussion on Wednesday at 2PM Eastern Time at the first (but certainly not the last) annual VulnCon conference; I would think she’ll say something about the problem then. There are still virtual seats available for $100 for the 3-day conference starting today, March 25. The conference is loaded with interesting presentations, at least if software vulnerabilities are important to what you do. You can sign up here. Note that anyone who has signed up for virtual access will be able to see streams of each day’s presentations, starting at the end of that day but continuing for the next 3-4 weeks. However, I was sorry to learn that virtual attendees aren’t allowed to attend the happy hour.

There were some very knowledgeable people in our meeting on Friday, including Pete Allor, Director of Product Security for Red Hat (and a member of the CVE.org board), Bruce Lowenthal, Director of Product Security for a major American software developer, and Steve Springett, leader of the OWASP CycloneDX and Dependency Track projects – whose side job is supervising software security for about 1,000 developers at ServiceNow. Other people contributed as well. Below is what I learned (I can’t guarantee these statements are completely accurate, but I’m sure I have the general picture right).

By the way, Pete is the founder and organizer of VulnCon. When he started musing last fall about having a conference just to discuss vulnerabilities, I thought they’d be lucky to get 100 people to discuss that topic. However, they sold out quickly for the onsite meeting (with at least 200 seats, I’m sure). There will also certainly be a lot of virtual attendees[i]. Pete deserves a lot of credit for putting this together; he thinks there could be at least 1,000 onsite attendees next year.

What caused the NVD’s problem? We won’t know until there’s some announcement to that effect, but around the time the slowdown began in mid-February, a new CVE Naming Authority (CNA) (see the post linked above for a description of what CNAs do) dumped a huge number of CVE reports on the NVD (it was in the thousands. Given that about 20,000 new CVEs are added to the NVD every year, this might have been almost half a year’s worth in one day). The reports were different from the normal ones, because they lacked a CVE number. Could this have caused the problem?

You might think it’s not surprising that a database would suffer serious problems if it suddenly had to process half a year’s worth of defective entries at once. But remember, databases are supposed to be resilient to these kinds of problems. Normally, I would expect the database to be able to reject invalid entries, rather than get bogged down trying to process them. Why didn’t that happen in this case?

But there’s a bigger question: My last post pointed out that every CVE report is submitted by a CNA. There are currently around 330 of these, all over the world. For example, Oracle and ServiceNow are both CNAs. Red Hat is one of five Root CNAs. The CNA reports to the CVE.org organization (popularly known as MITRE, since the staff is entirely composed of contractors from MITRE). The report describes a vulnerability that the CNA identified in one of its own products, or else in a product from another developer that isn’t a CNA, who came to the CNA for assistance in creating the report. The CNA’s primary job is to describe the vulnerability correctly in the report and assign a CVE number to it; they also need to describe the affected product(s) and version(s) accurately.

What the CNA doesn’t do is assign the CPE name (CPE is the software identifier on which the NVD is based) for the affected product and version, even though the CNA is usually the developer of the product. Instead, once the CNA has submitted their CVE report and it’s been entered in the CVE.org database, it goes to the NVD (which is part of NIST). There, a staff member creates the CPE using the supplier, product and version information entered on the report by the CNA. The NVD adds the CPE name(s) to the report, as well as the CVSS score and CWEs.

Here's the problem: It seems the NVD people who create the CPE name often make mistakes (or at least they make questionable decisions), and the CPE name ends up diverging from the CPE specification. Obviously, if the CPE name doesn’t follow the spec, it will be close to impossible to find it by a normal search.

Bruce Lowenthal has been pointing out regularly – in the 3+ years that I’ve known him – that the CNA, who developed the software in question, isn’t allowed to generate the CPE name. This creates a big problem for his company, because when the name created by NIST is incorrect, customers get mad at the developer, thinking they’re responsible for the error. So, the logical question is: Why not let the CNA create the CPE name and add it to the CVE report themselves? Why does the NVD have to be involved? It seems the NVD staff are literally contributing to the problem, not fixing it.

Bruce also makes similar comments about CVSS scores and CWE information. Those items are also added to the CVE report by NVD staff members, not the developer who reported the vulnerability in the first place. Creating those requires detailed understanding of the vulnerability; Bruce says his company could do a much better job of producing both CVSS cores and CWE information for their own products than the NVD staff members do. I’m sure many other developers would say the same thing.

Note that making this change would kill two birds with one stone. One is that it would result in more accurate CPE names. But the other is that it would go a long way toward fixing the immediate problem with the NVD – which is that they seem to be having trouble creating CPE names, period. Whatever is the cause of the problem, there’s no disputing that having the CNA create the CPE name – before the CVE report even goes to the NVD – would prevent the current problem from recurring.

But there’s another aspect of this problem: the fact that the NVD’s infrastructure, which is built on a foundation put in place more than two decades ago, is…how shall I say it?...not state of the art. For one thing, how many heavily used databases today aren’t totally redundant, so that if any piece of software or hardware fails, the database will be up and running very quickly, perhaps without users even knowing there was a problem? The answer? Not many. Yet I’ve heard for a couple years that there are multiple single points of failure in the NVD. Even though we don’t yet know what the NVD’s problem is, it’s clear that if the NVD had true high availability, they wouldn’t still be crippled by the problem, more than 40 days after it first appeared.

What’s ironic about this is there’s already another database that has all the information that the NVD does, yet it’s built on a much more robust, highly available infrastructure: CVE.org. All the information in the NVD is from CVE reports; those are first entered in CVE.org. Why should NIST continue to maintain the NVD and struggle (sometimes unsuccessfully, as in the past two months) to keep the thing running with duct tape and prayer, when they could just let CVE.org handle all the queries? That way, the NVD staff members could just concentrate on their other assigned task: adding CPE names, CVSS scores and CWE information to the CVE reports.

Of course, the answer to the question in the above paragraph should be clear when you read the last sentence. I pointed out earlier that Bruce Lowenthal and other CNAs want to take back responsibility for creating these three items from the NVD, since the developer that identified the vulnerability can create them much better than anybody else. Now I’m saying it would make much more sense for the NVD database to be shut down and replaced by an expanded CVE.org. What’s left for the NVD staff to do if they don’t do either of these things?

Good question. And that’s exactly why the NVD (and NIST) will fight any such proposal. But consider the following:

1.      Even if all their current responsibilities are taken away, the NVD staff members are all part of NIST. In case you didn’t know it, NIST does a huge amount of valuable work in the cybersecurity area. I strongly doubt those staff members would be laid off from NIST. Moreover, NIST will still receive the federal funding for the NVD, at least through the end of the fiscal year and maybe beyond that. Surely, they can figure out some creative way to keep the (now ex-) employees of the NVD gainfully employed for at least another few years.

2.      If CVE.org gets additional funding in future years, due to assuming the responsibility of operating “the database formerly known as NVD”, the NVD’s ex-employees will presumably get first consideration when they apply for a job at CVE.org.

3.      If all else fails and there is no longer any public employment available for ex-NVD staff members, there’s always the private sector. Given the experience and knowledge they’ve gained by working for the NVD, I doubt they’ll have any problem finding work.

If someone decided tomorrow that both above actions should be taken, how long would it take for them to be effective? Moving responsibility for CPE, CVSS and CWE additions from the NVD staff to the CNAs could be done very quickly. However, given there will need to be a lot of training as part of this, I think three months is a good estimate for it.

Moving the NVD to CVE.org will take more time, even though the overall structure of the database won’t change. This is because the querying facilities, data feeds, APIs, etc. that the NVD offers (not always successfully, though) will need to be reproduced in CVE.org. Since CVE.org can build on what is currently included in the NVD, I doubt this will take too long – perhaps six months. I think, based on technical considerations alone, the entire move from the NVD to CVE.org could be accomplished within six months.

Unfortunately, the problems in making this change won’t be technological but political. The NVD is part of the Department of Commerce and CVE.org is part of the Department of Homeland Security. Here’s Tom’s Law of Government: Whenever you have two Cabinet-level departments involved in a change, each with their own budget to protect, there will be resistance to that change from one or both of those departments.

The only governmental officer who can bang the appropriate heads together to effectuate that change is the one officer who outranks both Secretaries. That’s…you guessed it, the president! Even though the number of dollars and the number of staff members involved in this change isn’t even a rounding error in the tenth decimal point of the federal budget, the problem will probably require White House intervention. Only God knows how long that will take, but I’ll guess this will take at least 2-3 years (and 2-3 budget cycles, of course).

However, things aren’t as bad as they look. That’s because there’s nothing that either CVE.org or the NVD does that can’t be reproduced by an organization completely outside of government. For example, the CNAs are obviously a key part of this process, but they’re all volunteers. While the government has “CVE” trademarked, there’s nothing to prevent outside organizations from creating their own vulnerability identifier (perhaps “TOM”?) and having the CNAs create both “TOM Reports” and CVE Reports. They’ll forward the latter on to CVE.org (and through them, to the NVD), and combine the CVE and TOM data in a single vulnerability database. A good example of this is OSV, which was originally created by Google but is now a full-fledged open source vulnerability database. OSV has its own vulnerability identifiers, although the database also includes CVEs.

In fact, following this all-private path would bring an important added benefit: The new database could easily include purl identifiers as well as CPEs, because the OWASP SBOM Forum made a successful “pull request” to CVE.org in 2022. As a result, the most recent version of the “CVE JSON spec” (v 5.1) – which is already being used by some CNAs to create their CVE reports - allows purls to be added to the CVE reports, along with CPEs[ii]. That will solve a good portion of the naming problem in the NVD, which clearly will never be solved if the NVD is left to its own devices (it will be at least 2-3 years before the NVD will be prepared to adopt the 5.1 version of the CVE JSON spec, but a new private database would be able to adopt it from the start).

Of course, even if the entirely private route is taken, that alone will require at least a year, since we will need at least to give an all-government solution a chance to work first. There’s no question that this would be the best outcome. But I strongly doubt it will work.

 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

My book "Introduction to SBOM and VEX" is now available in paperback and Kindle versions! For background on the book and the link to order it, see this post.

---

[i] I’m finishing this post after I watched four of the sessions in Monday’s conference. It’s definitely worthwhile, and since you’ll be able to see all of the sessions via streaming, you don’t have to feel bad if you missed today’s sessions.

[ii] See this 2022 paper by the SBOM Forum.

So, you want to help the NVD?

There has been a lot of talk on LinkedIn lately about helping the NVD out of its problems, despite the fact that the NVD hasn’t announced they need help, or indeed announced anything at all about those problems – which have been going on for more than a month now. I have no problem with people trying to help the NVD, since I agree that in the short term, the NVD is essential to worldwide software security. However, I’ve seen this movie before – in fact, I was one of the stars!

In early 2022, the OWASP SBOM Forum (then just the SBOM Forum) put out a white paper that has held up very well and is still downloaded very regularly. It’s about changes we suggest for the NVD to fix the serious naming problems that CPE causes (those problems are described on pages 4-6 of the paper). We proposed to fix these problems by including alternative identifiers for software and intelligent devices in the NVD, without eliminating CPEs themselves. CPEs themselves can’t be reformed, but they will probably never die out; nor does that need to happen. The existing CVE reports in the NVD, which go back about two decades and almost all of which include CPE names, need to be preserved and continue to be available.

After initially receiving support from CISA for the suggestions in our paper, we were later disappointed to realize that support had waned. So, we reached out directly to Tanya Brewer of NIST, the leader of the NVD. We asked to meet with her to discuss how we might help the NVD with the naming problem (for those of you keeping score at home, NIST/NVD is part of the Dept. of Commerce, while CISA and CVE/MITRE are part of DHS).

Tanya was quite pleased to do that. She said they had read our paper and wanted to discuss it with us. We ended up having two meetings with her, which I described in four posts: first, second, third and fourth. The first two posts were written soon after the first meeting with Tanya. That meeting was quite upbeat, although Tanya specifically warned us about the following (the words below are not directly hers, but include my interpretations of what she said):

1.      Private organizations can’t provide money directly to federal agencies for use by the agency, to go into their general budget. What private organizations can do, at least with NIST, is negotiate and sign a Cooperative Research and Development Agreement (CRADA), which sets out specific research and/or development goals that NIST and the private organization will pursue together. However, no funds from the CRADA can go toward normal expenses of the agency (specifically the NVD in this case), meaning there will probably be no way for private organizations to provide money to NIST to alleviate their current crisis. If the problem is primarily due to lack of funds (which I doubt), the NVD will have to find the funds somewhere else, maybe as some sort of emergency allocation from NIST.

2.      Private organizations also can’t provide advice to federal agencies without following FACA, the Federal Advisory Commission Act. There are a huge number of hoops to jump through in order to form such a Commission: here they are. I imagine it would take at least a year to do that.

3.      Well, how about trying to get Congress to increase funding? That has three problems:

a.      Tanya made it clear that no federal employee can lobby Congress in any way, since they’ll almost surely be fired for doing so. They also can’t lobby through a third party. She didn’t forbid us from trying to lobby Congress, but her name needs to be entirely kept out of it.

b.      The government follows a fiscal year that starts in October. Since the budget is supposed to be in place at the beginning of the previous fiscal year, (it’s been years since that actually happened, of course), it’s too late to have any impact on budgets for the 2024-2025 FY. The earliest that any change could be made to the NVD’s funding would be starting in October of 2025. And you’d better get moving quickly if you want to be successful in doing that.

c.      Plus, if you do more than write a few letters to Congressmen or Senators, you have to be careful about not overstepping the line to becoming a lobbyist – without registering as one. A lot of people have gotten into legal trouble for this reason.

My third post was more downbeat, although not because of anything the NVD did or didn’t do. It was right at the moment last April/May when there was a serious possibility that the government would not only shut down, but default on its debts. This was just like a married couple arguing about whether they should pay their credit card bills. A last minute bipartisan agreement (between the President and the four leaders of Congress, from both major parties) was reached in May.

The agreement was supposed to cover the 2024-25 fiscal year, avoiding another possible government shutdown around now. However, within weeks, one party had already started to make clear they didn’t feel like following the agreement – with the result that, if a new agreement isn’t reached by Friday, most of the government, including the military, will shut down. This time, the Department of Commerce won’t shut down, so NIST and the NVD can continue to do what they’re doing.

However, the NVD’s current problem is entwined with CVE/MITRE. DHS will shut down, meaning all DHS employees (including CISA employees and MITRE contractors) are forbidden to work, even if they’re willing to do so without pay. So if there is a shutdown, CVE/MITRE will be closed, greatly complicating the task of fixing whatever the NVD’s problem is (which I’m sure is technical. It’s not due to the fact that they’re underfunded, since that’s been true for a decade or more).

There’s a specific twist that applies to MITRE: While government employees, including military personnel, are sure to receive back pay whenever the government reopens, contractors like those from MITRE will receive nada, although MITRE itself is presumably paying them a salary. However, and independent contractors (and the federal government is loaded with them) will be without any income (other than unemployment compensation) will never receive the pay that they missed during the shutdown. 

Let’s be clear: The fact that the government keeps having these shutdown scares (which happen regularly. The last was for 35 days in 2018-2019) must be having a debilitating effect on both the government employees who work for the NVD and especially the MITRE contractors who work for CVE.org. I don't think that's the primary cause for the recent problems, but it certainly must affect the motivation of the staff members (of CVE/MITRE and the NVD) who are supposed to be figuring out what the problem is. I would wonder, "Why the h___ do I need to bust my a__ to fix a problem with the NVD, when the country's elected representatives think the whole government should shut down?" And I would certainly be shopping my resume around at this time.

Some people talk loosely about how “we’ll all be better off if the government shuts down”. If you’re one of them, you should be ecstatic about the effect your loose talk is having in the real world – unless, of course, you think it’s important for there to be a well-managed, smoothly-running government-led vulnerability database. I honestly don’t know how either the NVD or MITRE can continue to hire new people, given that cybersecurity expertise is still in high demand in the private sector, when they know that at least once a year (and sometimes more often than that, like last year and probably this one as well), there will be a serious question whether they will even have a paycheck for some unknown number of weeks. Expect continuing problems like this in the NVD, unless everybody in Congress agrees that negotiations are the way to settle policy differences, not playing chicken with the well-being of the millions of people who work for the government.

My fourth post was written last June, after Tanya had returned to the SBOM Forum to tell us what she learned about public-private partnerships at NIST. She seemed to have good news: she described a Consortium she was going to set up, which would give private sector organizations a sounding board with the NVD and allow them to set up a CRADA to do research with the NVD, presumably on how to fix their 20+-year old infrastructure so it doesn’t keep crashing.

However, what she was clearly focusing on was a third way for the private sector to help: They could provide coders free of charge to the NVD. She wanted the coders to commit to spending six months there (I believe it had to be onsite). That wouldn’t be terrible for a big organization like Oracle or Microsoft, if it provided the coder a significant learning opportunity and an opportunity to do meaningful work. It would of course be out of the question for most smaller organizations to pay an employee to work for someone else for six months.

But the problem with this “offer” was that the “learning opportunity” was to learn an obscure old language I’d never heard of; evidently, this language is used in the foundations of much of the NVD. And the “meaningful work” (the words in quotes above are my language, not hers) would of course involve coding in that language. For a young coder (and I am neither of these), it might be an exciting opportunity just to work for 6 months in DC (although I don’t recommend the summers!). However, I wouldn’t call it a career-advancing step, unless your career is working for the NVD.

I consider this a bad sign for anyone who wants to help the NVD. That database has been around for about 25 years. Unfortunately, databases age like milk, not fine wine. There’s a lot of technical debt that has to be paid before they can even think of fixing the naming problem, etc. And my guess is that in February, that technical debt (or some of it) came due, resulting in the huge drop in their productivity. It’s a worse sign that, more than four weeks after this event must have happened, they still don’t know what the cause is – or if they do, they haven’t announced it to the vast unwashed masses like me and you.

And it’s an even worse sign that they haven’t bothered to make any announcement at all after more than 30 days, except to say that Tanya hopes (but doesn’t promise) that she’ll be able to say something this week.

Folks, the NVD clearly won’t be the foundation of the global vulnerability database that the world needs, although the NVD shouldn’t and won’t go away, either. Fortunately, I don’t think the GVD will have to be a huge project – but even if it is, it will be tiny compared to the project (and the risk) of building a great 21^st-century database on top of a creaking (and now crumbling) late 20^th century foundation called the NVD.

The NVD needs to remain in existence, and at least for now it needs to keep creating CPE names and adding them to CVE reports. But they haven’t been doing that for a month, and they haven’t even bothered to explain what the problem is. Until they’re willing and able to explain the problem, and until they ask for private sector help (and can identify a legal means to provide it), I think it’s a waste of time to even talk about providing help to them.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

My book "Introduction to SBOM and VEX" is now available in paperback and Kindle versions! For background on the book and the link to order it, see this post.

 

It’s time to move beyond the NVD

I learned about a week ago that, starting around the middle of February, the National Vulnerability Database (NVD), the most widely used vulnerability database in the world, seems to have stopped adding CPE names to the great majority of CVE records. CPE (common platform enumeration) is the only type of identifier for software and devices used by the NVD, and CVE of course refers to a vulnerability documented by CVE.org (formerly just referred to as “MITRE”).

What does this mean? Here’s how vulnerability data reaches the NVD normally:

1.      CVE.org (part of the Department of Homeland Security. It is funded by CISA and governed by a board that includes people from private industry and government) has approved over 300 CNAs, which stands for CVE Numbering Authority.

2.      When a software or device manufacturer identifies a new vulnerability in one of their products, they work with a CNA to create a CVE report. The report includes a CVE number (assigned by the CNA), a description of the vulnerability and an identification of the affected product (e.g., “Product X Version Y, produced by Developer/Manufacturer Z”).

3.      The contents of this report are added to the CVE.org database.

4.      The report is then passed on to the NVD. The NVD is part of NIST, which is in turn part of the Department of Commerce.

5.      A NIST employee that is part of the NVD team (which only has about 20 people, believe it or not) enhances the CVE report in a few ways, most crucially by assigning a CPE name(s) to the product(s) identified in the report.

6.      The CPE name is required for the new CVE to be linked to a product/version listed in the NVD. Unless there is a CPE name on the report, the new CVE will never appear in an NVD search – except when the user searches on the CVE itself (in which case they’ll just see what is already in the report).

And this is the problem: Even though a new vulnerability has been duly reported to CVE.org and the report includes a description of one or more products that are affected by the vulnerability, an NVD user will normally never learn about that vulnerability, unless a CPE name is assigned to every product/version in the report. This is because the great majority of NVD searches are intended to find vulnerabilities that apply to a product/version. If there is no CPE name for the product/version they are searching for, the user will never learn of any CVE that applies to it, even if one has been reported in a CVE report.

I believe (although I may be wrong about this) that the only way this user could learn about a “CPE-less” CVE report that applies to the product/version they are interested in is to do a string search for “Product XYZ Version 4.7” in the CVE.org database. Any CVE whose report includes that exact string will appear (but, for example, if the string in the report reads “Product XYZ, v4.70”, it will probably never show up in the search). The chance of success in this search is slim, and in any case that search can’t be automated. So, doing this for 1,000 product/versions is out of the question.

Therefore, the results of almost all NVD searches for vulnerabilities, that apply to any software or device product/version, are now suspect and need to be taken with a shakerful of salt. This is because there is no way to know whether there are new (since Feb. 15) vulnerabilities that apply to that product/version, but which aren’t linked to a CPE name for the product/version. Given the importance of the NVD in almost all vulnerability management activities today (including using SBOM and VEX for component vulnerability management purposes), this is as close to a catastrophe for software vulnerability management as anything I’ve seen.

If this situation continues, the software security community can’t count on the NVD to always have recent vulnerability information; any discussion of the results of an NVD search will need to have an asterisk pointing to “Does not include most CVEs identified after February 15, 2024.” The need for this will continue until the NVD starts adding CPE names to CVE reports again, and retroactively adds them to all reports that don’t already contain them. In other words, not only is the NVD in a hole now, but they’re enlarging the hole as fast as they can.

Assuming this continues, a lot of (perhaps most) processes and procedures having to do with vulnerability management can no longer be trusted to be comprehensive. This includes most vulnerability scans, since the scanner will usually look in the NVD for vulnerabilities that have been identified for the product/version being scanned. It also includes monthly patch releases, since the software developer can no longer be certain that the release any significant vulnerabilities that were identified after Feb. 15. It may be that some of these processes will need to be abandoned altogether, since getting an admittedly incomplete list of software vulnerabilities in a product/version might be considered in many use cases to be a waste of time.

When I heard about this last week, I thought it might be just a short-term glitch; so I put the problem out of my mind. However, this week the problem has become widely discussed on LinkedIn, in posts by Dan Lorenc of Chainguard (which has drawn a huge number of comments) Chris Hughes, and the company NetRise (I have commented on all three of these). The tenor of these posts, and of the comments, is something to the effect of “Holy s___. What do we do now?”

You may be wondering what the NVD is saying about this problem. As far as I know, the only notice they have put out on the problem is this one, which has been on the site for at least a week or two:

NIST is currently working to establish a consortium to address challenges in the NVD program and develop improved tools and methods. You will temporarily see delays in analysis efforts during this transition. We apologize for the inconvenience and ask for your patience as we work to improve the NVD program.

At first, this might seem comforting. After all, at least the notice shows that NIST understands the problem and is working on solving it. The first step in the solution seems to be establishing a “consortium” that will develop “improved tools and methods”. It’s unfortunate that there will be a temporary degradation of service – but at least there’s light at the end of the tunnel. Or so the announcement would have you believe.

Not so fast. I have a story to tell you:

Once upon a time (in the fall of 2022, to be exact), the OWASP SBOM Forum - at that time just called the SBOM Forum - published a white paper on how to ameliorate the naming problem in the NVD. While the paper got a lot of recognition, including an initial enthusiastic reception by a key person in CISA (as well as people in MITRE), it became clear to us in early 2023 that CISA had no intent of pursuing these ideas any further.

Since we considered this to be a serious problem, we reached out directly to the NVD (through one of our members who is on the board of CVE.org, which also includes NIST people) and asked Tanya Brewer, who leads the NVD, if she could meet with us. She was not only willing but pleased to do this. She said they had read our paper carefully and would like to discuss it with us.

We had three meetings with her (and several other NVD staff members). These three posts - first, second, and third – described our conversations with Tanya, all in the first half of 2023. In the third post, I relayed the news that NIST would soon come out with a video asking for suggestions on a public-private partnership (which she was calling a “consortium”) to fix the problems (as well as accomplish other goals, to be sure). After receiving and digesting the comments, NIST would probably post a Federal Register notice about their plans by late August 2023. Tanya thought the Consortium could be up and running by early 2024.

We heard from other people in government that this goal was probably overly ambitious, given the significant amount of red tape involved with forming a public-private partnership of any kind with a federal agency. However, we were impressed by her sincere desire to put this in place.

She said this to us in late June 2023. Since then, we have had some communication with her about technical issues with the NVD feeds – which she has been very proactive about addressing, along with appropriate staff members. However, we haven’t heard anything more about the consortium, and I assumed it was dead, until reading this notice.

But there was more than that. Not only did I assume the consortium was dead, but I wrote off the idea of working with the NVD to fix their problems. I was looking at the timeline below, which was implied by our discussions with Tanya. This timeline is almost certainly optimistic, yet even then, it’s appalling:

1.      The NVD announces the consortium in videos (which Tanya said would be her first step), then takes comments on those videos. They then analyze those comments and use the results to write a Federal Register notice about the Consortium (which in itself takes a month or two to be posted). This process is probably six months.

2.      Organizations interested in participating in the consortium have a few months to submit an application. The applications are evaluated; then each organization that is accepted will have to sign a Cooperative Research and Development Agreement (CRADA) with NIST. This will take six months at a minimum.

3.      It’s only at that point that discussions can even begin about a) what problems need to be addressed in the NVD and b) what needs to be done to address them. Let’s say these discussions take one year.

Let’s assume that the Consortium reads the OWASP SBOM Forum’s white paper, falls in love with it, and decides to implement it lock, stock and barrel. How long will it take to do that? I won’t discuss everything we proposed in the white paper, but let’s look at the most important part of our proposal: that purl identifiers be incorporated into the NVD. What would be the timeline for implementing that part alone?

1.      To implement purl in the NVD, purl identifiers have to be included in CVE reports. That means the CVE JSON spec needs to be changed to accommodate them. I believe CVE reports are now being produced in the CVE JSON 5.0 spec, but last year I heard the NVD was struggling to fully support that, so I’m not sure 5.0 is even supported yet.

2.      Two years ago, the SBOM Forum submitted a pull request to the group that maintains the CVE JSON format, to add purl identifiers to the format. The PR was accepted and will be designated v5.1. However, the format isn’t being used yet, since the NVD can’t accommodate it now (and if the NVD hasn’t fully implemented the 5.0 spec yet, they have to finish that before they can move on to 5.1). When will the NVD be able to do that? Given what’s happened in the last month, the answer is probably “never”. But let’s be charitable and say it will be one more year before CVE reports will be produced using the 5.1 format, and before they’ll be ingested by the NVD. That’s also wildly optimistic.

3.      Even when it’s possible to use purl identifiers for NVD searches due to the 5.1 spec being accepted by the NVD, that doesn’t mean the CNAs and the NVD staff will understand purl. Purl is a very different identifier from CPE (see the discussion of intrinsic identifiers like purl vs. extrinsic identifiers like CPE in the first part of the SBOM Forum’s white paper). There will have to be training on this, as well as a big ramp-up period. Let’s assume that full use of purls takes another year to accomplish.

Thus, to get to the point where at least a partial solution to the NVD’s problems is in place, it will take at least four years, and probably a good deal more than that (as evidenced by the fact that the NVD is already nine months behind the schedule Tanya outlined to the SBOM Forum last June). And this assumes the whole process starts today, which of course isn’t realistic.

Even worse, I strongly doubt that, even after going through all the above steps, the result will be worth the wait. Hanging over this are two things that I realized from our discussions with Tanya last year:

First, the NVD has a huge technical debt to pay off. Tanya seemed most interested in having Consortium members provide warm bodies (presumably with intelligent brains attached) to help them with making changes to their infrastructure. She said they want any such warm body for at least a six month commitment, mainly because they are going to have to learn a lot of arcane stuff about the NVD (it’s 2-3 decades old).

This includes learning some old language that seems to be fundamental to the NVD, that I’d never heard of (and I know something about old languages. The first program I ever wrote was in Fortran IV with the WATFOR compiler, for a mainframe at the University of Chicago that took up an entire building and probably had a tiny fraction of the memory on my cell phone. The program had to be typed on punch cards and submitted at a window in the building. Then, you would go back to your dorm and come back later that night to retrieve the printout along with the card deck, only to learn that you missed a comma in one of the lines in the program and it “abended”, meaning it abnormally ended. You then had to retype that card and resubmit the deck, until your program finished successfully. This was “interactive computing” in those days).

Second, Tanya is very concerned about strengthening CPE names and making them more widely used, not less. The SBOM Forum paper (in pages 4-6) shows that CPE is hopeless; it will never be fixable, since CPE names are created by human beings with imperfect guidance and - unlike purls - don’t line up with anything in the real world; moreover, that will never change. The SBOM Forum doesn’t want to end all use of CPEs, because there is so much vulnerability information already tied to them. But we also don’t want to strengthen or increase use of CPEs, any more than we want to promote new COBOL applications.

It seems that whoever wrote the NVD notice above had eaten too many magic mushrooms. Do they actually think it’s acceptable to have the vulnerability database the world relies on be down for over four years, with no clear replacement available? Are they thinking at all?

I’ll answer this question for them: No, this is not acceptable. The NVD needs to continue to operate at least in the near term, but it can no longer be accepted as the most important vulnerability database worldwide. There needs to be a short-term solution, a database in which new CVEs will include purl identifiers for open source software, and other identifiers for proprietary (closed source) software, as well as intelligent devices.

But there needs to be a long-term solution as well. It needs to be internationally supported, but it can't depend on government funding (although governments are welcome to participate in funding it). I’ve already proposed one such solution, that might be up and running in less than a year. I’m not saying we must go with that alternative, but I am saying there’s no longer any reason why we need to put up with the NVD’s continual delays and creaky infrastructure.

Before anything can happen, the vulnerability management community needs to discuss the NVD situation and what we can do about it, both in the short and long terms. I know the OWASP SBOM Forum will be discussing this in at least our next 2-3 meetings. If you aren’t a member and would like to join us (the calls are at 1PM ET on Fridays – which I know isn’t great for people in Europe), please drop me an email.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

My book "Introduction to SBOM and VEX" is now available in paperback and Kindle versions! For background on the book and the link to order it, see this post.

 

Will the lawyers squash SBOMs?

During the weekly meeting of the OWASP SBOM Forum this week, I had a discussion (although not quite an argument) with two well known people in the world of software supply chain security. We were discussing obstacles that are preventing regular distribution of SBOMs and VEX documents to software users. One of them brought up the fact that lawyers at software developers and intelligent device manufacturers are often resolutely opposed to distributing those documents, for fear of a customer suing the company because of a mistake or omission.

I pointed out that there’s nothing illegal about making a misstatement – or even an outright lie – on an SBOM, but we all agreed this wouldn’t stop someone who was determined to sue. This is especially true if they’re sure they can prove they have been damaged in some way.

I then remembered that I wrote a post last year about my realization that it will be at least a year (and now I’d say 2-3 years) before best practices regarding SBOM and VEX are clear enough that real contract language regarding them will even be possible. The only contractual term that should even be considered now is one stating that the supplier will provide SBOMs and VEX documents on an experimental basis. For their part, the customer needs to agree with this statement and promise they won’t base operational decisions on the contents of the documents.

In fact, I don’t think most suppliers will even start regularly distributing SBOMs and VEXes (i.e., not just providing a single SBOM) unless they have contractual provisions like this in place with their customers.

But at least one of my friends said that lawyers, at least those that work for publicly-traded commercial software and intelligent device suppliers, won’t even be satisfied with contractual provisions like this; they simply won’t let their companies distribute SBOMs or VEXes – not no way, not no how. At that point, I wondered why we were all spending so much time discussing SBOMs, if they’ll never be made available to end users (or to third party service providers that “process” the SBOMs and VEXes on behalf of the end users, while providing the users with up-to-date information about the exploitability status of component vulnerabilities in one or more product/versions they utilize. This idea, along with the idea of a proof of concept that the OWASP SBOM Forum hopes to initiate later this year or perhaps early in 2025, is discussed in Part 3 of my book, “Introduction to SBOM and VEX”).

It is indisputable that, because of this problem, some software companies (especially the large public companies) will be unable to distribute SBOMs and VEXes for their products for years; in fact, there are still a lot of companies – and especially intelligent device manufacturers – that have literally never reported a vulnerability for their products. Obviously, if they won’t even report vulnerabilities today, SBOM and VEX are out of the question.

However, it’s also indisputable that some companies will see the opportunity to demonstrate to their customers that they want to be transparent about the status of vulnerabilities in their products, whether due to code the supplier wrote or to code contained in third party components (this includes at least a few of the largest software and intelligent device suppliers). This will be especially true if they have customer contract provisions in place like the one I just described. The SBOM Forum believes that, once code is available to produce and parse “tight” VEX specifications for both the CSAF and CycloneDX VEX platforms – and Anthony Harrison of our group already has preliminary code for CSAF – it will be possible for one or more service providers to perform the SBOM and VEX “processing” service described above (and in Part 3 of my book).

We hope to demonstrate that this is indeed possible in a proof of concept this year or next. Assuming that is successful, we hope that both SBOMs and VEXes will be “launched” in the real world, even though only a small number of suppliers are providing them regularly. As other suppliers see that it’s possible to do this, and as they put the required contractual protections in place, we expect that number to grow – at first slowly and then rapidly. If you’re interested in participating in this effort, let me know.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

My book "Introduction to SBOM and VEX" is now available in paperback and Kindle versions! For background on the book and the link to order it, see this post.