For our regular meeting of the OWASP SBOM Forum last Friday, we had an open agenda as usual. Not surprisingly, everyone (including me) wanted to discuss the current problems with the National Vulnerability Database. For those who haven’t been keeping score at home, around mid-February the number of CVE reports added to the NVD dropped a lot – initially, to a small fraction of what they had been previously. Even more importantly, most of the reports that were added didn’t include the CPE identifier for the vulnerable product; this is needed to find out what software or devices a new CVE applies to. Having a CVE report without a CPE name (or any other identifier) for the affected product(s) is roughly equivalent to having a car without a steering wheel.
Fortunately, I’ve just heard that starting in March, most or
all of the CVE reports that are being added have CPE names, although the volume
of reports added is still less than a third of the previous average. This is
better, but considering how critical the NVD is to software vulnerability efforts
worldwide, some explanation should have been provided by now – but there has
been none.
Fortunately, Tanya Brewer of NIST, the head of the NVD team,
will be participating in a previously scheduled panel discussion on Wednesday
at 2PM Eastern Time at the first (but certainly not the last) annual VulnCon
conference; I would think she’ll say something about the problem then. There
are still virtual seats available for $100 for the 3-day conference starting
today, March 25. The conference is loaded with interesting presentations, at
least if software vulnerabilities are important to what you do. You can sign up
here. Note that anyone who
has signed up for virtual access will be able to see streams of each day’s
presentations, starting at the end of that day but continuing for the next 3-4
weeks. However, I was sorry to learn that virtual attendees aren’t allowed to
attend the happy hour.
There were some very knowledgeable
people in our meeting on Friday, including Pete Allor, Director of Product
Security for Red Hat (and a member of the CVE.org board), Bruce Lowenthal,
Director of Product Security for a major American software developer, and Steve
Springett, leader of the OWASP CycloneDX and Dependency Track projects – whose
side job is supervising software security for about 1,000 developers at
ServiceNow. Other people contributed as well. Below is what I learned (I can’t guarantee
these statements are completely accurate, but I’m sure I have the general
picture right).
By the way, Pete is the founder
and organizer of VulnCon. When he started musing last fall about having a
conference just to discuss vulnerabilities, I thought they’d be lucky to get 100
people to discuss that topic. However, they sold out quickly for the onsite
meeting (with at least 200 seats, I’m sure). There will also certainly be a lot
of virtual attendees[i]. Pete deserves a lot
of credit for putting this together; he thinks there could be at least 1,000
onsite attendees next year.
What caused the NVD’s problem? We
won’t know until there’s some announcement to that effect, but around the time
the slowdown began in mid-February, a new CVE Naming Authority (CNA) (see the
post linked above for a description of what CNAs do) dumped a huge number of CVE
reports on the NVD (it was in the thousands. Given that about 20,000 new CVEs
are added to the NVD every year, this might have been almost half a year’s
worth in one day). The reports were different from the normal ones, because
they lacked a CVE number. Could this have caused the problem?
You might think it’s not
surprising that a database would suffer serious problems if it suddenly had to process
half a year’s worth of defective entries at once. But remember, databases are
supposed to be resilient to these kinds of problems. Normally, I would expect
the database to be able to reject invalid entries, rather than get bogged down
trying to process them. Why didn’t that happen in this case?
But there’s a bigger question: My
last post pointed out that every CVE report is submitted by a CNA. There are currently
around 330 of these, all over the world. For example, Oracle and ServiceNow are
both CNAs. Red Hat is one of five Root CNAs. The CNA reports to the CVE.org
organization (popularly known as MITRE, since the staff is entirely composed of
contractors from MITRE). The report describes a vulnerability that the CNA
identified in one of its own products, or else in a product from another
developer that isn’t a CNA, who came to the CNA for assistance in creating the
report. The CNA’s primary job is to describe the vulnerability correctly in the
report and assign a CVE number to it; they also need to describe the affected
product(s) and version(s) accurately.
What the CNA doesn’t do is
assign the CPE name (CPE
is the software identifier on which the NVD is based) for the affected product
and version, even though the CNA is usually the developer of the product.
Instead, once the CNA has submitted their CVE report and it’s been entered in
the CVE.org database, it goes to the NVD (which is part of NIST). There, a
staff member creates the CPE using the supplier, product and version
information entered on the report by the CNA. The NVD adds the CPE name(s) to
the report, as well as the CVSS
score and CWEs.
Here's the problem: It seems the NVD
people who create the CPE name often make mistakes (or at least they make questionable
decisions), and the CPE name ends up diverging from the CPE specification. Obviously,
if the CPE name doesn’t follow the spec, it will be close to impossible to find
it by a normal search.
Bruce Lowenthal has been pointing
out regularly – in the 3+ years that I’ve known him – that the CNA, who
developed the software in question, isn’t allowed to generate the CPE name. This
creates a big problem for his company, because when the name created by NIST is
incorrect, customers get mad at the developer, thinking they’re responsible for
the error. So, the logical question is: Why not let the CNA create the CPE name
and add it to the CVE report themselves? Why does the NVD have to be involved? It
seems the NVD staff are literally contributing to the problem, not fixing it.
Bruce also makes similar comments
about CVSS scores and CWE information. Those items are also added to the CVE
report by NVD staff members, not the developer who reported the vulnerability
in the first place. Creating those requires detailed understanding of the
vulnerability; Bruce says his company could do a much better job of producing
both CVSS cores and CWE information for their own products than the NVD staff
members do. I’m sure many other developers would say the same thing.
Note that making this change would
kill two birds with one stone. One is that it would result in more accurate CPE
names. But the other is that it would go a long way toward fixing the immediate
problem with the NVD – which is that they seem to be having trouble creating
CPE names, period. Whatever is the cause of the problem, there’s no disputing
that having the CNA create the CPE name – before the CVE report even goes to
the NVD – would prevent the current problem from recurring.
But there’s another aspect of this
problem: the fact that the NVD’s infrastructure, which is built on a foundation
put in place more than two decades ago, is…how shall I say it?...not state of
the art. For one thing, how many heavily used databases today aren’t totally
redundant, so that if any piece of software or hardware fails, the database
will be up and running very quickly, perhaps without users even knowing there
was a problem? The answer? Not many. Yet I’ve heard for a couple years that
there are multiple single points of failure in the NVD. Even though we don’t yet
know what the NVD’s problem is, it’s clear that if the NVD had true high
availability, they wouldn’t still be crippled by the problem, more than 40 days
after it first appeared.
What’s ironic about this is
there’s already another database that has all the information that the NVD does,
yet it’s built on a much more robust, highly available infrastructure: CVE.org.
All the information in the NVD is from CVE reports; those are first entered in
CVE.org. Why should NIST continue to maintain the NVD and struggle (sometimes
unsuccessfully, as in the past two months) to keep the thing running with duct
tape and prayer, when they could just let CVE.org handle all the queries? That
way, the NVD staff members could just concentrate on their other assigned task:
adding CPE names, CVSS scores and CWE information to the CVE reports.
Of course, the answer to the question
in the above paragraph should be clear when you read the last sentence. I pointed
out earlier that Bruce Lowenthal and other CNAs want to take back
responsibility for creating these three items from the NVD, since the developer
that identified the vulnerability can create them much better than anybody
else. Now I’m saying it would make much more sense for the NVD database to be shut
down and replaced by an expanded CVE.org. What’s left for the NVD staff to do
if they don’t do either of these things?
Good question. And that’s exactly
why the NVD (and NIST) will fight any such proposal. But consider the
following:
1.
Even if all their current
responsibilities are taken away, the NVD staff members are all part of NIST. In
case you didn’t know it, NIST does a huge amount of valuable work in the
cybersecurity area. I strongly doubt those staff members would be laid off from
NIST. Moreover, NIST will still receive the federal funding for the NVD, at
least through the end of the fiscal year and maybe beyond that. Surely, they
can figure out some creative way to keep the (now ex-) employees of the NVD
gainfully employed for at least another few years.
2.
If CVE.org gets additional
funding in future years, due to assuming the responsibility of operating “the
database formerly known as NVD”, the NVD’s ex-employees will presumably get
first consideration when they apply for a job at CVE.org.
3.
If all else fails and
there is no longer any public employment available for ex-NVD staff members, there’s
always the private sector. Given the experience and knowledge they’ve gained by
working for the NVD, I doubt they’ll have any problem finding work.
If someone decided tomorrow that both
above actions should be taken, how long would it take for them to be effective?
Moving responsibility for CPE, CVSS and CWE additions from the NVD staff to the
CNAs could be done very quickly. However, given there will need to be a lot of
training as part of this, I think three months is a good estimate for it.
Moving the NVD to CVE.org will
take more time, even though the overall structure of the database won’t change.
This is because the querying facilities, data feeds, APIs, etc. that the NVD
offers (not always successfully, though) will need to be reproduced in CVE.org.
Since CVE.org can build on what is currently included in the NVD, I doubt this
will take too long – perhaps six months. I think, based on technical
considerations alone, the entire move from the NVD to CVE.org could be
accomplished within six months.
Unfortunately, the problems in
making this change won’t be technological but political. The NVD is part of the
Department of Commerce and CVE.org is part of the Department of Homeland
Security. Here’s Tom’s Law of Government: Whenever you have two Cabinet-level departments
involved in a change, each with their own budget to protect, there will be
resistance to that change from one or both of those departments.
The only governmental officer who
can bang the appropriate heads together to effectuate that change is the one officer
who outranks both Secretaries. That’s…you guessed it, the president! Even
though the number of dollars and the number of staff members involved in this
change isn’t even a rounding error in the tenth decimal point of the federal
budget, the problem will probably require White House intervention. Only God
knows how long that will take, but I’ll guess this will take at least 2-3 years
(and 2-3 budget cycles, of course).
However, things aren’t as bad as
they look. That’s because there’s nothing that either CVE.org or the NVD does
that can’t be reproduced by an organization completely outside of government. For
example, the CNAs are obviously a key part of this process, but they’re all
volunteers. While the government has “CVE” trademarked, there’s nothing to
prevent outside organizations from creating their own vulnerability identifier
(perhaps “TOM”?) and having the CNAs create both “TOM Reports” and CVE Reports.
They’ll forward the latter on to CVE.org (and through them, to the NVD), and
combine the CVE and TOM data in a single vulnerability database. A good example
of this is OSV,
which was originally created by Google but is now a full-fledged open source
vulnerability database. OSV has its own vulnerability identifiers, although the
database also includes CVEs.
In fact, following this all-private
path would bring an important added benefit: The new database could easily include
purl identifiers as well as CPEs, because the OWASP SBOM Forum made a
successful “pull request” to CVE.org in 2022. As a result, the most recent
version of the “CVE JSON spec” (v 5.1) – which is already being used by some CNAs
to create their CVE reports - allows purls to be added to the CVE reports,
along with CPEs[ii].
That will solve a good portion of the naming problem in the NVD, which
clearly will never be solved if the NVD is left to its own devices (it will be
at least 2-3 years before the NVD will be prepared to adopt the 5.1 version of
the CVE JSON spec, but a new private database would be able to adopt it from
the start).
Of course, even if the entirely
private route is taken, that alone will require at least a year, since we will
need at least to give an all-government solution a chance to work first. There’s
no question that this would be the best outcome. But I strongly doubt it will work.
Any opinions expressed in this
blog post are strictly mine and are not necessarily shared by any of the
clients of Tom Alrich LLC. If you would like to comment on what you have
read here, I would love to hear from you. Please email me at tom@tomalrich.com.
My book "Introduction to SBOM and VEX" is now available in paperback and Kindle versions! For background on the book and the link to order it, see this post.
No comments:
Post a Comment