I’ve figured out why software users aren’t requesting SBOMs

I realized about a year ago that the reason why SBOMs aren’t being used by non-developers for vulnerability management purposes – even though the developers themselves are using them very heavily for those purposes, as shown in this post – isn’t that most developers don’t want to supply them. Rather, it’s because the users aren’t asking for them.

I have until recently blamed that reticence on the fact that there are few low-cost or open source tools to “process” SBOMS (i.e. ingest them and look up component vulnerabilities in the NVD or another vulnerability database) available for end users; moreover, there’s only one tool (at any price point), Dependency-Track, that ingests SBOMs and VEX documents, and these tools have to do both. Of course,  the whole point of creating a machine-readable SBOM (as opposed to just a listing of components in a PDF) is that…strange but true!...it needs to be read by a machine. I can attest that SBOMs in JSON or XML can be read easily by someone with a little knowledge of English, but there’s still no reason to create such an SBOM if a machine isn’t going to read it.

And when you get to SBOMs with hundreds or thousands of components (the average software product now has about 150 third-party components, but some have thousands or even tens of thousands), there’s no question that you need to process it using a tool. That is, you probably have better things to do with your time than print out an SBOM and look up 150 components in the National Vulnerability Database (NVD) using “manual” searches.

But just having a tool that can “process” SBOMs isn’t enough. The end user should look not just for component vulnerabilities, but for the small subset of those that are exploitable in the product itself; those are the ones that pose a problem. VEX documents are required to separate the exploitable wheat from the unexploitable chaff. Since at least 90% of component vulnerabilities aren’t exploitable, an SBOM consumption tool that doesn’t also read VEXes is probably less than useful.

When Dependency-Track recently started supporting VEXes as well as SBOMs (it’s supported SBOMs since 2012, long before almost anybody else got interested in them), I happily announced that fact. And since D-T is open source and therefore free, I wondered if this development might at least get a few more end users interested in getting SBOMs from their suppliers. The result was (drumroll, please)…

Thunderous silence. Of course, when Dependency-Track (which was developed in 2012 by Steve Springett, co-leader of the CycloneDX SBOM format project) is downloaded, the person who did that doesn’t fill out a questionnaire saying what they’re going to use it for. However, Steve developed D-T for use by developers, and Steve says that what he hears from users is that they’re overwhelmingly developers.

To be honest, just changing some wording and some screens would make D-T a perfectly usable consumer product, but Steve has never had requests for that. He knows of a small number of non-developers who are using D-T to track vulnerabilities in products they use, but he doesn’t believe that number is growing.

This isn’t to say that, if someone wants to make the investment to create a consumer-oriented product that “processes” both SBOMs and VEXes, and they make it available at low cost, there won’t be takers for it. Until recently, I thought that was all it would take to make people start using SBOMs.

However, just asking myself a question recently – and answering it with some simple math – made it clear to me that the lack of a low-cost, end-user-friendly tool isn’t at all what’s holding back use of SBOMs.

The (rather long) question I asked myself was, “Let’s assume there’s a low-cost, end-user-friendly SBOM/VEX processing tool, and that a user feeds an SBOM into it. The tool looks up every component listed in the SBOM in the NVD. For every successful lookup, the user receives a list of any open vulnerabilities in the component. They then use any available VEX from the supplier, for the product and version that’s the subject of the SBOM, to determine which of these vulnerabilities are not exploitable. They remove them from the list of open vulnerabilities, ultimately yielding a list of only exploitable vulnerabilities.

“Given this, what is the probability that the user will a) learn about every exploitable component vulnerability in the product, but also b) know exactly which of those vulnerabilities are exploitable or not exploitable?” We will call these the two target probabilities.

There are three probabilities that need to be considered, to answer this question:

1.      The probability that the user will be able to find each component in the NVD when they look it up, and learn of any vulnerabilities (CVEs) that apply to the component;

2.      The probability that a vulnerability identified for a component will be exploitable; and

3.      The probability that there will be VEX documents available from the supplier, which will identify all non-exploitable component vulnerabilities. This will leave just the exploitable ones.

The first probability is based on the “naming problem”. This is the problem that the informal group I lead, the SBOM Forum, addressed in a recent paper; I discussed it in this post. Briefly, the aspect of the naming problem that we addressed can be stated as, “When a user looks up the components found in a newly-created SBOM in the NVD, they will find only a small fraction of them, unless they try other manual processes.” These processes include AI, fuzzy logic, references to other databases, etc. The point is that a purely automated process will only find a small fraction of the vulnerabilities in the NVD (six reasons for this are described in the paper); the person interpreting the SBOM will need to use these other processes, and mix in a good measure of dumb luck, in order to learn about any more than a small fraction of the vulnerabilities that apply to the product.

And what is that small fraction? Our paper quotes the Director of Product Security of a very large US-based software developer, who said that they can find fewer than 20% of the components that appear in one of their SBOMs (they’re creating SBOMs regularly for all their products, but they’re not distributing them now). However, in a recent discussion in one of our meetings, this person admitted that the real number, in their experience, is less than 10%, and maybe even less than 5%. Another participant in the meeting – who works for a very large US-based producer of both software and intelligent devices – said their experience shows that the number is under 5%.

What this means is that, even if an end user has a tool that ingests SBOMs and looks up all the components in the NVD, they will probably learn about vulnerabilities in fewer than 5% of the components. Of course, the SBOM Forum’s proposal aims to raise that percentage far higher – and it will ultimately do that. But I estimate that it will be one to two years before our proposal is fully implemented (I’m optimistic that it will be implemented, based on the reception it’s received so far).

Even after it’s implemented, it will mostly affect new vulnerability reports, not existing ones. So, it could be a number of years before we’ll be able to state that the percentage of successful NVD component searches is over say 70% (it will never be 100%, of course). The bottom line is that, for at least the next couple of years, the first probability above is no more than 10% and more likely below 5%.

I’ve already discussed the second probability, the percentage of component vulnerabilities that are exploitable, in previous blog posts. I normally use the figure of 90%, but most people I talk to believe it’s over 95%. In other words, once a user has looked up whatever components they can find in the NVD and listed all the vulnerabilities identified for those components, at least nine of every ten will be non-exploitable. This might seem like good news, until you consider that you don’t know which ones aren’t exploitable – so you have to treat them all as exploitable for the time being.

The third percentage, the probability that the supplier will produce enough VEX documents to identify every non-exploitable component vulnerability, is zero currently, since no supplier is producing VEXes, period (and BTW, don’t point to a vulnerability report and tell me that’s a VEX, even if it’s in the CSAF format). Will it become non-zero later?

I have recently come to believe there’s a good likelihood that VEX documents will never be produced, for various reasons I’ve discussed already and will in the future. This is why I’m now advocating for the idea of real-time VEX, although I’m also still working with the VEX committee as it makes slow progress toward producing a VEX playbook - which is needed no matter whether the VEX document or real-time VEX emerges as the winner. VEX information is required for SBOMs to be widely used, no matter how it ultimately gets distributed. And SBOMs will be widely used.

So, what’s my estimate for the third probability? It’s zero now, but in a couple years it will be nonzero. However, I want to remind you that the only way a user will be sure that the list of component vulnerabilities they have for a product only contains exploitable vulnerabilities is if they can be sure that the supplier has notified them of every non-exploitable component vulnerability in the product and version in question – that is, they have informed them which nine of every ten component vulnerabilities aren’t exploitable.

Now that we know these three probabilities, what about the two target probabilities we want to derive from them? These are the probabilities that the user will a) learn about every component vulnerability in the product, and b) learn exactly which of those vulnerabilities are exploitable or not exploitable.

Finding a) is easy, since it’s just the first probability listed above: the probability that the user will find every component in the NVD. We’ve already said that’s (conservatively) 10%. But finding b) is harder, since this requires receiving a VEX for each of the non-exploitable vulnerabilities in the product. So, if there are a total of ten vulnerabilities but only one is exploitable, the user will need to receive a VEX for each of the other nine vulnerabilities, saying it’s not exploitable (remember, the user initially needs to assume that every component vulnerability they identify is exploitable, until they receive a VEX stating otherwise).

Let’s say the user just receives seven VEX documents, each stating that one of the ten component vulnerabilities isn’t exploitable. Does the fact that only seven were received mean the remaining three vulnerabilities are exploitable? The supplier could always later end the uncertainty by sending another VEX, which states that the three are exploitable. However, would that be a good idea, especially if the supplier doesn’t have a patch for the vulnerabilities yet? It definitely wouldn’t be a good idea. Only if the supplier sends out a VEX stating that the vulnerabilities have all been patched, which includes the URL(s) of the patch(es), can the user be sure the remaining three vulnerabilities are exploitable in that product and version.[i]

Unfortunately, there’s no assurance that the supplier will always make sure to close the loop and inform their customers of the status of every vulnerability that the NVD lists for a component of a particular version of their product. Thus, it's very possible that a customer will waste time (theirs and the supplier’s) by contacting the supplier regarding non-exploitable vulnerabilities. This is one of the reasons why I now think real-time VEX is a better option than VEX documents.

So what is the value of target probability b)? As I said earlier, currently it’s zero, since no supplier is currently producing VEXes. However, I do think that the idea of a VEX API will ultimately be adopted. This will allow the customer to be certain whether or not each component vulnerability is exploitable. But, since we’re concerned with the next year or so at the moment, I can confidently say that b) is a very low percentage, and currently it’s zero percent.

Finally, we’re able to answer the big question: Why are so few users interested in receiving SBOMs for use in software vulnerability management – even though there’s one open source tool available that could help them in this job? There are at least two reasons:

1.      They have heard that it’s difficult to look up components from an SBOM in the NVD. An estimated maximum ten percent success rate doesn’t inspire them to try doing that, either.

2.      They know that, at least in the near term, they won’t be able to learn which component vulnerabilities are exploitable in the full product, without calling up the supplier’s help desk and running down their entire list of component vulnerabilities with the unfortunate person who answers their call, forcing them to find out whether each of those vulnerabilities is exploitable or not. Of course, this is one of the main reasons why suppliers aren’t keen on distributing SBOMs to their customers now; they suspect their help desks will be overwhelmed with calls about false positives.

You may notice a seeming contradiction in this post: I first talked about the fact that SOBMs have been hugely successful with developers, but now I’m implying they’re lying dead at the starting gate, as far as end users are concerned. Clearly, the developers know something that the users don’t. Can they share this?

As far as the first reason, the naming problem, goes, the suppliers definitely have knowledge. The naming problem has been known since the beginning of the NVD, but SBOMs have turned it from an annoyance into a crisis, because of the huge number of components that now need to be looked up. Developers and consultants have developed a whole array of tricks – including AI and fuzzy logic routines, databases that shed light on different aspects of software names, etc. – to get around the problem.

Since these tricks are very ad hoc, they can’t be shared as part of a tool. However, developers could apply them on behalf of their customers, so that the SBOMs they send out have valid identifiers (“CPE names”) that allow customers to look up components in the NVD (meaning the customers would probably still end up looking up the vulnerabilities in the NVD manually). But, more likely, the suppliers will just outsource the whole “SBOM processing” activity to third party service providers

The service providers will provide an “SBOM service” for end users. This service will take in SBOMs, as well as either VEX documents or real-time VEX information (whichever is available) and produce for their customers a list of exploitable component vulnerabilities in a product/version that is used by the customer. The customers won’t need even to look at an SBOM or VEX document, nor should they need to look up anything in the NVD on their own. They’ll receive what’s really the holy grail of SBOMs: continuously-updated lists of exploitable vulnerabilities in the products/versions in use by the user organization.

This is why I now think that, at least for the next few years, almost all “SBOM processing” for end users will be performed by third parties, not by standalone tools operated by the users themselves. My guess is the users will be quite happy with that arrangement.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] The fact that a patch has been released for a vulnerable version of a product doesn’t mean the status of the vulnerability changes in that version. Best practice dictates that application of the patch increments the version number of the instance of the product that’s patched. So the vulnerability will be fixed in the patched version but not in the unpatched version.

NERC's supply chain security guidelines

If you’re not part of the electric power industry, you may not have heard of NERC, the North American Electric Reliability Corporation. Or you may have heard of it, but all you know is the acronym.

NERC is a non-profit corporation owned by electric utilities and other power market participants. It develops and audits compliance with a number of standards for reliability of the North American Bulk Electric System (BES), including the famous (infamous?) NERC CIP standards for cybersecurity of the grid. But the regulatory muscle behind all NERC standards is provided by the Federal Energy Regulatory Commission (FERC), which is part of the Department of Energy. This unusual arrangement was mandated by the Electric Power Act of 2005.

NERC also plays a number of important educational roles, one of which is to provide guidelines on grid security, including supply chain cybersecurity. The supply chain guidelines are developed by the NERC Supply Chain Working Group, which was formed in 2018, as the industry was starting to prepare for implementation of NERC CIP-013-1, the supply chain cybersecurity risk management standard. It’s important to keep in mind that the guidelines are not aimed at compliance with CIP-013, but simply at good supply chain security practices.

I’ve been a member of the SCWG since shortly after its inception. In 2019, the group was asked to develop guidelines on (I believe) five topics related to cybersecurity. I volunteered to lead the group that developed a paper on supply chain cyber risk management lifecycle. After the first couple meetings of the group, I realized there should be a separate paper on vendor risk management lifecycle, so another group was formed to lead that – and nobody else volunteered to lead it, so I led that as well. The papers were published in 2019 and 2020 (in the end, there were 8 or 9 papers).

This year, the SCWG was asked by NERC to update the guidelines papers. Since nobody else stepped forward (hard to do in a WebEx, of course) I ended up leading both of those. Just as I’d found the experience of developing the 2019 papers to be quite intellectually stimulating, I found the same thing this year.

The two papers were published for comment by NERC on Monday; they’ll be up for 45 days, and then we’ll draft the final versions. The supply chain paper is here and the vendor paper is here. I think only NERC members can comment, but I’d love to hear any comments or questions. I’ll make sure we take them into account when we revise the documents, so your comments will be given as much attention as a NERC member’s.

I do want to point out that there is some material that was added to the papers before publication, that’s common to them as well as the other papers (which will be put up starting soon but continuing into 2023 – since a few were written in 2020, not 2019). That’s the material before the Executive Summary and after the table of participants. That material has nothing to do with the topics, but you may find it interesting anyway (I’ve always found the topic of the power grid in general to be quite interesting, and have written about it – i.e., not from a cybersecurity point of view – a number of times, like this one and this one).

And if you want to buff up on the NERC CIP standards, you’ll find a few posts of interest by searching on CIP in the search bar. There are probably about 4-500, so I wouldn’t plan on reading every one of them in one sitting.

If you’re with a NERC entity or an IT or OT supplier to the power industry, I’d love to have a discussion with you about CIP-013 and supply chain cybersecurity. Please drop me an email.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

I got a little carried away

Last week, I put up an upbeat post that pointed out that SBOMs are already a great success among software suppliers, since they are using them heavily (literally hundreds of millions of times a month, and billions of times a month if you want to count “use” in a different way) to learn about vulnerabilities in products they’re building. I noted that this was happening in the absence of any significant use by consumers, and I further noted that this wasn’t because at least some suppliers aren’t ready to supply them to their customers, but because their customers aren’t asking for them.

I didn’t go into detail on why consumers aren’t asking for SBOMs now, although I’ve dropped various hints in my posts – lack of free or low-cost tools to utilize SBOMs for vulnerability management, lack of support for SBOMs from vendors of vulnerability and configuration management software, lack of available VEX documents to notify consumers of non-exploitable component vulnerabilities, etc. But I did attribute the fact that so many software suppliers had in recent years “gotten religion” on the need to take responsibility for preventing or fixing vulnerabilities due to components in their software, to the fact that they knew SBOMs were coming. Moreover, they knew their customers were soon going to know about vulnerabilities found in components in their software and continually ask when they were going to fix them.

However, I realized today that there’s a kind of contradiction between those two ideas: That SBOMs aren’t being significantly utilized by consumers today and that suppliers are anticipating they will be heavily utilized by consumers in the near future. I resolve this by saying that suppliers are wrong if they anticipate SBOMs will be heavily used in say the next 2-3 years, but I anticipate they will be after that. I added a couple of paragraph's to last Monday's post to clarify this point.

I have a list of about 8 serious impediments (and there are many less-serious ones as well) to widespread SBOM use. I’m happy to say that one of them is on the road to being resolved, and the informal group responsible for that is now deciding what’s the next impediment we’ll take on. Anyone who’s already familiar with SBOMs and would like to join that effort should drop me an email.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

At long last…

When I first started attending the meetings of the NTIA Software Component Transparency Initiative (which of course was the name for the “SBOM initiative”) in the summer of 2020, I immediately started hearing about what was called the “naming problem”. I wrote a post about the problem in November 2020 and called it the “one problem that towers over the others”, as far as SBOMs are concerned.

In other words, at that time, I considered the naming problem to be the most serious roadblock in the path to widespread (or even narrowspread, to be honest) distribution and use of SBOMs by the general community. However, the whole group seemed to have decided that this was too hard a problem to address at the time, given that they were still trying to a) figure out how SBOMs were going to work in general, and b) interest the developer community in producing them. This was still the accepted view when the NTIA Initiative ended at the end of 2021, even though a lot of progress had been made in both of those areas.

I have good news and bad news. The good news is that I no longer consider the naming problem to be the biggest impediment to the spread of SBOMs. But the bad news is that I don’t say this because I think the problem has diminished. Rather, I’ve come to realize there are at least five or six just-as-serious impediments to SBOMs (and a host of not-so-serious impediments, which are impediments nevertheless); in other words, the naming problem now has company, instead of “towering” over the others. There’s progress for you!

Early this year, it seemed to me and a few others that it was time for the private sector to take the lead in addressing head-on the most serious problems for SBOMs. The idea was that we would meet weekly and discuss one problem until we at least understood how it might be solved. Then, if there was something we could do to put in motion the solution to the problem, we would do that. After that…on to the next problem.

Why were we doing this? I’ll admit that we all had selfish reasons. Our livelihoods are all tied in one way or another to the success of SBOMs. We will all benefit if SBOMs start to be widely used outside of the development community (where they’re already widely used for product risk management purposes).

The question came up, should the participants worry about helping potential competitors? After all, when one of these problems is solved, it will benefit their competitors, as well as themselves. My answer to this question (which I didn’t hear often, to be sure) is that ultimately the SBOM “market” will include every organization on the planet. It’s difficult to imagine any organization of any size in the world today that doesn’t use software (even if it’s just the software in one person’s smartphone); it will soon be literally impossible to imagine that.

Given this, it follows that today’s market for SBOM services and tools is an almost infinitesimal fraction of what it can be. Doesn’t it make sense to focus on expanding the market so it’s at least a significant fraction of what it ultimately will be, rather than worrying about how you and your competitors are going to divide up the very small market that’s there today?

Of course, it made sense to my friends to take the latter course. The only question was, which of the five or six serious problems would we start with? I wasn’t sure which one, but I was sure of one thing: it wouldn’t be the naming problem. I honestly thought that this problem would take a year to even understand, another year to develop at least a partial solution for it, and finally about eight long, grinding years – full of political battles – to see the near-solution implemented.

My friends and I created an informal group called the SBOM Forum and started meeting for an hour every Friday. In one of our earliest meetings, Tom Pace of NetRise described an eye-opening experience he had with the NVD, in which a device, that appeared in the NVD to have never had a vulnerability, in fact had at least 1,237; Tom found these with a simple scan of just two of the firmware products installed in the device. Moreover, Tom later came to realize that the same product has probably 40,000 unpatched vulnerabilities, even though an NVD search will find nary a one of them.

The problem Tom came across was just one of the many manifestations of the naming problem. The naming problem in general refers to the fact that software products have many names in the many locations in which they’re found. However, the branch of the naming problem that causes the most consternation for the software community is centered on the National Vulnerability Database (NVD), by far the most heavily used vulnerability database in the world. The NVD uses a very problematic naming system called CPE (common platform enumeration), which is the source of most aspects of the problem.

To my horror, after Tom’s discussions with our group, we stumbled into deciding to take on the naming problem first. But I was in for a big surprise: Instead of taking two years to identify the problem and document a solution to at least 70-80% of the problem, we took a little more than four months. We published the document this week.

The six most serious aspects of the problem with CPE are described on pages 5-7 of our document. I used to think that all those aspects were going to require their own specialized measures, which was why I was sure that any “solution” we proposed would be a godawful mess. However, I didn’t realize that our group was lucky enough to have the perfect Hero to slay the foul Naming Beast: Steve Springett, co-leader of the OWASP CycloneDX project and without much doubt the most creative person in the SBOM community.

As we started talking about the problem, Steve quickly realized that a big component (no pun intended) of any solution would have to be purl, a unique type of identifier that is already in wide use - although it isn’t used so far in the NVD (Steve is a maintainer of the purl project, which has helped our group a lot already). Purl has so far been used mostly with open source software (and used very extensively. Just one tool, Dependency-Track, is currently used about 27 billion times every month to look up a vulnerability for an open source product in Sonatype’s OSS Index vulnerability database, which is based on purl).

Steve realized that purl’s unique extensibility would allow us to “incorporate” two other identifiers into purl: SWID for proprietary software, and Software Heritage ID for legacy open source software (since our document explains the various naming schemes that our solution will utilize, I won’t explain them here).

The best thing about purl: for reasons described in the document (be sure to read the discussion of “intrinsic” vs. “extrinsic” identifiers), integrating purl into the NVD will not require any database linking; every open source product that’s in a package manager already has a purl. When searching for vulnerabilities in that version of the product, the user can easily create the correct purl – 100% of the time.

While purl, SWID and SWHID cover a good portion of the software universe, they don’t cover hardware. However, the NVD includes hardware, and devices also have CPE names (along with the problems belonging thereto). I would have been fine with declaring victory with software and moving on, but Steve insisted we could do hardware as well. He and Tony Turner of Fortress Information Security recognized that the two most widely used hardware identifiers, GTIN and GMN (both part of the “GS1” family of hardware standards), could be integrated with the NVD as well.

However, in order to allow those two identifiers to be used with the NVD, various databases need to be linked with the NVD. This is the only part of our proposal that requires substantial work. I think it would amount to just a few person-months, but since there will be various groups involved, it might take longer than that.

But the work will be worth it. One of the GTIN identifiers is the UPC code. Once our proposal is fully implemented, you should be able to find out about vulnerabilities in a device by entering its UPC code – either by scanning the code itself or entering the numbers by hand – in the NVD. Whoever thought using the NVD could be this much fun?

I’m hoping that, two years from today, our “proposal” will be fully implemented; Steve has already started discussing this with the group that will be most involved in the implementation: MITRE Corporation. CISA at some point (perhaps not until early next year) will hold a meeting to discuss the question of software naming in the NVD.

What can you – and your organization - do to advance this proposal? I’d say the first thing is to read the document. I’ll admit it’s dense. I’m going to put up two or three blog posts trying to shed light on some of the harder parts. And OWASP will have a webinar soon to discuss the proposal.

Second, you can read the blog post by Steve Springett. At the bottom, he has a “How to Help” section. To what he has there, I would add that we would like to put together a set of short testimonials (1-3 sentences) on why your organization supports this proposal, including how this will enable the organization to do whatever you do better and more efficiently.

For example, the document includes a statement from a very large software maker (identified in the document) to the effect that they regularly create SBMs for all of their products, but when they look at just the output of the tooling they use to create the SBOM, only one in five components can be identified in the NVD. Of course, this means they can’t “operationalize” their SBOM production process. While they can often find a lot of those components, they – like many other companies and consultants – need to employ AI, fuzzy logic, references to other databases, prayer…whatever will help. Being able to fully automate SBOM production will greatly increase the efficiency of their software development process.

But fully automating the SBOM production (or interpretation) process is out of the question for any organization, until the naming problem is solved.[i] Hopefully, that day is now closer than it’s ever been. 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] I won’t pretend that the naming problem will ever be “solved”, since there are so many “corner cases” that will never all be addressed in the structure outlined in our proposal. However, it can be close to solved for two important areas: open source components (which of course compose 90% of software components) and hardware devices. The one area where our proposal is weakest is proprietary software, since that will require commercial developers to create SWID tags for all of their products, both new and existing (although they will only have to enter 4 or 5 fields in the tag).

Sure, this will be a little work, and some developers will resist doing this. But the choice isn't necessarily theirs. If they don't put a SWID tag in their product, their users will be at the mercies of CPE names when they try to look up vulnerabilities for the product in the NVD. As with most consumer products, the consumers (whether organizations or individuals) "vote" for their favorite products in the marketplace, using dollars, euros, etc. A supplier that doesn't provide SWID tags may find themselves "losing" a lot of elections!

SBOMs are already a great success!*

 

You may know that I’ve said on various occasions (although not often in this blog) that it’s disappointing that, while SBOMs are being heavily used today for vulnerability management purposes, that usage is almost entirely by software suppliers (i.e. developers), not by end users. While it’s good to see SBOMs have been so successful with suppliers, just about everything you read about them in the general press – and the whole purpose of Executive Order 14028 – is how end users need them to secure their networks. Well, it seems the end users didn’t get that memo, since – I’ll be honest – almost no non-developer organizations are regularly receiving or using SBOMs today.

So, the title of this post may make you wonder if I’ve gone crazy: Despite what I just said, I’m certain that SBOMs have already had a huge amount of success. How could that be?

To understand this, it is important to understand the fundamental purpose of SBOMs, in the supply chain cybersecurity use case. That purpose is not to empower end users to secure the software they operate by their own efforts; this is simply not possible. Truly securing software requires being able to patch vulnerabilities. Other than in exceptional cases, end users (i.e. non-developers) can’t develop their own patches for software they purchased or downloaded; having SBOMs doesn’t change this situation.

Then why do end users need SBOMs (or at least the data from them)? What good does it do to provide users with lots of information on what makes up the software they operate, if they can’t use that information to make the software more secure? And, even if the end user could secure their software if they used SBOMs, I just said they’re not using them. How can I say SBOMs are successful?

I stand by this statement. The fundamental purpose of distributing SBOMs to end users of software was never to empower them to fix software’s problems on their own. Instead, it was to empower them to put pressure on suppliers to fix the problems. Being able to identify components listed in an SBOM, and to learn about vulnerabilities applicable to those components, empowers the user not just to make general exhortations to a supplier to follow good development practices, but to ask them what they’re doing with respect to specific component vulnerabilities. That gives users a lot more power. You might call this, “security by being a pain in the ___”.

In a 2017 study, Veracode found, “A mere 52 percent of companies reported they provide security fixes to components when new security vulnerabilities are discovered. Despite an average of 71 vulnerabilities per application introduced through the use of third-party components, only 23 percent reported testing for vulnerabilities in components at every release.” These numbers are pathetic, as I’ll bet you understand.

This study alone might have helped bring about the FDA’s statement in 2017 (I believe) that they intended to start requiring SBOMs for medical devices, which itself led to the NTIA’s Software Component Transparency Initiative starting operation in 2018. That Initiative (which ended last year) was sparked by some large hospital organizations (“healthcare delivery organizations”, to be accurate) and some large medical device makers, who decided they wanted to get ahead of the curve and start producing and using SBOMs, long before the FDA’s order would come out (it will come out sometime this year).

To my knowledge, the Veracode study hasn’t been repeated, but there is quite tangible evidence that a large percentage of software suppliers have gotten religion on the matter of software security. This evidence is something I’ve already written about: the statement this spring by Steve Springett, founder of the OWASP Dependency-Track open source project (and co-leader of the CycloneDX project), that Dependency-Track was used 200 million times in a single month to query the OSS Index vulnerability database for vulnerabilities found in components in an SBOM (although that statement from April needs to be updated. Steve says the number of queries from Dependency-Track is now up 35% to 270 million per month. And that’s total queries. If you look at the number of individual components searched on, it’s at least 100 times that, since the average software product has more than 100 components. So, this represents about 27 billion individual searches every month – all based on SBOMs and all just from one product that utilizes SBOMs. There are a number of other tools – generally called software composition analysis or SCA – that do the same thing as Dependency-Track, although these usually aren’t free).

Of course, this usage is almost certainly due to suppliers, not end users. However, it makes my point very well: a large percentage of software suppliers are now actively tracking vulnerabilities in components in their products (both those under development and those already in use in the field), so they can be proactive about securing them. They’re not waiting for the customers to start banging on their doors, demanding to know when CVE-2022-12345, with a CVSS score of 10.0, will be patched in their product; they do this because they know that component X in the product is vulnerable to that CVE. Before the customers come calling, the supplier wants to know about important component vulnerabilities, and either patch them or replace the components.

You might wonder why I attribute this huge increase in supplier interest in vulnerability management to SBOMs, since I’ve just said they’re not being distributed to, or used by, end users to any degree (and I’m sure that’s because the end users aren’t asking for them. A lot of their suppliers already have SBOMs and would love to share them when they see demand for them). After all, there could be lots of other explanations for this. Maybe the suppliers were visited by a pillar of fire and heard a voice telling them to start managing vulnerabilities. Or maybe their spouses all started hammering them about vulnerability management. How am I sure one of these isn’t the explanation?

As with many economic effects in the real world, I think this has to do with expectations. The suppliers know that users will start requiring SBOMs in the near future (in fact, US federal agencies have been required to ask for them since August 10). Therefore, whatever may have been a supplier’s security posture previously, not taking a proactive approach to identifying component vulnerabilities in their software and mitigating them (either by patching or by replacing the component) is no longer an option. They need to do something; hence the 27 billion Dependency-Track vulnerability searches. SBOMs are already working!

* However, I will point out that this success will be limited if users can't easily learn about vulnerabilites found in components in the software they utilize. As of the moment, given the lack of easily-to-install-and-use open source or low-cost tools that will list and track component vulnerabilities, and the VEX documents (or real-time VEX notifications) required to make those usable, I have to admit that users can't do this in any significant way today. But I anticipate that the 8 or so major impediments that I've identified, which prevent widespread distribution and use of SBOMs, will be removed in a few years. I'm happy to say that I believe one of them is now on its way to being resolved within two years or less.

The ultimate success of SBOMs depends on their widespread adoption by organizations that use software. If that doesn't happen, the implicit threat to developers that their customers will immediately learn about vulnerabilities in their products and use that knowledge to make disadvantageous comparisons between them and their peers in this regard, will prove empty. I'm sure a lot of developers will continue to track and remediate component vulnerabilities anyway; but a lot won't. We'll be back in 2017.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Real-time VEX

 

Tom’s note 6/30/2023: While I continue to believe that VEX's future is as a server/API, not as a document format, I no longer think what's described in this post is the right one. Rather, it's what's described in this post.

“Real-time VEX” is an API that will allow a device on a software user’s network to query a server that resides on a software supplier’s network (or perhaps is operated by a third party on the supplier’s behalf). The API allows the user to determine the exploitability status of a particular vulnerability (CVE) in a version of the supplier’s product, which is operated by the user.

The use case for real-time VEX is simple: When a user receives an SBOM for a software product they utilize from the product’s supplier, they (or a tool or third-party service) will search the NVD (or another vulnerability database) for vulnerabilities that apply to any of the components listed in that SBOM. Whenever they discover an open vulnerability (CVE) for one of those components, the user - or more accurately a tool operated by the user, or a third-party service provider acting on the user’s behalf - will assign the default status of “exploitable” to the vulnerability for that version of the product[ii].

Shortly thereafter, the user’s system will login to the supplier’s server and provide three pieces of information:

1. CVE number[iii]
2. Product name[iv]
3. Version string

The supplier’s server will send one of three responses:

1. This CVE is not exploitable in the product and version named.
2. The CVE is exploitable in the product and version named. Please immediately apply the patch located at…(URL).[v]
3. The exploitability status of this CVE, in the product and version named, is unknown.

The user or their system (or a third party service provider) will take one of three steps:

1. If the vulnerability is not exploitable, the CVE will be marked as such in the list of component vulnerabilities applicable to the version of the product. Because it is possible the supplier might later change the status of the CVE to exploitable, the user system will regularly query the supplier’s server regarding the CVE’s status.
2. If the vulnerability is exploitable, the CVE’s status in the vulnerability list will be left unchanged, since the default should be “exploitable” anyway. The organization’s patch management team will be notified to download and apply the patch to the product. After applying the patch, the CVE’s status in the vulnerability list will be changed to not exploitable.[vi]
3. If the status of the vulnerability is unknown, the supplier’s server will be queried regularly (probably multiple times a day) to learn if the status has changed.

What is the advantage of real-time VEX vs. relying on the supplier to prepare and transmit a VEX document that indicates the exploitability status of a CVE? It is time. When a new vulnerability appears in the NVD for a component included in a supplier’s product and they subsequently determine that the vulnerability is not exploitable, the product security team will not normally be inclined to drop everything they are doing to prepare and distribute a VEX for their customers.

Even for just a few products, this might happen ten or so times in one day. It would be very disruptive to getting work done. For that reason, it is likely that most suppliers will batch their VEX notifications and release them for example every three days. However, changing the status of the CVE in the real-time VEX server will literally require almost no time at all.

Why is time so important? As previously discussed, a component vulnerability should always be considered exploitable by the user organization when it is first identified. Since a user has no way of knowing beforehand which CVEs may later be designated “not affected” in a VEX document, they will need to investigate every one of them, especially those of higher severity. If a VEX saying the CVE is unexploitable arrives for example 2-3 days later, the user may already have wasted a lot of time on this; and if this same situation happens multiple times a week, this will be a significant problem for the organization. This is why the user organization needs to learn about non-exploitable vulnerabilities as soon as possible after the supplier learns about them.

With real-time VEX, the user will be able to learn about non-exploitable component vulnerabilities almost as soon as the supplier does – and just as importantly, the supplier will need to invest almost no time at all in the notification. Upon learning that a vulnerability is not exploitable, the supplier only needs to update the status of the CVE in the real-time VEX server (and this will likely be handled by an automatic update from the system used to determine exploitability of the CVE); the user will learn about the change the next time they query the server. The supplier will not have to do any more than this.

It is important to keep in mind that the supplier itself will benefit as much from real-time VEX as the customer. Especially for more severe component vulnerabilities, it is likely that users will not be content to sit back and wait for a supplier to issue a VEX to inform them that a vulnerability is not exploitable; they are likely to call or email the help desk regularly about this. The sooner the supplier can update the status information in the real-time VEX server, the less time their help desk will waste giving the same answer to caller after caller.

The savings of time and frustration that could be provided by real-time VEX, on both the user’s and the supplier’s side, are likely to be substantial. Remember that over 90% of component vulnerabilities are not exploitable in the full product. This means it is possible that over 90% of the time and effort required by both the supplier and end user, in responding to component vulnerabilities revealed by SBOMs, will be wasted – unless the supplier quickly determines[vii] the exploitability status of a CVE and communicates this to their customers.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] There are some other problems standing in the way of widespread SBOM use as well. Fortunately, I think one of the biggest – the “naming problem” – is at least on the road to being solved (or at least greatly mitigated). More on that in the next week or two.

[ii] If the user organization operates multiple versions of a single product, each version will have its own list of vulnerabilities.

[iii] There is no reason why a large number of these queries could not be entered at one time.

[iv] Because of the “naming problem” discussed earlier, it will not necessarily be clear what constitutes the product name. Perhaps a CPE name or purl will be required in lieu of a product name. The desired option will be chosen by the supplier and will probably vary by supplier and product.

[v] If the CVE is exploitable, yet there is no patch available, response 3 will be sent. To announce the CVE is exploitable without a patch being available is of course to court disaster.

[vi] This statement is not technically correct. When the patch is applied, the version number will be incremented, since two different codebases should never have the same version number. The CVE will still be “exploitable” in the unpatched version, but it will be “not exploitable” in the patched version.

[vii] Of course, third parties can also make judgments about exploitability; however, this is a different type of exploitability that isn’t applicable to particular products, but rather to the vulnerability itself. Because exploitability, in the VEX meaning of the word, requires knowledge of how the software was constructed, the supplier’s judgment in this matter will carry much more weight than that of a third party.

However, this is not to say that, if a supplier simply does not provide VEX documents or real-time VEX notifications to their customers, there is nothing the customers can do on their own about this problem. They can certainly scan the product with a vulnerability scanner; if a component vulnerability does not appear in the list of active vulnerabilities, this can be taken as evidence that it is not exploitable. But this finding can never be considered to be as authoritative as the supplier’s statement regarding the exploitability status of a particular CVE in a particular product and version number.

 

It’s time for some basic VEX education

 

I’ve written a number of posts about VEX, but there are only one or two posts that I’ve linked when I want to explain the term itself, so that someone who hasn’t heard of VEX can get a quick education. However, I recently went back and read those posts, and realized that they don’t actually give a basic explanation of VEX. Instead, they assume basic VEX knowledge to start out. Now, I’m writing the post I should have written a while ago. I’ll use this as my “VEX education” link in the future.

VEX is an acronym for “Vulnerability Exploitability eXchange”. VEX is a simple yet machine-readable pair of formats for reporting the status of vulnerabilities found in software products and intelligent devices. While it can be used for any vulnerability reporting, it was developed specifically for a use case specific to suppliers that provide SBOMs for their products.

VEX was developed to enable a supplier to notify its customers, in a machine-readable format, that a vulnerability they may believe is exploitable in their product is not in fact exploitable. The primary reason that customers would ask this question in the first place is if they have received an SBOM and looked up its components in a vulnerability database like the NVD, then learned about one or more serious vulnerabilities that apply to those components. They might waste a lot of time (and worry) trying to find and patch vulnerabilities that aren’t exploitable, and therefore pose no danger to them.

Another reason for a VEX is that a specific vulnerability or set of vulnerabilities, like the log4j or Ripple20 vulnerabilities, has been widely discussed in the news – and users want to know whether or not it applies to a software product they use.

The consensus in the software industry is that at least 90% of component vulnerabilities are not exploitable in the product itself, usually because of how the component was incorporated into the product. For example, a library may have four modules, but the supplier might have only installed the three modules that they needed. A serious vulnerability could be identified in the library, but it happens to be found in the fourth module, which the supplier didn’t install. Therefore, the vulnerability is not exploitable in the product itself. A VEX document will state that fact.

It is likely that the great majority of VEX documents will be based on the use case of identifying non-exploitable component vulnerabilities. However, the simplicity of the VEX idea enables a number of other powerful use cases for vulnerability notifications. These use cases go beyond the case for which VEX was originally developed.

The need for VEX arose from discussions in the Software Component Transparency Initiative (hereinafter, “the Initiative”). This was a “multistakeholder process” convened by the National Technology and Information Administration (NTIA) division of the US Department of Commerce, beginning in 2018 and ending in late 2021. The purpose of the Initiative was to get together private and public “stakeholders” in the software marketplace to discuss securing software and intelligent devices from risks arising from use of third-party and open source components; of course, the main tool for accomplishing this purpose is SBOMs.

Early on in those discussions, it became apparent that there was a lot of concern about the problem that was discussed above: the fact that a large percentage of vulnerabilities identified in components included in a software product are not in fact exploitable in the product itself.

At first, this might seem to be a good thing. After all, if 90% of vulnerabilities in the components of a product aren’t exploitable in the product itself, this means that the “attack surface” of the product is reduced by 90%, compared to what it would be if all component vulnerabilities were exploitable in the product itself.

This is true, but the problem is that the end user cannot normally determine, through their own actions, which component vulnerabilities are exploitable in the product, and which are not exploitable. Not having this information, the user needs to assume that every component vulnerability they have identified is exploitable. Thus, they may waste a lot of their time trying to determine whether these vulnerabilities are actually present in the product. Moreover, they will waste the supplier’s time by continually contacting their help desk to inquire when each of these vulnerabilities will be patched.

The NTIA group discussing this problem decided that the best way to prevent this from happening is for the supplier to notify the customer that a particular vulnerability is not in fact exploitable in the product, even though it is listed in the NVD (or another vulnerability database) as applicable to a component of the product.

Note that this is a type of vulnerability notification that has not been used very much, simply because it has not normally been needed. A “normal” vulnerability notification says, for example, that CVE-2022-12345 is exploitable in product ABC version X.Y.Z. We call this a “positive” vulnerability notification, since it states that a particular vulnerability is exploitable in the product.

Of course, if this example is a positive vulnerability notification, then the notification mentioned above – i.e. that a particular vulnerability found in a component of a product is not in fact exploitable in the product – should be called a negative vulnerability notification.

The supplier is normally the party that is best able to provide vulnerability exploitability information, since they have access to the source code and know in intimate detail how the product was built. Fortunately, it is also in the supplier’s interest to get this information to their customers, since this will eliminate a lot of “false positive” calls to their help desk.  However, because it is likely that many suppliers, at least initially, will not provide either SBOMs or VEX documents, third parties might need to step in to provide VEXes on their own.

There are currently two VEX formats. The first is based on the CycloneDX SBOM format (although it can refer to SPDX SBOMs as well). The second consists of a subset of the CSAF vulnerability reporting format. Unfortunately, as of now (September 2022), the CSAF VEX format isn’t completely specified (a partial specification, which will be enough for someone who already understands CSAF, is here). A “playbook” for creating and using VEX documents is currently being prepared by the CISA VEX working group.

Because of the delay in developing proper documentation for producing and using VEX documents, as well as the fact that end users will want to know about the exploitability status of component vulnerabilities as soon as they learn about them – not several days or weeks later, depending on how long it takes the supplier to produce a VEX document – I am now advocating for the idea of “real-time VEX”.

This is essentially an API where a user will be able to learn instantaneously what the supplier currently believes to be the exploitability status of a particular component vulnerability (of course, that status may be “We’re still investigating it”). My guess is that, if this is implemented, the need for a majority of VEX documents will disappear. VEXes will still be needed for more complex use cases like patching, but the urgent use cases can all be handled by the API.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.