Rethinking VEX

Recently, my friend Kevin Perry sent me this link to an article in CyberScoop. It gives a fairly good summary of where SBOMs currently stand: End user organizations (i.e. organizations whose primary business isn’t developing software, which even includes the business side of software developers) aren’t using SBOMs in any substantial way at all. But developers are using them very heavily to manage vulnerabilities found in components of the software they’re developing (the person who wrote the article didn’t know how successful developers have been in using them, since that’s not commonly known. However, it’s very possible that the majority of software developers still don’t utilize SBOMs to secure their products at all).

I’ve compared current SBOM utilization to a string which is lying flat on a table. Many developers, seeing how useful SBOMs have been for them, would like to distribute them to their end users, but the end users aren’t asking for them because they have no idea what they’ll do with SBOMs if they receive them. The developers are pushing on their end of the string, but without the end users pulling on their end, the string is going nowhere.

What will get end users to change their minds about SBOMs, so they start seeing some real value in them? It’s not just persuading end users that SBOMs will help them secure their networks; there’s been enough written about that subject – and attention paid to it - that this can no longer be the primary obstacle for end users, or even a major one.

One real obstacle for end users is the lack of easy-to-use and low cost tools and third party services to “process” SBOMs for vulnerability management purposes. The small number of tools that are available today address parts of the software component vulnerability management process, but what I consider the most important part – passing vulnerability data to the vulnerability and configuration management tools that are now deployed by end users – is currently being addressed by nobody.

However, this problem isn’t a fundamental one. The tools will come when there’s clarity about how SBOMs can be successfully utilized by end users. But there are still some big obstacles standing in the way of that clarity. The biggest of these obstacles has to do with the fact that, even though there are lots of tools (including open source tools like Dependency-Track) that will look up vulnerabilities for components listed in an SBOM, around 95% of those component vulnerabilities won’t in fact be exploitable in the product itself.

This means that, if the user takes every one of the component vulnerabilities it finds seriously and contacts the supplier’s help desk to find out when they will be patched (as well as looks for those vulnerabilities in the products where they’re supposedly found), 95% of their time (and the help desk’s time) in this effort will be wasted. As Tony Turner pointed out recently, given that security staff are already overwhelmed by the real vulnerabilities their scanners find on their networks, the last thing they need to do is waste time tracking down false positives.

Of course, this problem is precisely the one that VEX documents were supposed to solve. But VEXes are currently going nowhere. No supplier is regularly distributing them to users, whereas, if they were to work as intended, oceans of VEX documents would need to be coming out daily. There are specific reasons why this isn’t happening (like a lack of guidance on how to put a VEX together in either the CSAF or CycloneDX VEX formats). Those reasons do need to be addressed, since there are still important use cases for VEX documents.

However, I now realize – after having participated in just about every NTIA and CISA VEX workgroup meeting since they began in mid-2020 – that VEX as currently conceived (and even including my real-time VEX suggestion) will never achieve the primary purpose for which it was intended: shrinking the user’s list of component vulnerabilities, so that only the 3-5% that actually are exploitable will remain. Given the huge number of VEX documents that would need to be issued in order to achieve that goal (perhaps hundreds or even thousands, for a single SBOM), and the fact that the user will never be able to be sure there aren’t still some component vulnerabilities that aren’t exploitable - but for which a VEX statement has yet to be issued - this approach is simply not a good one.[i]

But the current VEX idea, which I now think is mistaken, didn’t spring out of nowhere; there are two assumptions underlying it that I now believe to be wrong. Only by removing those assumptions will we be able to identify an effective solution to the component vulnerability exploitability problem.

The more important of the two assumptions is that this is fundamentally a user problem. By that, I mean the assumption that managing vulnerabilities due to components in a software product they utilize, is the responsibility of the end user, not the supplier. In this view, it’s up to the users to look up component vulnerabilities after they receive an SBOM, then use the VEX information to winnow them down to just the exploitable vulnerabilities.

However, I no longer believe this is fundamentally a user problem. Users need the information found in SBOMs and VEXes, because suppliers haven’t previously been doing a good job of fixing vulnerabilities due to components in their products. A 2017 study by Veracode said “A mere 52 percent of companies reported they provide security fixes to components when new security vulnerabilities are discovered. Despite an average of 71 vulnerabilities per application introduced through the use of third-party components, only 23 percent reported testing for vulnerabilities in components at every release.” One hopes that this situation has improved in five years, but it’s certainly possible that not even a bare majority of suppliers are doing a good job in this regard now.

Who introduced the components into the products? It wasn’t the users. Yes, the users benefit from lower software costs and improved functionality, achieved through use of third-party components. But the suppliers certainly reap the lion’s share of the benefits of using components. Frankly, they should provide the lion’s share of the effort required to identify and fix exploitable vulnerabilities in those components, both before the software is distributed to customers and afterwards.

Moreover, the supplier’s costs for identifying component vulnerabilities should be close to zero, since they’re already identifying them now. And if they’re not, shame on them – they should be. By instead putting the burden of this on the user, the suppliers are essentially saying, “It’s more efficient to have each of our customers, even if there are thousands of them, perform exactly the same analysis that we’re already performing, than it is for us just to distribute the information we already have.”

Think about it: If done right, each user’s effort to identify component vulnerabilities in a product will yield exactly the same result as would the supplier’s effort, which they should already be performing. Why is it that, if there are 10,000 customers for a product, it’s expected that they should each perform a set of steps that are the same as what the supplier is performing now? Moreover, by intention all those users will at best achieve exactly the same results as the supplier has already achieved on their own. What’s wrong with this picture?

But, as I said earlier, around 95% of the component vulnerabilities that a user or supplier identifies (by searching for the component’s name in a vulnerability database like the NVD) will be false positives. VEX documents are supposed to fix that problem, since they’re intended to let the user (or specifically their component vulnerability management tool, no examples of which exist in complete form at the current time) separate the 5% wheat from the 95% chaff.

However, beside the fact that it’s highly unlikely that VEX documents will ever be produced in the volume that’s required to accomplish that purpose, the bigger question is why there needs to be a document at all. After all, it’s the supplier who knows how their software is put together and in most cases needs to make the judgment whether or not a particular vulnerability is exploitable in their product. If the supplier takes responsibility for identifying component vulnerabilities in their products, they can also easily take responsibility for removing the 95% of component vulnerabilities that aren’t exploitable, leaving just the exploitable 5%. Since the exploitability information will never have to leave the supplier (except to go to a service provider who may be performing this service on behalf of the supplier), it can be communicated using any secure internal method.

In other words, it would be far more efficient, and immensely more effective, if the supplier looks up the component vulnerabilities themselves (or outsources this effort to a third party services provider) and then removes the 95% that aren’t exploitable. Then, they would provide this machine-readable list of exploitable vulnerabilities to their customers. The customers could search diligently for every vulnerability on the list, knowing that – to the supplier’s current knowledge – every one of them is a real vulnerability in the product.

Note 12/26: When I put this post up on LinkedIn today, Steve Springett quickly pointed out that the VDR report described in the revised version of NIST 800-161 also lists exploitable vulnerabilities in a product. So why do we need another document that does that? I realized I’d been thinking the user would naturally assume that, when they receive the document I just described, it means that any other vulnerabilities they find for a component of the product are not exploitable.

However, I now realize that the user probably won’t, and moreover they shouldn’t, assume that all other component vulnerabilities they find aren’t exploitable; therefore, the document should list both exploitable and non-exploitable component vulnerabilities. For one thing, the document I describe will be produced at a certain day and time. Even if it were true at that time that all other component vulnerabilities in for example the NVD weren’t exploitable, it’s inevitable that more component vulnerabilities will appear later on.

The report will have to be regularly updated, but it’s inevitable that there will be periods of time in which there are more vulnerabilities applicable to components than the ones described. If the non-exploitable vulnerabilities aren’t also listed in the report, the user will be led to believe that any other component vulnerability they discover isn’t exploitable, when in fact that may not be the case.

Of course, anyone who knows anything about cybersecurity will now scream out, “Are you crazy? You want to have each supplier send out documents that list each exploitable component vulnerability in each of their products to every one of their customers? How many nanoseconds do you think it will take before each of these lists shows up on Vladimir Putin’s desk?”

And here’s where the second incorrect assumption appears: It’s the assumption that the only good way to provide this information to customers is to send out a document, even a machine-readable one. Instead, I think this information needs to be provided in a strongly authenticated, machine-to-machine transaction that is initiated by the customer, not the supplier. Moreover, there needs to be strong encryption of the transaction.

And if that isn’t good enough, there are certainly ways to add security to the transaction. For example, the supplier might send – through a different encrypted, customer initiated transaction, perhaps to a different device on the customer end – a list of the component vulnerabilities that can currently be found in the NVD (or other vulnerability database, like OSS Index) for the version of a product that is described in a recent SBOM. This list would consist of the CVE identifier for each vulnerability, as well as a unique short identifier for the vulnerability, consisting of randomly selected alphanumerics.

The list of exploitable vulnerabilities would be sent separately from the above list. It wouldn’t show the actual CVEs, but just the short identifiers for them, along with their status (exploitable or not); of course, this list could still be encrypted. And if that isn’t enough security, I’m sure there are more security layers that could be placed on top of that, when required by the nature or use of the software product.

There’s a lot more that can be said about this proposal, as well as many loose ends that will need to be tied up. I’d appreciate hearing anybody’s comments on this. This idea will be discussed in a more participatory context at some point. In the meantime, I’m not saying we need to drop the idea of VEX documents or of real-time VEX, since there are use cases for both types of notifications.

But I don’t want to play down the magnitude of the problem I’m trying to address. Without at least partial resolution of the vulnerability exploitability problem (whether it’s VEX or something else), SBOMs will never be widely (or probably even narrowly) used.

Period.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] I still believe my real-time VEX idea – which will be submitted as part of a standards application to the IETF next year – is the best way to handle VEX as currently conceived. It will be much easier on both suppliers and the user’s consumption tools if “not affected” status notifications can be provided through an API, rather than by preparing and distributing a machine-readable document.