The problem with VEX documents

 

From the beginning of the NTIA Software Component Transparency (SBOM) Initiative (“the Initiative”) in 2018, some of the private and public sector organizations represented in the workgroup meetings realized there was a serious problem that would need to be addressed before SBOMs could be distributed and used in volume. This problem was due to the fact that the majority of vulnerabilities that have been identified (in the National Vulnerability Database or another vulnerability database) in a component of a product are not exploitable in the product itself. That is, even though an attack on a component, which was based on that vulnerability, would probably succeed, it would not succeed if it were instead targeted at the product itself. For more information on this idea see my post titled “What is the purpose of VEX?”.

To address that problem, the Initiative described (but never specified) a new document format called VEX, which stands for Vulnerability Exploitability eXchange. Why did the effort initially focus on a document format? It was most likely because VEX was meant to be a “corrective” for SBOMs, and therefore needed to “follow” the distribution of SBOM documents.

The idea behind VEX was (and continues to be) to let the user know the exploitability status of each of the component vulnerabilities they have discovered in the National Vulnerability Database (NVD) or a similar database – that is, whether the component vulnerability is or isn’t exploitable in the product itself. In a VEX document, one or more vulnerabilities are listed, as well as the status of each vulnerability in one or more versions of a product.

When a component vulnerability isn’t exploitable in a particular product version, this may be due to various reasons. In some cases, it may be because the vulnerable component isn’t even included in that version (for example, the supplier has issued a VEX document saying that none of the versions of one of their products includes the log4j library, and therefore it isn’t vulnerable to the log4shell vulnerability). In other cases, it may be for another reason, like the supplier has already patched their product to remove that vulnerability (yet the vulnerable component remains in the product). Still another reason might be that the way the supplier implemented the vulnerable component in the product removed the vulnerability, either intentionally or inadvertently.

In these and similar cases, the supplier should notify their users that the vulnerability in question is not exploitable in their product, even though an SBOM for the product indicates the vulnerable component is included in the product/version. If the supplier doesn’t do this, their users may waste a lot of their and the help desk’s time trying to track down this vulnerability and demand to know the supplier will issue a patch for it. The more component vulnerabilities turn out not to be exploitable in the product itself, the more VEX documents will need to be issued, and with greater frequency.

What has been the experience with VEX so far? Given the above, many people might expect that suppliers would be issuing VEX documents to their end users (i.e., customers) in great quantities, since just about all such suppliers would be likely to feel the need to issue VEX notifications as often as possible.

However, suppliers today are not issuing VEX documents in anywhere near the volume required for VEX to be considered successful. In fact, it is likely that only a handful of suppliers are issuing VEXes to their customers at all. The fact that VEX documents aren’t being distributed in volume leads directly to the fact that SBOMs themselves are not being issued or consumed in anywhere near the volume required for them to be successful (except by developers, for their internal product vulnerability management purposes). Along with the “naming problem”, the author considers this to be one of the two “showstopper” problems preventing widespread distribution and use of SBOMs.

Why aren’t suppliers providing VEX documents to their customers? When asked this question, suppliers usually state that their customers aren’t asking for VEXes. Why aren’t the customers asking for them? The customers often point to the lack of easy-to-use, commercially supported tools as the reason for this. Specifically, there are no “complete” tools that a) ingest an SBOM for a particular software product/version, b) look up vulnerabilities for the components and track them over time, including checking daily for new vulnerabilities, and c) as VEX documents become available for that product/version, utilize them to determine which of the component vulnerabilities are exploitable in the product itself and which are not. While SBOMs can in some cases be used without a complete tool, there are few if any ways that a VEX document can be used without such a tool.

However, the lack of an SBOM/VEX consumption tool does not in itself explain why VEX documents are not being distributed to end users; instead, the lack of a tool is a symptom, not a cause of the problem. The author has identified two primary reasons why VEX documents aren’t being distributed.

First, the goal of VEX seems to have been misidentified. A software user’s primary concern, with respect to software component vulnerability management, should be identifying exploitable component vulnerabilities; after all, these are the vulnerabilities the user needs to worry about. However, the three VEX documents produced by CISA so far (all of which are available at https://www.cisa.gov/sbom), as well as an unpublished NTIA document (pages 8 and 9) from 2021, all describe VEX as primarily intended to identify component vulnerabilities that are not exploitable in a software product.

The reason for this assertion is probably that it would be risky to provide a document (machine readable or otherwise) to software customers that includes a list of exploitable (i.e., unpatched) vulnerabilities. No matter how many NDAs a customer may have signed, there is a non-zero chance that any document provided to any organization will fall into the wrong hands at some point. If the document lists exploitable vulnerabilities, the consequences of this happening might be very serious.

The second reason why VEX documents are not being distributed to end users is that, in order for the documents to fulfill their intended function (i.e., identifying non-exploitable component vulnerabilities in a software or intelligent device product/version), they will most likely need to be distributed in huge quantities. In fact, a single large supplier might need to issue thousands or even tens of thousands of VEX documents every day. Moreover, large software end users might have to receive and process thousands of VEX documents every day.

The following example illustrates why the above estimates are not at all unreasonable. The example assumes the following:

1.      A version of a software product (“product/version”) has 150 components (an industry average estimate), that are listed in a current SBOM.

2.      The National Vulnerability Database (or another major vulnerability database) lists one vulnerability for half of those components, for a total of 75 vulnerabilities.

3.      On any given day, the exploitability status of at least one of those vulnerabilities changes, for example from “under investigation” to “affected” or “not affected” (there can be changes in the list of vulnerabilities as well, including a new vulnerability for one of the components being identified in a vulnerability database). Since it is important that customers of a software product learn as soon as possible when any vulnerability that affects a component changes its exploitability status, this means the supplier should send out a new VEX document containing the notification of at least this status change as soon as they learn of the change. Thus, the supplier is likely to send out at least one VEX document for that product/version every day.

4.      A supplier needs to provide VEX information for every version of their product that is still being actively used. Even though the supplier might have previously issued hundreds of versions of a product (including patches and new builds, each of which should count as a new version), we will assume here that only ten of those versions are being actively used at any point in time. Since the supplier will issue one VEX document each day for each of those ten versions of the product, they will issue ten different VEX documents every day, for one product.

5.      Furthermore, suppose the supplier has 200 separate products (i.e., the products are separate enough to require their own SBOM. This includes different options within a single “product family”, e.g. multiuser vs. single user or English vs. Spanish). Since each of those products will require ten VEX documents a day, this means the supplier will need to issue 2,000 different VEX documents every day for those 200 products.

6.      Of course, having 200 separate products would probably put the supplier squarely in the smaller half of producers of software products and/or intelligent devices. If a supplier is five times as large and therefore sells 1,000 products, yet the other assumptions remain unchanged, this means the supplier will have to produce around 10,000 different VEX documents every day. Moreover, the very large software and device suppliers like Microsoft™ and Cisco™ will almost certainly need to produce some large multiple of that number.

 

Another consideration is that VEX documents will need to be “pushed” out to end users; it will never be enough simply to make them available on a portal. Moreover, to protect the information as much as possible, the documents should never be sent out via email; instead, they will need to be pushed directly to each customer’s SBOM/VEX consumption tool. Of course, doing this, even for 100 different VEX documents per day, will be a huge logistical challenge. But, if each of those documents needs to be sent to 1,000 customers, that means the supplier needs to send out 100,000 individual VEX documents – all to different IP addresses – every day. And if the supplier has to send out 10,000 individual documents per day (as described above) to 1,000 addresses each, that amounts to ten million individual VEX documents sent every day, mostly to different addresses. Clearly, this is not a sustainable situation.

Given these problems, the author has come to believe it was a mistake for VEX to be originally formulated as a document format. Instead, it should have been described as an internet-based service, which is accessible by an API utilized by an SBOM “consumption” tool (i.e., a tool utilized by software consumers for software component vulnerability management, not software suppliers – they have various tools available for that now).

Under this assumption, it will be reasonably safe for a supplier to pass information on exploitable, as well as non-exploitable, component vulnerabilities to their customers’ tools (which may be operated by a third party service provider).

Most importantly, this service would save software suppliers the huge amounts of time and effort required to send out perhaps tens or hundreds of thousands of individual VEX documents every day; in fact, given those huge amounts, it is hard to see that anything other than a service like this one would even be considered.

This solution would simply require a supplier to maintain and update a server, which the supplier’s customers could access at any time via an authenticated API. The server would store the VEX information to be provided through the API, including, for every product and version being actively used, each component vulnerability being tracked and its exploitability status (“affected”, “not affected”, “fixed” or “under investigation”), as well as status justifications and mitigations. The supplier would need to update this information only when it changes.

Of course, building and maintaining this server will require some effort from the supplier (although it will be several orders of magnitude less than what sending out huge numbers of VEX documents every day would require!). However, it is very likely that third party services will appear, that will construct cloud based “VEX servers”. These third party services will take the development and maintenance work out of the supplier’s hands; the third party will be able to utilize a single infrastructure to service hundreds or thousands of other suppliers as well. Of course, the supplier will need to update their own data themselves, but that is all they will have to do.

I believe that the “VEX Server” described above will enable software suppliers to share VEX information with their customers as frequently as needed (since the customer will decide how often they want to retrieve the information via the API), without imposing a huge logistics burden on the supplier. Of course, this will increase the availability and frequency of user access to VEX information, but even more importantly, it will increase distribution and use of SBOM information. This is because one of the two “showstopper” problems inhibiting widespread distribution and use of SBOMs by end users (along with the naming problem) is the fact that the document based VEX “solutions” currently being discussed are simply not realistic. I believe the solution described above is a realistic one.

Note there is a secondary VEX use case for which a document-based solution should be more than adequate. This is partly because this secondary case doesn’t carry the same urgency (and therefore the need for such frequently updated documents) as the primary use case described above. I’ll describe that case in a later post.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.