From the beginning of the NTIA Software Component
Transparency (SBOM) Initiative (“the Initiative”) in 2018, some of the private
and public sector organizations represented in the workgroup meetings realized
there was a serious problem that would need to be addressed before SBOMs could
be distributed and used in volume. This problem was due to the fact that the
majority of vulnerabilities that have been identified (in the National
Vulnerability Database or another vulnerability database) in a component of a
product are not exploitable in the product itself. That is, even though an
attack on a component, which was based on that vulnerability, would probably
succeed, it would not succeed if it were instead targeted at the product
itself. For more information on this idea see my post titled “What
is the purpose of VEX?”.
To address that problem, the Initiative described (but never
specified) a new document format called VEX, which stands for Vulnerability
Exploitability eXchange. Why did the effort initially focus on a document
format? It was most likely because VEX was meant to be a “corrective” for
SBOMs, and therefore needed to “follow” the distribution of SBOM documents.
The idea behind VEX was (and continues to be) to let the
user know the exploitability status of each of the component
vulnerabilities they have discovered in the National Vulnerability Database
(NVD) or a similar database – that is, whether the component vulnerability is
or isn’t exploitable in the product itself. In a VEX document, one or more vulnerabilities
are listed, as well as the status of each vulnerability in one or more versions
of a product.
When a component vulnerability isn’t exploitable in a
particular product version, this may be due to various reasons. In some cases,
it may be because the vulnerable component isn’t even included in that version
(for example, the supplier has issued a VEX document saying that none of the versions
of one of their products includes the log4j library, and therefore it isn’t vulnerable
to the log4shell vulnerability). In other cases, it may be for another reason,
like the supplier has already patched their product to remove that
vulnerability (yet the vulnerable component remains in the product). Still
another reason might be that the way the supplier implemented the vulnerable
component in the product removed the vulnerability, either intentionally or
inadvertently.
In these and similar cases, the supplier should notify their
users that the vulnerability in question is not exploitable in their product,
even though an SBOM for the product indicates the vulnerable component is
included in the product/version. If the supplier doesn’t do this, their users
may waste a lot of their and the help desk’s time trying to track down this
vulnerability and demand to know the supplier will issue a patch for it. The
more component vulnerabilities turn out not to be exploitable in the product
itself, the more VEX documents will need to be issued, and with greater
frequency.
What has been the experience with VEX so far? Given the
above, many people might expect that suppliers would be issuing VEX documents
to their end users (i.e., customers) in great quantities, since just about all
such suppliers would be likely to feel the need to issue VEX notifications as often
as possible.
However, suppliers today are not issuing VEX documents in
anywhere near the volume required for VEX to be considered successful. In fact,
it is likely that only a handful of suppliers are issuing VEXes to their
customers at all. The fact that VEX documents aren’t being distributed in
volume leads directly to the fact that SBOMs themselves are not being issued or
consumed in anywhere near the volume required for them to be successful (except
by developers, for their internal product vulnerability management purposes). Along
with the “naming problem”, the author considers this to be one of the two “showstopper”
problems preventing widespread distribution and use of SBOMs.
Why aren’t suppliers providing VEX documents to their
customers? When asked this question, suppliers usually state that their
customers aren’t asking for VEXes. Why aren’t the customers asking for them? The
customers often point to the lack of easy-to-use, commercially supported tools
as the reason for this. Specifically, there are no “complete” tools that a) ingest
an SBOM for a particular software product/version, b) look up vulnerabilities
for the components and track them over time, including checking daily
for new vulnerabilities, and c) as VEX documents become available for that
product/version, utilize them to determine which of the component vulnerabilities
are exploitable in the product itself and which are not. While SBOMs can in
some cases be used without a complete tool, there are few if any ways that a
VEX document can be used without such a tool.
However, the lack of an SBOM/VEX consumption tool does not
in itself explain why VEX documents are not being distributed to end users;
instead, the lack of a tool is a symptom, not a cause of the problem. The
author has identified two primary reasons why VEX documents aren’t being
distributed.
First, the goal of VEX seems to have been misidentified. A
software user’s primary concern, with respect to software component
vulnerability management, should be identifying exploitable component vulnerabilities;
after all, these are the vulnerabilities the user needs to worry about.
However, the three VEX documents produced by CISA so far (all of which are
available at https://www.cisa.gov/sbom),
as well as an unpublished
NTIA document (pages 8 and 9) from 2021, all describe VEX as primarily
intended to identify component vulnerabilities that are not exploitable
in a software product.
The reason for this assertion is probably that it would be
risky to provide a document (machine readable or otherwise) to software customers
that includes a list of exploitable (i.e., unpatched) vulnerabilities. No
matter how many NDAs a customer may have signed, there is a non-zero chance that
any document provided to any organization will fall into the wrong hands at
some point. If the document lists exploitable vulnerabilities, the consequences
of this happening might be very serious.
The second reason why VEX documents are not being distributed
to end users is that, in order for the documents to fulfill their intended
function (i.e., identifying non-exploitable component vulnerabilities in a
software or intelligent device product/version), they will most likely need to
be distributed in huge quantities. In fact, a single large supplier might need
to issue thousands or even tens of thousands of VEX documents every day.
Moreover, large software end users might have to receive and process thousands
of VEX documents every day.
The
following example illustrates why the above estimates are not at all
unreasonable. The example assumes the following:
1.
A version of a software product (“product/version”) has 150 components
(an industry average estimate), that are listed in a current SBOM.
2.
The National Vulnerability Database (or another major vulnerability
database) lists one vulnerability for half of those components, for a total of
75 vulnerabilities.
3.
On any given day, the exploitability status of at least one of those vulnerabilities
changes, for example from “under investigation” to “affected” or “not affected”
(there can be changes in the list of vulnerabilities as well, including a new vulnerability
for one of the components being identified in a vulnerability database). Since
it is important that customers of a software product learn as soon as possible
when any vulnerability that affects a component changes its exploitability
status, this means the supplier should send out a new VEX document containing
the notification of at least this status change as soon as they learn of the
change. Thus, the supplier is likely to send out at least one VEX document for
that product/version every day.
4.
A supplier needs to provide VEX information for every version of their
product that is still being actively used. Even though the supplier might have
previously issued hundreds of versions of a product (including patches and new
builds, each of which should count as a new version), we will assume here that
only ten of those versions are being actively used at any point in time. Since
the supplier will issue one VEX document each day for each of those ten
versions of the product, they will issue ten different VEX documents every day,
for one product.
5.
Furthermore, suppose the supplier has 200 separate products (i.e., the products
are separate enough to require their own SBOM. This includes different options within
a single “product family”, e.g. multiuser vs. single user or English vs.
Spanish). Since each of those products will require ten VEX documents a day,
this means the supplier will need to issue 2,000 different VEX documents
every day for those 200 products.
6.
Of course, having 200 separate products would probably put the supplier squarely
in the smaller half of producers of software products and/or intelligent
devices. If a supplier is five times as large and therefore sells 1,000
products, yet the other assumptions remain unchanged, this means the supplier will
have to produce around 10,000 different VEX documents every day. Moreover, the
very large software and device suppliers like Microsoft™ and Cisco™ will almost
certainly need to produce some large multiple of that number.
Another consideration is that VEX documents will need to be
“pushed” out to end users; it will never be enough simply to make them
available on a portal. Moreover, to protect the information as much as
possible, the documents should never be sent out via email; instead, they will
need to be pushed directly to each customer’s SBOM/VEX consumption
tool. Of course, doing this, even for 100 different VEX documents per day, will
be a huge logistical challenge. But, if each of those documents needs to be
sent to 1,000 customers, that means the supplier needs to send out 100,000
individual VEX documents – all to different IP addresses – every day. And if
the supplier has to send out 10,000 individual documents per day (as described
above) to 1,000 addresses each, that amounts to ten million individual
VEX documents sent every day, mostly to different addresses. Clearly, this is
not a sustainable situation.
Given these problems, the author has come to believe it was
a mistake for VEX to be originally formulated as a document format. Instead, it
should have been described as an internet-based service, which is accessible by
an API utilized by an SBOM “consumption” tool (i.e., a tool utilized by software
consumers for software component vulnerability management, not software
suppliers – they have various tools available for that now).
Under this assumption, it will be reasonably safe for a
supplier to pass information on exploitable, as well as non-exploitable,
component vulnerabilities to their customers’ tools (which may be operated by a
third party service provider).
Most importantly, this service would save software suppliers
the huge amounts of time and effort required to send out perhaps tens or
hundreds of thousands of individual VEX documents every day; in fact, given those
huge amounts, it is hard to see that anything other than a service like this
one would even be considered.
This solution would simply require a supplier to maintain and
update a server, which the supplier’s customers could access at any time via an
authenticated API. The server would store the VEX information to be provided
through the API, including, for every product and version being actively used, each
component vulnerability being tracked and its exploitability status (“affected”,
“not affected”, “fixed” or “under investigation”), as well as status
justifications and mitigations. The supplier would need to update this
information only when it changes.
Of course, building and maintaining this server will require
some effort from the supplier (although it will be several orders of magnitude
less than what sending out huge numbers of VEX documents every day would
require!). However, it is very likely that third party services will appear,
that will construct cloud based “VEX servers”. These third party services will
take the development and maintenance work out of the supplier’s hands; the
third party will be able to utilize a single infrastructure to service hundreds
or thousands of other suppliers as well. Of course, the supplier will need to
update their own data themselves, but that is all they will have to do.
I believe that the “VEX Server” described above will enable
software suppliers to share VEX information with their customers as frequently
as needed (since the customer will decide how often they want to retrieve the
information via the API), without imposing a huge logistics burden on the
supplier. Of course, this will increase the availability and frequency of user
access to VEX information, but even more importantly, it will increase distribution
and use of SBOM information. This is because one of the two “showstopper”
problems inhibiting widespread distribution and use of SBOMs by end users (along
with the naming problem) is the fact that the document based VEX “solutions”
currently being discussed are simply not realistic. I believe the solution
described above is a realistic one.
Note there is a secondary VEX use case for which a
document-based solution should be more than adequate. This is partly because
this secondary case doesn’t carry the same urgency (and therefore the need for
such frequently updated documents) as the primary use case described above. I’ll
describe that case in a later post.
Any opinions expressed in this
blog post are strictly mine and are not necessarily shared by any of the
clients of Tom Alrich LLC. If you would
like to comment on what you have read here, I would love to hear from you.
Please email me at tom@tomalrich.com.