VEXual confusion reigns!

My previous post described how the proposed OpenVEX format doesn’t have the capability to address the most important VEX use case, since it doesn’t support version ranges. However, OpenVEX has an even bigger problem with regard to that use case: It doesn’t address it at all. In fact, it seems that Chainguard, the organization that led the development of OpenVEX, doesn’t understand what VEX was meant to do, period.

However, the fact that Chainguard doesn’t understand VEX does not mean that OpenVEX isn’t a worthwhile endeavor; in fact, it seems to me that it fulfills an important need. It’s just that the need it fulfills has nothing to so with either SBOM or VEX.

What is the need that VEX was designed to fulfill? There’s no better description of that than this (unfortunately unpublished) document, which was drafted by Dr. Allan Friedman in 2021. At the time, he was part of the NTIA, although now he’s part of CISA. Here’s my summary of that use case:

1.      When a supplier provides a new SBOM for a version of a software product or intelligent device (hereinafter referred to collectively as “product”) to the customers of that product, the users will probably look in the National Vulnerability Database (NVD) for vulnerabilities that apply to product components listed in the SBOM (there are a number of tools that do this, although the leader is the open source Dependency-Track).

2.      Let’s say a user identifies 100 component vulnerabilities of varying levels of severity (indicated by the CVSS score) and exploitability (indicated by the EPSS score, although this sense of “exploitable” is very different from the VEX sense of the word). Furthermore, let’s say that the user decides that 20 of these have a high enough CVSS and/or EPSS score that they want to pursue them right away. They will jump on the phone and start pressing the supplier on all 20 of these: When will the supplier fix them?

3.      The user will find that the supplier’s help line has a long wait, since a lot of other users of the same product were able to identify the same vulnerabilities in the NVD, and also concluded that they needed to contact the supplier about these vulnerabilities.

4.      When the user finally reaches the help desk, they will already be hot under the collar (a result of their 25-minute wait, no doubt), and will be a little impolite as a result. They’ll demand to know when the supplier will fix each of these vulnerabilities, with “tomorrow” or “the day after tomorrow” being the only acceptable answers.

5.      However, instead of giving them the answer they want, the somewhat weary help desk person will answer – for the 57^th time that day – that there’s no need to worry about 19 of those 20 vulnerabilities, since they’re not “exploitable” in the product itself. This means that, if you consider the component as a standalone product, it’s subject to the vulnerability, but because of how the component was installed in the product (or sometimes because of the fact that it was not installed, yet showed up in the SBOM for another reason), the product itself isn’t vulnerable. And since a hacker is seldom able to directly attack just one component of a product, rather than the entire product, they will never be able to take advantage of this vulnerability. In other words, the 19 vulnerabilities the user believed to be exploitable were in fact false positives.

6.      Of course, the user will be glad to hear that the problem is only 1/20 the size of what they thought it was; on the other hand, they’ll be miffed that the supplier didn’t proactively inform them that 19/20 of the vulnerabilities in this product and version weren’t exploitable.

7.      The next day, the user receives an SBOM for a different product from a different supplier. However, the experience is roughly the same: The user identifies around ten important vulnerabilities attributable to components in the SBOM, but when they contact the supplier, they’re told that all but one or two of them aren’t exploitable in the product.

8.      Again, this is good news, but now the user starts to get angry that the suppliers are sending them SBOMs, but not warning them that most of the vulnerabilities they learn about aren’t exploitable. This is wasting a lot of their time, as well as the help desk’s time. Why can’t the supplier let them know about these unexploitable vulnerabilities, without their having to waste all this time to learn about them?

9.      This goes on for another few weeks. Finally, the user decides it’s not worth it to receive SBOMs anymore. Sure, it’s nice that there are so few exploitable component vulnerabilities, but the value of the time they waste in learning this outweighs the advantage of learning about the small number of exploitable component vulnerabilities. The next time a supplier offers to start sending them SBOMs, they tell the supplier what they can do with their SBOMs (it’s not nice, by the way).

When the NTIA Software Component Transparency Initiative began in 2018, the suppliers involved in it (mostly very large organizations) soon came to realize that simply distributing SBOMs to their customers, without at the same time implementing some means to inform them of non-exploitable component vulnerabilities that they were likely to find, would literally be counterproductive. The group decided that the best way to address this problem was to develop a machine-readable document format that could be utilized in conjunction with the machine-readable SBOM format. That format was called VEX – and once the acronym was decided on, the name Vulnerability Exploitability eXchange was developed to go with the acronym.

This is the use case that VEX was designed for; it is still by far the most important one. As you can see, it’s very tied to the idea of software component transparency. VEX only becomes necessary when suppliers start to distribute SBOMs; moreover, suppliers are unlikely to want to distribute SBOMs without a mechanism like VEX being in place, for fear of being overwhelmed with completely unnecessary support calls. Even worse, the suppliers are worried that their customers will waste huge amounts of their own time in calling the help desk, and they will blame the suppliers for this. There’s no doubt in my mind that one of the two or three primary reasons why SBOMs aren’t being widely (or even narrowly) distributed today is that there’s so much uncertainty around VEX.

Is this the use case that OpenVEX addresses? Let’s find out. First, we’ll look at the spec published on GitHub. However, the spec says nothing about the purpose of VEX, only that this spec meets “all requirements of a valid VEX implementation as defined in the Minimum Requirements for Vulnerability Exploitability eXchange (VEX) document published on XXX (sic) by the VEX working group coordinated by the Cybersecurity & Infrastructure Security Agency (CISA).”

This is an interesting statement, since the document they refer to has not only not been published, but the working group only finished with it yesterday, February 13 (the OpenVEX spec was created at least a month ago). And when the working group has finished it, it will probably be reviewed for a couple of months by CISA lawyers and others, so it will undoubtedly be changed further. Since the link provided by Chainguard, http://example.com, goes nowhere, here’s the link to the Google Docs version of the document (I wouldn’t normally link to an in-progress document, even though there are no restrictions on who can view it. However, since Chainguard seems to think the document is finished and published, I figured you at least ought to be able to see it).

Since the OpenVEX spec doesn’t say anything about the purpose of this format, we have to be satisfied with the single “Sample Scenario” that it provides:

“As an example, consider the following evolution of a hypothetical impact analysis:

1. A software author becomes aware of a new CVE related to their product. Immediately, the author starts to check if it affects them.
2. The investigation determines the product is affected.
3. To protect their users, the author mitigates the CVE impact via a patch or other method before the vulnerable component issues a patch.

Without VEX data, users scanning the author's software will simply get a third party alert with no details on how the status is evolving. Most critically, when the product is patched (in #3), the alert becomes a false positive.

To inform consumers downstream of the vulnerability evolution, the author can issue a VEX document (in #1) when the CVE is published to let their users know it is under investigation. In #2, when the product is known to be affected, the author can ship a new VEX document, stating the product is affected and possibly some additional advice, like temporary mitigation instructions. Finally when the product is patched, its SBOM can be complemented with a new VEX document informing it is no longer affected by the CVE. Scanners could consume this document and stop alerting about the CVE as it no longer impacts the product.”

(Audible sigh) Where to begin? Here are my questions:

1.      Why would a supplier advertise that their product might be affected by a vulnerability before they have a patch for the vulnerability? After all, when any document is published, even to a small group of customers that have all signed NDAs, you have to assume it will make it to the hacker community within hours. Wouldn’t it be better to…you know…say nothing at all until the patch (or an effective mitigation) is ready?

2.      The penultimate sentence says, “..when the product is patched, its SBOM can be complemented with a new VEX document informing it is no longer affected by the CVE.” In the world of commercial software that I’m most familiar with, and which I would hope the contributors to this document should know something about, that sentence simply makes no sense. The “product” is never “affected by the CVE”, and the “product” is never “patched”. Instead, one or more versions of the product is affected by the CVE, and one or more new versions contain the patch.

3.      Usually, the user of a supported version is given the option of upgrading to a patched version or applying a patch themselves. If they choose the latter option, applying the patch should increment the version number on the user’s instance, so that if the user needs to contact the help desk later and they’re asked what version of the software they have, the friendly help desk person will immediately know whether or not they’ve applied the patch. 

      The VEX document should list the status of each version that’s supported – whether it’s affected, not affected, or fixed (i.e. patched), or perhaps still under investigation. What good does it do to say “the product” is affected or not affected, unless it doesn’t have versions? And if it doesn’t have versions, then, to repeat my quote in the previous post from Bruce Lowenthal, Senior Director of Product Security at Oracle (and one of the originators of the VEX concept), any time there are multiple code bases that have the same version number (or no versioning at all), the product becomes unsupportable. I’m not sure I’ve ever seen a commercial product that wasn’t versioned.

Most importantly, what does this scenario have to do with the paradigmatic VEX use case above? That use case arises because an SBOM is distributed. However, in this scenario an SBOM is only mentioned in the last step, after the product has been patched. Even though the scenario indicates that three VEXes are distributed (my, this supplier seems to love distributing VEXes! I won’t ask exactly how they’re going to get them to all their customers, but it must be really easy, since they like to produce them so often), only the last one seems to have anything at all to do with the VEX use case I described.

But why, in step 3, would the scanner indicate that the vulnerability is present in the product, when in fact it has been patched? It seems this is because scanners often throw off false positives. When that happens, an OpenVEX document that says the product is “fixed” will certainly be helpful, since it will override this false positive designation. I have no argument with that idea.

And indeed, this seems to be the main use case for OpenVEX: Given that scanners throw off a lot of false positives, receiving documents stating that particular vulnerabilities aren’t in fact exploitable will reduce the amount of time that users waste chasing these false positives (as well as bothering the help desk about them). Indeed, this is a fairly good summary of the use case described by Dan Lorenc, the CEO of Chainguard, in a web presentation to an Open SSF working group on January 25 (more on that presentation, and the misstatements that were made by Dan and a couple of guest speakers, in another post soon).

This is a use case that I’d never thought of, and which – although I don’t know a lot about scanners – seems to be a very useful one. While it has nothing particularly to do with SBOMs or with the use case that VEX was developed to address, I certainly wish Chainguard good luck with whatever business they expect to generate out of OpenVEX (if nothing else, this has probably already generated them a lot of good publicity. Nothing wrong with good publicity, of course).

However, Chainguard is wrong in claiming that OpenVEX is in any way superior to, or even competing in the same league as, the two existing VEX specs: one based on CSAF, the other on CycloneDX VEX. It’s something like an up-and-coming hockey team claiming to be competitive in football with two NFL teams (of course, a good hockey team might have a decent chance against the alleged professional football team in my town, Chicago. Fortunately, they’ll be decamping for the suburbs in the next 4-5 years. Couldn’t come soon enough for me). Guys, you’ve defined a whole new product category: false positive reduction for scanning tools. This is a good opportunity. Don’t waste it by pretending to be something you’re not, which is a VEX format that’s comparable to the two existing formats. It’s not going to end well.

On the other hand, I have no problem if Chainguard wants to retain the “VEX” name. But I’m also fine if they want to name the format “Larry”, “Sally”, the “Magna Carta” or the “Tibetan Book of the Dead”. None of these names are trademarked, as far as I know.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.