Is a vulnerability exploitable when it’s not exploitable?

In my last post, I described an important issue regarding VEX that hasn’t yet been discussed in any published document (but should be): the question of how to handle the fact that how a user interprets a statement that a vulnerability is exploitable in a product or not (which of course is the purpose of VEX) is going to depend heavily on what kind of attacker they think they face. This is important, since if the user’s assumption on hacker skill level diverges substantially from the supplier’s assumption, they may draw exactly the wrong conclusion from what the supplier intended, when they designated a vulnerability as exploitable or not in their product.

There is another issue, which like the first hasn’t been discussed in print at all but is also important (although it was discussed a good deal within the NTIA-now-CISA VEX working group, as was the hacker skill level issue): the question of how the supplier, in preparing a VEX statement, should account for the environment in which it’s installed, especially any security controls. Here are some forms in which this question comes up:

1.      Say a vulnerability is normally not exploitable unless a default configuration setting is changed. If a user makes that change (either deliberately or inadvertently), does that make the vulnerability is exploitable in the product, meaning its status in a VEX statement should be “affected”?

2.      Now let’s say a vulnerability, which would otherwise be exploitable, will be rendered not exploitable by a product configuration setting recommended in writing by the supplier (but not made part of the default configuration). Should the supplier list the VEX status of that vulnerability in the product as “affected”? After all, the user might not follow their recommendation.

3.      If the supplier recommends their product not be connected to the internet and says a particular vulnerability isn’t exploitable, yet a user connects the product to the internet anyway and gets compromised through an exploit of that vulnerability, does this mean the vulnerability was in fact exploitable?

4.      What if the supplier says a vulnerability isn’t exploitable if their product isn’t connected to the internet, yet because of how the product operates, it would be difficult to use the product efficiently if it weren’t connected? If the vulnerability is exploited and the user suffers some loss because of that, is this the supplier’s fault or the user’s? After all, the supplier said the vulnerability isn’t exploitable if the product isn’t connected to the internet.

5.      What if a supplier recommends deploying their product securely but doesn’t describe what that means, and a user mistakenly doesn’t put the product behind a firewall? If the product gets compromised because of the lack of a firewall, is this the supplier’s fault for not explicitly stating the need for one?

6.      What if a supplier assumes that users will normally install a web application firewall, yet doesn’t state that explicitly in their installation instructions? A user has a normal stateful inspection firewall but not a WAF, and they get compromised through a vulnerability that the WAF would have protected against. Is this their fault or the supplier’s? Or is it neither’s fault?

7.      What if the supplier has issued a patch for a vulnerability? Is the vulnerability exploitable or not?

Here’s what I think the rule should be for cases like this (and they come up all the time): The supplier should assume that a) The user has followed any configuration or environment settings the supplier has recommended in writing, and b) the user hasn’t changed any default settings, and c) The user has followed basic security best practices (although the supplier doesn’t have to state exactly what they think those best practices are).

However, the supplier should, whenever they think there might be any question in the user’s mind about how the product should be configured, or what would constitute a security best practice in a particular case, always take an extra step and state explicitly what they have in mind. Here is how I would answer each question:

1.      A default configuration setting constitutes a very strong recommendation from the supplier that they want the user to leave it in place, meaning the supplier is justified in believing that the user won’t change the setting. Thus, the vulnerability would have a “not affected” (i.e., not exploitable) VEX status. However, it certainly might save both the supplier and their customers a lot of headaches if the supplier points out explicitly in their documentation that the configuration setting is required to prevent exploitation of that vulnerability, and if the user changes it, they risk being compromised because of their action.

2.      This is a lot like the first question, but the difference is that in this case, the configuration setting is just recommended, not made part of the default configuration. Nevertheless, in my opinion, the supplier is justified in assuming the user has followed the recommendation. Therefore, the VEX status should be “not affected”. Again, it would help if the supplier explicitly pointed out in their documentation that changing this default setting might make them vulnerable to an attack.

3.      This is in effect the same as the second question. The supplier has made an installation recommendation and is justified in assuming it will be followed. Again, the status for the vulnerability should be “not affected”.

4.      This is superficially the same as the third question, but is in effect very different. I would hope that no supplier would take such a cynical approach as to require the user do something that makes their product unusable, but I’m sure there are a few that can and will do this. In this case, the supplier is assigning a “not affected” status, but they’re basing that on the assumption that the user won't use the product as it's intended to be used - which is obviously horse manure. A customer who sees something like this happen more than once might want to re-evaluate whether they wish to stay with this supplier. Who knows what other shady steps they’re taking?

5.      In my opinion, a firewall is basic security best practice. No supplier should have to tell their users to put in a firewall. Therefore, the supplier should say the vulnerability isn’t exploitable (“not affected”).

6.      I don’t consider a WAF to be a basic security best practice. If the supplier thinks one is needed to protect their product, they should state it explicitly, not just assume it. Therefore, the supplier should list the status of the vulnerability as “affected” and list the WAF as a mitigation for the vulnerability.

7.      It’s important to keep in mind that the question is never whether the product itself is vulnerable, but whether a particular version of the product is vulnerable. If Product A version 2.0.0 is vulnerable to CVE-2023-12345 but a patch has been issued for that vulnerability, the patched version must have a new version string, say v2.0.1. Moreover, if the user applies the patch to v2.0.0, the version of their instance should change to 2.0.1. This is required both for VEX to work and for support reasons, since if multiple codebases have the same version number, the product becomes unsupportable. So, the status of v2.0.0 should be “affected”, but the status of v2.0.1 should be “fixed” or “not affected”. The mitigation that goes with the “affected” status for v2.0.0 might be “upgrade to v2.0.1” or “Apply patch XYZ, available at (URL)”, or perhaps both.

This issue, as well as the hacker skill level issue that I discussed in my previous post, is one that needs to be discussed and documented. With both issues, there isn’t one “right” answer, but there does need to be some articulated position, so that both suppliers and users will interpret the status settings correctly. It’s to address issues like this that a VEX playbook is needed. It might be developed by the CISA VEX working group; it might be developed by the SBOM Forum; or it might be developed by some other group where people discuss issues with SBOM and VEX. But it needs to be developed. 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

How do you account for hacker skill level in VEX?

The informal group I formed last spring to discuss problems that are holding back widespread (or even narrowspread) distribution and use of SBOMs has been focusing on VEX recently. This isn’t an accident, since I (and many others) now believe that the biggest factor inhibiting the success of SBOMs is the fact that VEX is such a poorly understood concept – mainly because there’s so little documentation of VEX, and what little there is contradicts itself. For example, the recently approved, but not yet published, VEX “Minimum Requirements” document directly contradicts the previous CISA Use Cases document on several key points, without even mentioning that fact. The user is left to discover this contradiction on their own.

The idea of VEX came about to address the case in which a user has received an SBOM and identified a number of vulnerabilities in the components listed in the SBOM, by looking them up in the NVD or another vulnerability database. Since there’s broad agreement in the software security community that over 90% of vulnerabilities in components of a product aren’t exploitable in the product itself, the supplier needs to have a means to let their customers know about those non-exploitable vulnerabilities, before they inundate the help desk with calls about both types of component vulnerabilities. VEX is a tool they can use to do that (although I and other members of the SBOM Forum now believe that VEX, as currently conceived, will never be able to accomplish this purpose. We’re now discussing how the VEX “delivery vehicle” can be improved to make it an effective one, although we’re not trying to change the basic use case for VEX itself).

One of the big issues that has come up multiple times in the SBOM Forum discussions (which also was discussed multiple times by the NTIA VEX Working Group in 2021. We thought we’d resolved the question then, but we didn’t put it into a document. That turns out to have been a big mistake) is how the supplier accounts for the attacker’s skill level in a VEX document.

The problem is that the supplier’s decision whether a particular component vulnerability is exploitable or not within one or more versions of one of their products will depend heavily on what they assume to be the likely skill level of the attacker. Consider three cases:

1.      The attacker has my skill level, which is somewhere around that of a (poorly) trained monkey. Using this assumption, almost no vulnerability should be considered exploitable.

2.      The attacker has a moderate skill level, meaning that vulnerabilities whose exploitation doesn’t require a “super hacker” should be considered exploitable, while others are not exploitable.

3.      The attacker is a super hacker, who could figure out a way to compromise a brick and make it do back flips.

Ruling out case number 1 as not at all realistic leaves cases 2 and 3. Which one of these should the supplier assume? The decision the VEX workgroup made in 2021 was that it doesn’t make sense to assume anything other than an attacker of average skills (with the understanding that none of these three levels is defined exactly and it would be impossible to do so). The problem with assuming a super hacker is that almost any vulnerability will be exploitable, so instead of saying that 90% of component vulnerabilities aren’t exploitable, we’ll have to say that literally all of them are exploitable. The supplier will have to patch every component vulnerability and their customers will have to apply all of them.

So, assuming case 3, there’s no need for VEX. Plus, this means that all the software security professionals who say that more than 90% of component vulnerabilities aren’t exploitable are grossly mistaken (and Chris Wysopal of Veracode was very mistaken when he said at RSA in 2017 that 97% of component vulnerabilities in Java apps aren’t exploitable in the app itself). Obviously, one should never rule out mass delusion, but I’d like more evidence of that before I make this assumption.

Thus, it seems that suppliers should assume that attackers will be of medium skill level, when they’re deciding whether or not to rule that a component vulnerability is exploitable in one or more versions of one of their products.

But here’s the problem: There are a number of users who assume, with a lot of justification, that they actually face the super-hackers. This can’t be dismissed as paranoia. After all, you’re not paranoid if everybody really is out to get you!

It’s not hard to find examples of this. Tim Walsh of the Mayo Clinic regularly states in the SBOM Forum meetings that he and his team prepare for the most skilled hackers. There’s a lot of justification for this assumption, since in the last couple of years, the ransomware gangs have figured out that they’re most likely to get a serious payout if they compromise a hospital, and the bigger the better (although I haven’t heard about big hospital attacks lately, so maybe their defenses are getting better. Sure hope so). Other obvious targets are electric utilities, oil refineries and military contractors (plus the military itself, of course). All these organizations would be quite justified in wanting suppliers to assume a much higher than average attacker skill level when they designate a vulnerability’s exploitability status in a VEX document.

However, it’s out of the question to require all suppliers to assume that all attackers are of the super-hacker level, unless the supplier only sells into one of the high security markets like hospital organizations (officially called “healthcare delivery organizations”, or HDOs, nowadays). After all, if 90% of component vulnerabilities aren’t exploitable by a hacker of average skill level, requiring the supplier to assume that all attackers are super-hackers means they will have to develop ten times as many patches as they would otherwise; and it will require their customers to invest ten times as much time to apply patches to the supplier’s products. Security always comes with a cost, and we shouldn’t require users and suppliers to invest ten times as much time (and therefore money, of course) in security measures, if we’re not convinced this is necessary.

How can a supplier who sells into all markets accommodate both the Mayo Clinics of the world and my local dry cleaners, which is – I’m just guessing here – perhaps not at the top of Vladimir Putin’s target list?

Fortunately, there is a vehicle to address this issue in VEX. It’s been available in CycloneDX for a couple of years, and in the spring of 2012 the CISA VEX working group published a document describing their idea for it. The document is called “VEX Status Justifications” (the CycloneDX capability is just called “justifications”).

Justifications are short, machine-readable explanations of why the supplier believes a particular vulnerability isn’t exploitable in a product/version they use; a justification is only used when the status of the vulnerability in the product is “not affected”, meaning it’s not exploitable. The justifications were developed based on the realization that some software users will not be satisfied with just learning that a vulnerability, in the supplier’s estimation, is not exploitable. Instead, they will want to know why the supplier believes this to be the case, so they can take a more conservative approach to determining exploitability than did the supplier.

For example, if a supplier is producing a CycloneDX VEX document for a version of one of their products and they list the status of CVE-2023-12345 in the product as “not affected”, they should examine the nine options for “justification”:

• code_not_present = the code has been removed or tree-shaked.
• code_not_reachable = the vulnerable code is not invoked at runtime.
• requires_configuration = exploitability requires a configurable option to be set/unset.
• requires_dependency = exploitability requires a dependency that is not present.
• requires_environment = exploitability requires a certain environment which is not present.
• protected_by_compiler = exploitability requires a compiler flag to be set/unset.
• protected_at_runtime = exploits are prevented at runtime.
• protected_at_perimeter = attacks are blocked at physical, logical, or network perimeter.
• protected_by_mitigating_control = preventative measures have been implemented that reduce the likelihood and/or impact of the vulnerability.

If the reason the supplier chose the “not affected” status for CVE-2023-12345 corresponds to one of these justifications, they should include that in the VEX document. However, if none of these describes their reason well, the supplier needs to include a textual description of their reason in the “details” field.

Of course, for Tim Walsh, just seeing any justification for a “not affected” vulnerability status isn’t enough to make him feel good. For example, I know that Tim assumes that Mayo’s networks are hostile, so the threats they face are never just external ones.

If the supplier lists the “protected_at_perimeter” option, my guess is Tim won't think that does him much good, since it assumes the only attackers will be external. So, Tim will modify the tool that interprets the VEX document to make it change the status of any vulnerability, for which the “protected_at_perimeter” justification is listed, from “not affected” to “affected”.

Tim might do the same with most of the other justifications as well. However, there is probably at least one justification that he would be willing to accept: “requires_dependency”. This means that the component that is the source of this vulnerability isn’t even present in the product (I assume this would happen when the tool that generates the SBOM mis-identifies a component altogether). He might do the same for the “code_not_present” justification, which means the component is included in the product, but the code that’s the source of the vulnerability isn’t included in the component.

Let’s say that Tim decides to accept these two justifications but reject the other seven. He’ll modify the tooling to change the status of a vulnerability from “not_affected” to “affected” if any of the seven unacceptable justifications is used, while leaving the status at “not_affected” when one of the two acceptable justifications is used.[1]

So, boys and girls, our story has a good ending after all: Tim is content, because he knows he can control the status designations of the vulnerabilities in the VEX documents he receives; they will now more closely reflect his understanding of those designations.[2] Meanwhile, the dry cleaners is content, since they don’t have to lay our serious cash to pay some security person to do ten times as much patching work on the software they use, as would otherwise be necessary.

And they all lived happily ever after.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[1] CycloneDX VEX uses different status designations than these. I’m using them here in the generic sense of the NTIA and CISA documents, in which they apply to both VEX formats. 

[2] Of course, this assumes that suppliers will always list a justification when it’s appropriate. Unfortunately, this may not be a good assumption for at least the next few years.

VEXual confusion reigns!

My previous post described how the proposed OpenVEX format doesn’t have the capability to address the most important VEX use case, since it doesn’t support version ranges. However, OpenVEX has an even bigger problem with regard to that use case: It doesn’t address it at all. In fact, it seems that Chainguard, the organization that led the development of OpenVEX, doesn’t understand what VEX was meant to do, period.

However, the fact that Chainguard doesn’t understand VEX does not mean that OpenVEX isn’t a worthwhile endeavor; in fact, it seems to me that it fulfills an important need. It’s just that the need it fulfills has nothing to so with either SBOM or VEX.

What is the need that VEX was designed to fulfill? There’s no better description of that than this (unfortunately unpublished) document, which was drafted by Dr. Allan Friedman in 2021. At the time, he was part of the NTIA, although now he’s part of CISA. Here’s my summary of that use case:

1.      When a supplier provides a new SBOM for a version of a software product or intelligent device (hereinafter referred to collectively as “product”) to the customers of that product, the users will probably look in the National Vulnerability Database (NVD) for vulnerabilities that apply to product components listed in the SBOM (there are a number of tools that do this, although the leader is the open source Dependency-Track).

2.      Let’s say a user identifies 100 component vulnerabilities of varying levels of severity (indicated by the CVSS score) and exploitability (indicated by the EPSS score, although this sense of “exploitable” is very different from the VEX sense of the word). Furthermore, let’s say that the user decides that 20 of these have a high enough CVSS and/or EPSS score that they want to pursue them right away. They will jump on the phone and start pressing the supplier on all 20 of these: When will the supplier fix them?

3.      The user will find that the supplier’s help line has a long wait, since a lot of other users of the same product were able to identify the same vulnerabilities in the NVD, and also concluded that they needed to contact the supplier about these vulnerabilities.

4.      When the user finally reaches the help desk, they will already be hot under the collar (a result of their 25-minute wait, no doubt), and will be a little impolite as a result. They’ll demand to know when the supplier will fix each of these vulnerabilities, with “tomorrow” or “the day after tomorrow” being the only acceptable answers.

5.      However, instead of giving them the answer they want, the somewhat weary help desk person will answer – for the 57^th time that day – that there’s no need to worry about 19 of those 20 vulnerabilities, since they’re not “exploitable” in the product itself. This means that, if you consider the component as a standalone product, it’s subject to the vulnerability, but because of how the component was installed in the product (or sometimes because of the fact that it was not installed, yet showed up in the SBOM for another reason), the product itself isn’t vulnerable. And since a hacker is seldom able to directly attack just one component of a product, rather than the entire product, they will never be able to take advantage of this vulnerability. In other words, the 19 vulnerabilities the user believed to be exploitable were in fact false positives.

6.      Of course, the user will be glad to hear that the problem is only 1/20 the size of what they thought it was; on the other hand, they’ll be miffed that the supplier didn’t proactively inform them that 19/20 of the vulnerabilities in this product and version weren’t exploitable.

7.      The next day, the user receives an SBOM for a different product from a different supplier. However, the experience is roughly the same: The user identifies around ten important vulnerabilities attributable to components in the SBOM, but when they contact the supplier, they’re told that all but one or two of them aren’t exploitable in the product.

8.      Again, this is good news, but now the user starts to get angry that the suppliers are sending them SBOMs, but not warning them that most of the vulnerabilities they learn about aren’t exploitable. This is wasting a lot of their time, as well as the help desk’s time. Why can’t the supplier let them know about these unexploitable vulnerabilities, without their having to waste all this time to learn about them?

9.      This goes on for another few weeks. Finally, the user decides it’s not worth it to receive SBOMs anymore. Sure, it’s nice that there are so few exploitable component vulnerabilities, but the value of the time they waste in learning this outweighs the advantage of learning about the small number of exploitable component vulnerabilities. The next time a supplier offers to start sending them SBOMs, they tell the supplier what they can do with their SBOMs (it’s not nice, by the way).

When the NTIA Software Component Transparency Initiative began in 2018, the suppliers involved in it (mostly very large organizations) soon came to realize that simply distributing SBOMs to their customers, without at the same time implementing some means to inform them of non-exploitable component vulnerabilities that they were likely to find, would literally be counterproductive. The group decided that the best way to address this problem was to develop a machine-readable document format that could be utilized in conjunction with the machine-readable SBOM format. That format was called VEX – and once the acronym was decided on, the name Vulnerability Exploitability eXchange was developed to go with the acronym.

This is the use case that VEX was designed for; it is still by far the most important one. As you can see, it’s very tied to the idea of software component transparency. VEX only becomes necessary when suppliers start to distribute SBOMs; moreover, suppliers are unlikely to want to distribute SBOMs without a mechanism like VEX being in place, for fear of being overwhelmed with completely unnecessary support calls. Even worse, the suppliers are worried that their customers will waste huge amounts of their own time in calling the help desk, and they will blame the suppliers for this. There’s no doubt in my mind that one of the two or three primary reasons why SBOMs aren’t being widely (or even narrowly) distributed today is that there’s so much uncertainty around VEX.

Is this the use case that OpenVEX addresses? Let’s find out. First, we’ll look at the spec published on GitHub. However, the spec says nothing about the purpose of VEX, only that this spec meets “all requirements of a valid VEX implementation as defined in the Minimum Requirements for Vulnerability Exploitability eXchange (VEX) document published on XXX (sic) by the VEX working group coordinated by the Cybersecurity & Infrastructure Security Agency (CISA).”

This is an interesting statement, since the document they refer to has not only not been published, but the working group only finished with it yesterday, February 13 (the OpenVEX spec was created at least a month ago). And when the working group has finished it, it will probably be reviewed for a couple of months by CISA lawyers and others, so it will undoubtedly be changed further. Since the link provided by Chainguard, http://example.com, goes nowhere, here’s the link to the Google Docs version of the document (I wouldn’t normally link to an in-progress document, even though there are no restrictions on who can view it. However, since Chainguard seems to think the document is finished and published, I figured you at least ought to be able to see it).

Since the OpenVEX spec doesn’t say anything about the purpose of this format, we have to be satisfied with the single “Sample Scenario” that it provides:

“As an example, consider the following evolution of a hypothetical impact analysis:

1. A software author becomes aware of a new CVE related to their product. Immediately, the author starts to check if it affects them.
2. The investigation determines the product is affected.
3. To protect their users, the author mitigates the CVE impact via a patch or other method before the vulnerable component issues a patch.

Without VEX data, users scanning the author's software will simply get a third party alert with no details on how the status is evolving. Most critically, when the product is patched (in #3), the alert becomes a false positive.

To inform consumers downstream of the vulnerability evolution, the author can issue a VEX document (in #1) when the CVE is published to let their users know it is under investigation. In #2, when the product is known to be affected, the author can ship a new VEX document, stating the product is affected and possibly some additional advice, like temporary mitigation instructions. Finally when the product is patched, its SBOM can be complemented with a new VEX document informing it is no longer affected by the CVE. Scanners could consume this document and stop alerting about the CVE as it no longer impacts the product.”

(Audible sigh) Where to begin? Here are my questions:

1.      Why would a supplier advertise that their product might be affected by a vulnerability before they have a patch for the vulnerability? After all, when any document is published, even to a small group of customers that have all signed NDAs, you have to assume it will make it to the hacker community within hours. Wouldn’t it be better to…you know…say nothing at all until the patch (or an effective mitigation) is ready?

2.      The penultimate sentence says, “..when the product is patched, its SBOM can be complemented with a new VEX document informing it is no longer affected by the CVE.” In the world of commercial software that I’m most familiar with, and which I would hope the contributors to this document should know something about, that sentence simply makes no sense. The “product” is never “affected by the CVE”, and the “product” is never “patched”. Instead, one or more versions of the product is affected by the CVE, and one or more new versions contain the patch.

3.      Usually, the user of a supported version is given the option of upgrading to a patched version or applying a patch themselves. If they choose the latter option, applying the patch should increment the version number on the user’s instance, so that if the user needs to contact the help desk later and they’re asked what version of the software they have, the friendly help desk person will immediately know whether or not they’ve applied the patch. 

      The VEX document should list the status of each version that’s supported – whether it’s affected, not affected, or fixed (i.e. patched), or perhaps still under investigation. What good does it do to say “the product” is affected or not affected, unless it doesn’t have versions? And if it doesn’t have versions, then, to repeat my quote in the previous post from Bruce Lowenthal, Senior Director of Product Security at Oracle (and one of the originators of the VEX concept), any time there are multiple code bases that have the same version number (or no versioning at all), the product becomes unsupportable. I’m not sure I’ve ever seen a commercial product that wasn’t versioned.

Most importantly, what does this scenario have to do with the paradigmatic VEX use case above? That use case arises because an SBOM is distributed. However, in this scenario an SBOM is only mentioned in the last step, after the product has been patched. Even though the scenario indicates that three VEXes are distributed (my, this supplier seems to love distributing VEXes! I won’t ask exactly how they’re going to get them to all their customers, but it must be really easy, since they like to produce them so often), only the last one seems to have anything at all to do with the VEX use case I described.

But why, in step 3, would the scanner indicate that the vulnerability is present in the product, when in fact it has been patched? It seems this is because scanners often throw off false positives. When that happens, an OpenVEX document that says the product is “fixed” will certainly be helpful, since it will override this false positive designation. I have no argument with that idea.

And indeed, this seems to be the main use case for OpenVEX: Given that scanners throw off a lot of false positives, receiving documents stating that particular vulnerabilities aren’t in fact exploitable will reduce the amount of time that users waste chasing these false positives (as well as bothering the help desk about them). Indeed, this is a fairly good summary of the use case described by Dan Lorenc, the CEO of Chainguard, in a web presentation to an Open SSF working group on January 25 (more on that presentation, and the misstatements that were made by Dan and a couple of guest speakers, in another post soon).

This is a use case that I’d never thought of, and which – although I don’t know a lot about scanners – seems to be a very useful one. While it has nothing particularly to do with SBOMs or with the use case that VEX was developed to address, I certainly wish Chainguard good luck with whatever business they expect to generate out of OpenVEX (if nothing else, this has probably already generated them a lot of good publicity. Nothing wrong with good publicity, of course).

However, Chainguard is wrong in claiming that OpenVEX is in any way superior to, or even competing in the same league as, the two existing VEX specs: one based on CSAF, the other on CycloneDX VEX. It’s something like an up-and-coming hockey team claiming to be competitive in football with two NFL teams (of course, a good hockey team might have a decent chance against the alleged professional football team in my town, Chicago. Fortunately, they’ll be decamping for the suburbs in the next 4-5 years. Couldn’t come soon enough for me). Guys, you’ve defined a whole new product category: false positive reduction for scanning tools. This is a good opportunity. Don’t waste it by pretending to be something you’re not, which is a VEX format that’s comparable to the two existing formats. It’s not going to end well.

On the other hand, I have no problem if Chainguard wants to retain the “VEX” name. But I’m also fine if they want to name the format “Larry”, “Sally”, the “Magna Carta” or the “Tibetan Book of the Dead”. None of these names are trademarked, as far as I know.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

OpenVEX

From Tom 9/20/2024: I noticed this post is suddenly getting a lot of views. I want to clarify what I said. Of the two problems I brought up, I still stand by the first one. However, on the second one, the idea that OpenVEX isn't really VEX, I need to apologize to Chainguard. 

I now realize I should have said that my idea of VEX is the one that the NTIA group came up with originally. However, the term can be used in multiple ways, and it certainly is now. In fact, besides "NTIA VEX" and OpenVEX, there's a third meaning of VEX, which is used by Cisco and Red Hat. All three are perfectly valid use cases, and none of the three can be said to be inherently better than the others. I provide a full discussion of this in Part 2 of my book, Introduction to SBOM and VEX.

Last week, a big announcement was made by Chainguard, a software supply chain security tools vendor, that they had developed, along with six other organizations, a specification for a new VEX format called OpenVEX. This was followed within a couple of days by a web presentation to the Open Source Software Foundation’s Vulnerability Disclosures Working Group, at which Chainguard’s founder Dan Lorenc, and the two leaders of the CISA VEX working group (one a CISA employee (!), the other now a private consultant) praised the new spec and made a (completely erroneous) derogatory comment about one of the two existing VEX formats. It was also followed by publication of a Security Week interview with Dan Lorenc, which contained at least four total falsehoods, although I doubt Dan even knew they were false.

There are two big problems with the OpenVEX spec:

1.      It’s not “open”, since it purports to be the only VEX format that’s completely “compliant” with the contents of an unfinished document being developed by the CISA VEX working group. Even when it’s finished and published, the document will not in any way be a “standard”, “requirement”, “specification”, “papal encyclical”, or anything else that’s compulsory. I’ll elaborate on this statement in another post, coming soon to an inbox near you.

2.      It’s not VEX, as I’ll discuss in this post and (hopefully) the next one.

If OpenVEX isn’t VEX, what is VEX? It would be nice if there were a published VEX definition or specification, but unfortunately, there isn’t one[i] – and the document that’s still unfinished won’t be anything close to that[ii]. The closest thing to an “official” VEX definition is what’s found on pages 1-3 of the VEX Use Cases document. But unofficially, by far the best “definition” of VEX is the document drafted by Dr. Allan Friedman in 2021, when the VEX working group was still under the auspices of the NTIA. Unfortunately, that document was never published (although I recommended last year to the working group – then as now under CISA – that it be updated and published. It would still be worthwhile to do that, but I’m sure it won’t happen).

Here's my VEX “definition”: VEX is a concept that was developed by members of the NTIA Software Component Transparency Initiative to address a specific problem that was identified as part of that initiative. The problem was that a huge percentage of vulnerabilities (almost certainly over 90%), that have been separately identified in components that are included in an SBOM for a software product or an intelligent device (collectively, a “product”), and which are publicly reported (usually to CVE.org), aren’t in fact exploitable in the product itself.

Therefore, if end users, after receiving an SBOM and identifying vulnerabilities for components, start searching for those vulnerabilities on their networks and contacting the product supplier about them, a huge amount of time and effort will be wasted by both software users and suppliers’ help desk employees in discussing these false positive observations (i.e., more than 9/10 of the time and effort will be wasted). This isn’t a sustainable situation, of course. Suppliers need to have a way of reliably and efficiently notifying their customers of non-exploitable component vulnerabilities in the products they operate. Until they have that, they won’t regularly distribute SBOMs to their customers.

It wasn’t foreordained that the solution to this problem would be a format. Yet, since the issue arose due to the imminence of distributing formatted SBOMs in volume to software customers[iii], it seemed natural that the solution to the problem would also be a format. While neither I nor anyone else questioned this at the time (I attended the second weekly meeting of the VEX working group in the summer of 2020 and have missed few meetings since then), it’s clear to me now that focusing immediately on documents was a mistake. I now think my real-time VEX API proposal is the best way to address the most important VEX use, which I just described. There are still other use cases for VEX documents, but if we can address by far the most important use case – and the one that VEX was designed to address – this year, we will have made a huge leap forward.

However, not only does OpenVEX not address the most important VEX use case at all (or even understand what it is), but it would actually set users back. This is because it will take away what will be a capability needed by probably a majority of suppliers. This is the ability to specify the exploitability status of a vulnerability in multiple versions of a product, not just the product itself – and especially in a range of versions. The reason this is necessary is simple: Almost any commercial product has versions. If there weren’t versions, the product would literally be unsupportable (as Bruce Lowenthal, Oracle’s Senior Director of Product Security, has pointed out regularly in past VEX working group meetings).

True, there are some open source projects that don’t have versions, but both private sector and government organizations run primarily on commercial software (which includes open source software that is supported by a commercial organization like Red Hat™). Essentially, private sector and government organizations want one throat to choke when there are problems; having to beg an open source project team to invest some of their own time to fix a problem just doesn’t appeal to most organizations. Since it’s almost never the case that all versions of a software product have the same status with respect to one vulnerability – i.e., they’re all affected or they’re all not affected – a VEX document that doesn’t support versions is literally useless for the great majority of private and public sector organizations.

However, not only must a VEX format support product versions, but it must support version ranges. After all, an exploitable vulnerability almost always is identified in a product at some point in its lifecycle – very seldom was it "baked into” the product when it was first developed. While it’s essential to make VEX statements like “Versions 2.1 and 2.2 are affected by CVE-2023-12345. All other versions are not affected”, that’s not sufficient. It’s also essential to be able to say, “All versions of Product A before version 6.5 are not affected…”

But OpenVEX doesn’t have a separate field for version numbers. To specify a specific version of a product, you need to construct a synthetic “product”, consisting of the product name and the version number.[iv] That is, in order to say “Version 3.7 of Product B”, you need to create a “product” named “B v3.7” or something like that, and then state the status of the vulnerability with respect to that “product”. Since OpenVEX allows only one product per statement, there needs to be a separate statement for each version.

For example, if you wanted to say that Product C Versions 3.1.5 and 3.2.0 are subject to a particular vulnerability, you would in effect need to write, “C 3.1.5 is affected by CVE-2021-54321” and “C 3.2.0 is affected by CVE-2021-54321”. Annoying, right? Well, now try to say that all versions before version 10.5.0 are subject to CVE-2022-98765. You would have to create a statement for every version.

How hard can that be? Quite hard, as it turns out. Consider that, for VEX purposes, every patch version, every new build, etc. needs to be a separate version number, along with of course every major and minor version. It’s not out of the question that a version range would include more than 1,000 versions. Guess how many statements that would take in OpenVEX? You got it…1,000. Now, how many statements would this take in CycloneDX VEX? One. You would have to write, in effect, “vers:generic/>=1.0.0|<=10.5.0 is not affected by CVE-2022-98765”.[v]

In my (hopefully) next post, I will discuss the second reason why OpenVEX isn’t VEX: the fact that it doesn’t address the main VEX use case.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] People often point to this one-page NTIA document as the “definition” or “specification” of VEX. It was indeed intended to be that at the time it was written (which I can say with confidence because I drafted it). However, we were all thinking then that there was no need to go into details. This was because we were sure there would soon be fully automated tools that would both create and consume VEX documents, without the user needing to know anything at all about the specification. Of course, those tools have never appeared, and none are currently on the horizon.

Since now there are two ways to create VEX – using CSAF or using CycloneDX VEX - which are very different from each other, there need to be two specifications. There will never be a single VEX uber-specification. Tools to produce and consume VEX documents in either format will need to wait for the specification for that format. Nobody will invest time or money in creating a tool without a firm specification to base it on. 

[ii] In fact, the unfinished document has had at least five titles. One of them was something like “VEX Specification”. When it became too obvious that the document was anything but that, the title was changed to “VEX Data Model”. And when that was shown to be a misnomer, the title was changed over the holiday break to “Minimum Requirements for VEX”. None of these title changes was ever discussed in advance with the working group – they simply appeared in the document, with little or no justification provided. There were other “magical” changes to wording in the document as well, presumably inserted by little green men from Mars. 

[iii] Unfortunately, distribution of SBOMs is just as “imminent” today as it was in the summer of 2020, when the VEX working group first started meeting. While use of SBOMs by suppliers – for their internal product risk management purposes – has skyrocketed in the last five years, it has yet to take off in any meaningful sense for the 99.5% of public and private organizations whose primary business isn’t developing software. 

[iv] This is also the case with CSAF. 

[v] The CycloneDX VEX format doesn’t support version ranges for all version formats – only the large number that are included in the Version Range Specifier (AKA “VERS”). Essentially, there needs to be some way to determine with precision whether a particular version falls within a range. For example, if the range is “4.2.0 to 5.0.9”, a version identified as “4.4.7” will definitely fall between 4.4.6 and 4.5.0; this version format is one of the formats supported by VERS. 

However, it’s impossible to precisely locate a version identified as “4.3a patch 015”. There will never be a way for a VEX consumption tool to properly interpret a range based on version strings like that one, unless the supplier itself creates code that can be incorporated into the consumption tools. This might happen at some point, but each supplier would have to create their own code and provide a way to update all VEX consumption tools that are available. This would also require that the supplier has consistently applied their proprietary versioning rules, which might not always be the case – especially for acquired products.

 

“None of our products is affected by the log4j vulnerabilities” and other fairy stories

The VEX working group is struggling with a document that makes a number of statements about what should be available in a VEX format (“should” isn’t meant in a regulatory sense, although that fact seems to have been lost on some people). In one section, we describe how a product or products can be identified in the format. One of the options that is currently included in the document is the ability to indicate a supplier name (say, ‘Supplier A’), which would be interpreted to mean, “the set of all products or components produced or sold by Supplier A”. Of course, Supplier A is usually the supplier that created the VEX.

The main reason usually cited for including this capability in a VEX format is because suppliers may want to make a statement like, “None of our products is affected by the log4shell vulnerabilities”. Of course, many suppliers said this in human-readable (or human-audible) form at the end of 2021 and the beginning of 2022. I’m sure it was very comforting for customers of these suppliers to hear they didn’t have to spend nights and weekends trying to figure out whether that supplier’s products, which were installed on their network, contained the Log4shell vulnerabilities.

So, isn’t it reasonable to make that same statement in machine-readable form as well? The point of VEX is that it’s a machine-readable document, even though there are few tools now available to read a VEX, and even fewer that can incorporate the information extracted from a VEX into an ongoing vulnerability management toolchain.

However, here’s the SBOM industry’s dirty little secret (well, one of them, anyway): The big problem with machine-readable formats is that they have to be read by…a machine. Unlike humans, machines take everything you feed them literally; common sense is foreign to them (although that’s also true of a lot of people I know). When making any statement in a machine-readable format, you always need to ask, “How will the user’s tooling interpret this statement?” along with its corollary, “Will the user tool arrive at the same interpretation as would an average human reading the document?

After all, just because a statement seems perfectly understandable when written in English, you should never assume it will be “understood” in the same way by a user tool that interprets a VEX document written in JSON.

Upon scanning the VEX statement “None of Supplier A’s products is affected by log4shell”, a user tool that ingests VEX documents will probably:

1.      Set up a rule to scan the supplier name of any product on the user’s network, to determine whether it matches the supplier name listed for the above VEX statement. Of course, it’s important to note there can be a variety of supplier names, even though the person making the above statement has elided that point in their announcement. For example, someone in the NTIA SBOM initiative once counted 27 different names for Microsoft, that were used by Microsoft employees to identify the company they work for.

2.      Apply whatever rule is set out in the VEX statement to each product whose name matches the supplier name in the statement. In this example, assuming the name of the supplier enunciating the VEX rule was “Supplier A”, the VEX statement might be, “No Version of any Product developed or sold by Supplier A is subject to CVE-2021-44228 or CVE-2021-45046”. In practice, the tool that interprets the VEX statement is likely to treat this as meaning that the status of every product/version pair, whose Supplier is Supplier A, is “not affected”.

Sounds simple, right? Yes, it is - to the computer. But think what it means in practice: This condition will apply to every version of every product whose supplier is Supplier A, no matter how long ago the product/version pair was developed. Moreover, it will apply to every version of every product developed by Supplier A that is acquired in the future – until this VEX statement is overruled by a future statement.

Of course, it may be that the supplier that made this statement intended for it to mean exactly that: Every version of every product they have every produced in the past, or will produce in the future, is not subject to the log4shell vulnerabilities.

However, my guess is that if whoever thought this statement is a great idea goes to their legal department and asks if they concur with that assessment, they may hear something different. They’re likely to be asked questions like:

1.      Have you assessed every product we make or have made in the past, and confirmed that it doesn’t contain any of the vulnerable versions of log4j?

2.      Of course, this also assumes that no component on any “level” of the product contains one of those versions…yea, unto the farthest levels. Have you confirmed this for every one of these products and versions, both past and present?

3.      Have you put in place sufficient controls on product development and product acquisition to ensure that no version of any product we develop or acquire in the future will ever contain one of the vulnerable log4j versions?

4.      Similarly, do you have controls in place to ensure that no component of any version of any product we develop or acquire in the future will ever contain one of the vulnerable log4j versions?

Finally, if I were one of these lawyers, I would ask,

1.      Will you personally indemnify the organization if it turns out that any current, past or future product of ours contains a vulnerable version of log4j and we get sued for it, and will you guarantee payment by posting a large bond? If your answer is no,which poses the larger risk to our organization: log4shell or you?

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.