Sunday, August 14, 2022

What's the difference between a VEX and a vulnerability notification?

Many people confuse VEXes with vulnerability notifications and wonder why there needs to be a VEX format at all. After all, software suppliers have been providing notifications of new vulnerabilities to their users for years. These notifications have been provided in many human-readable forms, including email text, PDF attachments, web pages, etc. They have also been provided in machine-readable forms, including the CVRF format[i] and its successor, CSAF[ii] (the general vulnerability reporting format on which one of the two VEX formats is based).

The VEX concept arose because of the need to provide negative vulnerability notifications: Instead of stating that a particular product and version(s) are affected by a particular vulnerability (CVE), a VEX indicates that the product and version(s) are not affected by the vulnerability. Is this the difference between a VEX and a “traditional” vulnerability notification?

The answer is no, for two reasons: First, A VEX document can provide a positive vulnerability notification, simply by changing the status designation in a negative statement like “Product X version Y is not_affected by CVE-2022-12345” to “Product X version Y is affected by CVE-2022-12345”.

Second, a negative notification can be provided in the same way as any positive notification is now delivered; a VEX isn’t required for this. For example, instead of issuing an email stating that a particular vulnerability is exploitable in their product (and therefore customers need to apply a patch or upgrade), a supplier can send an email stating, “None of our products is subject to the Ripple 20 vulnerabilities.” In fact, it is safe to say there is nothing that a VEX document can do that a “traditional” vulnerability notification cannot do; why are they both needed?

The difference between a VEX and a traditional vulnerability notification has nothing to do with their respective formats. Rather, the difference between the two is due to the fact that a VEX is designed to address a particular use case: the case in which a software user has utilized the list of components from an SBOM to find vulnerabilities that might be exploitable in a software product (or intelligent device) operated by their organization.

The VEX identifies vulnerabilities that are a) listed in a vulnerability database as applicable to a component of the product, yet b) are not exploitable in the product, taken as a whole. A software tool that ingests SBOMs and VEX documents would utilize the VEX documents to “trim down” the list of vulnerabilities identified for components found in the SBOM, so that the only vulnerabilities remaining on the list are exploitable ones[iii].

Another way that VEX documents differ from traditional vulnerability notifications is that a VEX only makes sense if it is “read” by a software tool. This is because, if a software product just contains a small number of components and the supplier only needs to notify their users that a few vulnerabilities are not exploitable in the product, it would be much easier for the supplier to do this in an email.

Under these assumptions, users could simply maintain a spreadsheet listing all the vulnerabilities identified (in the NVD or another vulnerability database) for components of the product. When they receive an email saying a vulnerability isn’t exploitable in the product, they would simply delete the line in the spreadsheet where that vulnerability is listed. Therefore, at any point in time, the spreadsheet would list all the component vulnerabilities currently deemed exploitable.

However, the average software product contains around 150 components and many products – including some commonly-used ones – contain thousands of components. Clearly, trying to manage exploitable component vulnerabilities using emails and spreadsheets will be impossible when products like these are encountered.

Of course, VEX documents can be used for other purposes than negative notifications (like the one just described). For example, a VEX document can be used to notify customers that one or more versions of a product are vulnerable to a serious vulnerability, as well as provide a notice of a patch and/or upgrade path to mitigate the vulnerability. However, given that almost every software supplier already has some means to notify its customers that a serious vulnerability is exploitable and a patch is available for it, why would a supplier want to use a VEX to convey this notification?

While this is speculation, it is possible that some suppliers will use VEXes for patch notifications like the above (as well as other positive vulnerability notifications), because they a) want to provide a machine-readable notification to their users and b) they are already providing VEXes for “negative vulnerability notification” purposes – meaning they have some experience with the format. Therefore, they might be more inclined to experiment with a new use case for a format (VEX) they already understand, rather than experiment with both a new use case and a new format.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at

[iii] Of course, this is an aspirational goal. It will require any supplier that provides SBOMs to their customers also to publish a VEX as soon as they discover that a vulnerability listed in the NVD for a component isn’t exploitable in the product. This will place a burden on suppliers and may dissuade some suppliers from producing SBOMs at all, at least in the near term. This is why I have proposed “real-time VEX”, as a way to greatly lessen the burden on suppliers of providing negative vulnerability notifications to their customers. See this post:

Wednesday, August 10, 2022

What’s the best SBOM? No SBOM. What’s the best VEX? No VEX.

Steve Springett, leader of the OWASP CycloneDX and Dependency-Track projects (as well as the SCVS project, which I haven’t written about previously but is also important for software security in general and SBOMs in particular), and I attend a weekly meeting where we and others discuss problems that are holding back widespread distribution and use of SBOMs. We’ve been working for five months on the naming problem, and will soon publish our proposed close-to-solution to the problem (plus, we’re getting the different parts in motion to implement that solution, although ultimately we can only present our case to the people who need to make it happen).

At last week’s meeting, Steve asked what we would tackle next (assuming we want to tackle anything next, but there’s no question that we want to do that. There are still some serious problems that need to be addressed before SBOMs can be widely used. No rest for the weary - although I find this quite interesting and invigorating).

I suggested an idea I brought up in a post a while ago (which was also based on something Steve said. No coincidence, there – BTW, what I’m talking about here appeared at the end of that post), which I call “real-time VEX”. IMHO, the idea is quite simple:

1.      Because about 90-95% of vulnerabilities in software components aren’t exploitable in the product itself, end users are going to be reluctant to utilize SBOMs for vulnerability management purposes (probably the most important security use case for SBOMs, although there are other use cases as well), unless they can be assured that they’re not spending 95% of their time chasing false positive vulnerability identifications.

2.      About two years ago, a working group within the NTIA Software Component Transparency Initiative started work on a format for documents that would allow a supplier (or perhaps another party) to notify customers that vulnerability CVE-2022-12345 isn’t exploitable in their Product X Version Y, even though the NVD shows the CVE is found in Component A, which is included in X.

3.      The name VEX was chosen (or more accurately, stumbled into) for the document; then the words “Vulnerability Exploitability Exchange” were identified as a good enough explanation of what VEX stands for (people never like it if you tell them that an acronym stands for NBI, Nothing but Initials, but that often happens. In fact, “CVE” is officially NBI now, even though it did stand for something initially).

4.      Now there are two standards for creating VEX documents: one based on CycloneDX and the other based on the OASIS CSAF vulnerability reporting standard. However, no software supplier is currently providing VEX documents to their customers, except perhaps in one or two isolated cases. The biggest reason for this is that – even more than SBOM – VEXes need to be consumed by automated tools. There is currently only one tool that consumes SBOMs and VEXes, which is Dependency-Track (I believe there will be one or two third-party services available in the near future, that will also do this). Another important reason why VEX isn’t exactly taking the world by storm is that there is no playbook currently available that describes how to create and consume VEXes (although the CISA VEX committee is starting work on one now).

5.      The big problem with the fact that VEX isn’t taking off is that this inhibits interest in SBOMs as well, for the simple reason that people don’t like the idea of wasting 95% of their time looking for vulnerabilities that aren’t there. Given that they can’t be sure that won’t happen if they start looking for vulnerabilities in components of a product, they aren’t very interested in SBOMs or VEXes.

I thought until recently that a good bit of education would help suppliers and consumers of software become comfortable with VEXes. That is coming and will help, but there’s another problem: While an SBOM will have to be produced whenever the software changes (including patches and new builds), VEXes will need to come out in much greater numbers, in order for users to be assured that the list of vulnerabilities that their SBOM/VEX tool shows them, for a particular product and version, consists mostly of exploitable vulnerabilities that are worth pursuing, rather than non-exploitable vulnerabilities that aren’t. As I discussed in this post, a large company would probably need to receive literally thousands of VEX documents every day (if not every hour), in order to reach the goal of narrowing down the component vulnerabilities that they take seriously, to only the exploitable ones.

The problem is made even worse by the fact that there is real urgency with VEXes (even more so than with SBOMs). Most users will want to know as soon as possible which component vulnerabilities aren’t exploitable, especially if these are serious vulnerabilities. If a supplier learns that a serious component vulnerability isn’t exploitable in one of their products, their users aren’t going to react favorably if the supplier waits until the end of the month, when they aggregate all of their VEX notices in one document; it’s likely that users will already have wasted a lot of time looking for that vulnerability in the product. In fact, the users probably won’t be satisfied if they hear the supplier issues a VEX once a week, or even once a day. They will want to know as soon as the supplier decides that a vulnerability isn’t exploitable.

But here’s the problem with that: Until there are automated tools to produce VEXes (and currently there are none, in either format), putting out a VEX is going to require a non-trivial amount of time. In fact, people are going to need to produce VEXes day and night, in order to keep up with demand (I believe the term for these people is “VEX workers”). My guess is that anybody with that job will either quit or commit suicide within months.

However, when I wrote the post I referred to earlier, I had just come to realize that there may be a solution to our problems. We don’t need a VEX (or any other document) for the primary VEX use case: informing users of non-exploitable component vulnerabilities found in products they use. Instead, we need:

1.      A “VEX server” at the supplier (or at a third party that offers this as a service on behalf of the supplier) will list, for each currently-used version of software products (or intelligent devices) produced by the supplier, all of the component vulnerabilities currently shown in the National Vulnerability Database (NVD), as well as perhaps other databases like OSS Index (which is better for open source components, I’m told). The components behind each list will be those in the SBOM for that version (since every change in the software should produce a new version as well as a new SBOM, there should always only be one SBOM for each version).      

      The VEX server will continually search the vuln databases for new vulnerabilities for each product and version (if this sounds like a lot of work, consider that Dependency-Track is now being used 200 million times a month to search OSS Index for vulnerabilities applicable to open source components – and that works out to about 20 billion SBOM-based vulnerability searches a month, just with that one open source product. Also, consider that security-conscious suppliers should already be continually searching for new component vulnerabilities in their products. They should already have this information), and continually update these lists.

2.      As the supplier identifies a component vulnerability that isn’t exploitable in a particular version of their product, they will set a “not exploitable” flag for that vulnerability, product, and version in the VEX server.

3.      Meanwhile, end users (or a third party performing this service on their behalf) will operate software tools that track all vulnerabilities listed for components of any software product/version used by their organization.  Whenever they identify a component vulnerability in the NVD, they will add it to a list (rather, their tool will do this) of vulnerabilities for the product/version (this list might be included in a vulnerability or configuration management database). Each vulnerability will initially be assumed to be exploitable (there is no other choice, if you think about it).

4.      There will be an API that can be called, as often as required, by software operated by end users (such as Dependency-Track), or by a third party acting on behalf of users. It will send three pieces of information to the appropriate server:

a.      Product name (Product A)

b.      Product version string (Version X.Y.Z)

c.      CVE number (CVE-2022-12345)

5.      These three pieces of information will be interpreted by the VEX server to mean, “What is the exploitability status of CVE-2022-12345 in Product A Version X.Y.Z?”

6.      The server will return one of two pieces of information in response:

a.      Not exploitable

b.      Under investigation

7.      If the answer is a, the user tool will flag CVE-2022-12345 as not exploitable in Product A Version X.Y.Z. If the answer is b, the tool will do nothing, and the CVE will continue to be listed as exploitable.

8.      It will be worthwhile for the user tool to use the API to query the VEX Server at least once a day, and more often if possible. This way, the list of exploitable component vulnerabilities will, at any particular time on any day, contain only the vulnerabilities that the supplier has not yet decided are not exploitable (note that I don't think the API should optionally return a third piece of information "Exploitable". Unless there's a patch available for the CVE in question - and it's been advertised to all customers of the product - that would be a dangerous thing to do).

Of course, there will always be some vulnerabilities that are currently listed as exploitable, that will later be found not to be so. Thus, the vulnerability management staff will still waste some time looking for non-exploitable vulnerabilities in the products the organization uses. However, they will at least know that the list they have reflects the supplier’s most current knowledge.

Note there are still a number of VEX use cases that need to be handled with a VEX document, for example a “positive” vulnerability announcement (e.g. “Product A Version B is vulnerable to CVE-2022-YYYYY and the patch is available at (URL)”), or a more complex scenario like patching a longstanding vulnerability in the product. But IMHO the most time-critical and high-volume use case, can be handled much more efficiently by “real-time VEX”.

To continue my story, when I’d described my idea in the meeting (in much less detail than above), Steve pointed out that he’s sometimes asked in interviews where he thinks SBOMs will be in ten years. He always says, “I hope they won’t be needed at all.” He went on to explain what he meant (I’m paraphrasing him): "What really matters is if the supplier (or a third party) has the data the consumer needs when they need it. If that can be exchanged purely using APIs, that’s great. We don’t need either SBOMs or VEXes.”

While I agree with Steve that the goal is not to need either SBOMs or VEXes, I think the low-hanging fruit at the moment is what I’ve just described. This may be one thing (along with solving the naming problem and the availability of user tools and third-party services) that opens the floodgates for use of SBOMs and VEXes for software vulnerability management. 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at


Monday, August 8, 2022

9 ½ years!

Fortress Information Security (a company with which I have a consulting relationship) has been analyzing a lot of SBOMs lately. They recently showed me the results of their analysis of the components in a particular security product (which one hopes would have better security than most software products). They were careful not to tell me that these results are somehow typical of the “average” software product, since determining that would require a much larger, controlled study. But they did say they’d examined a number of other products whose analysis yields similar results.

They had other statistics in their report, but what struck me most were the figures for “vulnerability age” – i.e., the number of days since the vulnerability was published (which I assume means when a CVE number was assigned to the vulnerability). The report said:

·        There were 19 vulnerable components in the product (i.e., they had at least one open vulnerability. I didn’t see the total number of components, but I’m assuming it’s a lot larger than that). Since there were 141 total vulnerabilities, this means there were about 13 unpatched vulnerabilities per component. That’s a lot.

·        Since the top 10 vulnerable components each had 5 open vulnerabilities, this means the 141 vulnerabilities must have been fairly evenly distributed among the components, with a lot of 5s, 4s, etc. It’s not like one or two bad actors were making the other 17 or 18 components look like slouchers. They were all bad actors, or at least not good ones.

·        Vulnerabilities were grouped by severity, based on their CVSS score category: low, medium, high or critical.

·        There were five low-severity vulnerabilities. Their average age was 2801 days or 7.67 years.

·        There were 61 medium-severity vulnerabilities, whose average age was 2,987 days or 8.18 years.

·        There were 2 high-severity vulnerabilities, whose average age was 3,058 days or 8.37 years.

·        There were 9 critical vulnerabilities, whose average age was 3,447 days or 9.44 years. In other words, the most serious vulnerabilities were the oldest, with the average serious vulnerability having been identified 9 ½ years ago.

Here are my conclusions from this data:

1.      Keep in mind that, if a vulnerability has been outstanding for 8 years but the component that contains the vulnerability wasn’t included in the product until yesterday, then the vulnerability in that product is one day old. Of course, unless this is a brand new product (which it’s not), at least some of the vulnerable components must have been in it for years.

2.      Also, keep in mind that these numbers probably haven’t been adjusted to remove non-exploitable vulnerabilities. Thus, the number of exploitable vulnerabilities could be expected to be much smaller than the 141 figure for instances of component vulnerabilities. It’s possible that, if Fortress had asked the product supplier about those 141 instances of vulnerabilities, they might have said something like, “We knew all along that (e.g.) only two instances of component vulnerabilities were exploitable and we immediately patched those, so the 141 that you see are all not exploitable.” It’s possible they’d say this, but I doubt it.

3.      If a component vulnerability has been around for more than say a year, one would hope the supplier would patch it, even though they still don’t believe it’s exploitable. After all, there are ways in which a component vulnerability that has been deemed non-exploitable by the product supplier might suddenly become exploitable, such as a code change that inadvertently made the vulnerability exploitable. So even if those 9 ½ critical vulnerability instances aren’t exploitable, they should have been patched long ago anyway.

4.      I don’t understand why the age goes up with component severity. One would generally think that the highest-severity vulnerabilities would be patched as soon as possible, not left to molder for years.

5.      Since it’s unlikely that the supplier acquired most of their components around 8 years ago, this means the creator of the component (and most were probably open source projects, since 90 percent of components are open source) probably sat on open vulnerabilities for years, before the supplier of the product that Fortress analyzed acquired or downloaded it. Hmm… that doesn’t say a lot for component suppliers, does it?

6.      However, since the supplier of the product hopefully didn’t acquire all their components yesterday, the fact that at least some of these vulnerable components were sitting in the product for years doesn’t say much for the supplier either, does it? Remember, this supplier makes a security product (in fact, one that’s advertised a lot in the ICS space). Who cares whether or not a security product is itself secure? It seems this supplier doesn’t…

7.      What I found most amazing is that the lowest average age for a component vulnerability is more than 7 ½ years. I would have guessed it would be, say, no more than one year – and even that seems like a lot to me.

Bottom line: If I were a user of this product and I learned that the majority of component vulnerabilities were known at least 8 years ago yet still hadn’t been patched, I would ask, “Does this supplier even look at component vulnerabilities, let alone patch them?” My conclusion would be, “Probably not.”

In 2017, Veracode performed a study[i] in which they found that “A mere 52 percent of companies reported they provide security fixes to components when new security vulnerabilities are discovered. Despite an average of 71 vulnerabilities per application introduced through the use of third-party components, only 23 percent reported testing for vulnerabilities in components at every release.”

I’ve always thought that this was a nice “before” quote, to be contrasted with the much-improved “after” quote from say last month. Certainly (I told myself), software suppliers’ practices would have improved a lot since 2017. I guess I was wrong.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at

Monday, August 1, 2022

Zero-days aren’t the worst of our problems, unfortunately


I want to thank Kevin Perry for forwarding a post by Patrick Miller from last week on the Ampere Security blog. You should read it, but I’ll summarize the post by saying it describes the dangers posed by two “zero-day” vulnerabilities that two researchers found in “…a Moxa Ethernet-to-serial converter with the newest firmware.” Of course, the problem with zero-days is they won’t be picked up by vulnerability scanners, so – absent intrepid researchers who are willing to reach out to the supplier before they exploit the zero-days they found (and aren’t planning on exploiting the vulnerabilities themselves, of course) – neither the supplier nor their customers will have any way of learning about those vulnerabilities.

I don’t want to downplay the importance of this development, since zero-days pose a big risk to any organization, and especially critical infrastructure organizations like electric utilities. An Ethernet-to-serial converter is very unlikely to be found in any but a critical infrastructure organization, since only these organizations are likely to be “blessed” (or cursed, one of the two) with serially-connected devices. If Ethernet-to-serial converters installed in multiple electric power transmission substations were to be compromised, this could potentially cause a serious outage, and potentially the worst kind of outage: a cascading one.

However, while this is a real problem, in May I learned, from discussions sparked by Tom Pace of NetRise in an informal group of SBOM practitioners that I’m part of, that there are at least two even worse problems than individual zero-day vulnerabilities. One is currently only theoretical, but could in the longer term (a few years – that’s not that long) turn into perhaps a huge issue worldwide, were some creative person or nation-state decide to devote a little time to exploit it. It’s hard to imagine any quick practical solution to this problem (perhaps banning production of open source software? That shows how serious the problem could turn out to be). That problem is described in this post

The other problem is an immediate one and is currently huge, although it will be hard even to learn how big it is until the evidence becomes all too clear through successful attacks. But this problem can be solved with relatively easy measures - easy technically, although I’m not sure about politically. This is one of those problems where the cause can be easily traced to the wetware between the seat and the keyboard. I’ll focus on this problem in this post.

In the first post of the series, I described Tom’s initial presentation to our group, in which he described his experience with one in a family of infrastructure devices from a certain manufacturer. These devices are used in critical infrastructure and in – shudder! – the military. Were you to do your due diligence on this device before procuring it, you might well think it had perfect security. After all, when you search on the name of either the device or its manufacturer in the National Vulnerability Database (NVD), you will receive this response: “There are 0 matching records.”

Sounds good, right? Before you purchased the product, you wanted to see if the NVD listed any serious vulnerabilities for it. Not only did no serious vulnerabilities appear, but no vulnerabilities at all appeared. However, in his initial presentation to our group, Tom said he knew that, in a single piece of firmware in the device in question, there are 1,237 unpatched vulnerabilities due to open source firmware components. In fact, Tom later calculated, this time through examining all of the firmware and software installed in the device, that there were at least 40,000 unpatched vulnerabilities[i] in the one device – which of course appears to have no vulnerabilities at all, if you just go by the NVD search results.

These 40,000 are all known vulnerabilities, not zero-days. But the problem is that end users will never learn about these vulnerabilities if they’re not listed in the NVD - and this manufacturer has never done so. As far as the users are concerned, these are zero-day vulnerabilities. In fact, this manufacturer appears to make around 50 devices and not a single one of them has any vulnerabilities listed in the NVD. Moreover, this supplier doesn’t even mention security or vulnerabilities on their web site. They obviously think they have the security problem licked in their devices, since they’ve never identified any vulnerabilities. Perhaps they missed the memo that said if you don’t look for vulnerabilities, you won’t find them…or perhaps their lawyers sent that memo.

What’s even worse is that the supplier itself isn’t hiding anything. They haven’t lied to the NVD or anyone else, since they have probably never identified any vulnerabilities in any of their products. Unfortunately, a search in the NVD that doesn’t find the product at all (as in this case) returns the same error message as if the product was found, but there were no vulnerabilities listed for it. Someone who hasn’t been schooled in the pitfalls of relying too heavily on what the NVD says will likely come away believing the product has zero unpatched vulnerabilities, and they should go ahead and buy it. However, their number is just a little off the true number of unpatched vulnerabilities – by about 40,000.

Here’s the reason why I say this is a bigger problem than any discovery of zero-day vulnerabilities. Even though these 40,000 vulnerabilities are known (meaning they have been assigned a CVE number or another recognized vulnerability identifier), the fact that they’re in this device (and lots of others, to be sure) will never be known to any end user, unless the supplier reports them to the NVD.

However, the supplier won’t report them until they’ve found them; but they’re not looking for them, so it seems they won’t find them. Darn the luck! To the users of this device, these might as well be 40,000 zero-days. But don’t worry; they’ll learn about them when they get hacked.[ii]

What can be done about this problem? I’m glad you asked. The group to which Tom Pace made this presentation (the group is completely informal, but we call ourselves the “SBOM Forum”) is now – largely because of that presentation – hot in pursuit of a solution to the problem behind this and other aspects of the “naming problem”. The naming problem is perhaps the biggest problem plaguing software supply chain security, and SBOMs in particular. It’s definitely a huge part of the reason why SBOMs aren’t distributed to customers in any more than a few trickles today (the exception to this is the European auto industry, which uses SBOMs for open source license management, not software vulnerability management. They are also exchanging their SBOMs through a unique arrangement that would probably – in my non-lawyerly opinion - not even be legal under US antitrust laws).

When we started to pursue a solution (although we didn’t dare use that word at the time) to the naming problem in May, I thought we should be realistic and break the problem into two parts: the part that can be addressed in the “near term” of 2-3 years and the part that will take 5-10 years to address. I advocated that we put off the latter for the moment.

But guess what: It turns out that the near-term solution is the same as the long-term one. And neither is technically hard at all: If everyone were agreed on what needs to be done and were on board with how to implement their part of the solution, it would be up and running in about a week – and I’m only saying that partially tongue-in-cheek.[iii] Of course, “everyone” will never be in agreement on everything, so I’m sure that there will never be a 100% solution to the naming problem. But, as we see it now, there are no big technical obstacles in the way of what we’re proposing, and the funding required (all of which will need to come from the government, of course) is much less than I imagined originally – although I’ll admit we’re not considering cost in our solution. The problem is already exacting a huge cost on the software industry, as well as the general public, due to not being able to easily (or even at all, in some cases) learn about most software supply chain vulnerabilities.

I’m hoping that in a couple weeks, I can share some details on what we’re proposing, although the formal push won’t come until September. We’re already starting to socialize it with people in the government and elsewhere who can help with the implementation (and we’re getting a good initial response). At the moment, we need to finalize the 12-page document we’re working on, before we can seriously discuss this with anybody. I hope we’ll have that done in a week or two.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at

[i] Of course, these are 40,000 vulnerabilities that are identified in software and firmware components found in the product. Since at least 95% of these vulnerabilities are probably not exploitable in the product itself, this means there are “only” 2,000 exploitable vulnerabilities in the device. And since maybe 2/3 (to be generous) of those vulnerabilities are perhaps not serious enough to patch, that leaves over 600 exploitable vulnerabilities that are serious enough to patch, and of course haven’t been patched yet – since this manufacturer evidently never even looks for vulnerabilities, let alone patches them. 

[ii] I know that Tom has recently been in contact with the manufacturer about this problem, although there hadn’t been any resolution when I heard from him more than a month ago. But the problem is that this manufacturer is far from alone. There are tons of products that have zero vulnerabilities listed in the NVD, mainly because the supplier has never reported a single one. This is why I said, in one of the posts linked above, that it’s actually safer only to purchase software products or intelligent devices that have at least some vulnerabilities listed in the NVD, since their suppliers obviously take vulnerability management seriously. This isn’t a joke. 

[iii] I’ll admit I’m glossing over the fact that we’re solving two naming problems: software and hardware. What I say above is related to software. The hardware side of the problem isn’t technically hard, but it will require a lot of grunt work merging databases. Nothing like that is required on the software side. However, since the NVD includes both hardware and software products and a lot of the components of the naming problem apply to both hardware and software, they both need to be solved, although not necessarily at the same time.

Thursday, July 28, 2022

SBOMs for devices vs. SBOMs for software

It has always struck me as odd that only one or two of the documents published by the NTIA Software Component Transparency Initiative (which ended last December) mentions SBOMs for intelligent devices, even though the Initiative got its start after the FDA announced in 2018 that they were going to require SBOMs for medical devices in the future (which turns out to be this year). SBOMs will be one small part of the “pre-market” approval process – meaning the device can’t be sold to hospitals unless the manufacturer has provided a satisfactory SBOM to the FDA.

In fact, the “laboratory” in which SBOM concepts were tested was (and still is) the Healthcare SBOM Proof of Concept, in which medical device manufacturers and some large hospital organizations (known in the industry as HDOs, which stands for “healthcare delivery organizations”) exchange SBOMs and document lessons learned. The ongoing work of that group was essential to the NTIA effort and is now informing the CISA effort (although I believe the PoC is now officially under the wing of the Healthcare ISAC). All the SBOMs exchanged in the PoC since 2018 are about medical devices, although the PoC would like at some point to start addressing standalone software products utilized by hospitals, of which there are many.

The reason that SBOMs for devices aren’t mentioned, in any but one or two documents published by the Healthcare PoC itself, is that there has been a dogma that there’s no difference between an SBOM for a device and an SBOM for a software product that isn’t tied to any particular device. And while it’s true that the form of the SBOM is the same in both cases, what is very different are the actions the manufacturer of a device needs to take to help their users identify and mitigate vulnerabilities found in the software and firmware products installed in the device. After all, those actions are what really matters, right?

This spring, I wrote an article with Isaac Dangana, who works for a client of mine in France, Red Alert Labs, which focuses on certification and standards compliance for IoT devices. The article addresses the question of how SBOMs for devices differ from SBOMs for “user-managed” software – which means everything that we normally refer to as “software”; it concludes with a set of actions that we believe IoT users should require (or at least request) of the manufacturers of IoT and IIoT devices that they use.

The article was for the Journal of Innovation, which is published by the Industrial IoT Consortium (IIC). It was published yesterday. You can download a PDF of the article here and you can view the entire issue here. I and Isaac will welcome any comments!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at


Wednesday, July 27, 2022

What SBOM contract language should we require? How about “None”?


As we approach the August 10 deadline for federal agencies to start requiring SBOMs from their “critical software” suppliers, the question comes up regularly[i], “What contract terms should we require for SBOMs?” I used to take such questions very seriously, stroking my chin and intoning, “That’s a good question. This is something that needs to be addressed soon” – or some incredibly wise statement like that.

I’ve said that, even though I’ve always been skeptical about the usefulness of cybersecurity contract language in procurements. I’ve always believed that the important thing is to get the vendor’s agreement to take steps to mitigate a particular cybersecurity risk (e.g., implement multi-factor authentication for the vendor’s remote access system). This could be through including terms in a contract, but it could also be through getting them to state their agreement in an email or letter.

However, I’ll admit that commitments made in letter or email are de facto contracts, and an organization can be held to those commitments almost as easily as when they’re included in a contract. If a vendor balks at agreeing to contract terms, they’re unlikely to agree to essentially the same terms in either a letter or an email.

This is why I think it’s fine, in most cases, to simply get the vendor’s verbal commitment to do something, then document who made that commitment and when. You won’t do this in order to nail the vendor in a court of law because of their verbal commitment; you probably can’t do that, and it would probably result in the mitigation not being made for years, anyway. However, it’s amazing how many people seem to think that the goal of supply chain cyber risk management is to get the vendor to agree to improve their cybersecurity in a contract, but not to get them actually to do what they promised. These people think their job is finished when the contract is signed.

Here’s some news: Getting a vendor to sign a contract mitigates zero risk. The risk only gets mitigated when they do what you asked them to do. That’s true, no matter whether you asked them to commit in contract language (signed in blood or just plain ink), an email, a letter, a phone call, a conversation at a restaurant, a message in a bottle, Morse code, cuneiform tablets, whatever. You have to follow up with them – often repeatedly – to ensure they do what they said they’d do. Otherwise, whatever time you spent getting them to sign the contract, or commit in any other way, is wasted.

But, if the vendor has agreed to do something in a contract, isn’t it likely they’ll keep their promise? That depends on the vendor, and what actions you take if it appears they haven’t done what they agreed to do. Is your company’s policy to threaten to sue a vendor as soon as they fall a day behind the commitment they made, then put more and more legal pressure on them until they capitulate? If so, you’d better have a lot of lawyers on staff with nothing better to do than harass vendors.

Here’s an experiment: Ask your procurement people how many times they’ve sued a vendor about anything, let alone a cybersecurity term. When I’ve done that (admittedly not with huge companies, since they won’t answer the question at all), the answer is always that they’ve never sued over cybersecurity terms (or anything else having to do with performance. It’s almost always financial). Moreover, in most cases, the company has sued any vendor for anything at all fewer than maybe five times. Lawsuits are only an enforcement mechanism of last resort; they should never be your first resort – and they shouldn’t be threatened as your first resort, since that just reduces your credibility with the vendor to zero.

The fact is, if a vendor is doing at least a reasonably good job for your organization and they’re liked by the engineers or whoever is dependent on the vendor’s product or service to get their job done, you’ll never sue them over anything. Consideration of the effort and cost of finding a new vendor – and the strong possibility that whatever new vendor you settle on won’t measure up to the one you just fired – will almost always lead to your organization settling whatever dispute you had with the vendor.

So why even pretend that lawyers need to be involved? Sit down with the vendor as soon as a cybersecurity issue comes up and figure out a solution you can both live with: That is, they’re sure they can achieve the objective, while you’re sure that the value of whatever you gave up in your negotiations will be far less than the cost of retooling or retraining for a new vendor.

However, with SBOMs, it’s even more cut-and-dried: Given that SBOMs are just starting to be distributed to customers in dribs and drabs (although software developers are using them heavily now for internal product risk management purposes) and end uses have just about zero experience using them in their own cyber risk management programs, it makes no sense now even to talk about contract terms for SBOMs and VEX documents. Before contract terms will ever be useful, there needs to be widespread experience with SBOM distribution and rough agreement on best practices for mitigating those risks. But that agreement is years away and needs to follow experience, not precede it. That will be years from now. I estimate it will be 5-10 years before SBOMs are being widely distributed and used, and the world has at least a couple years of experience with them. Then we can think about real contract language.

However, I know that most companies of medium-to-large size require a contract with every purchase, and more and more companies want to mention SBOMs in every contract for software; of course, that’s a good thing, and I recommend it as well. However, the term should read something like, “Vendor will provide software bills of materials in a format and frequency to be agreed upon between Vendor and (our organization).”

Once that’s agreed on, then the vendor’s product security team and the customer’s supply chain cyber risk management team need to discuss the details of what the vendor will actually deliver. What should the customer ask for? If SBOMs and VEXes were already widely distributed and widely used, I would ask for at least the following (I’m sure I’ll think of other terms later):

1.      A new SBOM needs to be provided whenever anything in the product has changed, including major and minor updates, patches, new builds of existing versions, etc.

2.      Whenever a new SBOM is produced, the supplier should provide both an SBOM produced as part of the “final build” of the product and an SBOM produced from the actual binaries that are distributed. These include not just the software product binaries, but any container, installation files, “runtime dependencies”, etc. – anything that will remain on the system after installation and which may develop vulnerabilities, just like the product itself can.[ii]

3.      The supplier should provide a valid CPE identifier for every proprietary component in the product, and a valid CPE or purl identifier for every open source component.

4.      The supplier should not provide an update to the product that contains any exploitable vulnerabilities that pose a serious risk (I’m sure most current contract language regarding software vulnerabilities currently relies on the CVSS score as a measure of risk. However, people who know more than I do tell me that CVSS scores don’t really measure risk, or anything else that’s useful. If so, we need another readily available score to take its place. The EPSS score might be that, although I don’t know how widespread its use is. It measures the likelihood that a vulnerability will be exploited, although I need to point out that this is different from the exploitability addressed in a VEX document. See this post for more information). If the supplier can’t avoid doing this in some case, they need to discuss with your organization mitigations that might address the risk posed by these vulnerabilities.

5.      The supplier will patch new vulnerabilities that pose high risk within (15-30) days of publication in the NVD, availability of exploit code, or some other acceptable event.

6.      The supplier will provide a VEX notification that a vulnerability identified in the NVD for a component of their product is in fact not exploitable in the product itself. This should be provided as soon as possible after the supplier determines the vulnerability is not exploitable.

These are all worth discussing with the supplier, but you shouldn’t expect to get them to agree to any of these items for a few years (except for numbers 4 and 5. These aren’t dependent on SBOMs being released, so the supplier should be making commitments like these already). But that’s OK. Find out what they can commit to and agree on that, even though it will just be a verbal agreement.

And for heaven’s sake, give the lawyers the day off. Tell them to come back in 5-10 years, when it will be time to discuss specific contract terms regarding SBOMs.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at

[i] It comes up from private industry, which of course isn’t subject to the EO; the federal agencies have to utilize the FAR (Federal Acquisition Regulation) and thus don’t have much, if any, control over their contract language.

[ii] I’ll admit that this particular “requirement” is something that I personally think is needed – not necessarily the NTIA, CISA, The Tibetan Book of the Dead, Baháʼu'lláh, or any other entity.

Saturday, July 23, 2022

Everybody needs to be more creative in addressing vulnerabilities


Almost predictably, Walter Haydock has produced another excellent post dealing with vulnerability management. I recommend you read it.

You will notice that the post addresses concerns of software developers, not end users. Since I don’t think too many developers read my post for development advice (for that matter, I don’t think too many hog producers read my blog for hog production advice, either. The two are about equally likely to occur), you may wonder why I’m suggesting that you, an end user of software, read it.

The first and perhaps more obvious reason is that end uses need to know what’s reasonable to require of suppliers regarding vulnerability management and what isn’t. If you look at Waler’s post from that perspective, you can read it as saying in general that requiring suppliers (in contract language or simply in an email) to follow rigid rules like “Never release a product with vulnerabilities having a CVSS score of >7.0” will sometimes be counter-productive (as Walter explains, this might mean the supplier will take longer to patch a serious vulnerability, if another serious vulnerability appears just before the deadline for the supplier to patch the first one). He suggests a more nuanced approach.

And while I’m at it, you should keep in mind that requiring a supplier never to release a product if there are any vulnerabilities in it could lead to the supplier ceasing to report vulnerabilities to the NVD at all (remember, by a large margin, most vulnerabilities are reported to the National Vulnerability Database by the supplier itself). Unless assiduous security sleuths are searching for and reporting vulnerabilities in that supplier’s products (and I doubt that happens for any but a tiny fraction of software products and intelligent devices today), soon there won’t be any vulnerabilities that show up in a scan of the product, since none will be found in the NVD. Problem solved (from the supplier’s point of view, anyway). The fact is that patching all vulnerabilities, regardless of severity or exploitability, is a fool's errand.

The second reason why a software user should read Walter’s post is the careful, balanced way he approaches vulnerability management. This applies both to this post and his posts on vuln management that are aimed at end users (note to Walter. Please put links to one or two posts focused on end user vuln management in the comment section below, as well as in the comments on this post in LinkedIn). 

Walter always keeps the North Star of maximum possible risk mitigation in front of him when he writes these posts, but he’s also careful not to become dogmatic about that. This is especially true when striving for maximum risk reduction would require a complicated process that would make Rube Goldberg blush with envy. It’s much better to provide advice that people can actually follow, as opposed to advice that might win kudos with the risk management freaks but would be impossible to follow in practice (or even counterproductive, as Walter regularly points out).

Tim Roxey emailed me the following commment on this post:

Failure of imagination was a blue ribbon panel finding for the 9/11 commission. It is a finding of many of the attacks I have studied both inside CONUS and on the events the US gov had asked me for support in other locations globally. Including war zones. 

It’s simplest observable is in the question “who would do that?  Or “wait, what? 
Cognitive dissonance. 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at