Rivers of SBOMs. Oceans of VEXes.

A lot of companies – both start-ups and established companies – are currently trying to figure out how they’ll be able to take advantage of the coming era of widely available and widely used SBOMs. I want to point out that this era won’t arrive for at least another 3-4 years. But the point is that it’s coming.

There are three types of organizations doing this figuring:

1.      End user organizations that will be able to greatly expand their efforts to combat software supply chain vulnerabilities – by identifying exploitable vulnerabilities in components of software they utilize.

2.      Software suppliers, who will voluntarily (or due to market pressure. Same thing) provide SBOMs and VEXes to their customers, so that they can perform this analysis.

3.      Third party tool and service providers, who are gearing up to enable all of this activity to take place.

However, there are also some people – again, at both startups and established companies – that think they already have SBOMs figured out, so their plans (and in some cases, actions) really don’t need to be rethought, just executed. These organizations think they understand SBOMs, and how the market will work for them.

I’m not one of these people. There’s only one thing I’m sure of: Whenever I think I understand SBOMs and the infant marketplace that’s growing around then, I’m always confronted with the fact that I don’t understand either one. As the great physicist Richard Feynman said about a different topic, “I think I can safely say that nobody really understands quantum mechanics.” Although I prefer the pithier (but unfortunately apocryphal) wording: “If you think you understand quantum mechanics, you don't understand quantum mechanics.”

I recently had the opportunity to be present when Dale Peterson did an interview with some guy for his podcast. Toward the end of the podcast, the guy brought up the Netscape IPO in 1995, and why its implications had been very different from what anyone predicted at the time. To elaborate on what he said in the podcast:

1.      It seemed in 1995 that a bunch of brand new companies - led by Netscape but also including WebVan, Baidu, Ariba, Yahoo! and others – would take over the world, because they obviously had their markets all figured out. Until they didn’t, of course.

2.      There were also companies that were in existence at the time but were considered quixotic and not worth paying much attention to. There was a guy named Jeff Bezos in Seattle, who had founded a small online bookseller but who kept talking about becoming an online marketplace for all consumer transactions. He also talked about the need to get big fast, even if it meant losing lots of money. He succeeded on both counts in Amazon’s early years: getting big fast and losing mountains of money. It seemed like the company would be a continual cash sink; I sold my modest amount of Amazon stock in the early 2000’s (one of a series of brilliant investing moves I made around that time).

3.      And there was a third type of company that was not even in existence in 1995, including companies like Facebook, Google, Alibaba, PayPal and Netflix. These companies got rich serving markets that didn’t exist in 1995, or at least were very poorly understood. In fact, as it turns out, Netscape itself didn’t understand its market well, since when Microsoft revealed its own browser the following year – and provided it free with Windows – Netscape was relegated to a small role from then on (Netscape’s spirit lives on in the Firefox browser, although the code base is completely different now).

The point is that, when a new technology market is in its very early stages, anyone who thinks they know what the market wants is most likely wrong. Especially if they’re trying to analogize from markets that are well understand to the new one. It’s only when there are lots of organizations pouring lots of money and effort into the new technology that it will gradually become clear what’s needed. Much money will be lost by companies that thought they’d figured out exactly what the new market needs and put in place a wonderful edifice based on that supposed knowledge, only to have that edifice disappear down the drain when it first confronts reality.

Meanwhile, entrepreneurs (or even executives of established companies who are searching for new opportunities) who retain some humility and don’t pretend to have everything figured out are much more likely to come across the real opportunities that don’t follow any previous model, like social media, online payments, intelligent search, and movie streaming. So how do these humble entrepreneurs discover these opportunities? Is it just the dumb luck of being in the right place at the right time?

While I never want to discount the importance of dumb luck, it’s possible to trace out the implications of facts that should be plain as day, in order to get an idea of areas where opportunities might lie in the infant SBOM marketplace. Here’s an example of such facts:

1.      A new SBOM is needed whenever any change occurs in any version of the product (although IMO it would be OK if this were just read to mean supported versions). This includes any change in the code that the supplier itself wrote, as well as any change in one of the components, including when a component is added, removed or upgraded. There should also be a new SBOM when the name of a component, or the component’s supplier’s name, changes. Etc.

2.      When the same change occurs in multiple versions of a product, such as when a certain vulnerability is patched in multiple versions, a new SBOM needs to be released for each of those changed versions (and of course, the version number itself needs to change, if only from v1.2.3 to v1.2.3a).

3.      So how often will new SBOMs come out? That will vary, of course, but I think the biggest predictor of how often a new SBOM will be needed is the number of components in the product. According to Sonatype, the average product has 135 components, but many products have thousands (and I’m just waiting to see how many components are in Windows 10). If a product has 5,000 components, I’d say there’s a good chance that a new SBOM will be needed daily, for every version of the product.

4.      And if you think daily SBOMs are a lot, you ain’t seen nuthin’ until you’ve seen VEXes. Generally, a new VEX will need to be put out any time the status of a component vulnerability has changed. So if a vulnerability has appeared in the NVD (or another vulnerability database, of course) for a component of a product, the supplier of the product should put out a VEX stating the status of the vulnerability in the product itself: exploitable, not exploitable, fixed (i.e. patched) or under investigation (which is “under triage” in the CycloneDX VEX format). And if the product’s supplier has patched the component vulnerability or has determined it isn’t exploitable (when before they thought it was), they need to issue a VEX indicating that changed status, and the version or versions (or version range) in which the status has changed.

5.      Of course, there doesn’t need to be a separate VEX for each status change of each component of each version of each product. If there are a number of changes in the same day, all in the same product, it should be fine to include them all in a single VEX, although I’ve come to believe  that, when there are changes involving multiple products, these should be handled in separate VEXes. However, it’s important that any change in vulnerability status (especially when a new exploitable component vulnerability is identified) be communicated the same day, rather than as part of say a weekly summary.

6.      Let’s go back to our product with 5,000 components. How often will a new VEX be needed for each version of that product? If we assume that it’s inevitable there will be some change in at least one of every thousand components every day, and if we assume there are 20 versions of the product still under support (including minor versions and bug fixes), this means there will need to be 5 (=5,000/1,000) X 20 = 100 new VEXes every day. And that’s for one product. If a supplier has ten different products, each with 5,000 components and 20 versions, there will need to be 1,000 new VEXes every day.

7.      And if an organization has five suppliers like this one, they’ll receive 5,000 VEXes every day.

8.      They’ll also receive 500 new SBOMs a day (= 5 suppliers X 5 products X 20 versions).

Of course, for most organizations there will be far fewer VEXes and SBOMs every day. But for others, there will be more than that. My point isn’t to try to estimate a precise number (which of course won’t be realized until 5-10 years in the future, when production and use of SBOMs are practically universal) of SBOMs and VEXes an organization is likely to receive; my point is that it could potentially be huge.

Anybody who tells you there’s a way that these volumes can be handled with anything but automated tooling is living in a fool’s paradise. Yet how many complete tools are available today? I define a complete tool to be one that will:

1.      Ingest SBOMs and store a set of components for each version of each product used by the organization in a database.

2.      Look up vulnerabilities (CVEs) applicable to each component in the NVD (and possibly other vulnerability databases) and store all of them under the product/version pairs that contain the vulnerable component – with a preliminary status of “exploitable” or “affected”.

3.      As VEXes come in that indicate a particular vulnerability isn’t in fact exploitable in a particular product/version pair, the status will change to “not exploitable” or “not affected”.

4.      Because of the above, the database associated with the tool will always contain, for each product/version pair, the most up-to-date listing of exploitable component vulnerabilities.

If you guessed that the answer to my question is zero, you’re right! There are currently no complete SBOM/VEX consumption tools. I have been told that one open source tool, Dependency-Track, will support ingestion of VEXes by the summer (D-T has been supporting steps 1 and 2 above since 2012, long before anybody was using the term “software bill of materials”); and I’m sure there will be others as well, both commercial and open source.

Let’s get back to what this post is about (I usually do that, sooner or later): how to identify opportunities in the burgeoning SBOM field. Since opportunities often arise from alleviating problems that might otherwise prevent adoption of a new technology, at least one source of opportunities should be quite clear: the fact that there will ultimately be “rivers of SBOMs and oceans of VEXes” clamoring to be processed, yet currently there are no tools available to do that. What are some specific opportunities that arise from this source?

The most obvious opportunity is consumer tools. If you’re a developer of software tools or you aspire to be one, I’ve just listed what your tool needs to do: Go thou and do likewise.

However, an equally important (or perhaps more important) opportunity is a service that provides the four steps I just described: The service ingests all available SBOMs and VEXes and outputs always-current (at least fresh daily, and hopefully more often than that) lists of exploitable software vulnerabilities in (ideally) every version of every commercial or open source software product used today. And this work is performed in an almost completely automated fashion, to make it affordable for organizations that don’t happen to have unlimited budgets for software vulnerability management.

In this case, I know one organization that’s rapidly gearing up to offer a service like the one described above (and they are providing at least parts of the service now to customers of their other services). This is Fortress Information Security, with their File Integrity Assurance tool. I want to make it clear that I have a consulting relationship with Fortress and have had it for more than two years, although I’m never been given views of services they’re developing.

I honestly don’t know of any other organization that is even contemplating a service like the above, but I’m sure Fortress will have competitors sooner or later. And just in case you’re about to suggest a firm that offers hourly consulting services as a competitor to Fortress in this space, I’ll point out that this won’t scale at all. The service is going to have to be completely automated, if it’s ultimately going to be able to handle the rivers of SBOMs and oceans of VEXes I’ve described.

Here's another opportunity: Almost all of the solutions that I’ve seen described for distribution of SBOMs and VEXes – whether startups offering such a service or the documents written by the NTIA Software Component Initiative (especially this one) – require a human being to be somewhere in the middle, usually identifying what SBOMs and VEXes are applicable to them.

The fact that a large organization may in the not-too-distant future have literally thousands of SBOMs and VEXes to deal with every day should quell any idea that a human can be involved. Some organization will need to take in streams of SBOMs and VEXes coming from software suppliers and produce custom feeds for particular organizations based on the products that they’re using, across the supplier spectrum.

This is probably the hardest opportunity (of the above three) to address from a technical point of view, since this deals entirely with data in motion (there will also be a database from which customers can choose individual SBOMs and VEXes, especially to help in pre-procurement analyses). The streaming service probably can’t depend on storage of the SBOMs and VEXes, even for a short period of time. It will literally be like having fire hoses shooting streams of liquids in primary colors at you. Your job is to – without even trying to divert those hoses into storage tanks – output a continuous stream of a particular mixed color for each of your customers.

Sounds hard, doesn’t it? I forgot to mention: Don’t even look for easy opportunities in SBOMs (or probably in any other technology discipline). If something looks easy, it’s almost certainly not a real opportunity, at least not anymore.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.