The Procurement use case for SBOMs

There’s lots of discussion of use cases for SBOMs, and one that always comes up is Procurement. At first it seems obvious that, when your organization is evaluating software or intelligent device suppliers for a procurement (whether you’re comparing multiple suppliers or simply deciding whether the supplier you’re already using merits retention), it would be great to get an SBOM (or even better a set of SBOMs from the past year or so) from each supplier. That way, you can find out what risks apply to the product, especially open vulnerabilities. This will certainly help you decide which supplier you prefer, or whether your current supplier should be retained.

Unfortunately, as happens often when you look at how to use SBOMs, the best-laid plans quickly run aground on data problems. One example is the naming problem, which is in my opinion is one of the two “showstopper” problems that is preventing SBOMs from being widely (or even narrowly, to be honest) distributed and used by organizations whose primary business isn’t developing software.

I’ve asked several major developers what percentage of component names in an SBOM – that was generated by an automated process without manual enhancement – can be found in the NVD; of course, if you can’t learn about vulnerabilities that apply to the components in an SBOM because you can’t find them in a vulnerability database, then the SBOM doesn’t do you a lot of good. One of those developers initially said the number was 20%, but when another expressed skepticism about that number, they agreed it’s actually…under 5%.

In other words, if an automatically-generated (usually as part of the software build process) SBOM lists 150 components (about the average number, according to Sonatype), you will be able to find just eight or fewer of those in the NVD, on average, without taking some extraordinary (and decidedly “manual”) steps to increase the percentage.

Of course, suppliers, consulting companies, etc. all have adopted various measures – comparing with other databases, doing manual searchers, looking through GitHub commits, fervent prayer, etc. – to increase the percentage, so that the SBOMs will be usable for their work. But will the supplier that you’re evaluating take all those steps before they provide you the SBOM (assuming they even know what steps to take)?

And suppose the supplier has something to hide…Wouldn’t it be easy to leave the useless names in the SBOM they provide for your evaluation even if they have better ones, so that you won’t be able to learn about the many vulnerabilities that apply to the components in their product?

Another problem: almost all vulnerabilities listed for a product in the NVD are reported by the supplier itself. What if the supplier you’re evaluating happens to be one that has never reported a single vulnerability? Tom Pace, of NetRise,  pointed out to the SBOM Forum about a year ago that he found one supplier of network devices used in critical infrastructure environments, and even the US military, that had never once reported a vulnerability to the NVD for any of the devices it makes (they make about 50 devices).

Tom looked at the firmware in those devices, and from identifying components in the firmware and knowing what vulnerabilities were present in those components, he decided that, in just one of those devices, the number of open vulnerabilities was a little more than the zero that had been reported. In fact, the number was 40,000, which I believe is 40,000 more than zero (if my math serves me correctly).

The worst part of the story, though, is that if a user that was evaluating the supplier looked up that device in the NVD, they would get a message that said, “There are 0 matching records.” This would also be the message the user would receive if the supplier diligently reported vulnerabilities for their products, but this product just didn’t have any. In other words, the message that indicates they haven’t reported any of the 40,000 vulnerabilities they have is the same message you would see if the product didn’t have any vulnerabilities at all.

Suppose you were someone who didn’t know how the NVD works (or doesn’t work, in this case), and you had been told to find the product (among the set of products being evaluated) with the smallest number of vulnerabilities. Suppose that the other suppliers had one or two vulnerabilities, but this one didn’t seem to have any at all. Would you be suspicious and investigate all 49 of that supplier’s other products, to find out if they had any vulnerabilities reported for them? And if you didn’t find any, would you call up the supplier to find out why they weren’t reporting vulnerabilities at all?

Of course not. You wouldn’t even think of doing that. Instead, you’re much more likely to circle the supplier with (supposedly) zero vulnerabilities on the list you hand back to your boss, and say, “Here’s the company you should buy from.” And then you’ll go home, so you’re not late for your daughter’s soccer game.

In the post I linked above, I said:

Steve Springett, who I’ve written about a number of times and who is tasked with helping 2,000 coders produce secure software in his day job, said last week that his company deliberately favors products for which there are a lot of reported vulnerabilities. They consider this a sign that the supplier is diligently seeking out vulnerabilities, not waiting for their product to be hacked.

Thus, in the hypothetical case I just described of the person who was anxious to select a supplier so they wouldn’t be late for a soccer game, that organization would literally have been better off if they’d told that person to identify the supplier with the most reported vulnerabilities in the NVD, not the least.

In the above case, the hypothetical evaluator selected a product that appeared to be vulnerability-free, but which was in fact anything but that. However, the opposite case also poses danger. Suppose you are assigned to evaluate suppliers of a big, complicated system like an electronic health record system for a hospital or an energy management system for an electric utility. Systems like this were typically first developed decades ago, and they have been added to and improved over the years.

But, as I described in this recent post, an SBOM will usually make systems like this appear to be riddled with vulnerable components, when in fact they are very well maintained and likely to be just as secure as other products that don’t have all of these “vulnerable” components. The problem is that these systems contain ancient components that can’t be removed without bringing the whole product crashing down, even though the components are well protected by the practice of “backporting” patches. Yet a standard SBOM analysis will seem to indicate that those ancient components, which are usually unsupported by their suppliers and therefore appear to have lots of vulnerabilities, are clear evidence that the supplier of the product doesn’t care one bit about security.

Cases like this show that it’s always a big mistake to evaluate a supplier’s security just based on technical details that could be misleading, including details regarding SBOMs. In fact, you should always require that a supplier you’re evaluating fill out a questionnaire that will elicit information about their actual practices regarding software security (and get some references on whether the supplier actually follows those practices). Then, if a technical analysis seems to indicate a big problem (like the one just described), but the questionnaire answers indicate that the supplier has a good vulnerability management program, you might not just dismiss them because of the technical issue.

Someday, a lot of the problems that currently prevent SBOMs from providing a complete picture of a supplier’s security practices will be solved. You will still need to rely on questionnaires, but you’ll also be able to trust the information that an SBOM provides you, to provide a good practical evaluation of those practices. However, that isn’t the case today.

In fact, I can think of just one item, related to an SBOM, that will provide unequivocal evidence of the supplier’s security understanding and practices: You can ask them for an SBOM. If they ask you what that is, you should dig deeper to find out how much they really know about software security. However, that’s about all you can do at the moment.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.