It’s time to retire the candy bar

Ever since SBOMs were invented by Thomas Edison (I think I’m correct in that statement), almost every introductory presentation has focused on the analogy to a list of ingredients in some processed food, often a candy bar.

The narrative always goes something like, “Just like you want to know the ingredients of a candy bar to learn if there’s something listed that could present risks to your physical health, so you want to learn about the ingredients (components) of the software you use, to see if they might present risks to your cyber health. If you find something like that, you might ask the supplier to mitigate that risk in some way by replacing a risky component. Or you might decide that continuing to use this software is too risky and you’ll find another supplier.”

There’s another implication to this narrative: that the supplier is negligent and acting against your best interests if they don’t replace a component that has a serious vulnerability – or even if they don’t regularly replace components that have reached a certain age. In fact, I would bet there are contracts today that include language requiring suppliers to take actions like these when certain conditions are met.

However, considerations like these are based on an assumption which I unconsciously made until last Friday: that the components of a software product or intelligent device are for the most part independent of each other. In other words, as long as a supplier can find a component of roughly equivalent functionality but which poses less of a risk, or perhaps develop the required code themselves, they should be expected, or even required, to replace the problematic component.

However, last Friday at our weekly meeting of the SBOM Forum, we discussed a problem that is well known among software developers, although it hasn’t been discussed at all (as far as I remember) in the NTIA or CISA meetings on SBOMs. This is the problem of “tight coupling” between components, meaning it is very hard to replace one component without at the same time replacing others – in some cases, many others. There can be various technical reasons for this, but the effect of all of them is the same.

This is especially the case for big, complex systems that have been built up over years, if not decades. They can have millions of lines of code and many thousands of components. They can literally get to the point where it’s almost impossible to replace a single component in them.

What are these big, complex systems? Hopefully, they’re not very important, so it’s not a big deal if the supplier can’t replace components anymore, right? Unfortunately, it’s just the opposite. Here are several examples of systems like this (without saying any particular system suffers from this problem):

1.      Huge military systems

2.      Energy management systems (EMS), which some electric utilities (and Independent System Operators/Regional Transmission Organizations) utilize for the most crucial function on the grid: balancing power supply with load in a certain area like a city (or 13 states, as in the case of the PJM RTO). This has to be done in real time. Unfortunately, with power, having too much supply at any moment is as bad as having too little. You can’t just put some power on the shelf for a time when it’s needed more, and it doesn’t evaporate. Both a surplus and a deficit of power supply (or an excess or deficiency of demand) will throw the grid out of synch, depending on how long the disturbance lasts (and we’re talking a few seconds at most).

3.      Electronic medical records (EMR) systems, which run an entire hospital. It’s nice that the FDA is going to require SBOMs for medical devices in hospitals, but if I – who has only limited experience with hospital systems, and that was years ago – were to point to the biggest source of cyber risk in a hospital today, I would point to the EMR system. Yes, an attack on an infusion pump might kill the person attached to it. But an attack on the EMR system - usually through ransomware - might disrupt an entire hospital, resulting in deaths. This has happened in Germany (with ransomware as the cause, not out-of-date components) and I’m told it’s almost certainly already happened in the US.

Thus, the only way a supplier of one of these systems can address a component vulnerability is to patch it (assuming the component vulnerability is exploitable in the product). Of course, patching a new vulnerability is something suppliers do all the time. What’s important in the case of tightly coupled systems is that, since components can’t usually be replaced or upgraded, the supplier will have to patch old vulnerabilities in components. In a less tightly coupled system, these would probably be fixed through replacing the component.

The practice of patching old vulnerabilities in old components (or in old products in general) is called backporting. A lot of suppliers have to do this. But having to do a lot of backporting can pose a serious perception problem for the supplier, when they start to release SBOMs.

Why does backporting patches pose a problem? Because, when the supplier releases an SBOM for a product like this, users (or a service provider or tool acting on their behalf) can be expected to look up the components in the NVD or another vulnerability database. They will probably be shocked when they find such old components, but they will be even more shocked when they find there are so many old vulnerabilities that apply to those old components, which have never been patched by the component supplier. The supplier may have removed the vulnerability with a backported patch, but that won’t show up when a tool looks up the component in the NVD.

Since the logical steps may not be clear, I’ll list them:

1.      Say a tightly coupled software product ABC includes component XYZ version 1.4. V1.4 was released by the component supplier (which might be an open source project) in 2017.

2.      Shortly after the component was released, the supplier reported to the NVD that CVE-2017-12345 was identified in XYZ v1.4. The supplier later released a new version of the component, v1.5, in which the vulnerability was fixed.

3.      Because ABC is tightly coupled, its supplier has not been able to upgrade or replace component XYZ v1.4. Instead, they patched CVE-2017-12345 in ABC.

4.      When ABC’s supplier starts to release SBOMs for their products, customers will notice that a) component XYZ is way behind the current version of the component, and b) the old CVE-2017-12345 is still found in XYZ v1.4 (since the NVD doesn’t know whether or not a particular vulnerable component has been patched in a particular product. In fact, the NVD doesn’t know which products include which components).

5.      The customers start complaining to ABC’s supplier about this old vulnerability. They have to be told one-by-one that CVE-2017-12345 isn’t exploitable in ABC, because the supplier backported a patch for it years ago.

Of course, if the above sequence were to be a rare occurrence, it might not pose a problem for a supplier. But, given that a tightly coupled product will likely contain a large number of backported patches, this won’t be a rare occurrence. In fact, a security person from a large service provider told the SBOM Forum last year that the fact that their main system contains a lot of backported patches was the number one reason they were holding back from releasing SBOMs for their products (fortunately, they believe they have solved this problem, and they are now getting ready to release SBOMs).

Thus, the main problem with a tightly coupled system is a perception problem, not a security one, since the software is presumably safe because of the backported patches. The problem becomes visible when the supplier starts to distribute SBOMs.

How can this perception problem be solved? Through communication, as is the case with most perception problems. Of course, the supplier can tell their customers about the problem through emailed PDFs, but can they also include the information in an SBOM? Since the SBOM itself led to the perception problem, can the problem also be corrected using an SBOM?

As it turns out, it can. The CycloneDX SBOM format includes a “pedigree” field, which can be used to record these changes. Note that, in order for the contents of this field to be automatically acted upon, the tool used to ingest the SBOM will need to include rules on how to react to the contents of the pedigree field. Since those rules are likely to be specific to the organization involved, they may always require customer development.[i]

In SPDX, this is done through Relationships, specifically using these two fields:

PATCH_FOR	Is to be used when SPDXRef-A is a patch file for (to be applied to) SPDXRef-B.	A SOURCE file foo.diff is a patch file for SOURCE file foo.c.
PATCH_APPLIED	Is to be used when SPDXRef-A is a patch file that has been applied to SPDXRef-B.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] Steve Springett, Chair of the CycloneDX SBOM Standard, Core Working Group, pointed out that CycloneDX also includes the OBOM (Operational bill of materials) format, which can be created separately from or together with SBOM information; he said that the backporting information could also be handled in OBOM, although I didn’t try to verify that.