Are the major SBOM formats interchangeable? Not very.

 

On Monday May 23, I had the wonderful opportunity to participate in a webinar on SBOMs with three people who have worked with and thought about SBOMs a lot. The panel was sponsored by six IEEE chapters in the NE Virginia/DC area, and also included Dr. Allan Friedman of CISA, Dr. Jean Camp of Indiana University, and Steve Pruskowski of CISA.

The group made some excellent observations, some of which were new to me. I recently listened to the recording, and I suggest it will be worthwhile for you to do that, too. Here is the link to the recording.

One good question had to do with SBOM formats – whether it’s sustainable to have multiple formats and whether they’ll easily interoperate with each other.  One of the other panelists said there were lots of tools for translating between the two major formats (SPDX and CycloneDX), and that it shouldn’t matter which format your SBOM is in – it will be usable in all tools and for all purposes, because it can always be translated to the other format. In fact, I would say this is the consensus opinion in the software community.

When I was asked to comment, I said I agreed with that answer to a point, but I added that it’s important to remember that the two formats both go far beyond the seven minimum fields that should be included in an SBOM. They each have 50-100 fields, and a lot of those aren’t replicated in the other format; so the information in those other fields will be lost in translation. This means that anyone who wants to convert between the two formats needs to make sure that all the fields in the original SBOM are available in the target format.

But I realized later that what I said falls far short of the real problem, which is that the two formats go about what they do in very different ways – so that addressing many of the differences between them requires doing a lot more than simply adding a field on one side or the other (if that’s all that were required, I’m sure all differences between the two formats would have disappeared a while ago).

Here’s an example of what I mean: One longstanding object of discussion (which Allan Friedman mentioned in the webinar) is the fact that a software supplier will sometimes make changes to the code of an open source component before incorporating it into their product. Of course, this is completely permissible, but a problem arises if the SBOM still lists the original name and version string of the component.

Because customers will normally look up the component in a vulnerability database like the NVD in order to find the vulnerabilities that apply to it, it’s possible – if the supplier did modify the code of the component – that they will either a) learn about – and possibly act upon – vulnerabilities that are no longer found in the component implemented in the product they use (because the supplier modified it), or b) not learn about a vulnerability that the supplier themselves inadvertently introduced into the component when they modified its code. Obviously, neither of these is a good thing.

CycloneDX (CDX) can represent this situation, as well as a few similar ones, using the “pedigree” field. Included in that field are five possible relationships: ancestors, descendants, variants, commits and patches.  Can all of these be translated into SPDX? SPDX includes 40 or so “Relationships”, and most of the CDX pedigree fields are included in those relationships.

However, there is a big difference in how the relationships are implemented in the two standards: Those “Relationships” in SPDX only apply to source code files, while “pedigrees” in CycloneDX also apply to libraries and entire operating systems. This means that, if a supplier has incorporated a modified library into their software product, they can represent that fact in a CycloneDX SBOM, but not in SPDX.

On the other hand, SPDX supports snippets, which are small bits of code that can be easily incorporated into a program in an integrated development environment. They can be similar to components, in that they’re sometimes developed by third parties; including them in software can save the programmer a lot of time. However, snippets sometimes come with licensing issues, which a developer needs to be aware of. CycloneDX doesn’t currently support snippets, meaning that, if an SPDX SBOM includes snippet information and is translated from SPDX to CycloneDX, it will not retain that information.

To sum up, you should never assume that an SPDX SBOM can be faithfully translated into the CycloneDX format or vice versa.[i] This means you need to think beforehand about what you need in a format before you standardize on one format or the other.

However, this might be a moot point. If your organization is a software supplier, I recommend you consider creating SBOMs in both formats, since it’s likely that some end-user tools that utilize SBOMs will only support one or the other format.

And if you’re an end user, I would strongly recommend that you not announce that you’ll only support SBOMs in one or the other format. You may miss out on SBOMs from some suppliers, who might feel they’re too small or under-resourced to support both formats. If you find a tool that only accepts one of the two formats,[ii] you’ll initially have to require SBOMs in only that format – and if one of your suppliers says they’re only going to provide SBOMs in the other format, you should have a discussion with them about whether they’ll change their mind.

But there’s another option on the software consumer side: In probably the next 3-6 months, there will be at least one subscription-type service available that will ingest SBOMs and VEX documents and perform risk analysis based on them (as well as on other information sources).[iii] Services like that will be able to handle both SBOM formats, so the Format Wars (and the need even to receive SBOMs) will go away forever, for their customers.

Unrelated note about the webinar: When asked about how SBOMs could gain acceptance by organizations in general, Jean Camp made the interesting point (starting at about 27:50 in the recording) that SBOMs might initially be positioned similarly to how the EnergyStar label was positioned: If a supplier is willing to provide SBOMs for their products, they obviously are committed to transparency about their security posture. Even if the user never does anything with the SBOM when they receive it, the fact that the supplier is willing to provide a regularly updated SBOM means they feel they have nothing to hide from their customers. This alone will be a great driver for improving software security.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] I need to point out that, while SPDX 2.2, the current version, is descended from an original version developed ten years ago for the purpose of open source licensing risk management, a new version 3.0 will appear later this year or next year, which is designed from the bottom up as a cybersecurity risk management tool – so it may be able to handle pedigree information for libraries. On the other hand, I’m told that an upcoming version 1.5 of CycloneDX will support snippets, so that difference between the formats will also go away. 

But I’m sure there will always be things that one format does and the other doesn’t, or at least that they do in significantly different ways. Let’s face it: The two formats are competing with each other. It’s not at all in their interest to become bland variants on each other. 

[ii] The only complete SBOM consuming tool that I know of (where “complete” means that it can ingest both SBOMs and VEX documents) is the next version of Dependency-Track, which will be available soon. This tool only supports CycloneDX SBOMs and (soon) VEX documents in the CycloneDX VEX format. 

[iii] There are currently services that will perform this type of work on an hourly consulting basis. Those are fine, but they won’t scale well, especially with significant increases in the volume of SBOMs and VEX documents being produced. What’s needed are subscription-based services, whose costs will decline in the future, as volume increases.

What fields should be in an SBOM?

Starting in August, federal agencies will have to comply with the provision in Section 4(e) of Executive Order number 14028, that requires agencies to start asking for SBOMs from their suppliers of software and intelligent devices. In addition, many private sector organizations are starting to put serious thought into how they will use SBOMs, once they become available from their suppliers and once the appropriate tools or services are available to utilize them. In all of these discussions, the question is always asked, “What fields should be included in an SBOM?”

The easy response to this question is, “Whatever fields are needed to achieve your purpose for using SBOMs. By the way, what are you planning to use SBOMs for?” Since the most important purpose for using SBOMs currently – and the only one that’s addressed in the Executive Order – can be described as “Managing risks that arise in procurement and use of software, which are related to components included in that software”, the answer to this second question is, “Whatever is required to manage risks arising from procurement and use of software.” Of course, at this point, the questioner isn’t much closer to having an answer to their question than when they started.

We can imagine a different Q&A occurring between a federal agency and a software developer. The agency says, "We need you to start providing us with SBOMs by this August." The developer asks, “What do you need in those SBOMs?” The agency asks, “What fields can you provide?” The developer points out that the SPDX SBOM format offers about 70 fields, and CycloneDX has more than 100, but it’s certain that the agency won’t need anywhere near this many fields. So which ones do they want included in the SBOM?

At this point, the agency is likely to ask what they’re “required” to have. The answer to that is that there isn’t a mandatory requirement for what the SBOM contains. The Executive Order simply required federal agencies to ask for “SBOMs”, but it also required that the National Technology and Information Administration of the US Department of Commerce identify “minimum elements” for an SBOM. The NTIA put out an excellent report on this topic last summer.

Besides listing seven minimum fields for an SBOM, the report contains a lot of other good recommendations. However, it also goes to great pains not to make those actually look like recommendations, so there’s been almost zero (that I’ve seen) written about them; I hope to remedy that deficiency soon. Meanwhile, it seems that most people who are following developments with SBOMs have heard of the minimum fields.

I’ll agree that each of those seven fields – Supplier Name, Component Name, Version of the Component, Other Unique Identifiers, Dependency Relationship, Author of SBOM Data, and Timestamp – should be included in every SBOM. But saying that these are what “should” be included is like saying that a car should include an engine, a steering wheel, four wheels, a drive train, and a place for the driver to sit. This car (plus some other basic items, of course) would perhaps get you from point A to point B, but it doesn’t fulfill the purposes that most of us are looking for in a car. What’s needed that goes beyond the minimum?

The real answer to this question is that what’s needed beyond the minimum fields are simply what the supplier and the user agree on. After all, SBOMs are a “good” (really, more of a service) that’s exchanged in the marketplace. If the supplier and the user agree on 60 fields that the user wants and the supplier can provide these, great. By the same token, if the supplier and user agree that only the seven minimum fields are needed, that’s great, too – although I would question how much the user will be able to accomplish in the way of software component risk management, with just those seven fields.

However, it’s also very unsatisfying to say something like, “Whatever the supplier and the customer agree on is fine.” I say this for two reasons:

1.      In many cases (and just about all cases when the supplier is a huge organization, but the user isn’t), there won’t be any negotiations on this – or any other – question. The user either accepts what the supplier is offering or they find a different supplier and product.

2.      When there is real negotiation, the user will almost always have no idea of what they need, since they probably only have a vague idea of how they’ll use the SBOMs. The fact is that there isn’t much written now about how SBOMs can be used for software component risk management purposes.

However, there are at least two documents that suggest additional data fields that users and suppliers should consider:

A.     The Minimum Elements document linked above identifies, on pages 14 and 15, three additional fields regarding software components: Lifecycle Phase, Other Component Relationships, and License Information.

B.     Fortress Information Security[i] published in April an excellent white paper that provides guidelines for using SBOMs. In addition to the seven minimum data fields, the paper recommends (starting on page 12) that these fields be included: Product Supplier Name, Component Type, Component CPE Name, Package URL (a software identifier that is important now and will likely become even more important in the coming years), Component Version String, Component Download Location, Licenses, Cryptographic Hash of the Component, and Document Digital Signature/Code Signing.

Which of these should you ask for? In general, if you’re not sure which of these you need, you should ask for them all; they’re all useful to have, although a couple of them, like Lifecycle Phase and Component Hash, may be difficult for suppliers to produce, in some cases.

If the supplier pushes back on one (or more) of these and you don’t have a strong reason why you want to have it, then don’t demand it. As you get more experience using SBOMs, you’ll get a better idea of what you need and don’t need, to meet your own requirements.

I hope to write another post soon, that discusses the other items in Minimum Elements, which aren’t as obvious – but are probably more important – than the seven minimum fields.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] Fortress is one of my clients, although I was not one of the direct authors of this document.

Bring me the broomstick of the wicked witch of the west!

Last week, someone brought to my attention this “position paper” on SBOMs from the “Cybersecurity Coalition” (a group I hadn’t heard of. But then, there are lots of legitimate groups I haven’t heard of). The person pointed out to me that the paper describes roadblocks preventing SBOMs from being widely used. I read it because I’m very interested in identifying these roadblocks. I’ve come to realize in the past year that widespread rollout of SBOMs will require solving some fundamental problems – and those problems need to be identified before they can be solved.

However, in reading this paper (it’s not very long), I realized that this isn’t an honest attempt to facilitate the rollout of SBOMs; it’s an attempt to impede it by asserting that this rollout can’t happen before a bunch of impossible things happen. It’s a lot like the Wizard of Oz ordering Dorothy to bring him the broomstick of the Wicked Witch of the West. The good wizard didn’t expect Dorothy to bring him the broomstick, and he certainly didn’t have any use for it when she did.

So what’s the broomstick that the Cybersecurity Coalition wants “SBOM” to bring back? Oh, just these four items:

·        Establish pilot programs involving software suppliers and agencies to demonstrate the effectiveness of SBOMs in improving vulnerability management practices, based on risk metrics, more rapidly and with less effort than existing tools and processes.

·        Drive to common standards for sharing, processing, and implementation of SBOMs and infrastructure to reduce potential confusion and inconsistency in outcomes.

·        Perform additional research and develop pilot programs to better refine how SBOMs can and should be used in cloud environments.

·        Create public/private workshops to discuss both the technical and non-technical aspects of SBOM sharing, including establishment of criteria for ensuring confidentiality where desired, and avoiding liability for software suppliers.

Of course, there’s nothing inherently bad about any of these things. But the NTIA Software Component Transparency Initiative went on for almost four years (from 2018 through the end of 2021) and took at least some of these steps. Plus, if the good people who wrote this position paper had joined the NTIA effort during that time and suggested these specific steps, I’m sure the group would have considered them. Why have they just recently realized that all of these things are needed?

Of course, the NTIA effort is over (although a couple of the groups that were working under the NTIA are continuing to work under CISA). Meanwhile, Executive Order 14028 (which the Cybersecurity Coalition expresses great love for in their document) – which was published a little more than one year ago - requires federal agencies to start asking their software suppliers for SBOMs less than three months from today. Isn’t it kind of late to suddenly decide that we need pilot programs and additional research and public/private workshops…and lions and tigers and bears (oh my!)? These people could have…you know, mentioned these things when the EO was being drafted or at least after it was released. And they might have contacted OMB to give them this input, since that agency is charged with implementing the EO.

But the items listed in the four bullet points above aren’t all the deficiencies these people have observed. I was shocked SHOCKED! to learn that:

1.      “Given that SBOMs are a relatively new and often misunderstood topic, they are not being produced routinely by many software producers, and changing that will require time and effort.” That’s certainly true. But what are producers required to do? The EO simply requires that federal agencies start asking producers for SBOMs, starting this August. If a producer says it’s too soon for them to produce SBOMs, they won’t be shot at dawn (or any other time). They’ll just need to tell the agencies when they will be ready. On the other hand, given that modern development tools can automatically produce a new SBOM with each build of the software – and given that one single SBOM-based tool used by developers is generating 20 billion requests to one vulnerability database every month - it’s a little hard to understand why there are developers who still think this is something new and difficult.

2.      “… it is important to recognize that, in the near term, there will be inconsistencies related to software produced before the Cyber EO versus software produced after.” One of the main purposes of SBOMs is to help suppliers and users understand how the components in software have changed from version to version and build to build. It seems to me that finding inconsistencies – and their causes – is one of the reasons for having SBOMs in the first place.

3.       “…it remains unclear how departments and agencies will receive and use SBOMs once they are shared with them.” My goodness! The authors seem to be saying that “departments and agencies” are used to receiving perfectly clear instructions that don’t require any thought at all, so they don’t know how to handle ambiguity. My guess is the agencies, had they been asked about this, would point out that dealing with ambiguity is a prerequisite for working in the government, just as it is in private industry. They can handle ambiguity just fine, thank you.

4.      “…overly stringent requirements related to the production and dissemination of SBOMs will require that resources be diverted from other activities.” In other words, the authors are saying, “Don’t get us wrong…we love SBOMs and we love the EO. But don’t expect us to, you know…have to balance competing resource priorities. We’ve never had to do that before.” Hmm…it seems that software developers are the only businesspeople on the planet who don’t have to make choices between (almost) equally valid uses for their funds. That’s news to me (and I expect to their employees as well). In any case, welcome to the real world.

The last two sections of the paper – titled “Automation” and “Challenges in cloud environments” – raise a couple of legitimate points, namely:

First, there isn’t much automation available now for use of SBOMs to manage vulnerabilities by software customers (although there are plenty of tools available for developers, contrary to what the paper says). Nobody has pointed this out more loudly than I have. But the paper suggests that what’s needed is “a series of pilots and test cases”, as if there haven’t been any so far (again, where were all these people while the NTIA Software Component Transparency Initiative – including several proofs of concept – was going on? In fact, the oldest of the PoCs, for healthcare, is still active. Any healthcare software or device supplier that wants to join that PoC should drop me an email; I’ll put you in touch with the leaders of the effort). 

The consumer tools will come; I’m sure of that. In fact, I just received word yesterday that Dependency-Track, the open source tool that is already widely used by software suppliers to manage component risks in their own software, now can ingest VEXes as well as SBOMs (it has ingested SBOMs for ten years, before the phrase “software bill of materials” was even being used); this makes it the first complete user tool for software component vulnerability risk management (I’ll have more to say on this subject soon).

Second, the last section points out that there are a lot of open issues having to do with SBOMs for SaaS. That’s absolutely true. However, the NTIA initiative made the calculated decision to address other issues before taking on SaaS SBOMs – so it’s not surprising these issues are still open. The CISA SBOM effort has this at the top of the agenda for this year, and even if that doesn’t happen, I’m sure another group will take it up.

But it’s simply not realistic to expect all of the capabilities that you would like to see in a new technology will be ready on day one. After all, did automobile buyers in 1910 demand that cars start without hand cranking, go at least 90 miles an hour, and by the way that there be plentiful roads and highways in place to handle cars going at those speeds? My guess is, if they’d demanded those things up front, we’d still be waiting for cars to be widely used today. New technologies need to grow in conjunction with use; no use, no technology growth.

And God help us if car buyers in 1910 had demanded Bluetooth! And especially if they’d demanded cell towers to allow them to make hands-free phone calls.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

IEEE SBOM panel discussion

Next Monday evening, I’ll participate in a virtual panel discussion on SBOMs sponsored by six IEEE chapters in the NE Virginia/DC area. My fellow panelists will include Allan Friedman of CISA and Jean Camp of Indiana University, who I’ve heard talk previously and seems to have a lot of interesting things to say. The panel discussion is open to anybody; you can register here. They’ll encourage audience questions, and if the discussion is still going on when the hour is up, they’ll let it continue. Since I hate to see discussions ended when there’s clearly interest in continuing it, I’m game for that! 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

Why would anyone want to bring down the US grid?

 

I’ve stated multiple times that I see just about zero likelihood of a cyberattack bringing down the US power grid (for example, in this post and this one). The reasons I’ve pointed to recently have always been related to the technical infeasibility of such an attack succeeding (although I should say “impossibility”, since I think it would be literally impossible to bring down the entire US grid with a cyberattack, no matter how massive).

But I recently remembered another reason for this opinion, which I stated in a post in 2020: Any cyberattack that succeeded in bringing down (or even substantially disabling) the US grid (actually grids, since there are three of them) would have to be carried out by a nation-state, or perhaps a huge and very sophisticated terrorist organization, like al Qaeda probably was before 9/11. Since there’s currently no such terrorist organization, the perpetrator would have to be a nation-state, like China, Russia, Iran, or North Korea.

If such an attack were carried out (whether successful or not), it would be considered an act of war, almost on the scale of Pearl Harbor. And if the US declared war on the perpetrator, there’s no way to know how far that might go – including a nuclear exchange.

Why would a nation-state, no matter what degree of hatred their leaders or people harbored for the US, risk possible total destruction, simply in order to cause a lot of damage to the US economy (and perhaps kill a lot of Americans) through an unprovoked attack? And don’t give me the “madman” story. Neither Vladimir Putin nor Kim Jong-Un, or any other world leader, is that mad that they’d risk destroying their life and the lives of their family members and friends.

If you want to worry about something that could cause a grid disaster, worry about an EMP attack or a massive solar storm (one did bring down the Quebec grid in 1989, and if the Carrington Event of 1859 had occurred today, that might well have brought down the entire US grid). Or worry about a pandemic. What would happen if one of those occurred, and the US response was so badly managed that a million people died? I agree that the idea is far-fetched, but I’d still say it’s a lot more likely than a cyberattack taking down the US grid.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

Tom Pace’s biggest problem

I recently wrote two posts (the second one is here) about a chilling revelation that Tom Pace of NetRise made at an informal meeting I recently attended. NetRise specializes in firmware security, and Tom has  noted that a huge percentage of software and firmware products aren’t registered at all in the National Vulnerability Database (NVD), meaning there’s no CPE name registered for the product. This means there has never been a single vulnerability reported for the product.

Thus, if you’re comparing competing IoT devices and you discover in the NVD that one device has a few serious software vulnerabilities while the other doesn’t even appear in the NVD (and thus has no identified vulnerabilities), you would be making a huge mistake if you ruled the first device out of consideration for that reason.

In fact, Tom produced software bills of materials for some firmware products (and no, an SBOM for firmware isn’t an FBOM, you naughty person!) that don’t have CPE names, and identified known vulnerabilities of components in those products. In one product (a member of a family of widely-used ICS products, that doesn’t even mention anything about security or vulnerabilities on its web site. If it’s a subject you’re embarrassed to talk about, better to not mention it at all, I guess...), he found 1,237 known vulnerabilities, and he could have found more if he’d had more time.[i]

Since I addressed the implications of Tom’s statements in the two posts on that problem, I won’t repeat them now – except to say that any supplier of products that aren’t listed in the NVD should be viewed with some degree of suspicion. Frankly, they should be considered guilty of poor security practices until they prove themselves innocent.

But at the end of both posts, I pointed out that this was only the second-most-serious problem identified by Tom Pace during his presentation, and I’d discuss the most serious one soon. Well, soon is now!

The 1,237 vulnerabilities that Tom found in one ICS product – and the probably tens of millions of vulnerabilities that are likely to be found (either by the good guys or the bad guys) in all other products that aren’t registered in the NVD – are all known, meaning they have previously been identified and have CVE numbers. These should be picked up by malware scans, so if you do those regularly, you should be able to at least bug the supplier of any product that contains them, to get them patched.

However, we all know that there’s a much more dangerous class of vulnerabilities called “zero-days”, that haven’t previously been identified and don’t have CVE numbers; therefore, scans won’t pick them up at all. The holy grail for a software attacker is to be able to implant a zero-day vulnerability in software or firmware during the software development process. This means that every organization that installs the software will potentially harbor this vulnerability without knowing it, allowing the attacker to penetrate their network and do bad things. This is exactly what happened with SolarWinds. The Russians planted the SUNBURST malware in seven updates of the Orion platoform. 18,000 automatically users downloaded – and presumably installed – the tainted files.

Of course, it’s not usually easy to plant malware – zero-day or otherwise – in software or firmware while it’s being developed. In fact, the Russians had, according to Microsoft, 1,000 people working in the team that penetrated SolarWinds’ development environment and planted the malware. They were in the environment for 15 months and – like all well-planned business projects - spent the first three of those doing a proof of concept (I imagine they’ll write a Harvard case study on their success one of these days).

Perhaps you’re comforted by the fact that such a big effort was required to plant Sunburst. If that’s what it takes to plant a zero-day vulnerability in a software product during development, that’s not very likely to happen often in the future, right? However, Tom discovered there’s a much easier way for bad guys to plant zero-days, which will be hard to prevent without some self-regulation (or actual regulation, although God help anyone who tries to impose regulations on open source! In comparison, building a house out of Jell-O™ would be easy). In the hopes that people in charge of open source projects will start to address these issues, Tom presented the following scenario:

1.      Suppose a particular open source project is regularly used as a component in other software or firmware, and that a malicious party identifies one such project.

2.      That party implants a vulnerability in the code. But not just any vulnerability: a zero-day. Perhaps they work for a nation-state that hoards zero-days for a rainy day, rather than notifying the supplier of the vulnerability (as they should do, of course). Or perhaps they’re just talented and dreamed one up on their own.

3.      The malware-laden code becomes part of the next update of the open source product, so it’s downloaded and included as a component in some end-user products. Since the vulnerability is a zero-day, the scanners don’t identify it.

4.      The party that planted the malware reaps what they sowed, and is able to penetrate end user environments that have downloaded a software product that includes the now-malicious component.

5.      The malicious party penetrates multiple organizations, using the zero-day vulnerability. Of course, these penetrations will probably only be detected after the attackers have succeeded in achieving whatever their goal was: disruption of a manufacturing plant, spreading ransomware, exfiltrating PII, etc.

6.      If we’re lucky, the vulnerability will be identified after the first few penetrations (rather than after the first 18,000, as in the case of Sunburst) and will be assigned a CVE number and reported to the NVD. After that, it will usually be picked up by scans, so proactive suppliers (i.e., the ones who actively look for vulnerabilities in their products and report these to the NVD, and who of course patch the vulnerabilities when they meet whatever severity threshold they set for patching) will protect their products. Of course, suppliers who have never reported a vulnerability in their products, and who don’t even register those products in the NVD, won’t even find this new CVE because they don’t look for vulnerabilities at all. Their mottos are “Do not seek, and ye shall not find” and “Ignorance is bliss, as well as solid legal protection.”

7.      Even after this vulnerability is identified in the NVD and the scanners start picking it up, it won’t immediately be obvious that it’s found in a component of the affected product(s), and, more importantly, in which component (remember, there are plenty of software products and intelligent devices – including ones you may have on your desktop – that have thousands of components). If you trace a vulnerability to a product, how will you trace it to a particular component? The answer: It will be very hard to do that with just a few known infections. As the number of known infections grows, it will be possible to narrow the list of suspect components down to just one. However, this is a hell of a way to identify a new vulnerability: wait until there have been a lot of infections, so that you can draw some statistically valid conclusions as to the source of the infections.

To be honest, I think Tom’s second-most-serious problem (the one I described in the two previous posts) is the more likely to cause damage, simply because of the huge numbers of software components – and software products themselves – that aren’t listed in the NVD at all, meaning that even already-known vulnerabilities won’t be identified in them. And to be honest, Tom didn’t rank these two problems himself (in fact, I don’t think he even identified them as separate problems in his presentation).

Regardless, this is a serious issue. The second-most-serious issue can be addressed “simply” by having software suppliers (especially including open source communities) register all of their products with the NVD and start reporting all vulnerabilities for them (however unlikely this is to happen on any sort of scale in real life). However, this new issue requires a lot more than that. For one thing, it might require controls on who can contribute to any open source project. And how will those ever be enforced?

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] Of course, just because a component of a product contains a vulnerability doesn’t automatically mean the vulnerability is exploitable in the product itself; that’s why we need VEX documents. In general, probably only 5-10% of component vulnerabilities are in fact exploitable in the product. But even that would amount to 60-120 exploitable vulnerabilities, in this case. Remember, even one exploitable zero-day vulnerability is a huge deal.

So, what can you do with SBOMs after you get them?

Last week, I put up a post about a webinar on SBOMs that Fortress Information Security will present next Friday, May 13, titled “Software Bill of Materials Overview”. While that title doesn’t tell you a lot in itself, the first sentence of the description of the webinar reads

Fortress VP Tony Turner will be presenting key use cases for any software consumers looking to get started using SBOMs for procurement, vulnerability management, (incident) response, and software assurance activities.

In my post, I commented that “Fortress has put together what looks like it will be the first webinar describing how people like you and I can use SBOMs to make our organizations more secure.” In other words, I was saying this will be the first webinar that discusses using SBOMs to secure your organization from vulnerabilities and risks found in the software you use.

This idea was similar to what I said in a post a month ago, after Fortress posted an important white paper on SBOMs. I said the paper, for the first time, “provides a comprehensive narrative of what an end user organization needs to do to obtain SBOMs and VEX documents and use them for the purpose of reducing the cybersecurity risks their organization faces – specifically risks due to components found in the software they utilize to achieve their mission.”

When I put up the above-linked post on LinkedIn last week, Ron Brash of aDolus (another organization very involved with SBOMs) took exception to my assertion that this would be the first webinar that describes how to use SBOMs to make your organization more secure. He pointed to two webinars that aDolus presented last year (one with Industrial Defender and one with Verve), that were both aimed at consumers. I had attended one of these (and took notes, which I just reread); I just watched the other.

Both were excellent webinars, and I recommend you watch at least one of them (they were both presented around the same time, so the subjects were similar). And I’ll agree that both of them were intended for use by software consumers, not software producers. But I stand by my original statement that neither webinar discussed how organizations can use SBOMs to make themselves more secure, although I’ll point out below that such an omission is the rule, not the exception, in SBOM messaging so far.

To understand why I say that, look at the topics discussed (and addressed very thoroughly) in the two webinars. They include:

1.      The National Vulnerability Database (NVD) doesn’t usually identify vulnerabilities found in components of a product, but just in the product itself. This is because the NVD has no way of knowing what components are included in a product. Of course, this is why you need an SBOM, since that tells you the components in the product and you can – in theory – look each component up in the NVD, in order to learn of the risks it poses.

2.      But it isn’t easy to find vulnerabilities for the products themselves in the NVD, let alone for the components. You really need to know the CPE (Common Platform Enumeration) name for the product, and there are a lot of problems with that (this is the “naming problem”, which I pointed out in 2020 is one of the two or three most serious problems preventing widespread use of SBOMs. It still is, although a group of people, which includes both me and Eric Byres of aDolus, has very recently resolved to understand the causes of the problems – which are actually not complex - and develop a suggestion for NIST to fix the problems moving forward. Of course, that in itself won’t fix the problematic data found in the NVD today, but doing that isn’t technically hard, either – once the revised structure is agreed upon).

3.      A big component (so to speak) of the naming problem is the fact that the supplier and name of a product change frequently, primarily due to mergers and acquisitions but also due to marketing changes, efforts to harmonize product lines, etc.

4.      SBOMs don’t normally provide information on open vulnerabilities in components. To learn about those, you need to look up the components in the NVD (or another vulnerability database), where – of course – you’ll once again run into the naming problem.

I agree that all of these are problems. However, these aren’t in themselves consumer issues – they’re supplier issues. In my opinion, the software supplier needs to take responsibility for addressing each of these problems as part of fulfilling their responsibility to produce SBOMs, and to alert users to vulnerabilities that are due to components in the product.

Why is this the supplier’s responsibility? Because they chose the components to include in their product. Including components (especially open source ones) in their software saves suppliers a lot of time and money, as opposed to having to develop or purchase the code themselves. But these components also bring risks to the users of the software – as we know all too well from Heartbleed, Apache Struts, Ripple20, Log4j, etc. While users have a role to play in managing those risks, the primary responsibility is the supplier’s, starting with creating and updating SBOMs for their products.

Since the four items listed above (and others) are the supplier’s responsibility, what are the users responsible for doing, to reduce the risk posed by components of the software they use? That’s exactly what – I believe – will be discussed in the webinar next Friday. However, you can get a preview of the webinar discussion in the white paper that I mentioned. Here are some of the topics addressed in that paper (I’m sure some of these will be discussed in the webinar as well):

·        SBOM USE CASES FOR VULNERABILITY MANAGEMENT  

·        UNDERSTANDING UNVERIFIED TRUST

·        PRODUCT VS. COMPONENT VULNERABILITIES

·        SBOM SOLUTION PROVIDERS AND VULNERABILITY MANAGEMENT  

·        NEW COMPONENT VULNERABILITIES  

·        NON-EXPLOITABLE COMPONENT VULNERABILITIES  

·        USING SBOMS IN PROCUREMENT  

·        CONSIDERATIONS BEFORE RECEIVING SBOMS

·        WHAT TO DO WITH SBOMS AFTER RECEIPT?  

·        WHEN SHOULD SBOMS BE UPDATED?  

·        COMPONENT MODIFICATIONS AND SOFTWARE PEDIGREE  

These are the topics that, in retrospect, should have been addressed in the two aDolus webinars more than a year ago, given that their focus was intended to be on software consumers, not suppliers. However, I certainly find it understandable that aDolus focused on problems that were really supplier issues, in a webinar that was intended to be for end users. When I say that there’s been no previous document that’s addressed how to use SBOMs, I’m including two government-published documents that were also supposed to address use of SBOMs by end users who aren’t also software developers (the developers’ use case for SBOMs is to evaluate risks posed by components they’ve included in their products. As I wrote a couple of weeks ago, there’s now ample proof that developers are already putting SBOMs to work, in a huge way, to help them reduce their own software development risks. So the developers don’t need much guidance on using SBOMs anymore – it’s the consumers who do):

1.      Before it shut down last year, the NTIA Software Component Transparency Initiative published a “Software Consumers Playbook”. It’s well-written (although the language will be hard to understand by non-developers, its intended audience), but it focuses almost entirely on how SBOMs should be produced – which, of course, has nothing to do with the question of how a consumer can use SBOMs after they’ve received them.

2.      In December, NIST put out preliminary guidelines for implementation of Section 4(e) of Executive Order 14028; this section includes the two or three provisions that mention SBOMs. In the post I wrote on the guidelines, I pointed out that, in the approximately four pages of guidance regarding SBOMs, there are only two sentences that even mention how software using organizations (i.e. the federal agencies that are subject to the EO, very few of which are in the primary business of developing software) can use SBOMs for component vulnerability risk management purposes. They are:

Develop risk monitoring and scoring components to dynamically monitor the impact of SBOMs’ vulnerability disclosures to the acquiring organization. Align with asset inventories for further risk exposure and criticality calculations.

Regarding those two sentences, I pointed out immediately below them, “…this is the only requirement in the entire SBOM section that relates to the agencies using the SBOMs. Every other requirement specifies something that the agency needs to ask their suppliers to do or provide. I hope NIST doesn’t expect the agency to simply drop an SBOM in the bit bucket as soon as they receive it – they need guidelines for how they will use it.”

Unfortunately, up until Fortress released their white paper last month, those two sentences constituted the entirety of the guidance available – from either public or private sources - on making use of SBOMs for software component vulnerability risk management purposes, at least that I know of. If anybody knows of other previous guidance, I would love to hear about it and will publish it.

Does this mean that I don’t think software users need to know anything about the various challenges that arise in developing SBOMS? No. That’s why I recommended that you watch at least one of the aDolus webinars. I also recommend that you try to at least skim through every document found here.

But as you do that, and you hear about the many problems developers face in preparing SBOMs, you need to keep in mind that these aren’t your problems. Your problem is what you do with that nice, shiny SBOM you just received. And keep in mind that there’s no good way to utilize SBOMs (or VEX documents) for risk management purposes, except through automation – either a tool that you run on-prem, or a third-party service that runs their own tools and provides you with the results; those results should include a continually-updated list of exploitable component vulnerabilities found in each of the software products used by your organization.

As of today, there’s no tool or third-party service (other than hourly consulting services, which aren’t scalable), that will ingest both SBOMs and VEXes and produce that continually-updated list for you. And don’t fool yourself into thinking that you could do that on your own, given enough time and effort and YouTube videos. There are probably products on your desktop today that have literally thousands of components (the average software product contains close to 150 components); are you really going to go through the steps shown in the video for every one of those components, especially since new SBOMs may, for some products with lots of components, arrive daily?

Sure, you can try looking up a few components from an SBOM in the NVD; you might even succeed in learning about a few vulnerabilities that apply to those components. But don’t mistake doing this for actually using SBOMs for vulnerability management purposes. If you want to learn about that, read the white paper and sign up for the Fortress webinar on Friday the 13^th.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

Tom Pace’s second-biggest problem

A recent post described a presentation I saw last Friday by Tom Pace of NetRise, describing what seems to be a huge security problem. To summarize it:

1.      Do you think products with a lot of open vulnerabilities - as indicated by CVE’s listed for the product in the National Vulnerability Database (NVD) - are dangerous and should be avoided? If so, you’re right.

2.      By the same token, do you think a product with no open vulnerabilities – and which has never even reported a vulnerability – is one you ought to buy? After all, what could be safer?

3.      It turns out that you’re probably better off with a product with lots of open vulnerabilities than with a product that’s never reported a vulnerability. Seriously.

4.      Why? As Tom pointed out, nobody can report a vulnerability (CVE) for a software product or intelligent device, unless the product has a CPE name. So, if the product doesn’t have a CPE name, no vulnerability has ever been reported for it.

5.      There are two reasons why a vulnerability might never have been reported for a product: A) The coders who created the product – as well as the third party coders who wrote each of the 135 components (on average) included in the product – never make mistakes; and B) The supplier of the product has never even bothered to look for vulnerabilities in it, perhaps because they believe A) is true. Of course, if this is their reasoning, it’s hard to argue with it: If you’re perfect, you don’t need to worry that you’ll ever make a mistake – after all, you wouldn’t be perfect if you made mistakes! Pretty compelling logic, no?  

6.      Tom came to realize that probably the majority of software products and intelligent devices sold in the US don’t have CPE names, meaning they don’t have any CVEs reported against them. So if your organization is comparing five different vendors in advance of procuring a particular type of software, and you decide to compare the vendors based in part on how many open vulnerabilities their product has, what will you do if you find that four of those vendors have some moderate number of open vulnerabilities, but the fifth doesn’t have any at all – in fact, they don’t even have a CPE number? Will you stop the evaluation and declare the fifth vendor the winner? After all, they’re not only good…They’re perfect…

7.      It turns out that Tom Pace doesn’t believe there’s such a thing as perfect software developers, so he decided to test this idea. He identified the supplier of a widely used line of devices found in many ICS environments and started looking for their devices in the NVD. He didn’t check on them all, but none of the ones he checked on had CPE names – and therefore they didn’t have any reported vulnerabilities. In fact, the supplier’s web site doesn’t mention anything about vulnerabilities or patches – or even security in general. It seems the subject never came up for them…

8.      So either this supplier is so perfect that they don’t even have to think about security, or they made the decision early on not even to look for vulnerabilities in their products, since that’s probably the best way to ensure that nobody will ever learn of vulnerabilities. They won’t have to report vulnerabilities to the NVD, post notices on their website, etc. Sounds good, right?

9.      Tom decided to test just one of this supplier’s products, by first creating an SBOM for it and then identifying all of the open vulnerabilities that apply to those components; if he found anyway, that would constitute proof that the supplier isn’t perfect.

10.   I regret to inform you that this supplier isn’t perfect, after all. How many vulnerabilities did Tom identify? 5? 10? He actually found…drumroll, please…1,237 vulnerabilities. And he could have found more if he’d had more time. Moreover, that was 1,237 vulnerabilities in just one of about 50 products sold by this supplier – and probably none of those other products have CPE names, either.

Of course, this strategy is pure genius: Since the   supplier never reports any vulnerabilities, they don’t have to worry about patching log4j, Ripple20, OpenSSL, or any other serious bugs that keep people up at night and ruin holidays. After all, this supplier is perfect.

But for those of us who live in the real world and realize there’s no perfection (especially in software), we need to take other steps to protect ourselves. Here are some suggestions:

1.      When you’re sending out an RFP for software, first check the website of each supplier you’re considering inviting to the RFP: What do they say about security and vulnerabilities? More importantly, do they say anything at all about those topics? If they don’t, drop them off your list.

2.      Ask the salesperson for each supplier you’re considering to give you the CPE name of their product. If they can’t give you anything (meaning there is none for the product), drop that supplier off your list.

3.      If the product has a CPE name, but the NVD doesn’t seem to list any vulnerabilities for any version of the product, ask the salesperson to tell you if they’ve ever reported any vulnerability for it (and if so, verify the CPE name. It’s possible they entered the CPE name wrong and reported vulnerabilities against that name. The NVD doesn’t validate CPE names, as far as I know).

4.      But, if they tell you, “We’re so dang good that we’ve never have a vulnerability!”, thank them for their time and drop them off your list. What they’re really saying is, “We’ve gone to great lengths to make sure we don’t know about any vulnerabilities in our products, so we can tell you truthfully that we know of no vulnerabilities. You can be sure that when a big vulnerability appears in one of our products, we’ll be the last to know about it! Are you ready to sign the contract?”

Again, thank them for their time and remove them from your list. By the time you’re done with this, I hope you’ll still have at least one supplier on your list. But even if you don’t, you’re much better off staying away from any supplier that’s so convinced of their perfection that they don’t want to take the chance that they’ll be proved wrong, by…you know, trying to find out if they actually might have some vulnerabilities.

At the end of my previous post, I noted that this problem was only the second most serious problem Tom has identified, having to do with CPE names. Coming soon to a blog near you: Tom Pace’s most serious problem.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

Finally – an SBOM webinar for the rest of us

Up until recently, almost the entire focus of SBOM messaging was on software developers. The reason for that – which I don’t argue with at all – was that SBOMs have to be both produced and used. Unless the software suppliers got onboard, it wouldn’t matter whether users want SBOMs or not – they won’t get them if they aren’t available in the first place. The users need to “pull” on the SBOM string, but there won’t be any string to pull on, if the suppliers aren’t “pushing” the SBOMs out on their end.

Well, that battle has been won. Software developers are using SBOMs for their own vulnerability management purposes, and heavily (as the 202 million calls from Dependency-Track to the OSS Index vulnerability database – in one month - attest). Now it’s time to get the users interested in receiving those SBOMs. That way, the string will actually move, as opposed to being pushed from behind into a tangled ball that doesn’t go anywhere, because nobody’s pulling on the other end.

But consumers aren’t interested in having SBOMs so they can a) impress their children that they know something about technology, too; b) print them out on waxy paper and use them to wrap fish; or c) print them out on contact paper and use them to paper the walls of their new den. They want to use them for vulnerability management purposes, so their organizations can be more secure, and so they don’t have to run around for months trying to find every instance of the next log4j.

In early April, Fortress Information Security[i] put out a white paper that was the first document (that I’ve seen) that discusses using SBOMs, not producing them. And in another first, it’s aimed entirely at software consumers (who can be assumed to know very little about SBOMs), rather than software developers (who can be assumed to know a lot about them).

Now, Fortress has put together what looks like it will be the first webinar describing how people like you and I can use SBOMs to make our organizations more secure, rather than to wrap fish. While I don’t know what will be in it, I’m sure it will be good. You can read about the webinar (on Thursday, May 12 at 11 AM EDT) and sign up for it here. See you there!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] Full disclosure: I provide consulting services to Fortress, although I’m promoting this webinar because it’s sorely needed.