Will VEX go both ways?

 

Probably the person I have learned the most from with regard to SBOMs is Steve Springett, chair of the OWASP CycloneDX working group, leader of the OWASP Dependency-Track project, and leader of the OWASP Software Component Verification Standard (SVCS) project (BTW, he has a day job as well!).

Steve is someone who is always far ahead of the “conventional wisdom”. One illustration of this is the fact that he developed Dependency-Track, which is a tool that ingests SBOMs (actually, BOMs in general) and identifies and tracks vulnerabilities applicable to components. And he did it in 2012, years before the term “software bill of materials” was invented, and especially before the idea became prevalent of using SBOMs for vulnerability management purposes.

Another illustration is something that happened this week. I’ve been corresponding with Steve by email regularly (he lives only about ten miles from me in suburban Chicago, and I’ve suggested we have lunch sometime. He said he’d be glad to do that when there’s a restaurant with outdoor seating where we can meet. However, for some reason I haven’t been able to find a restaurant in Chicago that offers outdoor seating in January or February. I’ll keep looking, though!).

Steve and I were corresponding on the subject of VEX in CycloneDX and Dependency-Track, when he wrote, “I’m troubled by the one-way nature of how VEX is positioned. The word ‘exchange’ means that two parties both give something and receive something. The case you’re framing is a simple one-way transfer, which may be useful, but will not work in the real world.”

This hit me like a bolt from out of the blue, since I’ve always thought of VEXes (and SBOMs, too) as one-way documents. They’re issued by a supplier (or perhaps another party) and provided to their customers. The VEX’s purpose is to tell the customer the status of a vulnerability (i.e. whether or not the vulnerability is exploitable in the product itself, not just in one of the components when considered as a standalone product) in one or more versions of one or more of the supplier’s products. And I’m not alone in believing that VEXes are only one-way documents. I’ve attended most of the VEX meetings (formerly under the NTIA and now under CISA) since their inception in September 2020, and I’ve never once heard any discussion of the VEX as anything but a one-way document.

So what did Steve mean by two-way transfer of VEXes? He continued, “Having spent most of my life working for software companies, I can tell you with absolute certainly that customers report issues all the time. Sometimes they’re for previously unknown vulnerabilities or they prove that a third-party library that was thought not to be exploitable, actually is. Many customers have internal teams that conduct offensive engagements, or they work with consultancies that perform the engagements for them, or software companies work with bug bounty firms like HackerOne.”

What Steve has in mind is customers creating VEXes that state a particular vulnerability is exploitable or not exploitable in a particular product. Of course, they do this both for the supplier’s and their own benefit, so the supplier will patch exploitable vulnerabilities they weren’t previously aware of.

Note that the activity – a customer reporting the exploitability status of a vulnerability to a supplier – is something that software customers are already doing. What would be innovative about Steve’s approach is that the customer would be able to deliver the “report” in a machine-readable format, which is in fact the same format that the supplier uses to report status of vulnerabilities to the customer.

Who will be the main beneficiary of this two-way communication? The customer will benefit, since two-way VEXes will make it easier for them to report an exploitable vulnerability to a supplier, once they get the hang of creating VEXes themselves. Now, the report could be created entirely by an automated process, instead of having to spend time explaining it to someone in tech support on the phone or in an email.

But two-way VEXes will make life a lot easier for the supplier, since getting a vulnerability report from a customer won’t require the help desk staff to spend a long time on the phone with the customer, then transcribe what they just learned into the supplier’s internal vulnerability management system. Since the supplier will already be familiar with the CycloneDX VEX format, they will be able to link that to their internal system, so that future VEX reports from customers can flow directly into the system.

But once we have two-way VEXes in place (and it sounds like doing that will just require some simple tooling on both the customer and supplier sides), are there any other processes that might be automated by this as well? Maybe even new ones we hadn’t considered before?

Coincidentally, earlier on Monday I had heard of a vulnerability management communication that would be very helpful, both to end users and suppliers. But since there’s currently no way now to do this communication cost-effectively, it simply isn’t being done at all – in fact, I hadn’t even thought of this communication at all until another person (who is very knowledgeable about SBOMs and communication channels) suggested it to me.

To understand this type of communication, think about the main use case for a VEX: A supplier tells a customer that a particular vulnerability – which is listed in the NVD as applicable to a component of a supplier’s product - isn’t in fact exploitable in their product. The reason the supplier should tell the customer that a vulnerability isn’t exploitable is that, by doing so, they’ll save both themselves and the customer a huge amount of wasted time and effort. The customer won’t waste time bugging the supplier about when they’ll patch a non-exploitable component vulnerability, and the supplier’s help desk won’t waste a lot of time talking to customers about those vulnerabilities.

The reason this is a big deal is that almost without doubt the great majority of software component vulnerabilities aren’t exploitable in the product itself – in fact, that number could be around 95%. In other words, if a customer called their supplier about 100 vulnerabilities identified in components of a product in the NVD, 95 of those calls would be a waste of both the customer’s and the supplier’s time.

However, for a VEX to work best and save the most time and money, it needs to be sent as soon as the non-exploitability is discovered – i.e. the same day. If that isn’t done, then both the supplier and the consumer are likely to waste significant amounts of time dealing with false positive product vulnerabilities.

If the need for the supplier to communicate that a component isn’t exploitable to their customers didn’t occur very often, it wouldn’t be hard to send a VEX to every customer very soon after the non-exploitability was discovered, so not much time would be wasted.

But consider the case of a supplier that has ten products, each with thousands of components (as in one of the examples in my previous post). The supplier might well discover multiple non-exploitable component vulnerabilities every hour of the day. Rather than be constantly sending VEXes out to their entire customer base multiple times an hour, they may aggregate these notices into one VEX every day or even every week.

The problem is that, when a customer discovers a component vulnerability in the NVD but hasn’t seen a VEX saying it’s not exploitable, they will need to assume that the vulnerability is exploitable. So even if a VEX arrives a few days later saying it’s not exploitable, the customer may already have wasted a lot of their and the supplier’s time, under the assumption that it is.

Wouldn’t it be nice if, upon discovering a component vulnerability in the NVD, a customer could immediately query the supplier about whether or not it’s exploitable and get an immediate answer – with no human involvement required on either end? Given that the customer might use hundreds of products with collectively tens or hundreds of thousands of components, this would enable them to learn right away about the exploitability status of every component vulnerability they discover in the NVD (or another vulnerability database, of course). Currently, they simply can’t do this. 

I think Steve's two-way VEXes could enable this problem to be solved as well, possibly introducing a lot of new efficiency into the software vulnerability management process, for both customers and suppliers of software.

So Steve has given us another gift, which will probably enable a whole cottage industry and make some people – if not rich – at least fully employed for a lifetime. Not bad for a day’s work.

By the way, would you like to know the technical name for this bi-directional VEX capability?...I didn’t think so, but I’ll tell you anyway: it’s biVEXuality. Catchy, no?

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

Rivers of SBOMs. Oceans of VEXes.

A lot of companies – both start-ups and established companies – are currently trying to figure out how they’ll be able to take advantage of the coming era of widely available and widely used SBOMs. I want to point out that this era won’t arrive for at least another 3-4 years. But the point is that it’s coming.

There are three types of organizations doing this figuring:

1.      End user organizations that will be able to greatly expand their efforts to combat software supply chain vulnerabilities – by identifying exploitable vulnerabilities in components of software they utilize.

2.      Software suppliers, who will voluntarily (or due to market pressure. Same thing) provide SBOMs and VEXes to their customers, so that they can perform this analysis.

3.      Third party tool and service providers, who are gearing up to enable all of this activity to take place.

However, there are also some people – again, at both startups and established companies – that think they already have SBOMs figured out, so their plans (and in some cases, actions) really don’t need to be rethought, just executed. These organizations think they understand SBOMs, and how the market will work for them.

I’m not one of these people. There’s only one thing I’m sure of: Whenever I think I understand SBOMs and the infant marketplace that’s growing around then, I’m always confronted with the fact that I don’t understand either one. As the great physicist Richard Feynman said about a different topic, “I think I can safely say that nobody really understands quantum mechanics.” Although I prefer the pithier (but unfortunately apocryphal) wording: “If you think you understand quantum mechanics, you don't understand quantum mechanics.”

I recently had the opportunity to be present when Dale Peterson did an interview with some guy for his podcast. Toward the end of the podcast, the guy brought up the Netscape IPO in 1995, and why its implications had been very different from what anyone predicted at the time. To elaborate on what he said in the podcast:

1.      It seemed in 1995 that a bunch of brand new companies - led by Netscape but also including WebVan, Baidu, Ariba, Yahoo! and others – would take over the world, because they obviously had their markets all figured out. Until they didn’t, of course.

2.      There were also companies that were in existence at the time but were considered quixotic and not worth paying much attention to. There was a guy named Jeff Bezos in Seattle, who had founded a small online bookseller but who kept talking about becoming an online marketplace for all consumer transactions. He also talked about the need to get big fast, even if it meant losing lots of money. He succeeded on both counts in Amazon’s early years: getting big fast and losing mountains of money. It seemed like the company would be a continual cash sink; I sold my modest amount of Amazon stock in the early 2000’s (one of a series of brilliant investing moves I made around that time).

3.      And there was a third type of company that was not even in existence in 1995, including companies like Facebook, Google, Alibaba, PayPal and Netflix. These companies got rich serving markets that didn’t exist in 1995, or at least were very poorly understood. In fact, as it turns out, Netscape itself didn’t understand its market well, since when Microsoft revealed its own browser the following year – and provided it free with Windows – Netscape was relegated to a small role from then on (Netscape’s spirit lives on in the Firefox browser, although the code base is completely different now).

The point is that, when a new technology market is in its very early stages, anyone who thinks they know what the market wants is most likely wrong. Especially if they’re trying to analogize from markets that are well understand to the new one. It’s only when there are lots of organizations pouring lots of money and effort into the new technology that it will gradually become clear what’s needed. Much money will be lost by companies that thought they’d figured out exactly what the new market needs and put in place a wonderful edifice based on that supposed knowledge, only to have that edifice disappear down the drain when it first confronts reality.

Meanwhile, entrepreneurs (or even executives of established companies who are searching for new opportunities) who retain some humility and don’t pretend to have everything figured out are much more likely to come across the real opportunities that don’t follow any previous model, like social media, online payments, intelligent search, and movie streaming. So how do these humble entrepreneurs discover these opportunities? Is it just the dumb luck of being in the right place at the right time?

While I never want to discount the importance of dumb luck, it’s possible to trace out the implications of facts that should be plain as day, in order to get an idea of areas where opportunities might lie in the infant SBOM marketplace. Here’s an example of such facts:

1.      A new SBOM is needed whenever any change occurs in any version of the product (although IMO it would be OK if this were just read to mean supported versions). This includes any change in the code that the supplier itself wrote, as well as any change in one of the components, including when a component is added, removed or upgraded. There should also be a new SBOM when the name of a component, or the component’s supplier’s name, changes. Etc.

2.      When the same change occurs in multiple versions of a product, such as when a certain vulnerability is patched in multiple versions, a new SBOM needs to be released for each of those changed versions (and of course, the version number itself needs to change, if only from v1.2.3 to v1.2.3a).

3.      So how often will new SBOMs come out? That will vary, of course, but I think the biggest predictor of how often a new SBOM will be needed is the number of components in the product. According to Sonatype, the average product has 135 components, but many products have thousands (and I’m just waiting to see how many components are in Windows 10). If a product has 5,000 components, I’d say there’s a good chance that a new SBOM will be needed daily, for every version of the product.

4.      And if you think daily SBOMs are a lot, you ain’t seen nuthin’ until you’ve seen VEXes. Generally, a new VEX will need to be put out any time the status of a component vulnerability has changed. So if a vulnerability has appeared in the NVD (or another vulnerability database, of course) for a component of a product, the supplier of the product should put out a VEX stating the status of the vulnerability in the product itself: exploitable, not exploitable, fixed (i.e. patched) or under investigation (which is “under triage” in the CycloneDX VEX format). And if the product’s supplier has patched the component vulnerability or has determined it isn’t exploitable (when before they thought it was), they need to issue a VEX indicating that changed status, and the version or versions (or version range) in which the status has changed.

5.      Of course, there doesn’t need to be a separate VEX for each status change of each component of each version of each product. If there are a number of changes in the same day, all in the same product, it should be fine to include them all in a single VEX, although I’ve come to believe  that, when there are changes involving multiple products, these should be handled in separate VEXes. However, it’s important that any change in vulnerability status (especially when a new exploitable component vulnerability is identified) be communicated the same day, rather than as part of say a weekly summary.

6.      Let’s go back to our product with 5,000 components. How often will a new VEX be needed for each version of that product? If we assume that it’s inevitable there will be some change in at least one of every thousand components every day, and if we assume there are 20 versions of the product still under support (including minor versions and bug fixes), this means there will need to be 5 (=5,000/1,000) X 20 = 100 new VEXes every day. And that’s for one product. If a supplier has ten different products, each with 5,000 components and 20 versions, there will need to be 1,000 new VEXes every day.

7.      And if an organization has five suppliers like this one, they’ll receive 5,000 VEXes every day.

8.      They’ll also receive 500 new SBOMs a day (= 5 suppliers X 5 products X 20 versions).

Of course, for most organizations there will be far fewer VEXes and SBOMs every day. But for others, there will be more than that. My point isn’t to try to estimate a precise number (which of course won’t be realized until 5-10 years in the future, when production and use of SBOMs are practically universal) of SBOMs and VEXes an organization is likely to receive; my point is that it could potentially be huge.

Anybody who tells you there’s a way that these volumes can be handled with anything but automated tooling is living in a fool’s paradise. Yet how many complete tools are available today? I define a complete tool to be one that will:

1.      Ingest SBOMs and store a set of components for each version of each product used by the organization in a database.

2.      Look up vulnerabilities (CVEs) applicable to each component in the NVD (and possibly other vulnerability databases) and store all of them under the product/version pairs that contain the vulnerable component – with a preliminary status of “exploitable” or “affected”.

3.      As VEXes come in that indicate a particular vulnerability isn’t in fact exploitable in a particular product/version pair, the status will change to “not exploitable” or “not affected”.

4.      Because of the above, the database associated with the tool will always contain, for each product/version pair, the most up-to-date listing of exploitable component vulnerabilities.

If you guessed that the answer to my question is zero, you’re right! There are currently no complete SBOM/VEX consumption tools. I have been told that one open source tool, Dependency-Track, will support ingestion of VEXes by the summer (D-T has been supporting steps 1 and 2 above since 2012, long before anybody was using the term “software bill of materials”); and I’m sure there will be others as well, both commercial and open source.

Let’s get back to what this post is about (I usually do that, sooner or later): how to identify opportunities in the burgeoning SBOM field. Since opportunities often arise from alleviating problems that might otherwise prevent adoption of a new technology, at least one source of opportunities should be quite clear: the fact that there will ultimately be “rivers of SBOMs and oceans of VEXes” clamoring to be processed, yet currently there are no tools available to do that. What are some specific opportunities that arise from this source?

The most obvious opportunity is consumer tools. If you’re a developer of software tools or you aspire to be one, I’ve just listed what your tool needs to do: Go thou and do likewise.

However, an equally important (or perhaps more important) opportunity is a service that provides the four steps I just described: The service ingests all available SBOMs and VEXes and outputs always-current (at least fresh daily, and hopefully more often than that) lists of exploitable software vulnerabilities in (ideally) every version of every commercial or open source software product used today. And this work is performed in an almost completely automated fashion, to make it affordable for organizations that don’t happen to have unlimited budgets for software vulnerability management.

In this case, I know one organization that’s rapidly gearing up to offer a service like the one described above (and they are providing at least parts of the service now to customers of their other services). This is Fortress Information Security, with their File Integrity Assurance tool. I want to make it clear that I have a consulting relationship with Fortress and have had it for more than two years, although I’m never been given views of services they’re developing.

I honestly don’t know of any other organization that is even contemplating a service like the above, but I’m sure Fortress will have competitors sooner or later. And just in case you’re about to suggest a firm that offers hourly consulting services as a competitor to Fortress in this space, I’ll point out that this won’t scale at all. The service is going to have to be completely automated, if it’s ultimately going to be able to handle the rivers of SBOMs and oceans of VEXes I’ve described.

Here's another opportunity: Almost all of the solutions that I’ve seen described for distribution of SBOMs and VEXes – whether startups offering such a service or the documents written by the NTIA Software Component Initiative (especially this one) – require a human being to be somewhere in the middle, usually identifying what SBOMs and VEXes are applicable to them.

The fact that a large organization may in the not-too-distant future have literally thousands of SBOMs and VEXes to deal with every day should quell any idea that a human can be involved. Some organization will need to take in streams of SBOMs and VEXes coming from software suppliers and produce custom feeds for particular organizations based on the products that they’re using, across the supplier spectrum.

This is probably the hardest opportunity (of the above three) to address from a technical point of view, since this deals entirely with data in motion (there will also be a database from which customers can choose individual SBOMs and VEXes, especially to help in pre-procurement analyses). The streaming service probably can’t depend on storage of the SBOMs and VEXes, even for a short period of time. It will literally be like having fire hoses shooting streams of liquids in primary colors at you. Your job is to – without even trying to divert those hoses into storage tanks – output a continuous stream of a particular mixed color for each of your customers.

Sounds hard, doesn’t it? I forgot to mention: Don’t even look for easy opportunities in SBOMs (or probably in any other technology discipline). If something looks easy, it’s almost certainly not a real opportunity, at least not anymore.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

Is it time for SBOMs to leave the nest?

On February 4, NIST released their promised guidance on the items in Section 4(e) of Executive Order 14028. The heading of this section says, “Such guidance shall include standards, procedures, or criteria regarding…” Regarding what? There are a number of items relating to software security, but since my primary concern is software bills of materials, there’s only one that is directly applicable to that:

“(vii) providing a purchaser a Software Bill of Materials (SBOM) for each product directly or by publishing it on a public website”.

Remember, the EO applies to federal agencies (and it is likely to be followed by a lot of private organizations as well, of course). It doesn’t apply to software developers, because the White House doesn’t have any inherent authority to regulate any organization other than federal agencies, without first getting Congress to give them the authority to regulate them (which of course they didn’t ask for, since this is an Executive Order).

However, you can see that item (vii) – like every other item in Section 4(e) – really applies to the software supplier. It says they need to give the federal agencies an SBOM for each product that they purchase. OK, that’s fine, but since the EO applies to the agencies, what are the agencies supposed to do when they get the SBOM? Are they only obliged to say “Thank you very much” when their supplier gives it to them?

Of course not. There are two important questions that need to be answered, regarding what a federal agency should do regarding SBOMs, while complying with the EO.

The first question is, “What is the agency supposed to ask the supplier to provide?” The new guidance document doesn’t offer anything new, but that question has already been addressed fairly completely in two other documents.  The first is the NTIA’s “Minimum Elements for a Software Bill of Materials”, a document specifically ordered by the EO. While there’s more that I think could have been included in this document, I think it did a good job of listing the most important elements that should be included in an SBOM, as well as minimum requirements for how often the SBOM should be produced, etc.

The second document is the draft set of guidelines that NIST put out in November, and which I wrote about in this post in December. As you can see in my post, I thought the document was overall good, although there were a couple parts that were real head-scratchers. My feeling is that, between these two documents, a federal agency should have a fairly good idea of what they should be expecting from their software suppliers – as long as they’re not looking for specific contract language, since that’s not in either of these documents (nor would I expect it to be).

Now to the second question that an agency needs to have answered: “What can we do with the SBOMs once we start receiving them? Should we frame them and put them on our wall? Should we print them on waxy paper and use that to wrap fish? More specifically, how can we use them to identify risks in software? After all, the EO called for SBOMS because they’re a tool that can be used to reduce software risks. But where’s the guidance on how we can do that?”

Aha, there’s the rub. In those two documents I just mentioned, there are more than 30 pages of text, yet they’re almost 100% dedicated to discussing what suppliers should do in providing SBOMs. Where’s the discussion of what the users (in this case, federal agencies) should do with the SBOMs? As far as I can see, there is no guidance for users in the Minimum Elements document, and there are literally just two sentences in the NIST guidance: “Develop risk monitoring and scoring components to dynamically monitor the impact of SBOMs’ vulnerability disclosures to the acquiring organization. Align with asset inventories for further risk exposure and criticality calculations.”

Now, NIST has never been the place to go for inspiring, uplifting prose, but I must say that I think the federal agencies will need a little more help than this, if they’re actually going to be able to make good use of SBOMs. In fact, since private industry will take its cue from what the Feds do regarding the EO, all of us will need more than these two sentences.

In fact, NIST isn’t alone in neglecting to provide guidance for software users; among all the documents published by the NTIA Software Component Transparency Initiative during its close to four years in existence, only one specifically purports to address the consumer, and even that document mostly discusses what suppliers should do in providing SBOMs to their customers – not how consumers can use the SBOMs to reduce software supply chain cybersecurity risk to their organizations.

Fortunately, I’m pleased to report that a private organization is readying a comprehensive guide to using SBOMs (and VEXes) for software risk management purposes (although the document isn’t in any way “definitive”. A definitive document about SBOMs is still far in the future); this will soon be made available without charge to any private or public sector organization that wants to see it. Noting that, for whatever reason, providing guidance on SBOM consumption is simply not the government’s cup of tea, this organization has stepped up to fill this particular void.

This is usually the point where I start to complain about what [insert name of government agency] is doing or – usually – not doing about [insert Tom’s Beef of the Day]. But this time, I’m going to surprise you. I now think that, as far as guidance on SBOMs goes, government has done about as much as it can be reasonably expected to do. Now the rest of us need to step up, as this privately-developed guidance will demonstrate.

We were very lucky that the NTIA (and in particular Allan Friedman) decided to take the little SBOM fledgling under its wing and nurture it…not to maturity but to the point where private industry, which of course will be the main beneficiary of having a robust SBOM ecosystem, can take over and make that ecosystem a reality.

And we’re also very lucky that the White House was persuaded to include SBOMs in EO 14028, even though SBOMs still have a long way to go before they can be widely used in the federal government. Just by “mandating” SBOMs, the WH made it clear that it’s time to have software component transparency. Now the private sector needs to make that happen. Moreover, it can make it happen now; two or three years ago, that would not have been possible.

How will the private sector do this? I’m beginning to have some ideas, but I’d like to hear what yours are as well.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they necessarily shared by CISA’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

My guest “appearance” on Dale Peterson’s podcast

Last Friday, I taped (not the right word anymore, of course) a podcast with Dale Peterson. It was put up today, and you can listen to it here. I want to thank Dale for inviting me to do this; it went very well. Dale has been very interested in SBOMs for a while and asked some great questions. We got into SBOM formats, VEXes, the EO, and especially how I see the SBOM marketplace shaping up (my answer: It’s like the Web marketplace in 1995, right after the Netscape IPO. Everyone knew the Web would be huge, but nobody could have identified the companies that are big players nowadays. In fact, most of them didn’t even exist then. The best is still to come on SBOMs, and there’s lots of opportunity).

So you may find the podcast to be interesting. One note: My “Road to Damascus” post that Dale and I discussed is here. It’s about why I think most organizations won’t want to do software risk analysis on their own, and I don’t think they should be forced to, either. I think the suppliers should do it, or they should engage a third party organization for that task (I named one organization that I know will be offering that service in the podcast. There may be others that will offer it as well, and if I hear of any, I’ll let you know. I will point out that this isn’t a service that can be performed using an hourly consulting model).

So end user organizations (who aren’t also software developers) don’t need SBOMs or VEXes; what they need is the analysis that will help them identify the risks latent in the components of the software they utilize. Most importantly, they need a list of exploitable component vulnerabilities for every software product that they utilize. And they need each of these lists updated regularly, ideally daily. In fact, once the supplier or third party has the tooling in place, there won’t in principle be a reason why they couldn’t update the analysis hourly – although that might be a little cumbersome, given the number of possible users for the service (i.e. ultimately every organization on the planet. I can’t speak for other planets at this time).

Of course, many organizations will still want to receive the SBOMs and VEXes themselves and do their own analysis. So we still need tools aimed at consumers of software, which will let them perform this analysis on their own. Those will come as well, although they’re later in arriving than most of us had hoped.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

A concise statement about open source component risk

I have been involved with the NERC Supply Chain Working Group (SCWG) since early 2019. Starting in 2019, the group developed a set of eight white papers on various aspects of supply chain cybersecurity risk management. Each was developed by a group of SCWG members. The white papers, and links to recordings of webinars about them, can be found here. It’s been three years (!) since we developed those white papers, and because – in the understatement of the year – there have been a number of new developments in the field of supply chain security in those three years, the SCWG decided we would update the papers this year.

One of the papers is “Risk Considerations for Open Source Software”. Development of that paper was led by George Masters of Schweitzer Engineering Laboratories. I participated in that group, and I learned a huge amount from George about a subject about which I at the time knew next to nothing.[i]

Two weeks ago, George convened the first meeting of the group that is going to revise this paper. In discussing changes we’d like to make to the paper, we decided we needed to address risks due to open source components in other software, not just risks due to use of open source software directly. Of course, the key tool needed to mitigate risks due to software components (both open source and proprietary) is SBOMs. George asked me to write a short summary of the risk due to open source components, and how it can be addressed in part by using SBOMs. Here is what I prepared (although I’m sure it will change by the time the paper is published):

 

Risks due to open source software components

The average software product today includes over 135 components[1]; some products include thousands of components. About 90% of those components are open source[2]. In other words, the greatest part of the code in almost any modern software product consists of open source components.

Of course, this also means that the greatest part of the risk in any modern software product is due to its open source components. Open source vulnerabilities like Apache Struts (source of the Equifax data breach), Heartbleed and Log4j have caused massive headaches for IT staff worldwide, as well as led to big financial and other losses.

Software users can sometimes download and patch these vulnerabilities directly (as in the case of Log4j), but more often they have to rely on the supplier to develop a patch themselves. In either case, the key to mitigating the risk is knowing “the software in your software” – i.e. the components included in the software your organization runs. An up-to-date “software bill of materials” (SBOM)[3] from your software supplier contains a list of those components.

Having an SBOM for a software product will enable a user organization to track vulnerabilities in the product’s components (both open source and proprietary), then coordinate with the supplier about mitigating the risk posed by those vulnerabilities. In most cases, the mitigation will simply be applying a patch provided by the supplier, although in some cases other mitigation measures will be required.

There is much to discuss about SBOMs, and at the current time the field is rapidly evolving. As of early 2022, the best information on SBOMs is available at https://www.ntia.doc.gov/SBOM. However, it is certain that much more up-to-date information will be available by the time you read this. As there is currently no central clearing house for SBOM information, you can email tom@tomalrich.com for an up-to-date list of SBOM information sources.

 

I didn’t want to include my email address in this document, but – since it will be close to a year before this document is published, and because the documents on the NTIA website won’t be updated any more, I really didn’t want people to try to figure out what’s going on with SBOMs in 2023 by going to the NTIA site. I’m sure there will be lots of new information in another year, but I can’t point to any site where people will be able to find that information. There’s a business opportunity for somebody!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they necessarily shared by CISA’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[1] https://www.sonatype.com/resources/white-paper-state-of-the-software-supply-chain-2020 

[2] Ibid 

[3] https://www.ntia.doc.gov/files/ntia/publications/sbom_at_a_glance_apr2021.pdf

---

[i] I led the development of two of the other papers, and we’re revising those as well. See this post.

A VEX use case: patching vulnerabilities

Tony Turner of Fortress Information Security suggested to me this week that one thing that has been sorely missing in both “official” and unofficial discussions of VEX is real use cases; he implied that there’s been far too much focus on formats and too little focus on why we need the formats in the first place. That made sense to me, and I decided to explore what I think will be one of the most important use cases for VEX: patching vulnerabilities. As you’ll see below, in the process I came to see that the number of steps that can be taken to describe what might seem to be a simple action can be huge, as can the benefits the supplier can realize from taking those steps.

Here is a scenario, along with steps that can (but don’t have to) be taken when a single serious vulnerability is patched in a single software product:

1.      Vulnerability CVE-2022-12345 has been identified in the NVD as exploitable in Bob’s Browser v4, which is a component of the supplier’s Product A.

2.      The supplier confirms, through their own testing, that this is indeed the case. They also confirm that Bob’s Browser v4 has been a component of Product A in every version, starting with v3.0.0 through the current version 3.5.2 (the supplier follows a best practice called semantic versioning, which requires a version string structure of [Major release].[Minor release].[Patch]).

3.      When they investigate the exploitability of CVE-2022-12345 in Product A, the supplier discovers that the vulnerability was not exploitable until version 3.2.0, due to a code change in version 3.2.0. They also discover that, in all versions from 3.2.0 through 3.5.2, the vulnerability is exploitable.

4.      The supplier develops a patch that fixes CVE-2022-12345 in versions 3.3.1 through 3.5.2 of Product A.

5.      At the same time, the supplier determines that a fix for versions 3.2.0 through 3.3.0 would be very difficult, and they decide to recommend that users of those versions upgrade to the current version as soon as possible.

6.      The new version created by applying the patch to version 3.5.2 is 3.5.3. When a customer patches their instance of 3.5.2, the version number is automatically incremented to 3.5.3.

7.      Similarly, when the patch is applied to any other version “A.B.C”, the version number will change to “A.B.(C+1)”. This is the normal procedure with semantic versioning.

8.      The supplier creates a VEX (in either the currently available CycloneDX VEX format or the upcoming CSAF-based VEX format) that indicates the following:

a)      CVE-2022-12345 is exploitable in Product A versions 3.3.1 through 3.5.2. The new patch should immediately be applied to all of these versions.

b)     CVE-2022-12345 is also exploitable in Product A versions 3.2.0 through 3.3.0. However, due to technical problems, the vulnerability will not be fixed in these versions. Users of these versions need to upgrade to the current version as soon as possible.

c)      CVE-2022-12345 is not exploitable in versions 3.0.0 up to (but not including) 3.2.0.

d)     The supplier has not tested for CVE-2022-12345 in versions before 3.0.0, since all of these versions are now unsupported. The supplier makes no assertion regarding whether CVE-2022-12345 is exploitable in unsupported versions of Product A.

Are you surprised that there are so many steps? After all, VEX has been advertised as a simple yet machine-readable way for a supplier to alert their customers that a vulnerability that is exploitable in a component of their product is not in fact exploitable in the current version of the product. These steps go far beyond that mission.

I agree with this analysis, but remember: The main reason why VEX is being developed is that some large suppliers are concerned that, when they start regularly providing SBOMs to their customers, their support lines will be overwhelmed with calls. This will happen when a serious vulnerability in a component in the current version of a product has been identified and customers want to know if it is exploitable in the product itself. If that were the only concern, a VEX would only need to state – in the above example – that CVE-2022-12345 is exploitable in the current version 3.5.2, and that a patch is available.

But should this be the supplier’s only concern? When customers learn that a vulnerable component has been included in the product through a number of versions, won’t they also call with questions like:

1.      “I’ve heard a lot about the new serious vulnerability CVE-2022-12345. Have you released a patch for it yet?”

2.      “I’m running [a non-current version] of Product A. Is CVE-2022-12345 exploitable in this version? And if it is, is there a patch for it?”

3.      “I work for an integrator, and we have customers running a number of versions of Product A. In which of those is CVE-2022-12345 exploitable, and in which is it not exploitable? Of those versions where it’s exploitable, for which ones is a patch available and what’s the URL to download it? And is the only solution for the other exploitable versions to upgrade to the latest release?”

4.      “I’m running [an unsupported version] of Product A. Will a patch for CVE-2022-12345 be released for my version?”

5.      “I’m running [a supported version] of Product A. I noticed you’re not patching this version, even though it’s under support. What’s the deal?”

All of these questions – and potentially a lot more - are likely to be asked, as a result of the above scenario. Without a VEX, the supplier will have no choice but to hold on and hope there aren’t too many of these calls. With VEX, the supplier will be able to answer some percentage of these questions, without tying up a single support person.

So it’s very likely that releasing a VEX will lead to a significant reduction in support calls for most products, after SBOMs have been made widely available. But here’s another consideration: What if Product A has 4,500 components, as some products do? There might be scenarios like this one every day, or even every hour. Without having recourse to VEXes, will the supplier of Product A even release an SBOM for it? After all, to do so might literally open the gates to a flood of support calls that could threaten the financial stability of the supplier.

This is why I have said repeatedly, “As long as suppliers can’t release VEXes in volume, they won’t release SBOMs in volume.” And that’s why getting VEX right is so important.

Here’s an extra credit question for you, based on an issue that has come up in VEX discussions: If the VEX weren’t able to handle version numbers and especially version ranges, would the above scenario be at all feasible? Of course not. There’s no way that the patch application use case could ever be addressed without having the ability to specify version numbers in the VEX.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they necessarily shared by CISA’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.