Dale Peterson’s interview with Steve Springett

Recently, Dale Peterson interviewed Steve Springett, leader of the OWASP Dependency Track and CycloneDX projects, for his podcast. I have watched a number of podcasts with Steve, and I never come away without a lot of notes; for this podcast, I took six pages’ worth. This was in part because Dale asked some really excellent questions, which showed he had done a lot of research beforehand. Between the two of them, it was quite a show.

I recommend you listen to the whole podcast, but I’d like to address three topics that came up – among many.

Contractual challenges to sharing SBOMs

Steve and Dale both agreed early in the podcast that, while SBOMs are being heavily used by software developers to learn about and manage vulnerabilities in products they’re developing, they’re hardly being distributed to or used by end user organizations at all (i.e., organizations whose primary business isn’t developing software, which is of course well over 99% of the organizations on the planet).

One of the reasons Steve pointed to for this problem (and it is a problem!) was contractual. He didn’t elaborate on that, but I can guess that, since there are no standards or official guidelines for producing or distributing SBOMs, any organization that tries to negotiate with suppliers on the terms under which they’ll provide their SBOMs won’t find this to be easy at all.

However, I’ve already discovered the solution to this problem. It’s a lot like the solution the doctor proposed to his patient who complained that if he performed an unusual arm motion, it hurt. The doctor’s solution? “Don’t do that.”

And that’s my solution to the problem of contracts for SBOMs being hard to negotiate: Don’t bother with them. Given the complete lack of standards or official guidelines for producing or distributing SBOMs, it’s simply way too early to even consider using contract language to “force” the supplier to give you exactly the type of SBOM you want, in exactly the way you want it.

However, let’s say the supplier gives you the perfect SBOM that you want. What are you going to do with it on your own? There are currently no low-cost, commercially supported tools that ingest SBOMs (and VEX documents, although they’re hardly being distributed at all today) and output a list of component risks for the user, including a list of exploitable component vulnerabilities. Yet, this is the use case that most end users have in mind for SBOMs.

This is why I recently suggested that we forget about using contract language for SBOMs for the time being (except for perhaps an NDA) and simply focus on mutual learning: the suppliers will produce the SBOMs they think their users want and their users will tell them whether and how they’re able to use them. This will be effectively a large-scale proof of concept and, as with any proof of concept exercise, there will need to be mechanisms for gathering and aggregating the lessons learned.

I know of only one active SBOM proof of concept that is exchanging SBOMs today - the healthcare PoC being sponsored by the Healthcare ISAC. Moreover, that PoC has just around ten participants (medical device makers and large hospital organizations). Wouldn’t it be great if we could get a lot more suppliers and users, in many industries, exchanging SBOMs and learning from them – and then sharing what they’ve learned from the experience? We can do that if we forget about contract language for now, and simply focus on what we can learn.

The only way SBOMs are going to provide value to end users in the near term

After the above discussion, Dale pointed to the one solution that can lead to SBOMs (or at least the information to be derived from them) being widely used by non-developers in the next few years. The solution won’t be to have low-cost, commercially supported tools that every organization can use. I used to think that those tools would someday magically appear, but they haven’t appeared yet and I know of none that are even on the horizon.

More importantly, there are serious issues like the “naming problem” (which Dale brought up later in the podcast) and how to get useful VEX information out (I plan a post on this problem next week), which are currently standing in the way of easy-to-use consumer tools. These problems won’t ever be completely solved, but they can be addressed to a degree that makes usable consumer tools possible. We’re simply not there yet, and won’t be for years.

However, Dale pointed out that he sees a more realistic path to SBOMs being usable to consumers in the near term, and that’s third party service providers; I agree with him 100% on this. The fact is that the tools required to analyze SBOMs and VEX information, and then report component risks to end users, are available now. However, they’re almost all not low cost or commercially supported or both – and that’s what will be needed for true consumer tools. Moreover, an end user today would need to string together several open source tools (and address data formatting issues, etc.), to get the required functionality. Few businesses or government agencies have the technical chops and available time to do this.

However, third party service providers are a different story. They know that whatever tooling they put in place will allow them to provide this service to many end user organizations. The economic picture will be very different for them.

Exactly who will these service providers be? I honestly don’t know now, but I do know that, given the high and growing interest in learning about software component vulnerabilities, they will appear. The important technical problems have all been solved in principle. Even the naming problem has been “solved” by hundreds or even thousands of software suppliers and their consultants, since there are lots of ad hoc workarounds available, including AI/ML routines, fuzzy logic, etc. No non-software suppliers will want to invest in developing these routines for their own use, but again, the economic picture changes drastically for service providers who can amortize the cost of doing so over hundreds or thousands of customers and products.

There’s one important aspect of these service providers that didn’t come up in the podcast, but is one that I’ve written about once or twice: It’s clear to me (at least) that end users shouldn’t be responsible for paying the service providers for their analysis. After all, the supplier is the one that chose the component and included it in their product. If they chose a component that poses big risks for end users, why should the end users have to find this out for themselves? More importantly, if there are 10,000 customers of a product, why should they each have to pay a service provider to tell them that CVE-2023-12345 is exploitable in the product they use, and they should immediately contact the supplier to find out when they’ll patch it?

Instead, why doesn’t the supplier pay the service provider to do this and distribute the results to their users? I remember when it first became clear in the 1990s that software vulnerabilities weren’t rare events (as had been the general opinion), but could be counted on to appear constantly in just about any product. At first, software suppliers tried to charge their users for the privilege of receiving patches for the vulnerabilities in their product – often through saying that only users who pay for maintenance would receive patches.

Of course, this idea quickly faded, and now I don’t know a single supplier that doesn’t develop and distribute security patches to their customers for free. I expect the same thing to happen with SBOM analysis. While I don’t think most suppliers will want to make the investment in providing the service I described above for their customers, I think they will gladly pay a third party to provide that service on their behalf. Especially when all their competitors start doing it and they realize they won’t be in business much longer unless they do as well. Just like what happened with security patches.

Side-channel attacks

Toward the end of the podcast, Dale brought up the topic of VEX. He had heard Steve say on a different podcast that “VEX is a missed opportunity because it doesn’t represent risk.” He asked what Steve meant by that.

Steve’s first response was fairly complicated (and led to a more complicated discussion than I want to address at the moment), but his second response was very straightforward: VEX considers a single vulnerability in isolation and provides a binary answer to the question of whether that vulnerability is exploitable in a particular product (and not just a particular product, but a particular version of that product).

However, Steve pointed out that hackers often don’t just try to exploit one vulnerability and then give up if they don’t succeed in compromising the product using that vulnerability. Instead, they often try to exploit a different vulnerability. Then, if they’re successful with that attack, they “pivot” to exploiting the vulnerability they had originally aimed for. This is known as a “side channel” or “chained” attack. When you consider these attacks, then VEX’s answer to whether a vulnerability is exploitable in the product or not becomes more complicated – since you need to consider which side channel attacks might be used to exploit that vulnerability, not just direct attacks.

This was a question that was discussed in the VEX working group when it was under NTIA (I don’t think we have discussed it much under CISA). Our feeling at the time was that, if we started to consider side channel attacks, then essentially every vulnerability would be exploitable. Why even have a VEX in that case? Instead of spending their time putting out VEX information, a supplier would just need to admit they need to patch every component vulnerability, even though they know that over 90% of them probably aren’t exploitable.

Moreover, end users will also need to admit that, even though they on average are currently only applying about 15% of the patches they receive for products installed on their network, they’ll now face a huge influx of additional patches, which they will have to add to their already voluminous patching queue. Since we (the VEX working group) didn’t want to provide those answers either to suppliers or end users, we decided to proceed on the assumption that only directly exploitable vulnerabilities need to be designated as exploitable in VEX.

However, enough people (besides Steve) have raised this issue that I now think we need to admit that there are more sophisticated hackers out there who know how to use side channel attacks. So, for some users (especially those in high assurance use cases, like military contractors, hospitals, and OT in general), it does make sense to patch any component vulnerability that is known to be present in the product, and for users to apply those patches.

Fortunately, there are some cases in which even a side channel attack won’t be able to exploit a particular vulnerability in a product, meaning that VEX can provide an acceptable solution, even for high assurance users. I’ll discuss that soon, hopefully next week.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

“British cuisine”, “Device security”, and other oxymorons

 

In the past couple of months, it seems events have been conspiring to tell me two things:

1.      Security in almost all types of intelligent devices is in terrible shape and borders on being nonexistent. However,

2.      Even if you want to take security matters for a device into your own hands (e.g., do your own vulnerability management), you’re going to have a hard time doing that.

The first event was when I wrote this post, which reported that a person involved in product security for a major medical device maker told me he had never reported a vulnerability for one of their devices, and in fact he wouldn’t know how to report one. Since most CVE reports (which form the basis for all listings in the NVD and the great majority of them in other vulnerability databases) are created by the developer of the product reporting the vulnerability (either a software product or an intelligent device), this probably means that none of that company’s products lists any vulnerabilities in the NVD.

In fact, since a CPE name is only created when a vulnerability is reported for a product (more correctly, a particular version of a product), this probably means nothing will be found when someone searches for that product in the NVD. This isn’t good, but it’s made worse by the fact that the message that displays when that happens is “There are 0 matching records.” This is the same message that displays when there is a CPE name for the product, but there aren’t any vulnerabilities reported against the product. In other words, a user that searches on a product and receives that message should either be elated that it’s a “vulnerability-free” product or worried that it might be loaded with vulnerabilities, but the supplier of the product doesn’t report them. How do you tell which is the correct conclusion to draw? Perhaps a coin flip?

My friend defended the fact that his company doesn’t report product vulnerabilities by saying they would only report a vulnerability if they had patched it. This would be comforting, if we were talking about a software product, and the developer would normally quickly release a patch for a serious vulnerability. 

But that's not how devices work. The user can't apply a patch themselves. They have to wait for the supplier to issue an update that includes the patch fix. And since releasing a device update requires a lot of work, device makers usually hold patches until they can be included with the next update. Even that wouldn't be terrible, if the device maker released an update every month (which I had naively assumed was the case). 

But my friend said his company only releases an update for their devices (which are medical devices, remember. They aren’t baby monitors or recipe servers) once a year. This changes everything. Let’s say a device manufacturer discovers a serious vulnerability in one of their devices the day after they released their annual update. Assuming they’re on an annual update schedule, this means the vulnerability won’t be patched in the field for 364 days.

Will the manufacturer let their customers know about this serious vulnerability and the fact that it won’t be fixed in the device they use for another 364 days – so they can at least apply some alternative mitigation to the device, even perhaps pulling it from their network altogether? If they’re like my friend’s company, the answer seems to be no. My friend said they won’t issue a notification for the vulnerability until it’s patched in the field. Where does that leave their customers if they’re attacked using that vulnerability during those 364 days? The answer seems to be "SOL".

I believe that an intelligent devicee manufacturer needs to promptly report vulnerabilities for the third-party software and firmware products included in their device, as well as for the code in the device that they wrote themselves (the first-party software). Moreover, they need to report all these vulnerabilities as applicable to a CPE that identifies the device, not to one of the myriad software or firmware products contained in the device.

The reason for doing this is obvious (to everyone except the device manufacturers, it seems): Just like for a “standalone” software product, the customer of an intelligent device needs to be able to learn about all the vulnerabilities in their device in a single search. Even if they have a current SBOM for the device (a dubious assumption today, to be sure), they shouldn’t have to invest all the time and effort required to look up each of those products in a vulnerability database (assuming they even have a searchable identifier, which means a purl or CPE). The supplier should be tracking all vulnerabilities for third- and first-party software and firmware in their device, and posting them all to the single CPE for the version of the device where those vulnerabilities are found.

What about the fact that a lot of the vulnerabilities that a manufacturer learns about (in a vulnerability database like the NVD) for the individual software and firmware products installed in their device won’t in fact be exploitable in the device itself – so they don’t actually pose a risk to their customers? Should the supplier still report every vulnerability regardless of its exploitability status, and then issue VEX information to let their customer know which of those vulnerabilities aren’t exploitable?

Were there any low cost, easy-to-use and commercially supported tools available for software users, which would use VEX information to narrow down a list of vulnerabilities in a particular product/version to just the exploitable ones, I would say by all means, that’s the way to do it. However, given that this is far from being the case, I’m willing to concede that the manufacturer can apply the exploitability information on their own and just report the small percentage of component vulnerabilities that are exploitable in their device, not the huge number (usually around twenty times the number of exploitable vulnerabilities) that aren’t exploitable. That will at least make the job much more manageable.

However, another thing that device suppliers need to do is provide much more regular updates to their devices. Quarterly updates would be a big improvement over annual ones, so maybe that’s what they should aim for today. But the manufacturers also need to let their customers know about serious new vulnerabilities in their devices and not wait until the next device update to do this. That way, if a customer wants to implement an alternative mitigation during the time remaining before the next update, they’ll be able to do this (the manufacturer should suggest alternative mitigations in their vulnerability report).

At the end of his recent presentation to the SBOM Forum, Tom Pace of NetRise admitted that, given the large number of vulnerabilities that are likely to be found in most intelligent devices (as he pointed out to our group last year, he examined one device – that had no vulnerabilities reported for it in the NVD – that very likely had at least 40,000 unpatched vulnerabilities), it was unrealistic to expect the manufacturer to be able to quickly patch all the vulnerabilities in the device; they will need to figure out a way to triage the vulnerabilities (using perhaps EPSS score or KEV ranking) and patch the most important ones first.

But he did point out that some device manufacturers have an alternative strategy for reducing the number of vulnerabilities they’ve found in their devices, without having to patch them at all. He showed us a message his company had received when they tried to follow up on a previous notification to a manufacturer about a serious vulnerability they’d identified in one of their devices. It read, “Your message to security@(manufacturername).com has been blocked.”

Yep, that’ll do the job, too. And it’s a lot cheaper than patching. 😊

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

A Global Vulnerability Database

 

…consistency is the hobgoblin of little minds.

- Emerson

Whenever you identify problems in an existing organization, program or device, you always face the question: Will it be easier to fix what is currently in place, rather than have to start over from scratch? Or will it be easier – or at least better – to start from scratch and build something of your own design, which hopefully avoids all the problems with the existing system?

Early in 2022, I got together a group of people who were committed to making SBOMs successful, but were concerned that this will never happen unless certain obstacles are removed from their path; we call ourselves the SBOM Forum. I knew there were a number of problems to address (including maybe 4 or 5 “showstoppers”), but I wasn’t terribly concerned about which one we started with – as long as we started somewhere.

However, there was one thing I was sure of: I didn’t want us to start with the “naming problem” – that is, the problem that a single software product can have many different names in different contexts, and these names can correspond to subtle (or not-so-subtle) differences in the product (small code variations, source code vs. compiled binaries, etc.). It seemed to me that, if we started with the naming problem, we’d work on it for five years and still be no closer to solving it than we were when we started.

But I didn’t want to constrain the conversations we had in our weekly meetings, so I let them go in whatever direction people wanted to take them. Guess what topic we ended up focusing on within a couple of weeks? You’re right…the naming problem. So much for planning, eh?

However, I was surprised by how much progress we made when we started discussing the naming problem. There are many aspects to the problem, but the one that’s probably on the minds of most people in the software security community is the National Vulnerability Database (NVD), which is by far the most heavily used vulnerability database worldwide. The NVD’s biggest problem is the software identifier it uses: CPE.

As we started discussing these problems, I was amazed that a) the problems became easy to identify and describe (I had thought it would take a year just to describe the problem we were addressing), and b) the solutions became apparent very quickly, at least at a high level (I’ll admit we had deployed a secret weapon that greatly speeded up identification of the solutions: Steve Springett, leader of the CycloneDX and Dependency Track projects and someone who – from what his family tells me – never goes to bed unless he’s solved at least one software security problem that day).

The result was that, five months after we started discussing the subject, we published a paper that has received a good amount of attention. It described what we saw as the major problems with CPE, as well as the broad outlines of a solution to the problem. Early this year, we started talking with Tanya Brewer, leader of NIST’s NVD team, about implementing our solution. She said they had read our paper with interest and would like to collaborate with us (as well as a few other organizations that have reached out and offered to help them).

These three posts - first, second, and third – described our conversations with Tanya. In the third post, I relayed the news that NIST would soon come out with a video asking for suggestions on a public-private partnership to fix the problems (as well as accomplish other goals, to be sure), and they would probably post a Federal Register notice about their plans in late August. Tanya thought the PPP could be up and running by early 2024.

How is that coming? Well, it’s now about six weeks after the date Tanya said she’d have the video out and…there’s been no word at all. This means the whole schedule is probably pushed back two months already…and counting.

What happened? I know they were having a serious problem that might have required “all hands on deck” (the last time we talked to her, her head count was down about 20%, since she hadn’t even begun to receive her 2023 funds – this was in late June. Of course, given that the US almost defaulted on its debt a few weeks earlier, I guess this isn’t a huge problem in the scheme of things), although the notification about that serious problem has been removed from their website, as well as the admonition not to ask them when it would be fixed. I also know that the video was first delayed by a couple of weeks because somebody who had to sign off on it was on vacation (does NASA work the same way? “Sorry, we’ll have to postpone the moon launch for ten months since Mr. Smith has to have surgery, and nobody else has authority to sign for him”).

Of course, delays like this are par for the course in the government game, and we haven’t given up on the NVD; we’ll pick up the phone whenever they want to talk. However, we’ve now decided that it’s time to work on Plan B, which is a Global Vulnerability Database.

Why have we moved from Plan A to Plan B? Because B has always been our goal. The whole world is becoming more and more concerned about software vulnerabilities.  We’re sure there will be a lot of interest in Europe, the UK and Japan (and perhaps other countries like Singapore), both in the public and private sectors, in having a single, robust global vulnerability database. In fact, we’ll have a first meeting with ENISA next week.

Given the upcoming (although still not finally approved) EU Cyber Resilience Act, as well as the rapid growth of SBOM use among software developers (plus the fact that SBOM use by organizations whose primary business isn’t software development has – to put it gently – nowhere to go but up. But it will go up, mark my words), it frankly makes no sense not to start mapping out a global vulnerability database.

The paper we put out last fall is no longer our wish list for the NVD (although we’d be glad to see it implemented at the NVD, bien sur), but rather the start of a blueprint for the GVD. Of course, we’re not going to walk away from the wealth of vulnerability information already in the NVD, namely the existing vulnerability listings. It’s easy to obtain; some organizations download the entire NVD multiple times a day, with time off for a leisurely lunch. And we’re not going to walk away from the CVE.org database, which is where the NVD’s vulnerability information ultimately comes from (in fact, one of the board members of CVE.org is a very active member of the SBOM Forum).

Fortunately, putting together this database doesn’t require solving any problem that hasn’t been solved hundreds or thousands of times already. And frankly, I think a lot of organizations worldwide feel a little guilty about never contributing to the NVD – which has never even asked anyone for money, as far as I know. I think the money to get the GVD up and running will be fairly easy to obtain.

Who will run the GVD long term? It could be a government body (although I’m not interested in trying to shake Washington for money, especially since there’s a good chance the government will be shut down through at least some of the fall. After all, shutting the US government down is becoming almost a pre-holiday tradition, like getting a Christmas tree) or even a private industry group. However, maybe the best long-term solution is IANA, which already assigns IP addresses and runs DNS. There are lots of options.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Today’s the anniversary!

August 14 is the 20-year anniversary of the 2003 Northeast blackout. This event had a profound impact on the world we live in today in many different ways; one of them was mandatory regulation of the power industry. Here is my summary of what happened:

·        The Wikipedia article states, “The blackout's proximate cause was a software bug in the alarm system at the control room of FirstEnergy.” Frankly, this is a stupid statement. It’s kind of like saying that the cause of World War II was the German invasion of Poland. There’s a big difference between a triggering event and a cause. This bug was a triggering event.

·        The causes were multiple: An important one was the fact that compliance with the NERC reliability standards at the time was voluntary. First Energy, as well as probably other utilities, had violated the NERC requirements for trimming trees under high voltage transmission lines. Since high levels of load cause lines to sag and the load levels were high on that hot day, the lines sagged into treetops, which caused them to be tripped by their protective relays. When one line shorted out, its load was automatically distributed to other lines, which themselves began to sag, encountered trees, and tripped. And so on.

·        Finally, when the last major line into the area tripped, northern Ohio became a virtual black hole for electric power, trying to suck in as much power as possible from all its neighbors. In turn, they tried to get all the power they could from their neighbors – and voila!, the cascading outage began. Within six minutes after the last line failed, the event was over. Much of the Northeastern US (excluding New England), Detroit and eastern Michigan and most of Ontario up to Hudson Bay had blacked out. 508 generating units at 265 power plants shut down during those six minutes.

·        But lack of tree trimming wasn’t the only cause. What made the blackout much worse than it had to be was that the protective relays protecting many generating units from grid instability were (in retrospect) set to trigger at a much lower level of instability than necessary. NERC’s monumental six-volume study of the blackout (which I skimmed through once, not that I could have understood most of it anyway) pointed out that (again in retrospect) many of those units would not have had to shut down if the settings had been more forgiving.

·        The resulting total blackout required activation of “blackstart plans” in many areas. Since most generating units require some amount of external power to start up, the blackstart plans lay out a complicated path for utilizing the small amount of power still available (e.g. power from hydroelectric plants or power from generating units that have diesel-powered emergency generators available) to one-by-one energize lines, then more generating units, then more lines, etc. – until the generation in the area covered by the blackstart plan is fully operational.

·        However, all of this could have been avoided if there had been better visibility into what was going on in Ohio. The software bug in the First Energy control center caused the alarms – which would otherwise have been flashing deep red – to be suppressed. The operators in the control center saw nothing but green screens and were happy as clams…until they weren’t. But even that might not have mattered, had a technician at what was called at the time the Midwest Independent Transmission System Operator (now the Midcontinent ISO) not left for lunch after fixing a problem with the “state estimator” system.

·        That system would have detected the growing divergence between electricity load and supply in northern Ohio (exacerbated by a couple of major plant outages in Cleveland) had it been active. However, the technician forgot to turn the system back on when he left. When it finally was turned on, it immediately became clear that northern Ohio was going to hell in a handbasket very quickly.

·        On the other hand, it’s not clear that the human beings monitoring these systems would have reacted correctly if they knew how bad things were. To end the imbalance between load and supply, they probably would have had to black out the entire city of Cleveland and the surrounding area. Would they have been authorized to do this and if not, would they have been able to get that authorization quickly enough to do any good, given how rapidly the imbalance between load and supply was increasing? Of course, Cleveland and many other cities ended up being blacked out, so it would clearly have been better if they system operators had taken this step.

Of course, all these causes (and others besides them) were correctable and since have been addressed with policies, standards, etc. But that required something else: mandatory standards for utilities to follow. That’s what Section 215 of the Electric Power Act (EPAct) of 2005 provided. It required the Federal Energy Regulatory Commission (FERC) to enforce mandatory reliability standards for the electric power industry.

To facilitate this, Section 215 ordered FERC to engage an “electric reliability organization” (ERO) to draft mandatory reliability standards as well as audit compliance with them, all under FERC’s oversight. Of course, there wasn’t much question that NERC was the only organization that could possibly be the ERO, and FERC chose them.

Section 215 also ordered that mandatory cybersecurity standards be developed for the power sector. At that time, the main NERC cybersecurity effort was a set of voluntary requirements called Urgent Action 1200, which applied just to large control centers. NERC was working on an update called Urgent Action 1300 but in 2016, NERC rolled that effort into development of version 1 of the CIP standards.

So the CIP standards, which were (I believe) the first cybersecurity standards that applied to Operational Technology (OT), outside of perhaps military standards, were very much a consequence of the Northeast blackout. While I have certainly expressed my disagreement with many aspects of CIP over the years (you might go back and read the at least 400 posts I’ve written on CIP, starting with my first post on this blog in early 2013), and while I continue to believe that compliance with the CIP standards is much more expensive than it should be, there’s no question in my mind that the standards have done a great job of securing the North American power grid against cyberattacks.

Moreover, I think all the NERC reliability standards (which cover all aspects of operating the power grid, including – thanks to the blackout – relay settings) have proven very important, not just the CIP standards. While local outages happen all the time (with squirrels being one of the main causes), there has not been a cascading outage since 2003, or even just a large outage that wasn’t caused by a natural disaster like a hurricane. The dedicated people of NERC (and the six NERC Regional Entities that explain and audit the standards) deserve all our thanks.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Are device makers reporting vulnerabilities? Not usually. Should they be? Of course!

One of my biggest eye-opening experiences of 2022 was when Tom Pace of NetRise described to the SBOM Forum (a group I lead) a maker of networking devices used by the military and critical infrastructure, which has never reported a single vulnerability for any of their devices – so naturally, a search on one of their devices in the NVD will leave many users with the impression that their devices are all vulnerability-free!

Unfortunately, the reality is different: Just a single device that Tom’s company examined likely had at least 40,000 unpatched vulnerabilities in components included in the device’s firmware. Of course, I was full of righteous indignation at this supplier (whose website doesn’t even mention “security” or “vulnerability”). How dare they shirk their obligation to report vulnerabilities to CVE.org (which is where the NVD, as well as any other database that uses CVEs, learns about its vulnerabilities)? After all, I’m sure (or was sure at the time) that all major intelligent device manufacturers (and software developers) diligently report all their vulnerabilities to CVE.org.  

Since a lot of people who work with vulnerabilities all the time don’t understand how they get into a vulnerability database (and I didn’t, until about a year ago), I’ll give a brief summary:

1.      CVE.org is part of DHS and is funded by CISA. It is staffed by the MITRE Corporation and used to be known just as “MITRE”. However, it is now run by an independent board which includes representatives of private organizations, CISA, NIST (which runs the NVD), and MITRE.

2.      New vulnerabilities are usually identified by the developer of a software product in their own software. If the developer is what’s known as a CNA (see the CVE.org website for a discussion of CNAs), they will be able to write up the CVE report and submit it to CVE.org. If they aren’t a CNA – and only about 200 organizations are CNAs; becoming a CNA requires training – they can go to a CNA for help if the CNA’s “scope” includes them. For example, Red Hat (a longstanding CNA) will help any open source project (I believe); GitHub will help any GitHub project; there are country- and industry-specific CNAs; etc.

3.      If a developer that wants to report a new vulnerability doesn’t think that their organization can be served by any of the CNAs, they can always approach MITRE directly (through CVE.org, of course), since they are the “CNA of last resort”. The CNA will help the developer (or device manufacturer) create the CVE report. The report will include information on the affected product, but not a CPE name; that name will be assigned by the NIST people later.

4.      The report is submitted to CVE.org and if approved, it will be passed to the NVD, where it automatically becomes part of the database. Around 20,000 new CVEs were identified in 2022.

5.      When the CVE report reaches the NVD, someone on the NIST team that runs the NVD will assign a CPE name to the product. Unfortunately, they don’t always do it correctly and the product becomes hard (if not impossible) to find in the NVD; the developer is left to apologize to their customers for NIST’s mistake. One of the changes the SBOM Forum has discussed with NIST is allowing the developer to assign the CPE name themselves.

Currently, there won’t be a CPE name for any product for which a CVE report hasn’t been submitted; that is, NIST currently only creates a CPE name when a vulnerability is reported for the product. This in itself wouldn’t be terrible were it not for the fact that, when a user creates a CPE name (following the specification) for a product they want to learn about and enters it on the search bar, they will receive the same message - “There are 0 matching records” - if no CPE exists with that name (which will be the case if the manufacturer has never reported a vulnerability for it, or if the NIST person made a mistake when they created it) as they will if the name is valid, but there are no vulnerabilities reported for it.

In other words, the user will have no way of knowing whether the product really has no vulnerabilities (and has a correct CPE name) or whether it has 40,000 vulnerabilities, but was entered under an incorrect CPE name.

(now, back to our regular programming) When I heard Tom Pace’s presentation last year, I initially thought that intelligent device manufacturers – at least the larger ones (including manufacturers of IoT, IIoT, medical devices, networking devices, etc.) - generally do a decent job reporting vulnerabilities in their devices, since I believe that’s the case for software developers. The main evidence that software developers are doing a good job is the number of software vulnerabilities that can be found in the NVD. Aren’t device manufacturers doing the same thing?

Some are. For example, when I just searched the NVD for “Cisco ASA” (Cisco’s firewall appliance), I received 581 CPE names for different models, different versions, etc. This means that Cisco™ has reported at least one vulnerability against each of these CPE names. Of course, the CPE name refers to “adaptive security appliance software”, not the device itself.

And this is how I think vulnerabilities should be reported for devices. They should be reported for the device’s software (and firmware) as a whole, not according to the individual software or firmware products installed in the device (since most devices include at least some third-party software, not just software developed by the manufacturer itself). If instead, vulnerabilities are only reported for the individual third-party products, then the only way someone will be able to learn about vulnerabilities in a particular device is to a) be a customer, b) receive a current SBOM that lists all the software in the device (and I know of no device manufacturer that is regularly releasing SBOMs to its customers. BTW, the new FDA requirement is unlikely to change that), and c) go through the work of looking up each of the third party products in the NVD – based on the infinitesimal-to-faint hope that the supplier of each of those third party products is diligently reporting its vulnerabilities.

This week, I was quite happy when I got a chance to have a one-on-one conversation with someone I’ve known for three years, who is involved with product security for medical devices made by one of the largest medical device makers (known as “MDMs” in the industry). One of the many questions I asked him was how often they report a vulnerability for one of their devices.

I wasn’t expecting him to say they report vulnerabilities all the time, but I didn’t expect the answer I received: that they have never reported a vulnerability for any of their devices, and he isn’t even sure how to report one.

He justified that answer by saying – and I believe him – that they have never found a vulnerability that they didn’t patch, so their products haven’t had vulnerabilities. But here he was simply wrong. Few if any software developers or device manufacturers would ever report a vulnerability unless they had a patch for it. The normal procedure is that the developer discovers the vulnerability, develops the patch, then submits a CVE report (and usually a report to their customers) that lists the vulnerability but also provides information that allows the user to find the patch and apply it. If the MDM doesn’t submit anything to CVE.org, nobody will be able to learn of the vulnerability.

If this were a standalone software product, not a device, there would be no question that the developer needs to report the vulnerability (as well as the patch), both to CVE.org and to their customers, since the customers need to apply the patch ASAP. However, usually there is no way for the customer to patch a device they utilize; instead, the supplier needs to do that.

Of course, if suppliers patched vulnerabilities as soon as they appeared in the device (or at least the serious vulnerabilities), there wouldn’t be any problem. I realized that this idea is unrealistic, since device manufacturers can’t be expected to push out each patch the day they create it. I assumed they push out monthly updates – which is probably frequent enough to catch all but the most severely exploited vulnerabilities before the bad guys can get into the device.

However, I asked my friend how often they update their devices, and was amazed to hear it was usually once a year. In other words, even if the manufacturer develops a patch for a vulnerability the minute after they learn of it, their users will never benefit from that patch for an average of six months. Even worse, since the manufacturer doesn’t report the vulnerability to the NVD (and presumably not to their customers either), I’m betting that the customers wouldn’t be pleased as punch to learn there was an important vulnerability in the device they rely on, but it won’t be fixed for six months (and remember: the devices my friend secures are used for sometimes life-sustaining patient care. They’re not baby monitors). In addition, no organization considering buying the product will be able to learn about this unpatched vulnerability before they buy it.

I’ll stop here and let you contemplate this situation. By the way, do you or a loved one have any surgery scheduled? Just wondering.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Playing pro ball vs. keeping score at home

As I made clear in this recent post, I no longer think it makes any sense for the end user of a software product or intelligent device to bear the responsibility for analyzing SBOM and VEX information to identify exploitable component vulnerabilities in a product they use. Instead, the supplier should bear this responsibility for five main reasons:

1.      The supplier chose to include the components in the product. They did this because utilizing pre-built components is now the only realistic way to create software, due to the huge savings of money and developer time that result from this practice. While some of the economic benefits of using components accrue to the end users of software because of the overall lower level of software prices that results from this widespread practice, it is hard to deny (at least for me) that the greater part of the benefits of using components accrues to the supplier. This is due to the savings that developers realize through accelerated development time and lower overall development costs.

2.      Currently, there are no tools available to software end users (which I define as consumers of software that are not utilizing the software they acquire to create downstream products, but instead utilize it to fulfill the mission of their organization) that are a) free or low-cost, b) commercially supported, c) easy to use, and d) complete[i]. By a “complete” tool, I mean one that:

                           i.          Ingests an SBOM for a particular software product and version, for example version 1.5 of product ABC;

                          ii.          Resolves any component naming issues, meaning it identifies a valid CPE or purl identifier for every component listed in the SBOM. Learning just the common name for a component is close to useless for a user concerned about component vulnerabilities in the software they use. Identifying a valid CPE for a component requires manual searching, fuzzy logic, dumb luck, etc. Identifying a purl for an open source component should be straightforward if the repository or package manager is known, along with the product name and version string registered in that repository. However, these should always be verified by the product supplier before including them in an SBOM);

                         iii.          Looks up each component CPE or purl in a vulnerability database like the NVD or OSS Index and includes every identified component vulnerability in a list of vulnerabilities found in ABC v1.5. Each vulnerability on the list is initially assigned an exploitability status of “under investigation”;

                         iv.          Repeats these lookups at least daily, adding any newly identified component vulnerabilities to the list with the “under investigation” status;  

                          v.          Receives VEX information from a supplier, either in the form of machine-readable documents or using an API that retrieves the information from a VEX server. VEX information consists of a) exploitability status for every vulnerability in the list for ABC v1.5 (“affected”, “not affected”, “fixed” or “under investigation”), b) a status justification for every “not affected” status designation, and c) mitigation information (e.g. patch or upgrade information) for every “affected” status designation;

                         vi.          Continually uses new VEX information to update the vulnerability list for ABC v1.5;

                       vii.          On demand from the user of the tool, provides a list of exploitable component vulnerabilities (i.e., those with “affected” status) in ABC v1.5, as well as mitigation information for each one (which is also retrieved from the VEX server)[ii]. The list can be provided in either a generic format or in a format required by a particular vulnerability or configuration management tool; and

                      viii.          Continues this process for ABC v1.5 until the user upgrades or removes the product from their network. If the user upgrades to ABC v1.6, the process will start over with step 1 above. If the user still has any instances of ABC v1.5 running, the tool will maintain the component vulnerability lists for both v1.5 and v1.6. Of course, the tool will be able to maintain vulnerability lists for many different products/versions from many suppliers.

3.      Despite the SBOM Forum’s recent incremental progress in getting NIST (which runs the NVD) to recognize the importance of addressing the NVD’s problems with CPEs, the naming problem is hardly solved. This means that, unless suppliers take it upon themselves to do the work required to identify a CPE or purl for as many components as possible, SBOMs (especially those created through an automated process) will continue to be mostly useless for the purpose of vulnerability management.

4.     VEX information is essential for the component vulnerability management use case, yet suppliers aren’t producing VEX documents for their customers. I know of only one supplier, Red Hat™, that makes documents called VEX available to customers now. However, much though I admire the great work Red Hat does in the area of vulnerability management and reporting (and the work that Pete Allor, Red Hat’s Director of Product Security, does as a member of the CVE.org board), because Red Hat has such a unique business model, the type of VEX documents they now produce would be of limited usefulness to the customers of any other supplier.[iii] Note that, if the supplier were doing the SBOM analysis themselves, they wouldn’t need to produce VEXes, since the exploitability information could just be passed – in any desired format, or even just as a list in an email - from the department that produces it to the department that uses it.

5.     Most importantly, it makes no sense to require that every customer, in order to learn about exploitable component vulnerabilities, perform exactly the same analysis on their own that the supplier should already be performing for themselves. Isn’t the supplier the one that needs to know first about exploitable component vulnerabilities – so they can patch them as soon as possible? Given that they have this information (or should have it), why can’t they just share it with their customers, rather than requiring each one of them to derive exactly the same information on their own?

Fortunately, I think that software users will start requiring that their suppliers perform this analysis and make the results available to them for free – just like they do for patches (suppliers initially resisted the idea of distributing security patches for free as well. That resistance quickly faded when they realized they would end up with exactly zero customers if they didn’t do it).

The suppliers will provide the analysis, although, unlike with patching, this work can be accomplished much more efficiently by third party service providers. Those providers will utilize tooling they have developed for their own use (and not for end users, which means it doesn’t have to be made foolproof or user-friendly. Given some big issues that need to be addressed, such as the naming problem, this is currently the only way to develop such tooling. My guess is it will be years before there will be any true consumer tools available for tracking software component vulnerability status). They will make the results (lists of exploitable vulnerabilities) available to the supplier’s customers for free, while charging the suppliers for their services.

Will the end users need to perform any component vulnerability analysis on their own? They will need to in cases where:

a.      The supplier simply refuses to pay a third party to perform the analysis, but cannot or will not perform it themselves; or

b.      The organization doesn’t trust the third party service provider to perform the analysis properly and wishes to do it themselves; or

c.      Someone at the user organization simply enjoys performing the analysis on their own (maybe utilizing open source tools they’ve chained together).

Thus, the “professionals” in the software component vulnerability analysis world will be the third parties that specialize in performing these services for suppliers. The organizations that perform the services on their own will be “keeping score at home” (not that there’s anything wrong with that, of course).

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] Steve Springett, the leader of the Dependency-Track project (as well as the CycloneDX project), showed me this week that D-T has all the capabilities required to be a complete tool, but they aren’t explicitly tied together to do that. I will work with him to explain the use case I have in mind, so D-T (which is already used 10 million-plus times per day to look up vulnerabilities for components listed in an SBOM) can truly be the first complete SBOM/VEX analysis tool. 

Steve and the CycloneDX project team are also working on a format-agnostic BOM exchange API called Project Koala. I believe (and hope) that this API will also ingest VEX information retrieved from a VEX server, since I no longer believe that VEX information will ever be shared in any quantity through machine-readable VEX documents, at least for software component vulnerability management purposes.

 

[ii] Of course, the tool will track the status of every component vulnerability listed for the product, not just those with “affected” status. The user will be able to obtain the complete list at any time, including component vulnerabilities with “not affected” status.

 

[iii] They are only producing one VEX document a year for a product, usually around July 1. However, to be truly useful, VEX information has to be updated at least daily, which is why I am pushing the idea of the VEX server.