Reporting vulnerabilities for intelligent devices

Note from Tom: This post is a slightly rewritten version of a chapter in my new book, "Introduction to SBOM and VEX".

It is a fact that almost all the documents that have been written about software bills of materials only discuss SBOMs for software, not SBOMs for devices. This is in spite of the fact that the need to have SBOMs for medical devices was one of the main drivers of the NTIA and CISA SBOM initiatives. However, even though the SBOMs produced by intelligent device manufacturers do not differ greatly from those produced by suppliers of user-managed software (i.e., almost everything we normally refer to as “software”), the vulnerability reporting responsibilities, and especially how they are implemented, differ significantly.

Please note that I use “intelligent device” to mean any hardware product that operates under the control of software and/or firmware. As far as this discussion goes, the device could be a baby monitor or an electronic relay controlling high voltage power lines; the characteristics and uses of the device make no difference to this discussion.

It is important to keep in mind the reason why SBOMs are needed by non-developers, whether for devices or for user-managed software: the user organization (or a service provider acting on their behalf) needs to be able to identify exploitable vulnerabilities in third-party components included in the software or device that, if not patched or otherwise remediated, pose a risk to the organization. What is needed for the user organization to achieve this goal, and how can the device manufacturer help their customers achieve it? There are four important types of actions the manufacturer can and should take to help their customers.

Reporting vulnerabilities

Vulnerability management, for both intelligent devices and software, is only possible if there is a way for the user to learn about vulnerabilities found in the device or software product. Of course, vulnerability databases are repositories for this information. But how does the information get into the vulnerability database? In most cases, information about vulnerabilities and what they apply to gets reported to CVE.org, an agency within the US Department of Homeland Security (DHS) that maintains the CVE database. This database was previously known as “MITRE”, since it is operated by the MITRE Corporation. MITRE contractors still maintain the database, but they now operate under the management of the board of the nonprofit CVE.org. This board is comprised of representatives of government agencies like NIST and CISA, as well as MITRE and other private-sector organizations.

CVE information is entered into the CVE.org database by means of “CVE reports”, which describe a vulnerability and the product or products it applies to. The reports are prepared by “CVE Numbering Authorities” (CNAs) – which are usually software suppliers authorized by CVE.org. The CNAs prepare reports both on their own behalf (i.e., describing vulnerabilities they have found in their own products) and on behalf of other suppliers that are not CNAs; as of early 2024, there are over 300 CNAs. Each CNA has a “scope” that indicates the types of suppliers it will assist in preparing CVE reports. A scope can comprise a country, an industry, etc.

The important thing to keep in mind is that information on a vulnerability (including the product or products affected by it) almost always gets entered in the CVE.org database by the supplier of the product (often, acting through a CNA). Obviously, this system relies heavily on the good faith of software suppliers. Fortunately, most larger software suppliers take this obligation very seriously, although the record is spotty for the smaller software suppliers.

Are device manufacturers also doing a good job of vulnerability reporting? Unfortunately, they could do a lot better. In fact, many major intelligent device manufacturers seem to have never reported a single vulnerability to CVE.org. This can be verified by entering the name of the manufacturer in the “CPE Search” bar in the National Vulnerability Database. If the search comes up empty (and the user has also tried a few possible variations on the manufacturer’s name), this means the manufacturer has never reported a vulnerability for any of their products (it is possible to say this because a CPE name for a product is only created - by NIST - when a CVE report is filed that lists the product as vulnerable to the CVE in the report).

I recently checked this assertion by searching the NVD for the top ten medical device manufacturers (whose devices are sold in the US). The results were, frankly, appalling:

1.      Four of the manufacturers were not listed in the NVD at all, meaning they had never reported a vulnerability for any of their devices (one of those four is the employer of a friend of mine, who is a product security manager at that manufacturer. He told me last year that they had never reported a vulnerability for any of their devices; the search confirmed his statement. This is one of the top medical device makers worldwide).

2.      Four of the manufacturers had only a small number of vulnerabilities listed for their devices (one of these manufacturers had only reported two vulnerabilities across all their devices).

3.      The remaining two manufacturers are part of very large, diversified companies that have reported many vulnerabilities. Because it would require a huge effort to determine which if any of those vulnerabilities apply to medical devices, I can’t make any statement about them.

What is most disturbing about these results is that medical devices have the most stringent cybersecurity regulations of almost any other devices or software (the regulations are by the US Food and Drug Administration or FDA. Most other countries regulate medical devices as well). Yet, for all practical purposes, it is close to impossible for hospitals (where most medical devices are installed, of course) to learn about vulnerabilities in the devices they use, without investing large amounts of time to do that (see below).

One defense I have heard from device manufacturers, when asked why they are not reporting vulnerabilities for their devices, is to point out (with a straight face, no less) that devices do not have vulnerabilities; rather, the software and firmware products installed in the device have vulnerabilities. Without the software and firmware, the device is simply an empty box made of sheet metal or plastic. It is up to the developers of the software and firmware products installed in the device to report vulnerabilities in their products; it isn’t the device maker’s responsibility.

There are three problems with this argument. First, to learn about vulnerabilities this way, users of the device would need to know all the software and firmware products installed in the device; that is, they would always need an up to date SBOM. Yet, it is doubtful that many device manufacturers are providing a new SBOM to their customers whenever the software in the device is updated (one exception to this statement may be Cisco™).

Second, even if users always had an up to date, detailed SBOM, tracking each software or firmware component in the device would be a big job. Some devices have hundreds or even thousands of software and firmware components installed in them. For the device customer to track vulnerabilities in all those components, the supplier of each component would need to report vulnerabilities regularly to CVE.org; that is not likely to be the case.

Third, why should every user of the device be responsible for tracking all vulnerabilities in the device, when the manufacturer should be (and hopefully is) doing that as well? After all, the manufacturer will never know about vulnerabilities in their device unless they do this. If there are 10,000 customers of a device, does it make sense that each of those customers would need to be constantly checking for vulnerabilities in every component, when the manufacturer could simply share the results of the analysis they are already doing?

Clearly, it makes no sense for device manufacturers to require their customers to bear the burden of learning about vulnerabilities for all the software components in the device. The manufacturer needs to monitor those vulnerabilities themselves (which they should be doing anyway), and report each of them to CVE.org, using the CPE name of the device (Cisco does this today for their networking devices). That way, their customers will be able (ideally) to learn about all the vulnerabilities that apply to any of the software or firmware components installed in the device with a single lookup in the NVD.

More specifically, the supplier needs to report all current vulnerabilities in any of the software components in the device to the CPE name for the current version of the device. Because device manufacturers usually update the software in their devices (including application of security patches) in a single periodic update package, each update should be treated just like a new version of a software product. That is, each update should have its own version number, SBOM and VEX documents, and CPE name.

Patching vulnerabilities

Like software suppliers, an intelligent device manufacturer should never report a vulnerability to CVE.org unless they have already made a patch for the vulnerability available to their customers. Is it possible this is the reason why device manufacturers are for the most part never reporting vulnerabilities in their devices – that they never develop patches for vulnerabilities?

It is not true that device makers are not patching vulnerabilities! A device maker’s regular updates to their devices include both functionality updates and security patches. Then, why wouldn’t the device maker also report each vulnerability that has been patched? My friend who works for a large medical device maker told me in 2023 that the reason why they don’t report vulnerabilities is they would never report them before they are patched – but once they have been patched, they are no longer vulnerabilities.

This might make sense, until one considers that software suppliers are often doing a good job of reporting vulnerabilities, yet the fact that a patched vulnerability is no longer a vulnerability is just as true for software as it is for an intelligent device. Why are software suppliers reporting vulnerabilities, while the device makers are mostly not reporting them?

I believe this anomaly is due to how patching happens in software vs. in a device. Usually, when a software supplier develops a security patch for one of their products, they notify their customers of it right away, at the same time as they notify them of the vulnerability (they will presumably also notify CVE.org of the vulnerability at the same time and provide the information about the patch or other mitigation).

The software customer then needs to make the decision to apply the patch. Sometimes, they may not apply it right away, due to some concern about impacting performance (for example, operators of power plants often hold all patches until they are able to schedule an outage for the plant. They are concerned that an unexpected “side effect" of applying the patch might cause a serious problem if the plant were running when it is applied – e.g., their $100,000,000 combustion turbine might start to vibrate uncontrollably and tear itself apart). But, since the software customer does have the option not to apply the patch, it is even more important for the supplier to provide them the information on the vulnerability, so they can decide whether the vulnerability poses enough of a risk that the need to apply it outweighs other concerns.

However, device makers often give their customers little or no choice as to whether or not to apply a patch: since the patch comes as part of an update that fixes other vulnerabilities and at the same time increases the functionality of the device, the customer will often not want to block the update from being applied; in fact, for many household devices like baby monitors and smart thermostats, the manufacturer remotely installs the update without even notifying the user that this is happening.

So, an important difference between a patch for a software product and a patch for an intelligent device is that application of the patch is almost completely under the customer's control in the case of a software product. However, that is much less likely to be true in the case of an intelligent device. For the moment, let’s assume the device customer has no control at all over whether the patch is applied – that all devices are like baby monitors, where each update is installed without any pre-notification of the customer.

If we make this assumption, then it might make sense for the device maker never to report a vulnerability to CVE.org, or even to their customers. After all, since the update has just been applied to all customer devices without exception, the vulnerability has ceased to exist anywhere in their customer base. There is therefore no reason to report the vulnerability now.

But there are two problems with this argument. The first is that, since device makers often give customers the option not to apply an update immediately – and this is especially true with more critical devices like medical or industrial devices – many of them will not apply it right away; moreover, they may decide to hold off indefinitely. Plus, if the update was pushed out, but the customer’s device was offline at the time, it also will not have been applied.

If the device maker does not notify both their customers and CVE.org of the vulnerability once the update has been applied, all the devices that didn’t receive the update will be vulnerable. Customers need to know about the vulnerability, so they can decide whether to apply some alternative mitigation – like removing the device from their network. Only the customer can decide whether an alternative mitigation is needed, but they won’t even be able to make that decision if they don’t even know about the vulnerability. Just as bad, new customers may buy the device without knowing about the vulnerability, because an NVD search on the device will not find it (and as stated earlier, it is likely that the search won’t find any vulnerabilities at all, assuming the manufacturer doesn’t report vulnerabilities for the device).

This situation is made more poignant by another fact that I learned in my discussion with the large medical device manufacturer in 2023: that manufacturer only performs a full device update once a year. This means that in some cases, a serious vulnerability might go unpatched in customer devices for hundreds of days, without the customers even being informed that the vulnerability is present in the device they use. Again, if customers knew of the vulnerability, they might apply alternative mitigation like removing the device from their network.

Contrast this with what happens when a software supplier learns of a serious vulnerability in one of their products. In most cases, the supplier will not immediately inform their customers of the vulnerability. However, they will start to work on a fix for the vulnerability as soon as they learn of it. As soon as the fix is available, they will post it for customers to download (or in some cases, the supplier will advise customers to upgrade to a patched version of the software); at the same time, they will report the vulnerability in their software to both their customers and to CVE.org.

My opinion is that device manufacturers should move to a model like that of the software developers. Here are some suggested best practices:

1.      Upon learning of a serious vulnerability in one of their devices, they should immediately develop a patch. As soon as the patch is ready, they should make it available to their customers, along with reporting the vulnerability to their customers and to CVE.org.

2.      If a manufacturer does not do full device updates at least every quarter, they should schedule regular “security updates” at least once a quarter, in which all outstanding security patches are applied. They then need to decide with their customers on criteria for when a patch needs to be made available immediately (as in the case of log4shell), vs. waiting for the next security update.

3.      If a new vulnerability does not meet the criteria for making the patch available immediately, the manufacturer should not report the vulnerability to CVE.org or to the customers of the affected device. Instead, they should schedule the patch for the next security update. With the update, they should notify both the customers of the product and CVE.org of the vulnerability.

What have we learned?

There are three main problems we have discussed regarding vulnerability management practices of intelligent device manufacturers:

1.      Device manufacturers are in many or most cases not reporting vulnerabilities in their products to CVE.org. While many software suppliers are also not reporting vulnerabilities, this appears to be a much bigger problem for device manufacturers.

2.      Except in the case of very serious vulnerabilities like log4shell, most device manufacturers do not immediately release a patch for a new vulnerability in one of their devices. Instead, they do not notify of the vulnerability and include application of the patch in the next full update, which might be up to one year later. 

3.      With the full update, they likely will include notification of the vulnerability in the patch notes. However, they seldom also report the vulnerability to CVE.org.

I recommend these practices to address these problems:

A.     Device manufacturers should put in place procedures to report future vulnerabilities to CVE.org (including making sure they understand how the process works, contacting a CNA who can help them when they need to report, etc.).

B.     Device manufacturers that do not provide at least quarterly full updates to their products should schedule at least quarterly security updates, during which all outstanding patches will be applied to a device.

C.      Each manufacturer needs to consult with its customers to determine appropriate criteria for deciding whether a newly discovered vulnerability in a device should be patched immediately, or whether it can wait for the next security update.

D.     When a patch for a vulnerability is made available to customers in a security update, the manufacturer should also report the vulnerability to CVE.org.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

I lead the OWASP SBOM Forum. If you would like to join or contribute to our group, please go here, or email me with any questions.