Monday, December 11, 2023

Few intelligent device manufacturers are reporting vulnerabilities. At all. Of course, that’s one way to have a clean record…



On behalf of my French client Red Alert Labs, I have been following developments in the US IoT device cybersecurity labeling program. This program was ordered (in very general terms) by Executive Order 14028 of May 2021. As required by the EO, NIST developed a good set of guidelines for IoT device security in 2022: NISTIR 8425. In July, the White House announced that the Federal Communications Commission (FCC) will oversee this program and that it will be based on NISTIR 8425. It will come into effect at the end of 2024, if then.

The FCC issued a Notice of Proposed Rulemaking (NPRM) for the program in August 2023. This was a preliminary document. There have been two rounds of comments on it, and the FCC will probably re-issue the NPRM, with changes based on those comments, in early 2024. The program will provide a certification (label) for consumer IoT devices, which will be based on an assessment conducted by an approved “CyberLAB”. While this is a voluntary program and therefore isn’t a “regulation”, the FCC makes clear that they expect the certification to be considered a valuable aid to marketing an IoT product, in that it will allow a consumer, who may not be a world-class cybersecurity expert, to trust a device they are considering purchasing. In many aspects, this program is modeled on the Department of Energy’s successful EnergyStar program (which is still ongoing).

Not too surprisingly, one of the main concerns listed in the NPRM is vulnerabilities that are identified in an IoT device once it is in use. The NPRM wonders what measures the FCC should take to ensure those vulnerabilities (which are inevitable of course, no matter how diligent the manufacturer was in patching vulnerabilities before the device shipped) are dealt with in a timely fashion.

Of course, this is a legitimate concern. However, in the past six months or so, I’ve come to realize that getting a device manufacturer to patch vulnerabilities in their products isn’t the main problem (although the timeliness – or lack thereof - with which those patches are applied is a big problem. I will deal with that in another post soon).

The main problem is that device manufacturers (including manufacturers of consumer IoT devices, Industrial IoT devices, medical devices, etc.) aren’t doing a good job of reporting vulnerabilities in the first place. Since the great majority of vulnerabilities (or at least CVEs) are reported by the supplier of the software or the device that’s affected by the vulnerability, this means that, if the supplier isn’t reporting any vulnerabilities, the device will usually appear to be vulnerability-free in the National Vulnerability Database (NVD). That is, a search for the device will yield the message “There are 0 matching records”. This could either mean that a) the device has never had a vulnerability, or b) it’s loaded with vulnerabilities, but they’ve never been reported. You take your choice.

Unfortunately, the latter interpretation is more frequently correct than the former. In fact, Tom Pace of NetRise reported in a presentation described in this post that he examined one device that had no reported vulnerabilities (the manufacturer had never reported a vulnerability for any of their devices). However, he found that it had over 40,000 vulnerabilities, not 0. Might this have been a rounding error?... I didn’t think so, either.

From what I can see, very few devices - even medical devices - have any reported vulnerabilities in the NVD (although I would like to hear from anyone who has looked at this question). Even worse, since device manufacturers usually don’t issue patches between full device updates, and it seems even some major medical device manufacturers only update the software and firmware in their devices once a year, this means that many if not most vulnerabilities will remain in the device for months before they’re fixed. Meanwhile, the users won’t be notified of the vulnerability, so they can’t at least put in place an alternate mitigation. Needless to say, this isn’t a good situation.

While I mostly blame the manufacturers for this situation, I will say that I’ve never seen any guidelines (from any organization, public or private) on how manufacturers should report device vulnerabilities. I have always assumed, and I believe it’s also been the assumption of people involved with the NTIA and CISA SBOM efforts, that the developers of each software or firmware product (component) installed in a device will report vulnerabilities for their product.

This means that a user who knows what software and firmware components are installed in a device they utilize should be able to find each component in the NVD and learn about any vulnerabilities in the component using a tool like Dependency Track or Daggerboard. The assumption at that point is that the device will be affected by all the component vulnerabilities, minus any vulnerabilities for which the status is later described as “not affected” in a VEX document.

But there are several problems with this assumption:

1.      Few if any intelligent device manufacturers (including medical device manufacturers or MDMs) now regularly provide SBOMs listing the components in the device to their customers. MDMs are required to provide an SBOM with the pre-market information package they must provide to the FDA, but that will never be made available outside of the FDA. The MDM faces no requirement to provide SBOMs to the hospitals that buy their devices (although this is a recommended practice). Of course, without an SBOM, a device user can’t learn what software and firmware components are present in their device. That’s why the FDA announced six or seven years ago that they were going to start requiring SBOMs for devices (although they didn’t list a date).

2.      Even if the user has an up to date SBOM for their device, they won’t find any vulnerabilities for the components in the NVD, unless the suppliers of the components regularly report their vulnerabilities to the NVD. For most software and firmware components, this is unlikely to be the case, especially considering that about 90% of software components are open source and many open source projects don’t report vulnerabilities at all.

3.      Even if the user had an SBOM and was able to find component vulnerabilities in the NVD, they still wouldn’t know which of these vulnerabilities are exploitable in the device without regular VEX documents. And I know of no device manufacturer that has issued VEXes in anything but a test case.

Does solving these three problems require that end users hound device manufacturers mercilessly, demanding that the manufacturers a) distribute SBOMs regularly to their device customers; b) turn around and hound their component suppliers to report all vulnerabilities to CVE.org (which then furnishes those reports to the NVD); and finally c) produce a new or updated VEX document as soon as they learn that any component vulnerability is not exploitable in their product?

While in theory this might be the best long-term solution, the fact is that the manufacturers have been regularly requested to do these things for at least a few years. Even if they suddenly started doing these things tomorrow, the users wouldn’t be able to use the information to learn about risks they face, since there are no complete consumer component vulnerability management tools that do this (and if you wonder what a complete consumer component vulnerability management tool does, see this document-in-progress from the OWASP SBOM Forum).

Fortunately, there is a way to avoid all these problems: The device manufacturer should take responsibility for performing these steps itself and report all exploitable vulnerabilities in the device to CVE.org. Specifically, the supplier needs to:

1.      Create an SBOM for their device, which lists every software and firmware product installed in the device. The SBOM needs to be updated whenever there is any change in the software, including major and minor version releases as well as patched versions; for the moment, the manufacturer doesn’t need to release these SBOMs to anyone outside of their organization.

2.      Using an appropriate tool, look up each component in the NVD and record any vulnerabilities listed for the component.

3.      If there are no vulnerabilities listed for a component, contact the supplier of the component and ask if they report vulnerabilities at all now; if not, strongly urge them to do so.

4.      Produce a list of all the component vulnerabilities in the current version of the product. These are the “third party” vulnerabilities.

5.      Identify any vulnerabilities (through scanning or otherwise) in the software that the manufacturer itself has written and installed in the device. These are the “first party” vulnerabilities.

6.      Develop a list that includes all the third party and first party vulnerabilities in the current version of the device. This is the complete set of device vulnerabilities.

7.      Report all these vulnerabilities to CVE.org (working through a Certified Numbering Authority, which will often be another developer. A complete list of CNAs is available at CVE.org).

8.      Develop and regularly update VEX documents that list the exploitability status of every vulnerability in the current version of the device. Initially, the status of a vulnerability will usually be “under investigation”, but whenever the manufacturer determines that one of the vulnerabilities either affects or doesn’t affect the device itself, they should change the status to “affected” or “not affected” respectively.[i] A new or revised VEX document should be provided to customers whenever the status of one or more vulnerabilities in the current product/version (or in a previous version that is still in use by some customers) has changed.

The end result of the above steps is that the manufacturer will have a continuously maintained list of all exploitable and non-exploitable vulnerabilities found in the current version of one of their devices. Of course, the manufacturer should already be maintaining a list like this for every version of their product that is in use by customers, since there is no excuse for a device manufacturer or a software developer not knowing what exploitable vulnerabilities are currently present in one of their products.

However, beyond tracking component vulnerabilities for their own product security management purposes, the manufacturer supplier should a) report every exploitable device vulnerability to CVE.org and b) make the list of device vulnerabilities and their status designations (i.e. a VEX document) available to their customers, updating the status designations at least once a month.

There is one caveat to this process: The manufacturer should usually not report an exploitable vulnerability to the NVD or to customers until the vulnerability has been patched and a new version number has been assigned to the patched version. For example, if a vulnerability was found in version 2.4.0 of a device and subsequently patched, the patched version might be named 2.4.1. The report to CVE.org, and the list of exploitable product vulnerabilities that is provided to customers, would describe v.2.4.0 as “affected” by the vulnerability but v2.4.1 as “fixed”. Moreover, both the CVE report and the list would include a link to download the patch, so that any user that has not applied it yet can download it.

There is also a shortcut that could drastically reduce the number of vulnerabilities that a device manufacturer needs to report to CVE.org. The manufacturer doesn’t need to report non-exploitable vulnerabilities, since they don’t affect the security of the device. Many software security professionals estimate that over 90% of component vulnerabilities aren’t exploitable in a device, meaning that up to 90% of component vulnerabilities don’t need to be reported.

On the other hand, all vulnerabilities, both exploitable and non-exploitable (including patched or “fixed” vulnerabilities) should be reported to customers. Because the reports to CVE.org will require a lot more work than will the VEX documents for customers, the manufacturer should be able to save a lot of time by taking this “shortcut”.

There is one “hole” in the above set of steps, which I have deliberately left for last: How does the manufacturer report vulnerabilities found in the device as a whole? It’s not hard to report a vulnerability in a single software or firmware product that’s installed in the device, but the device itself isn’t a software or firmware product. Other than the software and firmware, what in the device would be subject to vulnerabilities? The sheet metal or plastic that encloses the software and firmware?

This is where a little imagination is called for. Even though each vulnerability in the device is found in a particular software or firmware product installed in the device, the device itself performs one or more functions which can’t be attributed entirely to one of those software or firmware products. The device vulnerabilities need to be reported as applicable to the device itself, since otherwise users will never be able to learn about them.

And here’s where the big problem comes in: It seems few device manufacturers report vulnerabilities under the name of the device. For example, I talked to a product security manager with one of the largest medical device manufacturers last year, who told me that they had never reported a vulnerability for any of their devices (and I’m sure they have hundreds of families of devices and many thousands of individual device models). What this means is that no hospital or individual (whether a current or a prospective user of a device made by this huge manufacturer) will be able to learn about vulnerabilities in this manufacturer’s devices – since the manufacturer isn’t regularly distributing SBOMs to customers.

Fortunately, there are some device manufacturers that seem to be very concerned about reporting vulnerabilities. For example, below is part of just one of 29 pages obtained by searching the NVD for the Cisco ASA firewall appliance (one of many families of devices that Cisco makes); I apologize for the blurry text. Each of these CPE names refers to a particular Cisco ASA device.

So Cisco seems to have the right idea, when it comes to reporting device vulnerabilities. Now the other device manufacturers need to get the same religion. As the Good Book says, “Go thou and do likewise.”

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

I lead the OWASP SBOM Forum. If you would like to join or contribute to our group, please go here.


[i] There is also a status called “fixed”, meaning the vulnerability was present in a previous version of the device, but has been patched (or otherwise mitigated) in the current version.

No comments:

Post a Comment