Is fully automated vulnerability management even possible?

Note from Tom:

I have moved to Substack as my primary blog platform. If you want to see all my new posts, as well as my 1200+ legacy posts dating from 2013, please support me by becoming a paid subscriber to my Substack blog. The cost is $30 a year. Thanks!

 

A few years ago, I came to realize two things:

a.      There are a huge number of (now over 300,000) identified CVEs; and

b.      Some organizations have literally millions of intelligent devices that utilize software and firmware that can harbor vulnerabilities.

Thus, the only way for all but the smallest end user organizations to manage software vulnerabilities in all those devices is to have a fully automated process. Below are the high-level tasks[i] that I believe constitute vulnerability management:

One-time tasks (repeated periodically):

1.      Develop a CVE[ii] risk score based on information about the CVE, including a) presence or absence of the CVE in CISA’s KEV Catalog, b) CVSS Base score[iii], c) current EPSS score[iv], d) values of components of the CVSS vector string, etc. The organization needs to decide how each of these scores (and any other information deemed important by the organization, such as a measure of the criticality of a particular asset to the organization’s mission, if they have one) is incorporated into the CVE risk score.

2.      Identify a score that is the “do not fix” threshold. The organization will normally not apply a patch for a vulnerability whose risk score is below that threshold, although there will always be cases in which the vulnerability needs to be patched regardless of its score.

3.      Set up a database in which patches received from software or firmware providers are cross-referenced with the products that are reported to be vulnerable to that CVE. For example, if the record for CVE-2025-12345 lists product A version 2.0 and product B version 3.7 as vulnerable to the CVE, the user should be able to search the database for both product/versions and learn that they are vulnerable to that CVE. Of course, this capability is built into most asset and vulnerability management tools.

Ongoing tasks (repeated daily, if possible)

1.      To the extent possible, identify the product name and version number of every software and firmware product in use by the organization.

2.      To the extent possible, identify a machine readable software identifier – usually CPE or PURL – for each product/version in the inventory

3.      Search the National Vulnerability Database (NVD) and other vulnerability databases for vulnerabilities that apply to any of these product/versions.

4.      Assign a risk score to each vulnerability identified, based on the previously agreed-upon formula.

5.      Discard any CVE whose risk score is below the “do not fix” threshold.

6.      Match each patch received from a software producer with the product/version(s) in the inventory, to which that patch applies.

7.      Prioritize application of applicable patches according to the amount of risk mitigated by each patch, as measured by its risk score.

8.      Apply patches in order of priority, if possible.

9.      If a patch cannot be applied when it normally would be, determine what alternate mitigation(s) for the vulnerability or vulnerabilities addressed in the patch might be implemented. Leave those mitigations in place until the patch can be applied.

Until fairly recently (probably last year), I thought that developing a tool to perform all these steps automatically – i.e., by just setting some parameters and hitting “Start” - shouldn’t be hard; I was quite surprised when I realized there are no such tools, although there are lots of excellent tools that automate parts of this process (for example, the open source Dependency Track is used over 25 million times a day to look up an open source component shown in a software bill of materials (SBOM) in the Sonatype OSS Index vulnerability database). I was even more surprised to learn there doesn’t seem to be anyone working feverishly to be the first to produce such a tool.

At first, I attributed this lack of activity to security tool vendors not being aware of how important vulnerability management is. However, after at least a decade of major attacks that were enabled by unpatched software vulnerabilities - especially ransomware attacks - it became clear that there must be some other reason for this lack of tools.

Now the reason is clear as day to me: There are major obstacles, especially regarding data quality, that inhibit almost every step of the vulnerability management process. Of course, data quality is a problem in lots of disciplines; for example, economic data are often flawed and are never completely reliable (I used to work for an econometric forecasting firm where the unofficial motto was “We show our forecast values to six significant digits just to show we have a sense of humor”).

However, the data quality problem in vulnerability management is of such a magnitude that in many cases, trying to completely automate vulnerability management would most likely leave an organization thinking they were much better protected than they really were, due to all the false negative vulnerability information they would receive. Let’s look at four of the above vulnerability management tasks, and consider just one of the obstacles each task might face:

One-time tasks:

Task 1: All four of the data sources that I listed in this task (with the possible exception of presence in the KEV Catalog) have been the subject of lots of articles and blog posts stating that they are worthless for determining the true risk posed by a software vulnerability. That doesn’t mean that CVSS, EPSS, and KEV are in fact worthless; in fact, it’s certain that they have value if they’re utilized properly. However, an automated tool is unlikely to appreciate the nuance of “proper” utilization, meaning it’s not a great idea to utilize these to measure risk in an automated tool.

But what other risk data are publicly available and easy to access? Not much, so even though lots of security professionals understand that for example CVSS scores don’t give a true indication of risk, they stick with them as their measure of risk because nothing better is readily available. It’s like the man who walks out of his house at night and sees his neighbor on his hands and knees under the streetlight, searching for something:

Man: “What are you looking for?”

Neighbor: “My car keys.”

Man: “Where did you have them last?”

Neighbor (gesturing toward dark front lawn): “Over there.”

Man: “So, why are you looking for them here?”

Neighbor: “The light’s better here.”

 

Task 2: There is no good way to calculate the optimal “do not fix” threshold, since there’s no way to assign measurable values to the costs and benefits of using any particular value as the threshold, let alone comparing multiple values.

I suspect that most organizations say (in effect), “Our budget for patching vulnerabilities is $X. If we try to patch every vulnerability we receive (equivalent to setting the threshold at zero) for all of the software products we utilize, we will spend some huge multiple of that. On the other hand, if we set the threshold at the maximum value (meaning we won’t apply any patch we receive), we will greatly increase our likelihood of being hacked. What’s the threshold value that will allow us to stay within our budget, yet still apply what an auditor will believe is a reasonable percentage of the patches we receive?”

This reasoning makes sense in the world in which we live, although it doesn’t make sense in an ideal world in which cybersecurity measures need to be as comprehensive as possible, cost be damned. But what organization on the planet – outside of, perhaps, the military or a nuclear safety agency – could possibly even consider following the latter course?

Ongoing tasks:

Task 2: I have written a lot of blog posts on the two most widely used machine-readable software identifiers: CPE and PURL. While PURL is a very accurate identifier, it currently can only identify open source software distributed through package managers – which is not how most government and private organizations obtain the software they use. CPE, on the other hand, is a very unreliable identifier, due in part to reasons identified on pages 4-6 of this white paper by the OWASP SBOM Forum; however, it does identify commercial software.

Thus, for commercial software, there is no reliable machine-readable identifier (although there is a proposal on the table to extend PURL to cover commercial as well as open source software). This means that looking up a commercial product in the NVD, or another database based on the NVD, is unlikely to display a CVE if the CPE included in the CVE record doesn’t exactly match the one being searched for – even if both CPEs refer to exactly the same product. In other words, searching for vulnerabilities in the NVD yields an unacceptable number of false negative findings.

Task 3: Besides the problem with CPE just mentioned, there is an even greater source of unreliability when it comes to vulnerability databases based on the CPE identifier (primarily the NVD, of course). The additional unreliability is due to:

a.      As stated above, to locate a software product in the NVD, a fully automated process requires that the CPE number entered exactly match the CPE number of that product in the NVD.

b.      As I’ve just pointed out, CPE is an inherently unreliable identifier. This is primarily because of two fields in the CPE spec: “product” and “vendor”. Both products and vendors have many names at different times and different contexts. A friend of mine once asked people who worked for Microsoft the name of the company they worked for; she received over 20 different answers.

c.      The only way for a fully automated process to find a product in the NVD is to create a CPE name that follows the CPE specification, then search on that name. CPE names are originally created by the NVD, which employs contractors to perform this task (the NVD is part of NIST, which is part of the Department of Commerce). When they create a CPE name (which currently is for only about half of new CVE records), they have to make choices like whether the vendor name should be “Microsoft”, “Microsoft Europe”, “Microsoft, Inc.”, “Microsoft Inc.”, etc. Using any one of these options will produce a different CPE name from all the others. The NVD has no official guide to making these choices, so the contractor just takes their best guess regarding which to include in a particular CPE name.

d.      Thus, when an automated tool searches for a Microsoft product in the NVD or one of the other databases based on the NVD, it needs to guess which choice the contractor made for the vendor name. Of course, if there are one hundred options for Microsoft’s name (and I’m sure there are many thousands, when you consider companies Microsoft acquired, software versions in other languages, name changes for marketing purposes, etc.), the tool will have only a 1-in-100 likelihood of making the right choice.

e.      Even worse, when the searcher’s CPE name for the product doesn’t match the one the NVD contractor created (perhaps years ago), the searcher will receive the message, “There are 0 matching records”. This is the same message the searcher would receive if they had guessed the correct CPE name, but no vulnerabilities had been reported for that product. Of course, most people will take the more favorable interpretation and assume that no vulnerabilities have been reported in the product. Because of this, they may not apply a patch issued for the product, since they think it's not needed. This is probably the biggest danger posed by false positive vulnerability findings.

What’s the answer to the question in the title of this post? Just based on the obstacles I’ve pointed out in this post, the answer is clearly “No”. There are a lot more obstacles to discuss in future posts. 

 

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com or comment on this blog’s Substack community chat.

I’m now in the training business! See this post for more information.

[i] There could be a lot more steps in this list, depending on what a vulnerability management program applies to.

[ii] There are other types of vulnerabilities than CVE, including OSV, GitHub Security Advisories (GHSA), CISA’s ICS Advisories (ICSA), etc. However, there are far fewer of each of these other types than there are CVEs; plus, the other vulnerability types are often directly mapped to a CVE, meaning that a comprehensive CVE management program is likely to include them anyway.

[iii] The CVSS score is composed of three Metric Groups: Base, Temporal and Environmental. The CVSS scores published in the NVD just include the Base group. Temporal group metrics change over time, while Environmental metrics tailor the score to a specific user's context. If an organization wishes to use the latter two metric groups, they will normally have to create and include them in the CVSS scores themselves.

[iv] Since the EPSS scores for all 300,000+ CVE records are updated daily by FIRST.org, it is always a good idea to use the most recent score available.