Sunday, June 4, 2023

How do you prioritize vulnerability patching when VEX is a consideration?

 

It’s safe to say that most organizations today that are concerned about patching vulnerable software on their networks have many more patches to apply than they have time and personnel to apply them. Of course, this isn’t a new problem, but it’s gotten worse. In fact, Chris Hughes published an (as usual) excellent blog post recently that mentioned “…a report from Rezilion and the Ponemon Institute showing that…more than half of organizations have more than 100,000 open vulnerabilities in their backlog…”

However, what is a new problem for these organizations is that, within the next 2-3 years, they will be receiving SBOMs regularly from at least some of their software suppliers. And they know those SBOMs are only going to make this problem worse, not better.

Why do they say this? To simplify the analysis, let’s say that each software product or software component will on average be found to have one vulnerability in an NVD search. If we start with the situation that almost 100% of organizations are in today – that they are not regularly receiving SBOMs for the software products used on their network – this means that, if the organization uses 1,000 products, they will usually have 1,000 vulnerabilities on their “to be patched” list. Of course, that doesn’t mean 1,000 devices to be patched. If each of those vulnerabilities is found on ten devices, there are actually 10,000 devices to be patched.

Since we’ve already stipulated that the organization is overwhelmed with their current vulnerability load, this means that 1,000 vulnerabilities is too many for them to handle at one time. Therefore, they’re probably doing what most organizations would do; performing triage on the vulnerabilities they have to patch, perhaps by breaking them up into three categories – high, medium and low priority. They will move heaven and earth to patch vulnerabilities in the high category. They’ll do their best to patch vulnerabilities in the medium category. And they’ll get around to the low priority vulnerabilities when and if they get the chance to do so, with no guarantee that they will be able to patch the lows at any time in the future.

Now, let’s assume the organization suddenly starts receiving regular SBOMs from their suppliers for one quarter of the software products installed on their network – that is, for 250 products. And let’s say the average SBOM identifies 50 components in a product, which is much less than the average of around 150. Since we’re assuming that components have on average one vulnerability, as do the products themselves (i.e., the “first party” code in the product), this means the organization now knows of 250 X 50 = 12,500 new vulnerabilities, plus the 1,000 vulnerabilities they already knew about. And, if our previous assumption that every vulnerability is found on ten devices still holds, this means there are now (12,500 + 1,000) * 10 = 135,000 devices to be patched (one hopes the organization has an automated solution for this, but, as any experienced patch management person can tell you, it’s impossible to automate 100% of patch applications).

Remember, we’ve already stipulated that the organization considers their current load of 1,000 unpatched vulnerabilities to be overwhelming, meaning there is no good way the security team will ever be able to patch all of them (this is because more vulnerabilities are being identified every day. By the time they patch the high priority vulnerabilities and maybe some of the medium ones, they’ll have a new batch of highs, which demands their attention before anything else). What will their security team say (that’s printable in a family blog post) when their boss lets them know that, because of this new thing called SBOMs, their number of unpatched vulnerabilities has grown from 1,000 to 13,500? And keep in mind that the above analysis makes assumptions that could turn out to be too low. The actual number could be much higher.

This shows that the only thing that, once SBOMs start being distributed to end user organizations (by which I mean all organizations except those whose primary business is developing software or maufacturing intelligent devices) at the needed frequency (i.e., with every new major or minor version), the only thing that will save vulnerability management teams in end user organizations  will be developing a strict methodology for triaging the vulnerabilities they face. While there are many such methodologies, one that a lot of organizations are using now is to triage vulnerabilities according to their EPSS scores or whether they’re in CISA’s KEV catalog. Both of these measures are based at least in part on whether actual exploits based on the vulnerability in question are taking place.

So now, the security team that thought 1,000 unpatched vulnerabilities was the workload from hell is faced with 13,500 unpatched vulnerabilities. What will they do? They’ll do the only two things they can do: a) ask for more head count, and b) re-prioritize. They might do the latter by first prioritizing the 13,500 vulnerabilities into high, medium and low; then they’ll take just the highs and re-prioritize them into high/high, medium/high and low high. Finally, they’ll concentrate on the high/highs and perhaps a few of the medium/highs. Those are probably all they’ll ever be able to patch.[i]

What happens when we introduce VEX into this picture? You might think that, since VEX has to do with exploitability, it won’t have much to add to the EPSS score or the Kevin catalog. However, if you think that, you’re wrong. This is because there are two types of exploitability. One is the VEX sense, which applies just to a single product and is binary; it answers the question, is CVE-2023-12345 exploitable in product A version 2.5? The answer is yes or no. Note that it doesn’t apply to more than one version of a product, unless the statement applies to a version range (which is currently only an option in the CycloneDX VEX format).

The second type is the exploitability discussed above, measured using (among other things) the EPSS score and KEV catalog. This is a percentage (0 to 100%) and applies to a vulnerability, not a product; indeed, this type of exploitability applies across all products.[ii] It answers the question, given the most up-to-date information on exploitation and availability of exploit code, how likely is it that CVE-2023-12345 can be used to compromise any software product containing that vulnerability?

Let’s get back to the original question: If a security team needs to prioritize the vulnerabilities it faces and just patch the most exploitable ones (in the second sense of the word), how can VEX help them do that, since it deals with a different type of exploitability? More succinctly, can VEX provide much help at all?

My answer to this question might surprise you: VEX can only play a secondary role in addressing the problem faced by the intrepid security team we’ve described above. The team needs to use EPSS, KEV, etc. to prioritize the vulnerabilities it’s addressing; VEX won’t help with that job.

However, VEX can play a secondary role that could in theory (but not in practice today, unfortunately) make the team’s problem more manageable than it would be otherwise. Remember that at least 90% (and maybe 95-97%) of vulnerabilities identified in components of a product that are listed in an SBOM will not be found to be exploitable in the product itself. If our team could learn, for every version of every software product that their organization uses, which component vulnerabilities (i.e., vulnerabilities in components listed in an SBOM, that can be found in the NVD or another vulnerability database) are not exploitable in every software product used by the organization, they could then remove those versions from the list of those to be patched for the non-exploitable vulnerability.

Let’s illustrate this with an example:

1.      Suppose the security team has identified and prioritized 1,000 vulnerabilities for devices on their network, as well as 12,500 vulnerabilities for components listed in an SBOM for one of the software product/versions installed on those devices (as in the case we described above). They have identified 250 “high/high” vulnerabilities, based on EPSS scores and the KEV catalog. Using our estimate of ten vulnerable devices for each vulnerability, this means the team needs to patch 2,500 devices, in order to patch every instance of each high/high vulnerability.

2.      Their initial patching push for the month will focus just on the high/highs. They will start by identifying every device on their network(s) on which a software product subject to at least one of the high/high vulnerabilities is installed. They will plan to patch each of these until they have patched every one of the high/high vulnerabilities, on every device on which that vulnerability is found. In other words, when they have finished the initial phase of the effort, they will have patched every instance of every high/high vulnerability.

3.      However, what if the organization were receiving, from the supplier of each of the 1,000 software products installed on their network, a revised SBOM corresponding to at least every major new version of the product? And on top of that, what if the supplier were sending a new VEX document whenever they have determined that a particular component vulnerability is exploitable (or not) in the product in which the component is included (hey, a guy can fantasize, can’t he)?

4.      Moreover, suppose each of the organization’s software suppliers is really thorough and, within say one month of the release of a new version of a product, they have determined the exploitability status (in the VEX sense, of course) of each component vulnerability in that version.

5.      Finally, after aggregating all the VEX information it has received about each product/version installed on its network, the security team realizes that 95% of all their component vulnerabilities are not exploitable in the full product; of course, this is probably close to the average number.

6.      When the security team realizes this, there will be great rejoicing. After all, they had at first thought they’d have to patch 2,500 machines just in order to patch all the high/high vulnerabilities. However, it turns out they only need to patch 5% of those machines, or 125. A nice improvement, n’est-ce pas?

7.      Furthermore, instead of having to patch 135,000 devices in order to eliminate every vulnerability in their backlog from the network, the team “only” has to patch 135,000 * 5% = 6,750 machines. That’s still a lot, of course, but what’s most striking is that the security team is now at the point where they can seriously discuss completely eliminating their vulnerability backlog, vs. at best patching all of the high/high and medium/high vulnerabilities.

Of course, I’ve already admitted that this vision is a fantasy. Neither SBOMs nor VEX documents are being provided to end users whenever needed (an SBOM is needed at least whenever a new version of the software is released, while a VEX document is needed whenever the exploitability status of a component vulnerability has changed).

Moreover, I don’t see that changing until both types of documents are regularly being released.[iii] VEX without SBOM is irrelevant, while SBOMs without VEX will never be accepted, because of the huge false positive vulnerability problem. This isn’t a chicken-or-egg problem. After the VEX problem is solved and the naming problem is substantially ameliorated (through changes to the NVD, primarily), SBOMs will start to be distributed in the required frequency and quantity. Fortunately, progress is being made on both fronts.



[i] There’s another way to triage their work, and that’s according to the importance of the device on which the software is running. For example, say that CVE-2023-12345 is at the top of the prioritization list, but it’s found on three types of devices. One is the computers used by the team that plans next month’s lunchroom meals (are there still lunchrooms?). The second is computers used in the finance department, although not for trading. The third type is devices that control the manufacturing process which brings in literally every dollar the company makes. 

It might seem like a no-brainer that even though the team would start their day by patching CVE-2023-12345, they would first patch the manufacturing machines, then the machines used by finance and lastly the menu people. However, even that isn’t a sure thing. There are all sorts of reasons why individual machines will be patched in a different order than others, which have nothing to do with prioritization by CVE or type of machine. 

[ii] Of course, in both types of exploitability, the words “other things being equal” apply. For example, if a device is isolated from the internet and located in a room that requires a retina scan and a smart key to enter, it’s likely it will be close to unexploitable in either sense. This means that in both types of exploitability, protections that need to be separately applied and will not always be in place in all environments should be ignored. 

[iii] I have given up on the idea that VEX documents will ever be released in the quantities that will ultimately be needed, which might be tens or even hundreds of VEX documents for a single product every day. This is why I’m focusing on the idea of an API by which users can query VEX information from a server maintained by a supplier, freeing the supplier from having to push out any VEX documents at all.

No comments:

Post a Comment