Tuesday, December 17, 2024

What should a NERC CIP vulnerability management requirement look like?


In October, I wrote this post titled “NERC CIP needs vulnerability management, not patch management”. That post made the argument – which I’ve made before – that compliance with Requirement CIP-007 R2 is hugely expensive relative to the security benefit it provides. My most important evidence for this was the fact that CIP-007 R2 requires the NERC entity, every 35 days, to identify every security patch that has been issued for each version of each software or firmware product installed within their ESP, that has been issued in the previous 35 days. It doesn’t matter whether the vulnerability or vulnerabilities mitigated by the patch pose high, medium or zero risk; the patch needs to be evaluated and applied within 35 days. If it can’t be applied in that time, the Responsible Entity must develop and implement a mitigation plan for the vulnerabilities addressed in the plan.

When the original patch management requirement CIP-007-1 R3 appeared with NERC CIP version 1 in 2008, the requirement wasn’t spelled out in such exquisite detail, but it was very similar: The NERC entity needed to “establish and document a security patch management program for tracking, evaluating, testing, and installing applicable cyber security software patches for all Cyber Assets within the Electronic Security Perimeter(s).” As is the case with the current version, there wasn’t a word about the degree of risk posed by the vulnerability addressed in the patch.

However, in 2008 this wasn’t as big a problem as it is today, since far fewer vulnerabilities were identified at that time; therefore, far fewer patches were issued. The first vulnerabilities identified with a CVE were reported in 1999; a little more than 300 were identified in that year. Even 15 years later in 2014, only 7,928 CVEs had been reported in total. How many CVEs have been reported as of last week? 274,095, of which 38,000 have been reported so far this year (a huge jump from last year, by the way). In other words, almost five times as many new CVEs have been reported so far in 2024 alone as were reported in total during the period 1999 – 2014.

In fact, any company that tries today to apply every patch they receive for every one of the software products they use is sure to fail; yet that is exactly what CIP-007 R2 requires. How do organizations not subject to CIP compliance keep from being overwhelmed by patches? They must prioritize them. Of course, it’s best to prioritize them by the degree of risk posed by the vulnerability or vulnerabilities that are mitigated by each patch. What’s the best way to do that?

A lot of organizations prioritize vulnerabilities based on CVSS score; in fact, that used to be considered the best way to do it. However, CVSS doesn’t measure risk; it measures severity of impact of the vulnerability – and even that is very hard to measure in a single score, since the severity of impact will vary greatly by the importance of the affected system, how well the network is protected, etc.

There is now a growing consensus in the software security community that the best measure of the risk posed by a vulnerability is whether it is currently being exploited by attackers. There’s a big reason for using that measure: Only about six percent of CVEs are ever exploited in the wild.[i]

Of course, this means that, in retrospect, an organization that tries to apply every patch will be wasting at least 80-90% of the time and effort they spend in patching. While it’s not possible to reduce this wasted time to zero, it’s certainly possible to reduce it substantially by prioritizing patch application for vulnerabilities in CISA’s Known Exploited Vulnerabilities catalog or values close to 1.0 in FIRST’s EPSS score[ii] - as well as using other measures like the impact if the system is compromised.

However, there’s one type of organization that’s unable to prioritize patch applications: that’s a NERC entity with high or medium impact systems.[iii] This is the most important reason why the CIP-007 R2 patch management requirement needs to be replaced with a risk-based vulnerability management requirement. What would the new CIP-007 R2 look like? Very roughly, I think it should require the Responsible Entity to develop and implement a vulnerability management plan to:

1.      Identify new vulnerabilities that apply to software and firmware products installed on medium or high impact BES Cyber Systems, EACMS, PACS or PCAs that they operate.

2.      Assign a risk score to each vulnerability, based on criteria identified in the plan.

3.      Identify a score that is the “do not fix” threshold. The NERC entity will normally not apply a patch for a vulnerability whose score is below that threshold, although there will always be special cases in which the vulnerability needs to be patched regardless of its score.

4.      Regularly investigate security patches released by the vendors of those products.

5.      Prioritize application of those patches according to criteria listed in the plan; apply patches in order of priority, if possible.[iv]

6.      If a patch cannot be applied when it normally would be, determine what alternate mitigation(s) for the vulnerability addressed in the patch might be applied.

How will compliance with this new requirement be audited? The entity will need to provide two types of evidence:

A.     The plan itself, including a narrative explaining how the vulnerability management plan makes much more effective use of resources and mitigates more risk than would a plan to apply every applicable patch, regardless of the degree of risk mitigated.

B.     Evidence that the plan was implemented and that it significantly lowered the entity’s level of software cyber risk. This could include information such as a comparison of a group of vulnerabilities that were patched vs. a group that were not patched, along with the average risk scores of the two groups; of course, the comparison should show that the scores of the patched vulnerabilities were much higher than those of the unpatched vulnerabilities.

C.      Along with the quantitative evidence, qualitative evidence that the plan was implemented and was effective in lowering the entity’s level of software risk. For example, there could be a narrative like, “An example of the success of our plan is CVE-2024-12345. This vulnerability was on CISA’s KEV list when the patch was released. We assigned the vulnerability our highest risk score of 5 and applied the patch the day it was released. After we applied the patch, three major cyber incidents were reported in which CVE-2024-12345 was the primary attack vector.”

Note that, unlike the current CIP-007 R2, compliance with this vulnerability management requirement will not mandate that the NERC entity be able to provide evidence of every instance of compliance – e.g., evidence that the entity (or their tool) checked with every software or firmware vendor in scope every 35 days to determine whether any new security patches were available.

Even more importantly, the NERC entity will not need to provide evidence on an individual device basis, as is usually required for CIP-007 R2 compliance. Instead of having to identify individual BES Cyber Assets that were patched, the entity will identify – for example – vulnerabilities on the CISA KEV list that were patched, without having to point to individual devices.

This is especially important for one reason: Cloud service providers will never be able to provide compliance evidence on a device basis, since they’re not equipped to do that. This means that the current CIP-007 R2 (and CIP-010 R1, which also requires evidence on a device basis) will continue to be an obstacle to cloud use by NERC entities with high and medium impact BES Cyber Systems until CIP-007 R2 is changed to a risk-based (or “objectives-based”, to use NERC’s preferred term) requirement. The risk that patch management mitigates is the risk posed by unpatched vulnerabilities, so the only way that CIP-007 R2 can be made “cloud friendly” is to replace it with a vulnerability management requirement.

Of course, we can assume that all of CIP will ultimately be rewritten (or replaced) as part of the huge effort required to make use of the cloud fully “legal” for NERC entities. However, as I pointed out recently, it will likely be 6-7 years before that is accomplished. Does the power industry have no choice but to wait that long?

What I just wrote about CIP-007 R2 points to a possible interim solution, which may “only” require rewriting two requirements: CIP-007 R2 and CIP-010 R1 (configuration management). I’ve just described what needs to be done with the former requirement. As for the latter, I think CIP-010 R1 can remain a configuration management requirement, but it can be reworded so that it is no longer device-based and so that it focuses on the risk of inadvertent misconfiguration.

I think both of these requirements could be rewritten and gain approval from NERC and FERC in 3 to 3 ½ years, which is half the time required for the full “CIP in the cloud” revisions that the Project 2023-09 Risk Management for Third-Party Cloud Services Standards Drafting Team is now working on. The full set of revisions will ultimately be needed, but I believe most cloud use by NERC entities could be made “legal” just by rewriting these two requirements.

However, doing this will require a different SDT – one that is just focused on these two requirements. Besides accelerating the development of the replacements for CIP-007 R2 and CIP-010 R1, constituting a second SDT will allow the current SDT to finish their work at least a year or two earlier than 2031. 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

My book "Introduction to SBOM and VEX" is available! For context, see this post.


[i] I owe this observation to Chris Hughes, who writes the excellent blog on software security called Resilient Cyber. I recommend you subscribe to it, if software security is a concern of yours. The post where I found this figure is here.

[ii] EPSS scores can change daily, so it is important to update them daily, if you’re basing your patch prioritization in part on EPSS.

[iii] Of course, NERC auditors are human beings; they’re unlikely to issue a Notice of Potential Violation (NPV) if an entity convinces them they don’t have the resources required to apply or mitigate every patch that’s been released for every piece of software found in their ESP. But it’s a big waste of time for a NERC entity to have to prove it. The fact is that today, an organization that does have those resources is the rare exception that proves the rule.

[iv] There may be reasons why some patches should not be applied strictly according to their assigned priority. For example, if the target systems are found in different physical locations, and it will save a lot of time if all patches due to be applied in one location are applied at once.

No comments:

Post a Comment