Monday, November 23, 2015

Tom's Lessons Learned No. 4: “Firmware is Software”


Did you know that, under CIP-007 R2, you must evaluate patches (or updates) to firmware in the same way you track “pure” software patches? I must admit I didn’t realize this until recently, and it seems many others didn’t either. Here is how I was led to this conclusion, as well as a few other issues that came up along the way.

In my post on RF’s CIP v5 workshop in early October, I included this paragraph, discussing a presentation by Todd Thompson, one of the RF auditors:

“Todd Thompson addressed the monster requirement, CIP-007. After Todd’s presentation, a questioner asked if an entity must upgrade their firmware if the upgrade will assist with compliance – for example, it will enable longer passwords – given that firmware upgrades can sometimes lead to unanticipated issues. Todd answered that the entity needs to test to make sure the upgrade won’t cause a problem. If it does, they don’t need to apply the upgrade, but they do need to document why they couldn’t apply it.”

I must admit I didn’t think too much about the full meaning of this question when I wrote the post. I knew that a number of NERC entities were making a concerted effort to upgrade firmware on lots of devices in substations in advance of the v5 compliance date, April 1, 2016. I knew there was no requirement to have the latest firmware as of the compliance date, but I assumed the entities were doing this as a good practice, perhaps with a nudge from their NERC Region. I assumed the questioner was referring to this process.

However, one reader with whom I’ve exchanged other emails about interpretation issues in the past, Brandon Workentin of EnergySec, saw something different in there. He thought this was really a CIP-007-6 R2 question; in other words, the real meaning of the question was “I know that R2 covers firmware patches as well as pure ‘software’ ones. But am I still obligated to apply one if I believe it might lead to a reliability problem?” Firmware patches are nowhere explicitly mentioned in R2, nor are they explicitly mentioned in the Guidance and Technical Basis for R2.

He pointed out that the auditor’s answer just addressed compliance with R2.2; he was saying that, if you evaluate a firmware patch and determine it will cause a reliability problem, you need to document that finding but you don’t have to apply the patch. However, Brandon pointed out that you still need to comply with R2.3, which requires the entity to develop a mitigation plan that addresses the vulnerability the patch was meant to fix.

In other words, the questioner was probably pretty happy with the auditor’s response, since he may have thought that he had no further obligation for a particular firmware patch, as long as he could document that applying it would impact reliability. But the questioner would probably not be so happy if he had received the full answer, which is that he would still have to go through the full mitigation plan process for firmware patches, just like pure “software” patches.

This struck me as potentially a lot of work, and before I advised people that they need to treat firmware patches the same as pure “software” ones, I decided to check with some people I know on whether they agreed with this. Two of those people were auditors (different regions); they both agreed that firmware patches (although they’re usually called “updates”) are covered by CIP-007-6 R2, even though the requirement does not explicitly say that (as one auditor stated, “Firmware is software”).[i]

I also asked several entities (large and small), as well as one consultant, whether they knew that “Firmware is software”. Only one of them (the small entity) said he knew about this. He said he’d heard about it at a WECC (Western Electric Coordinating Council) meeting a year or two ago, but couldn’t remember which one. I have attended a lot of regional meetings, but I couldn’t remember hearing this at any of them. However, I did go back and read the presentation on CIP-007-6 that was given at WECC’s Advanced CIP Training in early September. While I can’t remember this being stated in the actual meeting, slides 66, 67 and 72 do say that firmware patches need to be treated like pure “software” ones.[ii]

Of course, just the fact that WECC has stated this in one or two meetings doesn’t mean that all NERC entities in the US and Canada have thereby received notification that firmware is software. In case you want to counter that by saying “Since firmware is software, the burden isn’t on NERC or the regions to notify people of this; they should have known that”, I wish to counter you by pointing out that Brandon did an excellent analysis (see the end note[iii]) of the wording in the Guidance and Technical Basis. His analysis indicates that even the Standards Drafting Team members who wrote the Guidance didn’t seem to always understand this. I think it can quite legitimately be argued that NERC entities have not been sufficiently made aware that they should treat firmware as software under CIP-007-6 R2, for them to be held to strict compliance with that particular “implied requirement” on April 1, 2016.  Even if NERC (or FERC) comes out with guidance on this issue tomorrow, I think it's too late for entities to be expected to be in compliance on that date.

Before I go, I’d like to point out three other items that came up in these discussions:

First, for both pure “software” and firmware patches (or “updates”), CIP-007-6 R2 requires the entity to distinguish security patches from non-security ones; you only need to deal with the former. However, as one of the auditors pointed out to me, these security patches are usually identified as such by the vendor releasing them, but not always. So for a vendor that doesn’t distinguish between the two patch types, it is important for the entity to read the description of every patch released to see if it is a security patch or not.

Second, at GridSecCon I met Monta Elkins of FoxGuard Solutions (who gave an excellent presentation on patch management). He pointed out to me that servers and workstations will usually have different pieces of hardware – video card, I/O controller, etc. – that are made by different vendors. Does this mean you have to inventory each component of each server or workstation and monitor each component vendor for firmware patches? This would lead to a huge effort.

I think the answer to that issue, and Monta agreed with me, is you should confirm with the server or workstation vendor that they will provide firmware patches for any of the third-party devices in your system. This way, you still just have to monitor one patch source for all of those patches.

Third, in Brandon Workentin’s analysis (in end note iii below) of the Guidance and Technical Basis for CIP-007-6 R2, he pointed out another issue on which the Standards Drafting Team left contradictory guidance. The first paragraph of the R2 guidance reads:

“The SDT’s intent of Requirement R2 is to require entities to know, track, and mitigate the known software vulnerabilities associated with their BES Cyber Assets. It is not strictly an ‘install every security patch’ requirement; the main intention is to ‘be aware of in a timely manner and manage all known vulnerabilities’ requirement.”

I, like probably most people, didn’t notice on first reading this that it raises a big problem, but after Brandon’s email it stands out to me like a sore thumb: My understanding (and I think most others’) was that R2 requires entities to evaluate security patches and make sure the vulnerabilities they address are mitigated, either through applying the patch or (if the patch can’t be applied) through alternative measures (described in the mitigation plan).

But here the SDT is saying something quite different. They’re saying that the entity is responsible for knowing, tracking and mitigating all of the “known” software vulnerabilities associated with BES Cyber Assets! How do they do this? Clearly, just considering all patches that are released isn’t sufficient, since there are many “known” vulnerabilities – in probably most software products – that are never patched, either because the software vendor doesn’t consider them serious vulnerabilities, or because they have determined that the cost of patching them is too high.

This wording seems to imply that entities must continually scan all of the many sources of vulnerability information for any mention of a vulnerability in any software or firmware that’s installed anywhere in their ESPs. When they find mention of a vulnerability they didn’t know about, they need to first see if there’s been a patch released that addresses it. Presumably, if a patch has already been released, the entity has either been patched or mitigated the vulnerability, through complying with R2. However, this can’t automatically be assumed.

But what if no patch has been released? According to this wording, the entity is still responsible for mitigating any “known” vulnerability. And how would your auditor determine whether or not you had mitigated all of those? Will NERC scan all of the literature and publish a continuously-updated list? Clearly, that’s not going to happen. Also quite clearly, R2 is for patching, not for vulnerability management.

I think it’s safe to say you don’t need to think that your CIP v5 compliance burden has just taken another significant leap up, and that you are now responsible for assessing all vulnerabilities, as well as patches. As everywhere in CIP v5, only the language of the Requirement is auditable, and that clearly states that this is a requirement for patch management, not vulnerability management. But it’s quite striking that the Guidance could be so very off base in this case.


The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte Advisory.


[i] One of the auditors pointed out that CIP-010-2 R1.1.1 says configurations need to be tracked for “Operating system(s) (including version) or firmware where no independent operating system exists”. Of course, this doesn’t necessarily mean that firmware needs to be covered in CIP-007 R2. It also indicates that firmware configurations do not need to be tracked when the device runs an O/S.

[ii] Slide 91 also mentions that a device’s lack of External Routable Connectivity, while not getting the device in question off the hook for compliance with R2, will make the compliance burden for that device easier. This is because some patches may simply not be applicable to it, due to the device not having ERC.

[iii]This is the analysis Brandon provided, to show that the Guidance and Technical Basis for CIP-007-6 R2 seems to be confused as to whether or not firmware patches are in scope for the requirement:
·       R2 exclusively uses the term "security patches." There are various places this term is attempted to be defined, but using the definition from those various places can lead to differing conclusions.

·       Guidance and Technical Basis for Requirement R2 ¶ 1: "The SDT’s intent of Requirement R2 is to require entities to know, track, and mitigate the known software vulnerabilities associated with their BES Cyber Assets...the main intention is to 'be aware of in a timely manner and manage all known vulnerabilities'"
o   (Let’s take the example of a firmware patch that addresses security, such as increasing the password length.)  Password length functionality isn't generally considered a "vulnerability." This would imply that a firmware update which adds functionality, even if that functionality is security-related, wouldn't be applicable. A firmware update which removed/addressed a vulnerability, however, would. (An example of this is if an undocumented admin account were hard-coded into the firmware. When that happens and is discovered, those are generally addressed as vulnerabilities in the sense that they get an ICS-CERT alert or something similar.)

·       G&TB §2.1 ¶ 1: "The requirement applies to patches only, which are fixes released to handle a specific vulnerability in a hardware or software product."
o   Again, is the inability to have long passwords a vulnerability? I would say no. It would qualify as an "insecure-by-design" feature, but I don't think it's a "vulnerability."
o   Tom’s comment: Here, the SDT seems to move away from what they said about the focus of R2 being vulnerabilities. I would take this as the more authoritative statement, not the quote from the first paragraph of the Guidance that I cited in the last section of the post.

·       (A major software vendor defines “vulnerability” as) “a security exposure that results from a product weakness that the product developer did not intend to introduce and should fix once it is discovered." They formally define it as, "a weakness in a product that could allow an attacker to compromise the integrity, availability, or confidentiality of that product," and state, "Security vulnerabilities involve inadvertent weaknesses; by-design weaknesses may sometimes occur in a product, but these aren't security vulnerabilities."
o   Tom’s comment: I can’t believe that the vendor is saying that the same piece of code would be a vulnerability if not intended, but wouldn’t be one if intended!

·        G&TB §2.1 ¶ 1: "The requirement covers only patches that involve cyber security fixes and does not cover patches that are purely functionality related with no cyber security impact."
o   A firmware update which allows for longer passwords or more complex passwords pretty clearly has a "cyber security impact" (even though it would technically be a functionality update). If you ignore that this uses the term "patches" again, this sentence would imply that such a firmware update would be "in-scope" for the requirement.

·       G&TB §2.1 ¶ 1: "The National Vulnerability Database, Operating System vendors, or Control System vendors could all be sources to monitor for release of security related patches, hotfixes, and/or updates."
o   This one pretty clearly states that "updates" would need to go through the R2 process.

·       G&TB §2.1 ¶ 1: "A patch source is not required for Cyber Assets that have no updateable software or firmware (there is no user accessible way to update the internal software or firmware executing on the Cyber Asset)"
o   This part pretty clearly says that updatable firmware needs to be tracked, if possible.

·       G&TB § 2.2 ¶ 1: the first several sentences all use the term "patch," which as discussed above seems to have only a limited definition in this document.

·       G&TB § 2.2 ¶ 1: "Considerable care must be taken in applying security related patches, hotfixes, and/or updates or applying compensating measures to BES Cyber Systems or BES Cyber Assets that are no longer supported by vendors."
o   Again, uses the term "updates" and assumes they are included in the R2 process.

·       G&TB § 2.3: This section again uses "patch" and "vulnerability," so it implicitly includes the limitation those terms imply.

No comments:

Post a Comment