Did you know
that, under CIP-007 R2, you must evaluate patches (or updates) to firmware in
the same way you track “pure” software patches? I must admit I didn’t realize this
until recently, and it seems many others didn’t either. Here is how I was led
to this conclusion, as well as a few other issues that came up along the way.
In my post
on RF’s CIP v5 workshop in early October, I included this paragraph, discussing
a presentation by Todd Thompson, one of the RF auditors:
“Todd
Thompson addressed the monster requirement, CIP-007. After Todd’s presentation,
a questioner asked if an entity must upgrade their firmware if the upgrade will
assist with compliance – for example, it will enable longer passwords – given
that firmware upgrades can sometimes lead to unanticipated issues. Todd
answered that the entity needs to test to make sure the upgrade won’t cause a
problem. If it does, they don’t need to apply the upgrade, but they do need to
document why they couldn’t apply it.”
I must admit
I didn’t think too much about the full meaning of this question when I wrote
the post. I knew that a number of NERC entities were making a concerted effort
to upgrade firmware on lots of devices in substations in advance of the v5
compliance date, April 1, 2016. I knew there was no requirement to have the
latest firmware as of the compliance date, but I assumed the entities were
doing this as a good practice, perhaps with a nudge from their NERC Region. I
assumed the questioner was referring to this process.
However, one
reader with whom I’ve exchanged other emails about interpretation issues in the
past, Brandon Workentin of EnergySec, saw something different in there. He thought
this was really a CIP-007-6 R2 question; in other words, the real meaning of
the question was “I know that R2 covers firmware patches as well as pure
‘software’ ones. But am I still obligated to apply one if I believe it might
lead to a reliability problem?” Firmware patches are nowhere explicitly
mentioned in R2, nor are they explicitly mentioned in the Guidance and
Technical Basis for R2.
He pointed
out that the auditor’s answer just addressed compliance with R2.2; he was
saying that, if you evaluate a firmware patch and determine it will cause a
reliability problem, you need to document that finding but you don’t have to
apply the patch. However, Brandon pointed out that you still need to comply
with R2.3, which requires the entity to develop a mitigation plan that
addresses the vulnerability the patch was meant to fix.
In other
words, the questioner was probably pretty happy with the auditor’s response,
since he may have thought that he had no further obligation for a particular
firmware patch, as long as he could document that applying it would impact
reliability. But the questioner would probably not be so happy if he had
received the full answer, which is that he would still have to go through the
full mitigation plan process for firmware patches, just like pure “software”
patches.
This struck
me as potentially a lot of work, and before I advised people that they need to
treat firmware patches the same as pure “software” ones, I decided to check
with some people I know on whether they agreed with this. Two of those people
were auditors (different regions); they both agreed that firmware patches
(although they’re usually called “updates”) are covered by CIP-007-6 R2, even
though the requirement does not explicitly say that (as one auditor stated,
“Firmware is software”).[i]
I also asked
several entities (large and small), as well as one consultant, whether they
knew that “Firmware is software”. Only one of them (the small entity) said he knew about this. He said he’d heard about it at a WECC (Western Electric Coordinating Council) meeting
a year or two ago, but couldn’t remember which one. I have attended a lot of
regional meetings, but I couldn’t remember hearing this at any of them.
However, I did go back and read the presentation
on CIP-007-6 that was given at WECC’s Advanced CIP Training in early September.
While I can’t remember this being stated in the actual meeting, slides 66, 67
and 72 do say that firmware patches need to be treated like pure “software”
ones.[ii]
Of course,
just the fact that WECC has stated this in one or two meetings doesn’t mean
that all NERC entities in the US and Canada have thereby received notification
that firmware is software. In case you want to counter that by saying “Since
firmware is software, the burden isn’t on NERC or the regions to notify people
of this; they should have known that”, I wish to counter you by pointing out
that Brandon did an excellent analysis (see the end note[iii]) of
the wording in the Guidance and Technical Basis. His analysis indicates that
even the Standards Drafting Team members who wrote the Guidance didn’t seem to always understand
this. I think it can quite legitimately be argued that NERC entities have not
been sufficiently made aware that they should treat firmware as software under
CIP-007-6 R2, for them to be held to strict compliance with that particular
“implied requirement” on April 1, 2016.
Even if NERC (or FERC)
comes out with guidance on this issue tomorrow, I think it's too late for
entities to be expected to be in compliance on that date.
Before I go,
I’d like to point out three other items that came up in these discussions:
First, for
both pure “software” and firmware patches (or “updates”), CIP-007-6 R2 requires
the entity to distinguish security patches from non-security ones; you only
need to deal with the former. However, as one of the auditors pointed out to
me, these security patches are usually identified as such by the vendor
releasing them, but not always. So for a vendor that doesn’t distinguish
between the two patch types, it is important for the entity to read the
description of every patch released to see if it is a security patch or not.
Second, at
GridSecCon I met Monta Elkins of FoxGuard Solutions (who gave an excellent
presentation on patch management). He pointed out to me that servers and
workstations will usually have different pieces of hardware – video card, I/O
controller, etc. – that are made by different vendors. Does this mean you have
to inventory each component of each server or workstation and monitor each
component vendor for firmware patches? This would lead to a huge effort.
I think the
answer to that issue, and Monta agreed with me, is you should confirm with the
server or workstation vendor that they will provide firmware patches for any of
the third-party devices in your system. This way, you still just have to
monitor one patch source for all of those patches.
Third, in
Brandon Workentin’s analysis (in end note iii below) of the Guidance and
Technical Basis for CIP-007-6 R2, he pointed out another issue on which the
Standards Drafting Team left contradictory guidance. The first paragraph of the
R2 guidance reads:
“The SDT’s
intent of Requirement R2 is to require entities to know, track, and mitigate
the known software vulnerabilities associated with their BES Cyber Assets. It
is not strictly an ‘install every security patch’ requirement; the main
intention is to ‘be aware of in a timely manner and manage all known
vulnerabilities’ requirement.”
I, like
probably most people, didn’t notice on first reading this that it raises a big
problem, but after Brandon’s email it stands out to me like a sore thumb: My
understanding (and I think most others’) was that R2 requires entities to
evaluate security patches and make sure the vulnerabilities they address are
mitigated, either through applying the patch or (if the patch can’t be applied)
through alternative measures (described in the mitigation plan).
But here the
SDT is saying something quite different. They’re saying that the entity is
responsible for knowing, tracking and mitigating all of the “known” software
vulnerabilities associated with BES Cyber Assets! How do they do this? Clearly,
just considering all patches that are released isn’t sufficient, since there
are many “known” vulnerabilities – in probably most software products – that
are never patched, either because the software vendor doesn’t consider them
serious vulnerabilities, or because they have determined that the cost of
patching them is too high.
This wording
seems to imply that entities must continually scan all of the many sources of
vulnerability information for any mention of a vulnerability in any software or
firmware that’s installed anywhere in their ESPs. When they find mention of a
vulnerability they didn’t know about, they need to first see if there’s been a
patch released that addresses it. Presumably, if a patch has already been
released, the entity has either been patched or mitigated the vulnerability,
through complying with R2. However, this can’t automatically be assumed.
But what if
no patch has been released? According to this wording, the entity is still
responsible for mitigating any “known” vulnerability. And how would your
auditor determine whether or not you had mitigated all of those? Will NERC scan
all of the literature and publish a continuously-updated list? Clearly, that’s
not going to happen. Also quite clearly, R2 is for patching, not for
vulnerability management.
I think it’s
safe to say you don’t need to think that your CIP v5 compliance burden has just
taken another significant leap up, and that you are now responsible for
assessing all vulnerabilities, as well as patches. As everywhere in CIP v5,
only the language of the Requirement is auditable, and that clearly states that
this is a requirement for patch management, not vulnerability management. But
it’s quite striking that the Guidance could be so very off base in this case.
The views and opinions expressed here are my own and don’t
necessarily represent the views or opinions of Deloitte Advisory.
[i]
One of the auditors pointed out that CIP-010-2 R1.1.1 says configurations need
to be tracked for “Operating system(s) (including version) or firmware where no
independent operating system exists”. Of course, this doesn’t necessarily mean
that firmware needs to be covered in CIP-007 R2. It also indicates that
firmware configurations do not need to be tracked when the device runs an O/S.
[ii]
Slide 91 also mentions that a device’s lack of External Routable Connectivity,
while not getting the device in question off the hook for compliance with R2,
will make the compliance burden for that device easier. This is because some
patches may simply not be applicable to it, due to the device not having ERC.
[iii]This
is the analysis Brandon provided, to show that the Guidance and Technical Basis
for CIP-007-6 R2 seems to be confused as to whether or not firmware patches are
in scope for the requirement:
·
R2 exclusively uses the term "security
patches." There are various places this term is attempted to be defined,
but using the definition from those various places can lead to differing
conclusions.
·
Guidance and Technical Basis for Requirement R2
¶ 1: "The SDT’s intent of Requirement R2 is to require entities to know,
track, and mitigate the known software vulnerabilities associated with their
BES Cyber Assets...the main intention is to 'be aware of in a timely manner and
manage all known vulnerabilities'"
o
(Let’s take the example of a firmware patch that
addresses security, such as increasing the password length.) Password length functionality isn't generally
considered a "vulnerability." This would imply that a firmware update
which adds functionality, even if that functionality is security-related,
wouldn't be applicable. A firmware update which removed/addressed a
vulnerability, however, would. (An example of this is if an undocumented admin
account were hard-coded into the firmware. When that happens and is discovered,
those are generally addressed as vulnerabilities in the sense that they get an
ICS-CERT alert or something similar.)
·
G&TB §2.1 ¶ 1: "The requirement applies
to patches only, which are fixes released to handle a specific vulnerability in
a hardware or software product."
o
Again, is the inability to have long passwords a
vulnerability? I would say no. It would qualify as an
"insecure-by-design" feature, but I don't think it's a
"vulnerability."
o
Tom’s
comment: Here, the SDT seems to move away from what they said about the focus
of R2 being vulnerabilities. I would take this as the more authoritative
statement, not the quote from the first paragraph of the Guidance that I cited
in the last section of the post.
·
(A major software vendor defines “vulnerability”
as) “a security exposure that results from a product weakness that the product
developer did not intend to introduce and should fix once it is
discovered." They formally define it as, "a weakness in a product
that could allow an attacker to compromise the integrity, availability, or
confidentiality of that product," and state, "Security vulnerabilities
involve inadvertent weaknesses; by-design weaknesses may sometimes occur in a
product, but these aren't security vulnerabilities."
o
Tom’s
comment: I can’t believe that the vendor is saying that the same piece of code
would be a vulnerability if not intended, but wouldn’t be one if intended!
·
G&TB §2.1 ¶ 1: "The requirement
covers only patches that involve cyber security fixes and does not cover
patches that are purely functionality related with no cyber security
impact."
o
A firmware update which allows for longer
passwords or more complex passwords pretty clearly has a "cyber security
impact" (even though it would technically be a functionality update). If
you ignore that this uses the term "patches" again, this
sentence would imply that such a firmware update would be "in-scope"
for the requirement.
·
G&TB §2.1 ¶ 1: "The National
Vulnerability Database, Operating System vendors, or Control System vendors
could all be sources to monitor for release of security related patches,
hotfixes, and/or updates."
o
This one pretty clearly states that
"updates" would need to go through the R2 process.
·
G&TB §2.1 ¶ 1: "A patch source is not
required for Cyber Assets that have no updateable software or firmware (there
is no user accessible way to update the internal software or firmware executing
on the Cyber Asset)"
o
This part pretty clearly says that updatable
firmware needs to be tracked, if possible.
·
G&TB § 2.2 ¶ 1: the first several
sentences all use the term "patch," which as discussed above seems to
have only a limited definition in this document.
·
G&TB § 2.2 ¶ 1: "Considerable care
must be taken in applying security related patches, hotfixes, and/or updates or
applying compensating measures to BES Cyber Systems or BES Cyber Assets that
are no longer supported by vendors."
o
Again, uses the term "updates" and
assumes they are included in the R2 process.
·
G&TB § 2.3: This section again uses
"patch" and "vulnerability," so it implicitly includes the
limitation those terms imply.
No comments:
Post a Comment