Tuesday, November 24, 2015

An Auditor Responds to my Firmware Post


Judging by the large number of page views, my recent post on firmware – and the fact that firmware patches are subject to CIP-007-6 R2 – has clearly struck a chord. The day after the post appeared, I received an email from a well-known CIP auditor. As usual, this person had some very good points to make. I reproduce his email in full, with my comments interspersed in italics.


Auditor: First, firmware is nothing more than software burned onto a chip.  It might be updatable (through a process of “flashing” the PROM), or it might require replacement of the chip to implement the update.  However, to emphasize what you referenced in your blog, it is software and it is subject to CIP-007-6 until such time as an official, FERC-approved definition of Programmable Electronic Device determines otherwise.

Tom: It hadn’t occurred to me that the definition of “Programmable” – or rather the lack thereof – would have anything to do with this. In any case, I wouldn’t count on a definition coming out any time soon, and especially one that ruled out firmware updates being subject to CIP-007-6 R2.

A: Probably the biggest question is whether an update that provides for configuring and entering a complex password constitutes a security patch.  While I would dearly like to assert that it is, I cannot.  A security patch is an update that corrects a vulnerability in the affected code.  Modifying the code to allow a better password is a feature update and does not address a vulnerability.  Therefore, the patch that allows for stronger passwords is not subject to the CIP standards.  Having said that, I am not convinced that all Regional auditors share the same view, so the entity would be well advised to have this discussion with their Region sooner rather than later.

T: This point was also made by Brandon Workentin of EnergySec, in my long footnote quoting him. The bottom line is that, if you want to be safe, you need to consider a firmware patch adding a feature like stronger passwords to be a security update, even though it isn’t one from a purely technical point of view. Of course, like all CIP questions, you can always discuss this with your region. And like all questions since a few weeks ago, you probably can’t count on your region’s answer (if you get one) guiding your auditor, since your auditor may now be from FERC, not your region. In general, it’s better to take the more “hawkish” interpretation, until FERC comes out with guidance on these and a lot of other issues (and there is no assurance now that they’ll come out with guidance on any issue regarding CIP v5 interpretation).

A: And, to confirm what others have stated, you have to assess security patches for applicability.  You do not have to implement them as long as you have a mitigation plan that addresses the vulnerabilities covered by the patch.  I disagree that you have to test and determine with certainty that implementation of the patch “breaks” the system before you can choose to not implement the patch.  It is well established that outside of the traditional servers and workstations found in the Control Center, (1) updates are often cumulative, meaning you cannot pick and choose what you are going to implement, (2) vendors to this point have very frequently not identified whether a patch or update includes security fixes, (3) vendors have been reluctant to go back and try to provide information necessary to identify security patches for updates released over the past many years, and (4) even if the next released update does explicitly include a security patch, the fact that updates are generally cumulative and the vast majority of the previously released, unimplemented updates are feature releases still causes an entity to appropriately be wary of installing the update.  Extensive testing would be required and even then there is no guarantee that some feature or function change won’t be overlooked or misunderstood, introducing reliability risks that exceed whatever the security patch may be addressing.

T: The second part of this paragraph brings up something I’ve heard elsewhere: namely, that the patching process is very different once you move away from the standard IT OS platforms and go to devices like relays. As the auditor points out, the fact that the firmware updates are often cumulative, and that security updates are often not explicitly identified as such, means the entity is probably justified in being very cautious about applying even updates that are identified as being for security purposes. So having a way of mitigating the risk addressed by a security update – without actually applying the update – is especially important for non-standard OS’s.

A: The bottom line is that the entities need to make sure they are on the top of their game, that they have implemented as many defensive controls as they can regardless of whether there is a pending security patch, and that they are focused on cyber security rather than the absolute minimum necessary to demonstrate compliance.

T: This paragraph makes yet another good point: That it is important to have defensive controls (like IDS or firewall rules) that go beyond immediate compliance needs. You never know when a patch will come out that can’t be installed without impacting reliability (or for another reason). This will require you to have an alternative means of mitigating the risk, and those controls may turn out to be absolutely necessary for compliance.

T: I do have another point to make. I got an email from a compliance professional that referred to the part near the end of the post where I mentioned that firmware for video cards, etc. needs to be updated as well.  This person said:

“In line with this question I feel like there’s the potential for this to bring device drivers into scope…It could be argued that on top of checking my source for firmware updates for my network card, I also need to check my source for driver updates for the same NIC.”

I really don’t see why this wouldn’t be the case. Device drivers would seem to have the same status as firmware updates – they’re “software” updates that affect the basic operation of the system.[i] I can hear 1,000 hearts stop beating as I’ve just added yet another item to your list of “Things I Need to Address in my CIP v5 Compliance Program.”

Believe me, I’m not writing this post (and the previous one) because I’m trying to make your job harder. But that is the way CIP v5 has been ever since its approval in 2013 (two years and two days before the day I’m writing this): as entities (and auditors) get further into the implementation effort, they discover more and more “implicit requirements”. These were in theory implied by CIP v5 all along, but just haven’t been discovered until now. And guess what: These will continue, probably long after April 1, 2016.

Here’s a tip: I’m soon going to start writing about how I think CIP v5 needs to be rewritten, so we can finally get off this hamster wheel of constantly-increasing requirements, explicit or implicit. And I’ll be talking about it (although “preaching” might be a better word) at Digital Bond’s S4X16 conference in Miami Beach in January.


The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte Advisory.

[i] I showed this comment to the auditor. He agreed with it, and had the following additional comments: “The CIP standard is explicit in that the registered entity is not expected to separately baseline drivers and other components that come with the operating system (see the discussion found in the Guidelines and Technical Basis section of CIP-010-1/2).  But, video drivers and network drivers are often provided by different vendors from the operating system vendor and are separately installed from the operating system.  In other words, these drivers are not part of the installed operating system.  So, yes, you must baseline, track, and patch the nvidia display driver and the Intel NIC driver software packages that were installed on the PC (and show up as installed, versioned software under Programs and Features on a Windows system).  This auditor, at least, is applying the same concepts to installed third-party software as apply to the operating system.  You must baseline and manage the package.  You do not have to separately baseline all of the drivers and other components comprising the package.  Again, registered entities should check with their Regions to make sure their auditors share the same view.”

No comments:

Post a Comment