More on Windows XP and CIP v5

Since my post yesterday about Windows XP and CIP Version 5, I’ve corresponded with two auditors (different regions) and one NERC entity about two interesting topics.  I’ll address the shorter one first.

Topic 1

An auditor (not the same one I quoted in the original post) made this very good point:

“One comment about your recent blog post – the topic of Windows XP on BESCAs/PACS/EACMS/PCAs came up in a seminar recently, and the response given there was similar to the one you had, except the presenter (CIP auditor) stated that when that information was pulled in as part of the pre-audit scoping processes related to the shiny new NERC risk-based auditing framework, it would heighten the entity’s risk profile and thus expose them to an audit of additional CIP standards/requirements, more samples of performance evidence requested, etc.

 

“IMHO - I’d expect similar treatment for folks with ‘Applicable Systems’ running Open VMS, Solaris, OS/2, OS/400, backlevel Cisco IOS/CatOS…basically any deprecated or EOL version of an OS.”

Of course, when the auditor refers to the new “risk-based auditing framework”, he means what most of the rest of us call RAI – the Reliability Assurance Initiative (that name has officially been retired, since it’s no longer just an initiative).  Essentially, this means that an entity with a lot of XP might be considered more risky.  Therefore, that entity might not be moved as quickly to the new risk-based auditing framework as one that had very little or no XP (of course, this doesn’t apply if the reason they don’t have XP is that they haven’t yet been able to upgrade from Windows NT!  I hope there aren’t any entities in that situation).

The auditor who commented in the original post had this to say, when I asked his opinion: “Yes, running on obsolete, unsupported systems would be a risk factor that could, not necessarily would, increase the level of scrutiny directed to the entity’s compliance program.  That is an auditor discretion issue.”

Topic 2

The second topic arose when a compliance person at a large IOU wrote in with this question:

“Tom, a couple of questions were raised when I read your last post regarding patching of XP. The first is around Entities that have bought the extended service from Microsoft and thus still receive XP patches; are you required to patch? I take at this point you must evaluate those patches since they are available - but what about Entities that don't subscribe to that service; patches are available sort of, it's just they don't want to pay for them?

This gets a bit more complicated with this question. So you have certain systems from a vendor like GE, Honeywell, or Emerson that are running XP but those vendors are not releasing patches for their XP systems (not saying that all these vendors are not releasing XP patches). Due to some contract agreements, the Entity is not allowed to patch those systems independently - rather the vendor must provide the patch for those systems...and if the vendor isn't (providing patches) because they don't subscribe to the service - it puts the Entity in a precarious position. From one standpoint they evaluate all patches from the vendor that are available and install them (it's just there are none); yet then there are those patches they are receiving from Microsoft...but the Entity really can't install those either.”

The First Auditor’s Response:

I first ran this by the auditor from the original post, who said:

“In my view (and I cannot guarantee all auditors will take the same stance), there is nothing in the CIP standards requiring an entity to subscribe to post-end-of-life for-fee support.  There are no more XP patches publicly available from a vendor that routinely makes its patches publicly available for supported systems.  I would not demand that an entity pay huge dollars for continued, limited security support.  But if they did, then they are expected to monitor for and install any applicable patches that might be released under the support agreement.

“As far as the GE, Emerson, etc., these are typically turnkey system deliveries.  As such, especially with a contractual requirement that the customer not try to implement patches out of band, not provided by the vendor, the entity appropriately relies upon the third-party turnkey system provider for all patches and updates.

“Finally, as we are in the CIP V5 Transition period, the entity can assert compliance with the CIP V5 requirement, designate the patch source, and proceed accordingly.”

This last is a good point.  Under CIP v5, this issue wouldn’t even come up, since the entity chooses, for each system and software package, which vendor it will follow for patches.   So if your vendor is XYZ and they’re no longer releasing XP patches because Microsoft isn’t providing them for free anymore, there’s no obligation to go beyond what XYZ offers.

The Second Auditor’s Response:

Since I had the second auditor’s attention (with the first issue), I decided to press my luck and ask him to respond to the NERC entity’s questions.  I’m very glad I did, since he provided some very good guidance not only on this specific issue but on patch management in CIP v5 in general.  Here are his comments; note that he inserted them into the text of the entity’s original email, which he italicized (of course, I didn’t send either auditor the actual email, so they don’t know who sent it to me.  And I won’t reveal that information, even if Dick Cheney himself waterboards me).

“Tom, a couple of questions were raised when I read your last post regarding patching of XP. The first is around Entities that have bought the extended service from Microsoft and thus still receive XP patches; are you required to patch?

• You’re required to execute your CIP-007-5 R2.2-2.4 processes to perform the following:
•  R2.2 – Is this patch:
•  Applicable to one of the “Applicable Systems” for your entity; if it is an application specific patch (say for Microsoft Windows Media Player for XP), is that application installed on the “Applicable System”?
• Is it a security patch?  Smart folks will treat every patch as a security patch until the patch source (vendor) explicitly tells them otherwise or it is explicit in the README or release notes with the patch (especially anything that is a “Service Pack” for multiple issues).  If it’s ambiguous, treat it as a security patch and start processing it – you can always inquire in parallel to the vendor to get clarification, but you can’t turn the 35 day hourglass back over if you find out on day 30 it was a security patch all along. 
• R2.3/R2.4
• Apply the patch to all “Applicable Systems” within 35 calendar days, or
• Create a mitigation plan to mitigate the vulnerabilities addressed by each security patch and follow it (R2.4).  
• The most common delay in patching is due to operational concerns or constraints (have to take an outage on all units controlled by the DCS, need to test it on one smaller plant before rolling it out to all plants in the fleet) then the expectation is probably that there would be something put in place in the meantime to “mitigate” (maybe an additional IDS/IPS signature, more frequent review of logs, etc) but that the patch would ultimately be applied. 
• If the patch is deferred “forever”, there would be some evidence expected as to why the patch could not be applied (testing showing failure is the easiest and most common piece).  Simply expressing a fear of patching due to an ambiguous danger to reliability without evidence to back that up would be a dicey play.  Even if your RE didn’t write you up, (you might end up by) having a long mitigation plan with multiple milestones to trip you up if you don’t “implement” it on time (R2.4).  Did I mention having to have TFEs for deferred patches?  In the V3-V5 Transition Guidance NERC released, they identified R2.4 as a TFEable requirement.  All in all, this is a short term gain, long term loss.

This gets a bit more complicated with this question. So you have certain systems from a vendor like GE, Honeywell, or Emerson that are running XP but those vendors are not releasing patches for their XP systems (not saying that all these vendors are not releasing XP patches). Due to some contract agreements, the Entity is not allowed to patch those systems independently - rather the vendor must provide the patch for those systems...and if the vendor isn't because they don't subscribe to the service - it puts the Entity in a precarious position.

• Some of this was why in V5 the Registered Entity is allowed to define (and they should date, define, and document this in writing so that it is above board) their patch source(s) in R2.1.  So if they identify that their DCS vendor is their patch source instead of Microsoft because of their contractual obligation, then they would be able to rely on the vendor for patches. 
• However if they then did not pay for the support agreement/subscription to receive those patches from the same source they had identified in R2.1(whether it was Microsoft or a DCS/EMS/PLC vendor), I don’t think they’d have a leg to stand on – I’d see a PV on CIP-007-5 R2, Part 2.2.  To me, it’s not a far cry from saying that they didn’t receive any patch notifications from their vendors because to save money they didn’t pay the bill for their corporate internet connection for email.  I understand that those agreements aren’t cheap, but this isn’t a cost/benefit model at this stage – that was in the stage where FERC issued the NOPR. Positive note:  It’s something to consider in the procurement cycle for a lot of these systems – if they are going to have a long upgrade cycle and the ongoing cost is considerable or the support agreement is thorny, factor it in to your selection.”

  

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Honeywell.