My most recent post discussed the important question of the role of procurement contract language in CIP-013. My general point in the post was that I think it is a big mistake for a NERC entity to focus their CIP-013 compliance efforts solely, or even primarily, on getting the vendor to agree to incorporate particular language in their contract. After that post came out, an auditor wrote in to suggest some of my statements needed clarification. We exchanged a number of emails, and I’ve identified four important points that he raised.
I. Tales from Beyond the Scope
Early in the post, I quoted from a note that appears after CIP-013 R2: “the following issues are beyond the scope of Requirement R2: (1) the actual terms and conditions of a procurement contract; and (2) vendor performance and adherence to a contract.” Regarding (1), I had this to say: “The first part of the second sentence…says that the terms and conditions of a procurement contract are ‘beyond the scope’ of R2. What does this phrase mean? It specifically means that contract language is not the subject of R2 – and if an auditor asks to see an entity’s vendor contract, they can be turned down because it isn’t in scope.”
It turns out the truth is more nuanced. To quote from the auditor’s email: “For compliance finding purposes (i.e., determination of a PNC) the contract is out of scope. However, if the entity states that the vendor will (or has promised to) do something, such as advise its customers within x days of identification of a vulnerability in its products, I want to see something from the vendor that demonstrates that, not just a verbal statement of assurance from the audited entity. In many cases, the MSA, SLA, or contract T&C will address that expectation and it is appropriate for the entity to provide the written vendor assurances.”
What does this mean? It means that, if you assert that a vendor has promised to do X in a contract, the auditor can (and most likely will) ask for evidence of this. If you refuse to show him the contact and point to phrase (1) above, the auditor will probably say “OK, Mr. Smart Guy. Show me some other evidence they’ve made this promise.” At this point, you’ll wish you had a letter or email from the vendor making the same promise; presumably, you won’t feel you’re violating some legal principle by not showing him one of these.
But what if your only evidence is the contract – and you’ve been told by your lawyers never to show the contract to the auditors, even if they jump up and down and promise divine retribution for this refusal? In this case, the auditor technically can’t issue a PNC (potential non-compliance) finding, but they can certainly list in their report that you were unable to substantiate your assertion regarding the promise by vendor Y to do X.
So if you are going to be at all constrained in your ability to show a contract to the auditors, why even take the contract route in the first place? Get a letter from your vendor. This constitutes obtaining a vendor promise just as much as contract language does, and it will make your audit go much more smoothly. As the auditor pointed out above, other documents from the procurement process, such as an RFP stating terms the vendor agreed to, a Master Services Agreement or a Service Level Agreement, might also constitute evidence of the vendor’s promise.
II. Trying is Everything
What happens if you beg your vendor to promise to do something and they refuse – yet dropping them as a vendor is out of the question because of the cost of making the change, the unavailability of close substitutes, etc? Will you receive a PNC because of this? The auditor further states in the same email: “The key point in the contractual language exclusion is that the auditor cannot hold the entity at fault with a PNC if the entity was not successful in getting the vendor to agree to a CIP-013 requirement expectation. And, failure of the vendor to comply with its contractual obligations or promises does not constitute a compliance failure on the part of the audited entity. But the entity needs to demonstrate how it addressed the expectation during its procurement process. The Plan should address how the entity will go about asking and the pre-contract negotiations and perhaps elements of the contract itself will demonstrate the Plan was followed.”
In other words, your supply chain cyber security risk management plan (which R1 requires you to develop, of course) must describe how you will go about requesting that your vendor promise to do something, as well as the different vehicles available for the vendor to document that promise. For each vendor, you should document how you tried to get them to make the promise. If they agreed to make it, then you need some document to verify that. If they didn’t agree, you should document that fact as well. To quote the auditor again: “The entity does need to keep records of any communication with the vendor; preferably date, time, with whom, and a summary of the discussion or agreement. I like emails better than phone logs for the obvious reason; emails are better evidence than one-sided hand-written records.”
The auditor also points out that you don’t have to simply throw up your hands if the vendor tells you “no” the first time you ask for something, even though you know that finding another vendor for the same product isn’t an option, for whatever reason. How about negotiating, using tools that you (as a compliance or cyber security person, not a supply chain or legal person) have access to? The auditor also says “…if I cannot get, for example, prompt notifications of vendor staff terminations, then maybe I should reconsider my willingness to allow electronic or unescorted physical access to my BES Cyber Assets. That is called a negotiation point.” Good idea!
Before I leave this topic, I want to point out something else: Your supply chain cyber security risk management plan must list risks and how you intend to mitigate them. Your first step to mitigate a particular vendor risk may well be getting the vendor to promise to take certain steps that will mitigate the risk. But if the vendor absolutely refuses to do this, are you now off the hook for that particular risk? Will your mitigation be “Well, we tried our best, but that vendor is as stubborn as a senior citizen mule!”?
Unfortunately, this isn’t mitigation – this is making an excuse for non-mitigation. There should usually be some step you can take to mitigate the risk in question, which doesn’t require the vendor to do anything at all. Let’s go back to the example the auditor just used, the risk that a vendor won’t provide prompt notification when an employee leaves who had access to your systems (this is the risk addressed by R1.2.3). The auditor suggested that, if the vendor refused to promise to give you this notification, you might then refuse to give their employees electronic access and/or unescorted physical access to their systems at your facilities. This isn’t just a negotiation tactic. It’s also a legitimate mitigation for this risk, since an employee who had just been fired and wanted to take out his anger on your systems would have a hard time doing so without electronic or unescorted physical access. He/she would have to come onsite and be escorted to the systems!
III. Just Do It!
In my last post, I made this assertion: “I don’t think the entity needs to show any procurement documents at all, because I don’t think the entity needs to prove that they ‘initiated correspondence’ with the vendor to implement certain controls, if they can show other evidence of compliance. And what other evidence can the entity show? The best evidence will demonstrate that the vendor is actually doing what they were supposed to do! After all, isn’t the goal of CIP-013 to protect the BES against supply chain risks, not simply document that you’ve asked your vendors to do certain things?”
In this statement, it seems I was flat out wrong. While it is important that you gather some evidence to show the vendor is actually keeping a particular promise[i], that evidence alone isn’t sufficient. In his email, the auditor further stated “I do have a problem with a “handshake” agreement. I cannot evaluate that – it is essentially an attestation in the absence of supporting evidence. Yes, the entity can show me that they received a notification of a vendor staff termination. But, what prompted the notification? Was there a contractual expectation or a vendor declaration that such (notification) would be sent by the vendor consistently and in a timely manner? How do I know that the vendor will inform me of every applicable termination or reassignment? Similarly, how do I know that the vendor will apprise me of every security vulnerability in their products?”
In other words, if a vendor makes a promise, it needs to be made in writing in some way (although contract language is just one of those ways). You also need to document either a) that the vendor is keeping that promise (using emails, etc); b) that you tried to get the vendor to promise something but you were turned down; or c) the vendor promised to do something but didn’t keep the promise. If either b) or c) is the case, you also need to document some other mitigation steps you took to address the risk in question – assuming there is a way you could mitigate the risk without the vendor’s cooperation (obviously, as required in R1.2.4, if the vendor absolutely refuses to tell customers about vulnerabilities it knows about but hasn’t yet made public, you will not be able to take steps to address those vulnerabilities - except perhaps by simply tightening up the normal security measures that would be applied to that system).
IV. The Big Guys
Finally, the auditor also made a good point about how you mitigate vendor risks when it comes to big vendors like Microsoft, who obviously aren’t going to change their contract language for a single electric utility, no matter how large it might be. He says “…the plan can certainly address how procurement from the huge international players like Microsoft, Red Hat, Cisco, and Oracle will be handled. I would prefer to see you get your networking gear from a Cisco channel partner, not eBay or some unknown used equipment reseller. I would expect you to download software from a known, reputable source, not Sammy’s Software Shack. Hopefully the vendor provides a digital signature (MD5 or SHA(2)-256 hash) of its software for verification purposes. It is all about risk management. You cannot eliminate all risk, but you can certainly reasonably mitigate it.”
If you would like to comment on what you have read here, I would love to hear from you. Please email me at firstname.lastname@example.org. Please keep in mind that Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post. To discuss this, you can email me at the same address or call me at 312-515-8996.
[i] Although you definitely don’t have to document every instance where the vendor kept the particular promise in question, as would be the case if we were talking about a prescriptive requirement like CIP-007 R2.