Caveat Emptor

 Note from Tom: I haven’t mentioned it for a few months, but I’m still working on a book, tentatively titled “Supply chain cybersecurity for critical infrastructure”, with Steven Briggs. The book is going well and we hope to publish it by late January. This is the text of an appendix to the chapter on contract language. Note that I didn’t bother to remove references to other chapters, etc.

In early 2019, the Edison Electric Institute (EEI) issued a document entitled “Model Procurement Contract Language Addressing Cybersecurity Supply Chain Risk”. In May 2020, EEI released version 2.0 of this document, which remains the current version as of the writing of this Chapter. Many NERC entities have paid close attention to this document, although the authors do not know how many have actually decided to include it in contracts for Procurements of BES Cyber Systems and related Products and Services.

There are six sections in the document, addressing the six requirement parts of CIP-013-1 R1.2.1 – R1.2.6; each of those sections is called a “term”, although as we will discuss below, each section in fact contains many additional terms that should be each negotiated separately (if they are in fact really needed at all), not together with many others. While the authors admire the work that was put into preparing this document, we believe that no NERC Entity should incorporate even one of the six terms into their contracts in its entirety, without first considering two points we made earlier in this Chapter:

1.      Every term that you propose for a contract with a Vendor or Supplier should be based on a Risk that you have identified as part of the process described in Chapter xx “Identifying Risks”. Moreover, each term should address only one Risk.

2.      If in your reading you identify a particular term that you would like to include in your contracts, yet there currently is no Risk that corresponds to it, you should first reword the term into a Risk, then add this to your existing list of Risks. Only after that has been done should you add the term to your contract. You need to do this so that you can keep track of the number of Risks that you are actually addressing in your CIP-013-1 compliance program. If you adopt the EEI document in its entirety, you will be adding 40-60 new Risks (by the authors’ estimate) to your CIP-013-1 program.

Remember, for every Risk on your list, you need to a) take steps to assess the Likelihood of the Risk being present in the Supplier’s or Vendor’s environment, usually through a question on a questionnaire; and b) take steps to ensure the Vendor mitigates that Risk, first by obtaining their commitment to do so (in contract terms or via other means like a letter or phone call – which you document afterwards) and second by following up to make sure they keep their commitment.

The main problem with the contract language in the EEI document is that it goes well beyond what is required to address the specific Risks addressed in CIP-013-1 R1.2.1 through R1.2.6. Remember that each “extra” term you require a Vendor to include a) leads to additional legal time and effort, since the vendor is almost always guaranteed to want to negotiate any contract term that you require of them; b) requires you to take steps to verify the vendor is in fact complying with the term; and c) puts you at compliance risk if you do not monitor the Supplier’s compliance with every cybersecurity term in their contract, and follow up to make sure the Supplier complies with each one.

Here is just the first section of the term for R1.2.5, which is titled “Hardware, Firmware, Software, and Patch Integrity and Authenticity”:

(a) Contractor shall establish, document, and implement risk management practices for supply chain delivery of hardware, software (including patches), and firmware provided under this Agreement. Contractor shall provide documentation on its: chain-of-custody practices, inventory management program (including the location and protection of spare parts), information protection practices, integrity management program for components provided by sub-suppliers, instructions on how to request replacement parts, and commitments to ensure that for [negotiated time period] spare parts shall be made available by Contractor.

(b) Contractor shall specify how digital delivery for procured products (e.g., software and data) including patches will be validated and monitored to ensure the digital delivery remains as specified. If Company deems that it is warranted, Contractor shall apply encryption technology to protect procured products throughout the delivery process.

(c) If Contractor provides software or patches to Company, Contractor shall publish or provide a hash conforming to the Federal Information Processing Standard (FIPS) Security Requirements for Cryptographic Modules (FIPS 140-2) or similar standard information on the software and patches to enable Company to use the hash value as a checksum to independently verify the integrity of the software and patches.

(d) Contractor shall identify or provide Company with a method to identify the country (or countries) of origin of the procured Contractor product and its components (including hardware, software, and firmware). Contractor will identify the countries where the development, manufacturing, maintenance, and service for the Contractor product are provided. Contractor will notify Company of changes in the list of countries where 11 product maintenance or other services are provided in support of the procured Contractor product. This notification in writing shall occur at least 180 days prior to initiating a change in the list of countries.

(e) Contractor shall provide a software bill of materials for procured (including licensed) products consisting of a list of components and associated metadata that make up a component.

(f) Contractor shall use or arrange for the use of trusted channels to ship procured products, such as U.S. registered mail and/or tamper-evident packaging for physical deliveries.

(g) Contractor shall demonstrate a capability for detecting unauthorized access throughout the delivery process.

(h) Contractor shall demonstrate chain-of-custody documentation for procured products as determined by Company in its sole discretion and require tamper-evident packaging for the delivery of this hardware.

This is followed by four other sections, three of which are of almost equal length to the first one. They are titled “Patching Governance”, “Viruses, Firmware and Malware”, “End of Life Operating Systems” and “Cryptographic Requirements”.  Remember, all of this is in theory included to address just Requirement Part 1.2.5, which contains about 25 words in total.

One interesting thing the authors have noticed is that in EEI’s entire term for R1.2.5, there seems to be no requirement for the Supplier to provide a means of verifying software authenticity – usually a digital signature – although there is a requirement to verify integrity using a hash value. This omission is very puzzling, since this is the single best Mitigation for the Risk addressed in R1.2.5. So the EEI term for R1.2.5 seems to address at least 40 cybersecurity Mitigations, but not the one that is most essential for compliance with R1.2.5!

On the other hand, we also want to point out that there are many provisions within the EEI document which, even though they do not directly address the two Risks behind R1.2.5 (lack of software authenticity and integrity), nevertheless do address important Risks. Perhaps our favorite is this provision, found in the “Patch Governance” section of the R1.2.5 term:

(e) When third-party hardware, software (including open-source software), and firmware is provided by Contractor to Company, Contractor shall provide or arrange for the provision of appropriate hardware, software, and/or firmware updates to remediate newly discovered vulnerabilities or weaknesses, if applicable to the Company’s use of the third-party product in its system environment, within 30 days of availability from the original supplier and/or patching source. Updates to remediate critical vulnerabilities applicable to the Contractor’s use of the third-party product in its system environment shall be provided within a shorter period than other updates, within [a negotiated time period (e.g., 30, 60, or 90 days)] of availability from the original supplier and/or patching source. If applicable third-party updates cannot be integrated, tested, and made available by Contractor within these time periods, Contractor shall provide or arrange for the provision of recommended mitigations and/or workarounds within 30 days.

Note that this specifically applies to software components. The authors believe it is the first explicit acknowledgement from a power industry organization of the Risks posed by software components. What is the most important prerequisite for monitoring the Supplier’s compliance with this provision? Software bills of materials![i]

It is tempting to say “Well, all of the provisions required in this term are certainly best practices.” That is correct. However, this overlooks the fact that each of the 40-60 provisions in this term should be treated as a separate Risk; and if your organization does not in fact consider each one of these to be a Risk, why are you asking the Supplier to agree to all of these provisions in their contract? As we discussed earlier in this Chapter, you should never ask a Supplier to comply with a contract term – or even ask them a question in a questionnaire – if it does not address a Risk that your organization has identified as important. If the term does not address an important Risk, why are you requiring the term in the first place, especially given how expensive it is to negotiate any contract term?

EEI’s R1.2.5 term requires far more than what CIP-013-1 R1.2.5 itself requires, which is for the Supplier or Vendor to provide a means to verify authenticity and integrity of downloaded software and patches. For example, just paragraph (a) above requires the Supplier to agree to the following provisions, none of which have any direct bearing on software integrity or authenticity:

1.      Chain-of-custody practices;

2.      Inventory management program (including the location and protection of spare parts);

3.      Information protection practices;

4.      Integrity management program for components provided by sub-suppliers; and

5.      Commitments to ensure that, for a negotiated time period, spare parts will be made available by the Contractor. 

These provisions are all just in the first paragraph of the first section of the R1.2.5 contract term. There seem to be over ten additional provisions in the first section. Plus there are four more sections in just the R1.2.5 term, with probably 40-60 total provisions in this one term.^[ii] None of these have any direct connection to software integrity and authenticity.

Remember, each one of these provisions must be negotiated back and forth between you, your lawyers, the Supplier’s lawyers, and the Supplier’s product and technical teams. Supplier or Vendor lawyers will almost never accept a new contract term – that’s not already part of their “standard” contract – without putting up a fight. If you assume there will be at least five hours of legal back-and-forth required for each provision in the term, the contract negotiation for this single term (R1.2.5) will require at least 200 hours of time, or $20,000 at $100 per hour, since the authors believe there are at least 40 implicit provisions in EEI’s R1.2.5 term.

For example, let’s look at the first of the provisions listed above: chain-of-custody practices. This refers to the Risk that a hardware Product will be hijacked at some point in the process of shipment to the customer and a counterfeit Product with malware inserted will be substituted (or maybe the original Product will be opened and a counterfeit hardware component with malware will be inserted). This is certainly a valid Risk, but is it a likely one? It might be, but that is for your organization to decide. If you decide that the Likelihood of this Risk occurring for this Supplier is high, then by all means you might want to include this provision in your contract.[iii]

However, in the authors’ opinion, each provision should be introduced as a separate term, not included in a single term with 59 other provisions. In this case, the term might read “You will provide our organization with chain-of-custody records for any shipment of a hardware Product to us.” That way, the term can be addressed on its own merits in your negotiations. Perhaps the Supplier’s lawyers will say “We can absolutely provide chain-of-custody information for every shipment. But this will make the shipment more expensive by a factor of X (e.g. 50%).” At that point, you and your legal team will be able to make the decision whether or not this term is worth including in the contract, given the tradeoff between cost and risk reduction. If this provision is included in a single term with 40-60 others (as in the case of the EEI term), it is very hard, if not impossible, to make decisions like this.

However, negotiating the contract is just the beginning of what your organization will need to do to address each term that you have been able to include in the contract. Using the above example, you also need to follow up with the Vendor to make sure they have done what they promised to do. You will need to make sure that the Vendor provides chain-of-custody data for every shipment to your organization. And if you find that in any shipment, they have not done this, you will need to contact them to either get the documentation or reprimand them for not collecting chain-of-custody information for this particular shipment.

Moreover, if your organization has to comply with NERC CIP-013-1, you will need to make sure you have documentation that the Vendor provided chain-of-custody data for every shipment to you – and that if they did not do this for a shipment, you followed up with them to request it. Of course, you will not be in violation of CIP-013-1 if the Vendor did not provide the data even after you contacted them, or if they told you that in this case they had neglected to obtain that information. However, if you did not even follow up with them about this matter, then you might well be found in violation, because your supply chain cybersecurity risk management Plan said – or should have said – that you will follow up in any case where the Vendor or Supplier does not fulfill a provision they agreed to.

To summarize this Appendix, you need to be very careful about requiring that Suppliers or Vendors comply with the EEI contract terms, because they go far beyond what Requirement Parts R1.2.1 – R1.2.6 require. You need to look at each of the provisions in each of the terms and decide whether or not the Risk that it addresses has a high Likelihood of being present with a Supplier or Vendor. If the Likelihood is high, this might justify your going through the expense of time and money required to negotiate that particular provision, and then following up later on with the Supplier or Vendor to make sure they actually do what they agreed to. And if you do not believe the Risk addressed by a provision is important enough to warrant the required time and expense of including it in the contract, then by all means you should not ask the Supplier to agree to it in the first place.

---

[i] Note that there is an explicit requirement for SBoMs in paragraph (e) of the excerpt earlier in this Appendix. Tom has been told that NERC entities are getting a lot of pushback from Suppliers on this particular requirement. This is understandable, given that there are many questions about how SBoMs will be produced, managed and implemented. The answers to most of these questions will hopefully be clarified in the upcoming proof of concept for use of SBoMs in the electric power industry. 

[ii] There are some very technical requirements embedded in this term, like: 

(i) The system implementation includes the capability for configurable cryptoperiods (the life span of cryptographic key usage) in accordance with the Suggested Cryptoperiods for Key Types found in Table 1 of NIST 800-57 Part 1, as may be amended.

(ii) The key update method supports remote re-keying of all devices within [a negotiated time period(s)] as part of normal system operations.

(iii) Emergency re-keying of all devices can be remotely performed within 30 days. 

Just complying with these three requirements will place a big burden on the Supplier. As with all requirements you place on a Supplier, you need to assume there will be some cost to their complying with every term. You also have to assume that your organization will end up paying that cost in one way or another. As Tom’s former economics professor Dr. Milton Friedman always said, “There is no such thing as a free lunch.” 

[iii] Note that the Likelihood of this Risk occurring for a software Supplier is zero, since chain-of-custody only applies to physical shipments. It has been a long time since a Supplier needed to ship you a box, in order for you to be able to install software that you have paid for! 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.