Caveat Emptor

 Note from Tom: I haven’t mentioned it for a few months, but I’m still working on a book, tentatively titled “Supply chain cybersecurity for critical infrastructure”, with Steven Briggs. The book is going well and we hope to publish it by late January. This is the text of an appendix to the chapter on contract language. Note that I didn’t bother to remove references to other chapters, etc.

In early 2019, the Edison Electric Institute (EEI) issued a document entitled “Model Procurement Contract Language Addressing Cybersecurity Supply Chain Risk”. In May 2020, EEI released version 2.0 of this document, which remains the current version as of the writing of this Chapter. Many NERC entities have paid close attention to this document, although the authors do not know how many have actually decided to include it in contracts for Procurements of BES Cyber Systems and related Products and Services.

There are six sections in the document, addressing the six requirement parts of CIP-013-1 R1.2.1 – R1.2.6; each of those sections is called a “term”, although as we will discuss below, each section in fact contains many additional terms that should be each negotiated separately (if they are in fact really needed at all), not together with many others. While the authors admire the work that was put into preparing this document, we believe that no NERC Entity should incorporate even one of the six terms into their contracts in its entirety, without first considering two points we made earlier in this Chapter:

1.      Every term that you propose for a contract with a Vendor or Supplier should be based on a Risk that you have identified as part of the process described in Chapter xx “Identifying Risks”. Moreover, each term should address only one Risk.

2.      If in your reading you identify a particular term that you would like to include in your contracts, yet there currently is no Risk that corresponds to it, you should first reword the term into a Risk, then add this to your existing list of Risks. Only after that has been done should you add the term to your contract. You need to do this so that you can keep track of the number of Risks that you are actually addressing in your CIP-013-1 compliance program. If you adopt the EEI document in its entirety, you will be adding 40-60 new Risks (by the authors’ estimate) to your CIP-013-1 program.

Remember, for every Risk on your list, you need to a) take steps to assess the Likelihood of the Risk being present in the Supplier’s or Vendor’s environment, usually through a question on a questionnaire; and b) take steps to ensure the Vendor mitigates that Risk, first by obtaining their commitment to do so (in contract terms or via other means like a letter or phone call – which you document afterwards) and second by following up to make sure they keep their commitment.

The main problem with the contract language in the EEI document is that it goes well beyond what is required to address the specific Risks addressed in CIP-013-1 R1.2.1 through R1.2.6. Remember that each “extra” term you require a Vendor to include a) leads to additional legal time and effort, since the vendor is almost always guaranteed to want to negotiate any contract term that you require of them; b) requires you to take steps to verify the vendor is in fact complying with the term; and c) puts you at compliance risk if you do not monitor the Supplier’s compliance with every cybersecurity term in their contract, and follow up to make sure the Supplier complies with each one.

Here is just the first section of the term for R1.2.5, which is titled “Hardware, Firmware, Software, and Patch Integrity and Authenticity”:

(a) Contractor shall establish, document, and implement risk management practices for supply chain delivery of hardware, software (including patches), and firmware provided under this Agreement. Contractor shall provide documentation on its: chain-of-custody practices, inventory management program (including the location and protection of spare parts), information protection practices, integrity management program for components provided by sub-suppliers, instructions on how to request replacement parts, and commitments to ensure that for [negotiated time period] spare parts shall be made available by Contractor.

(b) Contractor shall specify how digital delivery for procured products (e.g., software and data) including patches will be validated and monitored to ensure the digital delivery remains as specified. If Company deems that it is warranted, Contractor shall apply encryption technology to protect procured products throughout the delivery process.

(c) If Contractor provides software or patches to Company, Contractor shall publish or provide a hash conforming to the Federal Information Processing Standard (FIPS) Security Requirements for Cryptographic Modules (FIPS 140-2) or similar standard information on the software and patches to enable Company to use the hash value as a checksum to independently verify the integrity of the software and patches.

(d) Contractor shall identify or provide Company with a method to identify the country (or countries) of origin of the procured Contractor product and its components (including hardware, software, and firmware). Contractor will identify the countries where the development, manufacturing, maintenance, and service for the Contractor product are provided. Contractor will notify Company of changes in the list of countries where 11 product maintenance or other services are provided in support of the procured Contractor product. This notification in writing shall occur at least 180 days prior to initiating a change in the list of countries.

(e) Contractor shall provide a software bill of materials for procured (including licensed) products consisting of a list of components and associated metadata that make up a component.

(f) Contractor shall use or arrange for the use of trusted channels to ship procured products, such as U.S. registered mail and/or tamper-evident packaging for physical deliveries.

(g) Contractor shall demonstrate a capability for detecting unauthorized access throughout the delivery process.

(h) Contractor shall demonstrate chain-of-custody documentation for procured products as determined by Company in its sole discretion and require tamper-evident packaging for the delivery of this hardware.

This is followed by four other sections, three of which are of almost equal length to the first one. They are titled “Patching Governance”, “Viruses, Firmware and Malware”, “End of Life Operating Systems” and “Cryptographic Requirements”.  Remember, all of this is in theory included to address just Requirement Part 1.2.5, which contains about 25 words in total.

One interesting thing the authors have noticed is that in EEI’s entire term for R1.2.5, there seems to be no requirement for the Supplier to provide a means of verifying software authenticity – usually a digital signature – although there is a requirement to verify integrity using a hash value. This omission is very puzzling, since this is the single best Mitigation for the Risk addressed in R1.2.5. So the EEI term for R1.2.5 seems to address at least 40 cybersecurity Mitigations, but not the one that is most essential for compliance with R1.2.5!

On the other hand, we also want to point out that there are many provisions within the EEI document which, even though they do not directly address the two Risks behind R1.2.5 (lack of software authenticity and integrity), nevertheless do address important Risks. Perhaps our favorite is this provision, found in the “Patch Governance” section of the R1.2.5 term:

(e) When third-party hardware, software (including open-source software), and firmware is provided by Contractor to Company, Contractor shall provide or arrange for the provision of appropriate hardware, software, and/or firmware updates to remediate newly discovered vulnerabilities or weaknesses, if applicable to the Company’s use of the third-party product in its system environment, within 30 days of availability from the original supplier and/or patching source. Updates to remediate critical vulnerabilities applicable to the Contractor’s use of the third-party product in its system environment shall be provided within a shorter period than other updates, within [a negotiated time period (e.g., 30, 60, or 90 days)] of availability from the original supplier and/or patching source. If applicable third-party updates cannot be integrated, tested, and made available by Contractor within these time periods, Contractor shall provide or arrange for the provision of recommended mitigations and/or workarounds within 30 days.

Note that this specifically applies to software components. The authors believe it is the first explicit acknowledgement from a power industry organization of the Risks posed by software components. What is the most important prerequisite for monitoring the Supplier’s compliance with this provision? Software bills of materials![i]

It is tempting to say “Well, all of the provisions required in this term are certainly best practices.” That is correct. However, this overlooks the fact that each of the 40-60 provisions in this term should be treated as a separate Risk; and if your organization does not in fact consider each one of these to be a Risk, why are you asking the Supplier to agree to all of these provisions in their contract? As we discussed earlier in this Chapter, you should never ask a Supplier to comply with a contract term – or even ask them a question in a questionnaire – if it does not address a Risk that your organization has identified as important. If the term does not address an important Risk, why are you requiring the term in the first place, especially given how expensive it is to negotiate any contract term?

EEI’s R1.2.5 term requires far more than what CIP-013-1 R1.2.5 itself requires, which is for the Supplier or Vendor to provide a means to verify authenticity and integrity of downloaded software and patches. For example, just paragraph (a) above requires the Supplier to agree to the following provisions, none of which have any direct bearing on software integrity or authenticity:

1.      Chain-of-custody practices;

2.      Inventory management program (including the location and protection of spare parts);

3.      Information protection practices;

4.      Integrity management program for components provided by sub-suppliers; and

5.      Commitments to ensure that, for a negotiated time period, spare parts will be made available by the Contractor. 

These provisions are all just in the first paragraph of the first section of the R1.2.5 contract term. There seem to be over ten additional provisions in the first section. Plus there are four more sections in just the R1.2.5 term, with probably 40-60 total provisions in this one term.^[ii] None of these have any direct connection to software integrity and authenticity.

Remember, each one of these provisions must be negotiated back and forth between you, your lawyers, the Supplier’s lawyers, and the Supplier’s product and technical teams. Supplier or Vendor lawyers will almost never accept a new contract term – that’s not already part of their “standard” contract – without putting up a fight. If you assume there will be at least five hours of legal back-and-forth required for each provision in the term, the contract negotiation for this single term (R1.2.5) will require at least 200 hours of time, or $20,000 at $100 per hour, since the authors believe there are at least 40 implicit provisions in EEI’s R1.2.5 term.

For example, let’s look at the first of the provisions listed above: chain-of-custody practices. This refers to the Risk that a hardware Product will be hijacked at some point in the process of shipment to the customer and a counterfeit Product with malware inserted will be substituted (or maybe the original Product will be opened and a counterfeit hardware component with malware will be inserted). This is certainly a valid Risk, but is it a likely one? It might be, but that is for your organization to decide. If you decide that the Likelihood of this Risk occurring for this Supplier is high, then by all means you might want to include this provision in your contract.[iii]

However, in the authors’ opinion, each provision should be introduced as a separate term, not included in a single term with 59 other provisions. In this case, the term might read “You will provide our organization with chain-of-custody records for any shipment of a hardware Product to us.” That way, the term can be addressed on its own merits in your negotiations. Perhaps the Supplier’s lawyers will say “We can absolutely provide chain-of-custody information for every shipment. But this will make the shipment more expensive by a factor of X (e.g. 50%).” At that point, you and your legal team will be able to make the decision whether or not this term is worth including in the contract, given the tradeoff between cost and risk reduction. If this provision is included in a single term with 40-60 others (as in the case of the EEI term), it is very hard, if not impossible, to make decisions like this.

However, negotiating the contract is just the beginning of what your organization will need to do to address each term that you have been able to include in the contract. Using the above example, you also need to follow up with the Vendor to make sure they have done what they promised to do. You will need to make sure that the Vendor provides chain-of-custody data for every shipment to your organization. And if you find that in any shipment, they have not done this, you will need to contact them to either get the documentation or reprimand them for not collecting chain-of-custody information for this particular shipment.

Moreover, if your organization has to comply with NERC CIP-013-1, you will need to make sure you have documentation that the Vendor provided chain-of-custody data for every shipment to you – and that if they did not do this for a shipment, you followed up with them to request it. Of course, you will not be in violation of CIP-013-1 if the Vendor did not provide the data even after you contacted them, or if they told you that in this case they had neglected to obtain that information. However, if you did not even follow up with them about this matter, then you might well be found in violation, because your supply chain cybersecurity risk management Plan said – or should have said – that you will follow up in any case where the Vendor or Supplier does not fulfill a provision they agreed to.

To summarize this Appendix, you need to be very careful about requiring that Suppliers or Vendors comply with the EEI contract terms, because they go far beyond what Requirement Parts R1.2.1 – R1.2.6 require. You need to look at each of the provisions in each of the terms and decide whether or not the Risk that it addresses has a high Likelihood of being present with a Supplier or Vendor. If the Likelihood is high, this might justify your going through the expense of time and money required to negotiate that particular provision, and then following up later on with the Supplier or Vendor to make sure they actually do what they agreed to. And if you do not believe the Risk addressed by a provision is important enough to warrant the required time and expense of including it in the contract, then by all means you should not ask the Supplier to agree to it in the first place.

---

[i] Note that there is an explicit requirement for SBoMs in paragraph (e) of the excerpt earlier in this Appendix. Tom has been told that NERC entities are getting a lot of pushback from Suppliers on this particular requirement. This is understandable, given that there are many questions about how SBoMs will be produced, managed and implemented. The answers to most of these questions will hopefully be clarified in the upcoming proof of concept for use of SBoMs in the electric power industry. 

[ii] There are some very technical requirements embedded in this term, like: 

(i) The system implementation includes the capability for configurable cryptoperiods (the life span of cryptographic key usage) in accordance with the Suggested Cryptoperiods for Key Types found in Table 1 of NIST 800-57 Part 1, as may be amended.

(ii) The key update method supports remote re-keying of all devices within [a negotiated time period(s)] as part of normal system operations.

(iii) Emergency re-keying of all devices can be remotely performed within 30 days. 

Just complying with these three requirements will place a big burden on the Supplier. As with all requirements you place on a Supplier, you need to assume there will be some cost to their complying with every term. You also have to assume that your organization will end up paying that cost in one way or another. As Tom’s former economics professor Dr. Milton Friedman always said, “There is no such thing as a free lunch.” 

[iii] Note that the Likelihood of this Risk occurring for a software Supplier is zero, since chain-of-custody only applies to physical shipments. It has been a long time since a Supplier needed to ship you a box, in order for you to be able to install software that you have paid for! 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

An opportunity to be part of the solution

In my post last Thursday, I explained why one of the biggest cybersecurity risks worldwide today is the risk posed by vulnerabilities in software components. It is very likely that the majority of these risks aren’t being patched today because software consumers – and very likely most suppliers – can’t easily track those vulnerabilities. Thus, they simply don’t know they’re present. A good example of this is the Ripple 20 vulnerabilities, which are estimated to have affected hundreds of millions of devices worldwide – yet there was almost no way that a supplier could learn whether the Treck IP stack component was included in a given software package, since knowing that would require knowing all of the components of the components of the components, etc…of their software.

But there is hope. As I’ve written a lot lately, there is now recognition that the widespread availability and acceptance of software bills of materials (SBoMs) will finally allow both suppliers and consumers of software to understand what the software packages they use are made of, then be able to learn about vulnerabilities found in the components of the software packages. SBoMs are being used for some limited purposes today, and there is agreement on their formats (there are three primary SBoM formats now, with one or two more being developed. Each format can be easily transformed to each of the others, so there is no need to standardize on a single format for all uses).

So the good news is that there aren’t any technical barriers to the widespread use of SBoMs. However, there are definitely barriers, and they’re quite simple to describe:

1.      Producing and continually updating SBoMs will require a sustained effort by all software suppliers, of all sizes. The suppliers are reluctant to make that effort until they are sure that their customers (both current and future) will see value in having SBoMs.

2.      Organizations that use software (i.e. just about every organization on the planet) aren’t sure how they could use SBoMs if they were available.

It’s easy to see the solution here: The software suppliers need to be convinced there’s demand for SBoMs, and the organizations that use software need to be convinced that having SBoMs will allow them to greatly increase the security of the software they use. That’s the good news. The bad news is that this is a classic chicken-and-egg situation: Suppliers won’t produce SBoMs in quantity until they know their customers will use them, and customers are reluctant to invest the time required to learn how to use SBoMs if they’re not sure their suppliers will in fact start producing them regularly.

This is a rough description of the problem facing the National Technology and Information Administration (NTIA) of the US Department of Commerce. I described the important work that group is doing in this post. Specifically, Dr. Allan Friedman of the NTIA is leading a “multistakeholder process” called the Software Component Transparency Initiative, whose goal is to make the production and use of SBoMs a normal process, not a rare occurrence like a sighting of Bigfoot.

As I described in that post, this Initiative will not issue regulations or even guidelines for production and consumption of SBoMs. Rather their main tool is a Proof of Concept (PoC), in which a small number of software suppliers and users within a particular industry develop and test procedures for producing and using SBoMs, and share what they’ve learned with all other parties involved in the NTIA initiative.

The primary use case for SBoMs so far in the PoCs has been for an end user organization to learn about vulnerabilities that have been identified in components of software packages used by the organization. So far, there have been two successful PoCs in the healthcare industry (focused on medical devices used in hospitals, like infusion pumps); a third (more specifically, the second iteration of the second PoC, if you’re keeping score at home) is now under way. A PoC for the automotive industry (led by the Auto-ISAC) is just getting underway.

Why do the PoCs focus on particular industries, rather than some other way of dividing up the problem – e.g. having separate proofs of concept for organizations whose names begin with the letters A, B, etc? The reason is that this allows NTIA to break through the chicken-and-egg problem described above, by having software suppliers for a particular industry sitting across a (virtual) table from software users from the same industry. The suppliers will hopefully become convinced that their customers want to see SBoMs; in other words, “If you build them, they will come.” And the customers will see that they can use SBoMs when they’re widely available.

I hinted in the post linked above that a PoC would probably be coming soon for the electric power industry. Now I don’t have to hint: A PoC is coming under the auspices of the NTIA (although one or two other organizations may play a leadership/sponsorship role as well – not that the PoC is in any way an expensive undertaking).

The PoC itself could start early in 2021, but right now I believe that Dr. Friedman of NTIA is trying to identify organizations that are interested in learning more about this (he’s certainly not asking for commitments now, of course!). I see there to be the need and opportunity for four types of participants:

1.      Suppliers of software used in the OT side of the power industry (including software used in BES Cyber Systems, of course – but hardly limited to BCS software). This can include suppliers of hardware devices that include software - relays, RTUs, etc.

2.      Power industry organizations that use OT software. This includes electric utilities (of any size) and independent power producers, as well as government-owned power marketing organizations. It also includes associations of those organizations, such as the E-ISAC, NATF, EEI, NRECA, APPA, and EPSA.[i]

3.      Service providers, who may sit “in the middle” between suppliers and users of SBoMs. This group plays an important role. As I described in two posts three weeks ago (found here and here), there are some important problems that need to be solved before SBoMs can be widely used – and there are other problems besides these two. Rather than have every electric utility solve these problems on its own, service providers can address them for all of their customers at once. They will not “solve” them once and for all, but they can at least provide a good enough workaround so that use of SBoMs can move forward, at the same time as research continues on the ultimate solutions.

4.      Observers. The above three types of participants will all need to sign an NDA (to be worked out among the participants at the beginning of the PoC). In return, they will be the only organizations allowed to see the SBoMs produced for the PoC. However, the results and lessons learned from the PoC will be shared with anybody who is interested, with no restrictions other than that they must be a user of electricity (and I assume that everyone who reads this post is in fact a user of electricity, not living off the grid in some place like the Yukon). There will be regular meetings of the PoC group to which all observers are invited (the healthcare PoC conducts these meetings weekly), where issues and solutions will be discussed. The PoC group will also draw up documents to inform the outside world of what they’ve learned.

I want to point out that, while the NTIA PoCs focus on particular industries, there is really nothing very significant that’s unique to any industry, when it comes to production or use of SBoMs, In fact, this was the conclusion of the first healthcare PoC, since when they started working in 2018 it wasn’t clear that this would be the case.

The most important implication of this is that the power industry PoC won’t start from ground zero, like the first healthcare PoC did. The lessons learned in the healthcare PoCs were all carefully documented; the power PoC can build on what they have done and go beyond it. Moreover, the people participating in the healthcare PoC are more than willing to coach subsequent PoCs on what they did and the mistakes they made (in fact, they’re providing coaching sessions for the Autos PoC now. The first was last week). In turn for receiving all of this help, the power PoC participants will be expected to share what they learned with the other groups participating in the NTIA initiative.

However, the primary goal of the power PoC won’t be to advance the frontiers of SboM practice in general: it will be to demonstrate a workable use case for SBoMs in the power industry, that can be implemented by the industry now. It will definitely not be a fully automated use case, so it won’t immediately scale to say hundreds of suppliers. But because the suppliers have to learn about this, too (in fact, they have much more learning to do than the users), this may be enough for the moment. 

One good thing is that it’s useful just to have some SboM information for some software products; you don’t need complete information, and you don’t need it for all products that you use (e.g. just knowing the names of ten components of a software package you use regularly is better than not knowing any. The average software product has 135 components, according to a 2020 study by Sonatype). Just that will allow an organization to learn much more about the vulnerabilities they face in software components than they know now (which is very little, of course).

There’s another important reason why you should consider joining the power PoC: You will be part of the group defining the solution to the software transparency problem. SBoMs are coming, without a doubt. I quoted Gartner in the first part of this post: “By 2024, the provision of a detailed, regularly updated software bill of materials by software vendors will be a non-negotiable requirement for at least half of enterprise software buyers..”

I suspect there will be a webinar in December for interested parties. If you want to hear about that and any other PoC information, please drop an email to Allan at afriedman@ntia.doc.gov. If you want to cc me, you can do that, but you don’t have to. And if you’d like to have a conversation about this now, let me know. I’d be glad to do that.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] Note I didn’t list NERC here because I’m assuming they may – mistakenly, IMHO – convince themselves that their Rules of Procedure don’t allow them to participate in the PoC. I am sure NTIA would love to see NERC participate directly in the PoC if they can, but in any case, I hope they will at least be observers.

It looks like there’s been a big power grid hack in India

Kevin Perry just forwarded me this story about a major grid outage in Mumbai, India on October 13. It has now been identified as caused by a cyberattack after a big investigation (it’s nice that the authorities didn’t declare it a cyberattack up front, but actually did the hard legwork to confirm their suspicion before they announced it). It sounds like there will be a news conference soon that will reveal more information.

I note in one of the two Indian news stories linked in the Security Week article that “hackers have been trying to target the country’s power utilities since February”. In June, there was “a swarm of 40,000-plus hacking attacks by non-state groups (I wonder how they know they were non-state groups) purportedly operating from China..”

If so, this is the first I’ve heard of Chinese actors even seriously targeting a power grid, let alone causing an outage. I thought that was Russia’s job.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

An opportunity – part I

Note from Tom 12/8/2021: I am currently reconsidering my opinions on VEX. I'm not going to withdraw any of my posts on the subject, but frankly I don't recommend that you spend much - if any - time reading this or my other posts that addressed VEX. I expect to update this notice within at most a month.

I am pleased to announce that members of the electric power cybersecurity community are invited to participate in an important upcoming event: A Proof of Concept for software bills of materials (SBoMs) in the power industry. I will describe the PoC in part II of this post. In part I, I want to make the case (better than I have before) for why this is important.

I recently wrote that I have lately fallen in with a bad crowd: the Software Transparency Initiative of the National Telecommunications and Information Administration (NTIA). The NTIA’s job is to promote usage of important new technologies, without trying to regulate them in any way. They do this by convening “multistakeholder processes” that gather the different stakeholders in the technology, to figure out what are the barriers to adopting the technology and how they can be overcome.

The Software Transparency Initiative is focused on an important problem in cybersecurity: The average software product contains a large number of components; one recent study by Sonatype said that number is 135, but some software products have thousands of components – and 90% of these are open source. In fact, believe it or not, a different recent study by Synopsys found that 70% of the average software product consists of open source code. This means that, rather than speak of components as isolated “islands” of code in a sea of code written by the supplier of the product, it is almost more accurate to speak of a software product as consisting of a large number of components, with some “glue” provided by the product supplier to hold the components together.

Of course, this in itself doesn’t constitute a problem. The problem arises because, just as the code written by the supplier itself has vulnerabilities, each of the components can have vulnerabilities as well. Let’s say the product itself has a 1% likelihood of having a serious vulnerability at a given time, and that each of its 135 components has that same likelihood[i]. If you don’t consider the components, you would say that the software product has a 1% likelihood of having a serious vulnerability. But if each of the components also has a 1% likelihood, this means the likelihood there is a serious vulnerability in the product is 136% - i.e. it’s a certainty.

Fortunately, as discussed in this post, probably only about one in 20 of those component vulnerabilities is actually exploitable. This means the likelihood that there will be a serious vulnerability in this product or one of its components is about 8%. This is still eight times more than the likelihood that the product itself will have a serious vulnerability, if you don’t take the components into account at all.

But even this increased likelihood doesn’t itself constitute a problem, as long as you can count on your supplier to identify and patch all of these component vulnerabilities. And as long as you’re sure that, if there ever is a vulnerability that the supplier can’t immediately patch for some reason, you can count on their letting you know about it and at least providing you with some workarounds to mitigate the vulnerability.

However, the problem is that you can’t count on your supplier to a) quickly learn about new vulnerabilities in any one of the 135 components in their product, and b) take quick action to mitigate it, through a patch or other means. Here are some reasons why that is the case:

1.      Sometimes (hopefully not too often), the supplier doesn’t even know all of the components in their software.

2.      If they do know all of the components, they don’t receive prompt notice of a vulnerability in each component. Remember, 90% of components are open source, so there is no company that sends them a notice (and especially not a patch) whenever there’s a new vulnerability in the component. It’s up to the software developer to regularly visit the site of the community that developed the product to determine if there are any new vulnerabilities identified, as well as any new patches available (although sometimes they may get an automatic notice).

3.      In theory, all of these vulnerabilities in open source and third party components should appear in the National Vulnerability Database (NVD), meaning that the supplier should easily be able to find at any time if a component has a vulnerability, right? Wrong. One of the biggest problems with components is the naming problem, which I wrote about in this post. The name that a supplier uses for a component is almost never the CPE name that is used in the NVD. Since that name requires knowledge of the exact version number and the exact string of words that describes the product, you will get a null result if you’re just one or two characters off. Say that you know you’re looking for version 2.3.4 and you entered “vsn. 2.3.4”. But the CPE name actually reads “v. 2.3.4”.  You won’t find anything in your search.

4.      And suppose the supplier does find the right component name and finds there is a serious vulnerability (CVE) for that component. Will they immediately develop a patch and send it to you? Some suppliers might and some suppliers might not. A 2017 study by Veracode found that “A mere 52 percent of companies reported they provide security fixes to components when new security vulnerabilities are discovered.” In other words, there’s just as much chance that a supplier won’t provide a fix for a vulnerability in a component as that they will, even if they know about the vulnerability.

5.      Now, why would a supplier treat vulnerabilities in components differently from vulnerabilities in their own code? After all, most software suppliers nowadays do a good job of patching vulnerabilities in their code soon after they are identified. Could it perhaps be that their customers can quickly learn about any vulnerability in their own code by looking up the product in the NVD? But how is the customer going to learn about a vulnerability in a component if they don’t know what components are in the software they use? Until customers know about the components of their software, and are able to quickly learn about vulnerabilities in them, they will be left in the dark about what might be the biggest source of cyber risk on their networks: the components of the software they use.

This is where SBoMs come in. An SboM tells you the components of your software, just like the label on a candy bar tells you its ingredients (OK, there are some differences between an SboM and an ingredients label, I’ll admit). If you can identify the CPE names of the components, you can look up vulnerabilities that apply to them in the NVD. This can make your organization more secure in several ways:

1.      When purchasing software, you can learn about vulnerabilities that apply to each component of a software package you’re interested in, not just the vulnerabilities that apply to the software itself. This information might help you make a better-informed choice of one package over another. And it can definitely give you important knowledge that will help your negotiations with the supplier. If for example you know there are five serious vulnerabilities in components of the package, you can require that all five be remediated before you will sign the contract.

2.      Once you own the software, you can track vulnerabilities in components like you currently track them in the software itself. If you learn of an important component vulnerability and the supplier doesn’t issue a patch or a notice about that quickly, you can contact the supplier to find out when they will patch the vulnerability. Of course, it’s very possible the supplier will tell you that the vulnerability isn’t exploitable in their product, because of the way the component has been implemented (e.g. the component is a library with three modules, but the supplier only implemented two of those modules. The vulnerability is in the third module). That’s fine, but at least you will be able to cross this off of your list of things to worry about. And quite frankly, the supplier is much more likely to patch component vulnerabilities quickly if they know their customers are tracking them and will be on the phone soon if they don’t do something about a new serious vulnerability, than if the customers are totally in the dark, as they are now.

3.      The supplier will also benefit from widespread availability of SBoMs. After all, the supplier needs to know about the components of the components they have included in their software, since serious vulnerabilities in those second-level components might very well be exploitable in their software. They can track vulnerabilities in the second-level components, and – just like their customers do with them – they can put pressure on the suppliers of their first-level components, including open source communities, to patch those second-level component vulnerabilities.

4.      And what happens when the next Ripple 20 comes along? This is a set of serious vulnerabilities that have been in a particular component – the Treck IP stack – since the late 1990s. The vulnerabilities are estimated to be found in hundreds of millions of devices worldwide. The vulnerabilities might be found four or five component layers down. Software suppliers worldwide have struggled even to tell their customers whether or not their products are susceptible to the Ripple20 vulnerabilities, since the suppliers generally have no idea of what their second-level components are, let alone their fifth- and sixth-level components. I heard that Jsoft, the Israeli company that discovered these vulnerabilities, resorted to combing through LinkedIn to find references to Treck, for example in resumes of developers, in order to find companies to warn about this. They knew of no other way to find what products might be affected by Ripple 20. Of course, were SBoMs widely available and multiple layers deep, this wouldn’t have been hard to do. But right now the idea of widely-available multilevel SBoMs is pretty much a pipe dream. Hopefully, in a few years one-level SBoMs will be at least somewhat widely available. That alone would be a huge help.

It’s getting late, and I’ll close this post with a quote from Gartner[ii]: “By 2024, the provision of a detailed, regularly updated software bill of materials by software vendors will be a non-negotiable requirement for at least half of enterprise software buyers, up from less than 5% in 2019.” They’re coming!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] With open source software, the chance that there will be vulnerabilities is higher. In fact, the Sonatype study found that on average 11% of the open source components in any given software product have at least one serious vulnerability. And a 2020 study of developers found that 21% of them had experienced a breach due to a vulnerability in an open source component in the past 12 months. This was down from the 31% figure in 2018. 

[ii] From their “2020 Market Guide for Software Composition Analysis”, available at  https://www.synopsys.com/software-integrity/resources/analyst-reports/software-composition-analysis-market-guide.html

Chris Krebs goes gently into that good night

So now Chris Krebs has been fired. The only refreshing part of this news is that Trump didn’t even pretend it was because of misconduct or incompetence, instead saying:

"The recent statement by Chris Krebs on the security of the 2020 Election was highly inaccurate, in that there were massive improprieties and fraud - including dead people voting, Poll Watchers not allowed into polling locations, “glitches” in the voting machines which changed votes from Trump to Biden, late voting, and many more. Therefore, effective immediately, Chris Krebs has been terminated as Director of the Cybersecurity and Infrastructure Security Agency."

In other words, he was fired for telling the truth. The fact that this is no longer surprising doesn’t make it any the less shocking.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

How do we assess risks in the products we buy?

This was originally going to be the third part of a set of three posts about a great supply chain security presentation by Matt Wyckhouse of Finite State at the MRO (virtual) Security Conference last month. After the conference, Matt and I exchanged some emails and I decided to break the single post I was going to write into three. You can read the first two posts here and here.

This third post was always going to be different from the others – and it will live up to that billing. The first two are discussions of what Matt said (and my email exchange with him afterwards helped me better understand what he said – especially since I just had the slides to work with, not the recording). However, in that email exchange we traded views about a subject on which we have seemingly different opinions: the role of supplier assessments in managing product security risks.

For this third post, I was originally going to simply reproduce what we had said about supplier assessments in his emails to me. However, last week I noticed an article on the Finite State web site, which explained Matt’s position on this issue better than he’d been able to in our relatively short email exchange. So I decided to make this third post a response to his article.

Here’s my summary of Matt’s article, using liberal quotes from the article (although Matt, if I’ve missed something, send me an email and I’ll put it up as another post), along with my responses to some of his points in red:

1.      In the past, users of critical infrastructure have been able to rely on vendor questionnaires to determine what risks might be present in an OT device we purchase. “But the rapid adoption of connected devices—the internet of things (IoT), operational technology (OT), and other embedded systems—has completely changed the source of device risk.”

2.      Here’s the problem: “Whereas before, having a deep understanding of our vendors was the primary option for assessing risk, it is now clearer than ever that these third party risk assessments are falling short of where we need to be to prevent our networks and infrastructure from being compromised.”

3.      What’s the solution? “Both device manufacturers and asset owners need a scalable solution that takes inherent device risk into account. And while these assessments give us key information, we cannot see the full picture without looking directly into the devices themselves.”

4.      In the second section of the article, Matt discusses third party (supplier) risk assessments. He says, “Through lengthy questionnaires that evaluate a vendor’s security posture, reputation, privacy risk, and so on, these assessments attempt to answer the question, ‘should I trust this company with my data?’”

I totally agree with Matt that most supplier risk questionnaires ask questions about how the supplier protects customer data. That’s great if you’re assessing the risk of say a cloud provider or a payroll services company. In those cases, the whole question is how they will handle your data. However, that shouldn’t be the case with OT supplier assessments. With a couple notable exceptions[i], OT suppliers hold or process little or none of their customers’ data. Why should say a supplier of relays hold a lot of data on an electric utility’s operations? If they need to troubleshoot a problem, they can get all of the data they need at that time. When they no longer need the data, they should delete it.

However, I have yet to see a questionnaire that focuses solely on OT risks, other than the one I have developed with my clients. This summer, I wrote four posts about the questionnaire NATF introduced in a webinar in May, including this one that pointed out that a significant number of the questions in that questionnaire don’t address OT risks at all. This includes questions about data privacy and how the supplier assesses the companies with which they share data (both important questions for a data services supplier, yet close to irrelevant when it comes to BES Cyber Sytems).

While there are some data-oriented questions that do address legitimate OT risks (including all of the data risks listed in the NATF Criteria), it’s a huge waste of your time and your supplier’s time to ask them questions about how they handle data, when they will be handling very little of your data, if any. So I agree with Matt that asking suppliers about data is in most cases not useful. But I disagree with his implication that data questions are inevitable in a supplier questionnaire. It’s certainly possible to design a questionnaire without them (or more correctly, without the basic set of questions that are necessary, such as those in the NATF Criteria). In fact, I’ve done it!

5.      Matt continues: “A lack of hard data means that you must rely on trust. There’s something strange about asking a vendor questions about their own trustworthiness. If you’re worried that you can’t trust a company with your data, why should you trust their responses? This leads to situations where your team may have to rely on gut instinct to know when something just feels off.”

This is another example where I think Matt is starting from the wrong premise. What is the purpose of a supplier questionnaire? Is it to help your organization make the decision whether or not to purchase from this supplier? I’m sure that the great majority of cyber risk management professionals in any industry, including the power industry, will say that is the case. In fact, NATF says that the purpose of the supplier assessment process is to make a purchase decision.

This is almost certainly true for purchases of IT systems. However, I’m certain that, for OT systems, in at least 95% of procurements (really, asymptotically approaching 100%) the decision on who to purchase from was made a long time ago. One municipal utility estimated to me that for their OT systems, they consider changing a supplier no more than once every five years, and probably a lot less often than that.

Let’s say you don’t like your EMS vendor’s answer to a couple questions on the latest security questionnaire you sent to them. Are you immediately going to put out an RFP for a new EMS supplier? How about the supplier for the turbines in your generating plants? Or the relays in your substations? It just doesn’t happen. Consider: There are some organizations that have been buying products from GE for close to 100 years (i.e. since just a few decades after Thomas Edison founded the predecessor company). Are they going to tell GE to take a hike if they think there are a couple areas where their cybersecurity leaves something to be desired?

Of course not. So why do organizations send cybersecurity questionnaires to OT suppliers? In some cases, the answer may be “Because that’s our policy”. But the real answer should be “Because we have decided to purchase the product from supplier X, we want to make sure we understand their current posture with regard to the supply chain cybersecurity risks that we think are important. If their answers indicate that they may not have mitigated a particular risk to our satisfaction (for example, they haven’t implemented multi-factor authentication for their own remote access systems. DHS said in 2018 that over 200 suppliers to the power industry had been penetrated by the Russians, usually through their remote access systems), we will talk with them about how they can rectify this problem.”

So the question of supplier trust isn’t the issue here, as Matt seems to be saying. We don’t – or at least we shouldn’t – ask the supplier questions in order to form an opinion on whether they’re trustworthy or not. We should ask them questions about their policies and practices because we want to see them improve their security over time. After all, they’re our supplier and that’s not likely to change.

6.      Matt goes on “You must assume that the vendor being assessed understands their products, firmware, and supply chain. Even if you deem a vendor to be trustworthy, you are then relying on their processes and on every member of their team to know what they’re doing and to function flawlessly. The vendor in question may very well believe that their products are secure, but if they don’t understand their own supply chain risk, or if their suppliers don’t understand the security of their components, then how can you trust their answers?”

I’ll paraphrase this: “Risks that arise from the supplier’s own supplier chain are just as important as risks that are due to the supplier’s own policies and practices. A questionnaire can’t truly assess risks that come through the supplier’s own supply chain, since the supplier itself usually doesn’t understand those risks.”

Once again, I don’t disagree at all with the statement that suppliers don’t sufficiently understand their own supply chain risks, meaning the risks that arise from third-party and open source components they have included in their software and firmware products. Both suppliers and consumers need to understand those risks – the problem is they don’t normally have any information on the components of the products they purchase; if they did, they could research those risks. What’s the ultimate solution to this problem? SBOMs!

7.      Matt concludes this section thus: “Even if we do our best to minimize these issues, what are we actually measuring? Again, we are asking, “should I trust this company with my data?” but that doesn’t help us understand actual product risk. It doesn’t give asset owners the ability to see what vulnerabilities are inside their devices or what issues are introduced by their supply chains. What if in addition, we asked, “should I trust this device on my network?” How would we assess risk then?”

Again, I want to paraphrase what he said. I’ve already pointed out that questionnaires for OT products a) shouldn’t ask a lot of questions about data, and b) aren’t used to determine whether to trust the supplier – they’re already trusted, which is why they’ve been the RTU supplier for 30 years. The question we should be asking in OT supplier risk assessments is “What are the supply chain risks that could arise through this supplier, that could have an adverse impact on our ability to control and/or monitor the Bulk Electric System?” Even though a lot of NERC entities don’t explicitly have this question in mind when they send a questionnaire to a supplier, this is IMHO the real justification for doing so.

However, I’m in total agreement with the last two sentences of the above quotation. We do need to ask whether we should trust a new device on our network, and we need to assess the risks that may apply if we do connect the device to our network, before we connect it. The point is this is a completely different question from the question that we’re trying to answer with our questionnaire. The questionnaire is trying to identify risks that may be present in a supplier that can affect products they produce in the future. Mitigating them is important because we don’t want to have bad experiences with the supplier’s products in the future.

But getting the supplier to for example implement background checks for all software developers won’t make the product we buy today any more secure than it was yesterday. We also need to assess the product itself. The moral of this story: We need both to assess supplier risks through questionnaires, and assess product risks through testing, before we install a product.

The last section of the article describes what it means to assess risk in a device that you’re planning to install on your network. I don’t disagree with any of that, except to say that assessing risk must always itself be a risk-based decision. If you’re about to install the 50^th HP server in your Control Center, you probably don’t have to conduct an in-depth assessment like Matt describes. On the other hand, if you’re installing the first of a new type of relay – and the supplier rushed development to get this into your hands by your deadline – you should definitely give it a thorough looking-over before you install it.

Now that I’ve finally finished discussing Matt’s MRO presentation, maybe one of these days I’ll get to what I consider the most interesting topic: How he came up with the name for his company, and why he made a puzzling (to me, anyway) choice in arriving at that name. But that post won’t help you at all with CIP-013 compliance (although it might help us all understand what “programmable” means).

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] E.g. a turbine manufacturer holding a lot of performance data for predictive maintenance purposes.

Everyone should listen to this advice, regardless of party

This morning, the Washington Post carried this op-ed written by Richard Danzig, who served as secretary of the Navy under President Bill Clinton; James Lawler, an infectious disease physician and co-director of the Global Center for Health Security at the University of Nebraska Medical Center; and Thomas P. Bossert, who served as homeland security adviser under President Trump.

They start with a warning: “On the present trajectory, before we inaugurate a new president and before we deploy vaccines, the United States will experience unprecedented trauma from the coronavirus.” Why do they say this? Just look at the daily new infection numbers. They were around 30,000 at the height of the first wave in April. They were around 60,000 at the height of the second wave in August. Yesterday and the day before, new cases were about 140,000.[i] In other words, the third wave is already more than four times as high as the first wave, and more than twice as high as the second.

But it’s not going to stop there. They say “On the present course, by Inauguration Day we can expect to see 400,000 Americans infected each day, a rate equal to that recently experienced in Belgium and the Czech Republic. Even with the lower fatality rates witnessed over the summer, such increases could result in deaths on a scale not seen since the 1918 influenza pandemic.” In other words, by January 20 daily new cases will be, if we keep on the current course, close to three times as high as the current level, and more than six times as high as the peak of the second wave. And of course, the authors chose January 20 because it’s Inauguration Day, not because they think that will actually be the peak of this wave.

What does this mean for deaths? While death is hardly the only serious consequence of Covid-19, it is certainly the most visible one. I have been following deaths as a percentage of closed cases since Worldometers.info started publishing the latter number in late March. At that time, the ratio was above 40%, meaning four of every ten Covid cases ended in death. Yesterday, the ratio was 3.6%, meaning fewer than four of every 100 cases ended in death.

Why has the ratio fallen so much? Two reasons: huge advances in treatment and much less hospital overcrowding. Is the ratio likely to keep falling in the near term, so it might be much lower on January 20? Not at all. Yesterday, about 65,000 people were hospitalized in the US, about the same as at the peak in April, and hospitals in some states like Utah and North Dakota are essentially full. Given how quickly new cases are increasing, we’ll soon (probably within a couple of weeks) see a lot of the dreadful phenomena we saw in New York in April, including ERTs being forced to triage sick people at home, deciding on their own whether they could be saved if they were brought to the hospital.

Most harrowing of all, we’ll see the refrigerator trucks lined up at hospital doors, waiting for their inevitable grim cargo. In fact, that is already happening in El Paso.

So what are the authors calling on the country to do? Are they advocating another sweeping lockdown (although even in April the lockdown was hardly that, in many states)? No. “Increased testing, contact tracing, isolation and quarantine, more personal protective equipment, and other interventions are important, but they will not address the immediate challenges of the surge. In addition, they require spending authority and complicated logistical improvements.”

They’re not calling for anything that can’t be implemented with just a little will, and especially with a nudge from Washington:

“As a foundation for emergency response actions, the emphasis should be on three interventions, executed in concert, in any region with case counts over 20 per 100,000 persons per day. These are to temporarily 1) restrict all indoor gatherings of adults to no more than 10 people; 2) close indoor restaurants, bars and clubs; and 3) mandate universal mask-wearing in public.

These steps will not eradicate the coronavirus. But by reducing super-spreading events, they could reverse the present trajectory of community transmission, halving (or better) the growth rate between now and Inauguration Day, and buying time for additional interventions and vaccines.

Such a plan would require sacrifices, but the president-elect and Congress could alleviate the necessary pain by subsidizing affected businesses in a targeted way as part of a coronavirus relief act. Without that assistance, the practical economic needs of affected individuals will compete with the collective needs of the public health.”

Will this happen? It’s going to take a change of hearts in the White House and Congress, for sure. But I would hope the needs of the country would now take precedence over what are (wrongly, to be sure) considered to be the political interests of one party and one person.

I would love to hear any comments or questions you have on this post. Drop me an email at tom@tomalrich.com. If you would like to see my other posts on the pandemic, go here.

---

[i] According to Worldometers.info.

Software and Sisyphus

A friend who is knowledgeable about software security recently sent me an article that has created a lot of waves in the software community (it is available here, but you’ll have to pay $15 to download it. If you want to email me, I’ll send you my copy).

You can read the paper on your own, although the authors are careful to summarize their conclusions in the first three pages. They set out to answer a particular question: “Is the security of software releases increasing over time?” This is an important question, because there’s a generally accepted view that, as a software offering matures and offers new versions, each version will fix problems that have been found in previous versions. This means that the overall level of security of a software package will increase over time.

However, they found the opposite to be the case, after studying various software releases in the Debian Linux “ecosystem”. They say “Even in popular and ‘stable’ releases, the fixing of bugs does not seem to reduce the rate of newly identified vulnerabilities.” Does this mean we should stop trying to fix software vulnerabilities? Obviously not, since then the problem would keep growing until finally most software was unsafe to use. We need to keep fixing vulnerabilities, but we shouldn’t expect the need for this work to decrease.

There is a reason why the title of the article refers to the tip of the iceberg. Since about nine tenths of an iceberg is underwater, no matter how much ice you chip off the part that’s above water, it won’t help at all. More of the underwater part of the iceberg will emerge from the water, so the total amount above water will remain about one tenth. It’s like the ancient Greek myth of Sisyphus, who was doomed for eternity to roll a huge rock up a hill, only to have it roll to the bottom every time.

What I found most interesting about this article were the two examples the authors used of serious vulnerabilities that caused a huge amount of consternation worldwide, and which were exploited to cause damage: Shellshock and Heartbleed. The main reason both of these were so serious is that they were found in lots of software components – and organizations were faced with the task of trying to figure out, for almost every piece of software they owned, whether it had vulnerable components, whether those components had vulnerable components, etc. Just like the Ripple 20 vulnerabilities more recently.

What does this show? It shows we need SBoMs!

 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.