Sunday, November 8, 2020

A great supply chain security presentation – part II

If you're looking for my pandemic posts, go here.

More than two weeks ago, I put up the first of three posts discussing a presentation that Matt Wyckhouse of Finite State gave at the MRO (virtual) Security Conference a few weeks ago. Because Matt covered a lot of ground, I’ve broken it up into three separate (but related) topics, which I’m addressing one-by-one in these posts.

In my first post, I discussed the first part of Matt’s presentation, when he made the point (which I totally agree with) that nation-state supply chain attacks shouldn’t be a big priority in supply chain security. Other than an attack the US carried out in 1982 on the Soviet Union, there’s no evidence of successful attacks (and little evidence of unsuccessful ones, either). Moreover, given the defenses deployed to protect the Bulk Electric System (not the least of which are the CIP-005 R1 and R2 requirements) exactly how such an attack would be carried out on the US power grid remains unclear at best.

Matt then pointed out (and I again totally agree with this) that the real supply chain security problem is vulnerabilities in software or firmware, no matter where they come from. A few of these are deliberately planted by deliberate attackers (almost always aiming for financial gain, and coming from all over the world, including the US of course); a few more are deliberately planted by suppliers, in order to make it easier for them to troubleshoot problems. Of course, when a vulnerability is deliberately planted by anyone, supplier or ne’er-do-well, it is called a backdoor.

However, the vast majority of software and firmware vulnerabilities are due to much more mundane causes, such as insecure coding practices or simply bad timing (meaning that e.g. a particular sequence of commands hadn’t yet been identified as a vulnerability. Any developer lacking perfect knowledge of the future would probably see no reason not to include that sequence in a new software package). But this doesn’t make them any less dangerous, and indeed some of the most destructive software attacks (NotPetya, Wannacry, and Equifax, for example) exploited completely unintentional software vulnerabilities. These are what we need to focus on, even though addressing them doesn’t have the same kind of sex appeal as does fending off potentially devastating (but rarely occurring, if at all) supply chain attacks by China.

How does Matt suggest we address software and firmware vulnerabilities? Here are (IMO) his most important points:

·        What are we really trying to prevent by banning components from adversary countries? Backdoors, unintentional vulnerabilities, intentionally and unintentionally placed malicious code, and the potential for future insertion of malicious code or vulnerabilities. These are all embedded threats and vulnerabilities. We need to look for these, regardless of their country of origin. The good news is we can detect these if we know where to look.

·        How about pen testing products? Is that worthwhile? It’s a good point in time solution, but it’s not scalable for all devices due to cost, time required, and most importantly lack of completeness.

·        Organizations need a scalable way to rapidly screen all the devices, firmware and software entering their networks.

·        Only 5-25% of the code in a device is proprietary. A complex supply chain of libraries, platform code, etc. can lead to risks even the manufacturer is unaware of. Of course, these are the software component vulnerabilities I’ve been discussing a lot lately. The main tool for addressing these vulnerabilities is the software bill of materials.

·        Most threats live in the software and firmware, not the hardware – which means continuous screening of software and firmware updates is critical.

·        Everything you want to verify for firmware is in the firmware update package. You can get it from the vendor’s support site, by asking the vendor for it, and by using more technical methods (Note from Tom: If you have my level of software/firmware literacy, or even a level far above mine, you may find this to be a daunting task).

Matt provides five examples of devices that Finite State examined and found to have serious firmware vulnerabilities. Here are my two favorites:

·        First, his company found a local privilege escalation backdoor in the restricted shell interface for a popular wireless access point. This allows anyone who knows the “secret code” to gain root access to the device by telnet or SSH.

o   He and his colleagues were suspicious that this vulnerability resulted from somewhere in the upstream supply chain. They confirmed that suspicion and found that this same vulnerability was sound in 787 firmware implementations by several vendors. Of course, that means that someone knowing the secret code could penetrate a huge number of devices.

o   The vulnerability (in this case a backdoor that was almost certainly deliberately inserted) was introduced by a chipset vendor named Senao. The backdoor was introduced in the software development kit (SDK) for the chipset. Any product using that SDK is now vulnerable to a serious privilege escalation vulnerability.

o   The manufacturers who incorporated the chipset didn’t find this fairly-easy-to-discover backdoor because they didn’t do third party code verification (i.e. verifying authenticity and integrity, as required by CIP-013-1 R1.2.5).

·        Second, they examined a widely deployed smart meter and identified two hardcoded account passwords, including an FTP user.

o   They also found numerous “critical and high severity vulnerabilities” in the smart meter.

Risk-based product security

Matt concluded the presentation by saying that we need to move toward risk-based product decisions that factor in all sources of risk, including:

·        Vendor geopolitical, regulatory and compliance risk.

·        Supply chain risk.

·        Software vulnerabilities.

·        Configuration vulnerabilities.

·        Device hardening measures.

·        Known threats.

And here’s where I differ from Matt. I think product risks are important, and we can screen for them to a certain extent (although a lot of this takes a level of expertise that most of us simply don’t have). But there are plenty of supply chain risks that are due to the vendor’s practices, not their products. Trying to protect against these solely by screening products strikes me as the wrong approach.

To mitigate these risks, you need to first assess the supplier, usually through questionnaires. And if you believe a risk is present, you need to try to get the supplier to mitigate that risk, using contract language, asking for a letter, or simply getting them to agree on a phone call. Here are some of the supply chain risks that can only be mitigated by working with the supplier:

  • Misguided policies on the part of the supplier, such as not bothering to tell their customers about vulnerabilities in software components (Veracode says 52% of software suppliers don't do this).
  • Lack of good security practices, such as not doing background checks on employees involved in software development.
  • Lack of a good supply chain risk management program (i.e. a program for managing risks posed by suppliers of software components included in the supplier’s product).
  • Improper controls on remote access to the supplier’s own networks.
  • Insecure practices for product shipment to customers.
  • Lack of notification that a product, or a component of a product, is now out of support.
  • Inadequate controls over customer data in their possession.

As you’ll see in the exciting conclusion to this three-part series of posts, Matt disagrees with me on the relative importance (and usefulness) of working with your suppliers to mitigate risks, rather than focusing primarily on detecting and mitigating them at the product level. I will reproduce an email dialogue we had a few weeks ago. Watch for this, coming soon to a blog near you!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

No comments:

Post a Comment