Supply chain security advice from the Washington Post

I’m not used to reading good cybersecurity advice in major newspapers, although the three I read – the NY Times, the Wall Street Journal and the Washington Post - all provide good reporting on cybersecurity issues. But WaPo really surprised me today, when I read an editorial that provided more clear-headed advice about lessons to be learned from the SolarWinds attacks, than anything I’ve read yet in the “cybersecurity press”.

Being an editorial, it is short compared to normal articles about cybersecurity. But every word shows an excellent understanding of what the SolarWinds attacks mean; and the advice provided (which targets the federal government, since after all that’s WaPo’s backyard beat) is excellent. The first paragraph sums up the problem perfectly:

THE SOLARWINDS hack, it seems, was worse than initially feared — and initial fears were already alarming enough. The manner in which the U.S. government left itself vulnerable to the attack demands a reckoning that runs the policy gamut. But the best place to start may be the least flashy: security of the software supply chain.

The last three words say it all: First, this was a supply chain attack. Just these words show a lot more understanding than other articles I’ve read, that seem to think the attacks show that the US needs to beef up the front doors protecting its systems. The whole problem is that the attackers knew the front door was well guarded and went around to the back, where the door used by vendors was easily breached. Sure, we need to maintain the front door protections. But they’re already pretty good.

What we really need to do is secure the back door just like the front door. But protecting against supply chain attacks is a completely different problem from protecting against “front door” assaults. This is because my front door/back door analogy isn’t really accurate. There are actually a huge number of back doors to protect – one for each supplier of products or services that could be subject to a cyberattack, or for that matter one for each supplier that employs human beings that use computers (I’m told there are still a few of those left), since human beings can succumb to phishing attacks that allow an attacker to send well-crafted phishing emails to the customer, who is of course the real target (of course, there are hundreds of other possible vectors for supply chain attacks).

Second, this was an attack that came through software, not hardware. Unfortunately, lots of people in the press and the power industry – as well as in government – have come to believe that what we really need to protect against is Chinese geniuses embedding attacks in hardware, when in fact there has never been a pure hardware attack – i.e. one in which the microcode of a microprocessor was altered. This would be extremely difficult. 

The only other way hardware could be attacked is through firmware. I know of only one major firmware attack, the Mirai attack that turned a huge number of IoT devices into a Botnet used to attack various targets, including Brian Krebs’ website. Of course, the devices penetrated had close to zero security, which isn’t the case at all with devices that are the big concern, including those that play a role in managing or monitoring the power grid.

Devices that don’t have a microprocessor (or perhaps an FPGA) can’t be victims of a cyberattack, just as my cheap steam iron can’t be a victim of a cyberattack. However, about 20 of the roughly 25 devices listed on last May’s Executive Order don’t have microprocessors – why were they even in the order? This includes transformers[i], which became a rallying cry for those in the press who are ready to believe that the grid is totally unprotected from cyberattacks and could be knocked over with one press of a button in Beijing.

The most important reason why I don’t lose sleep over hardware supply chain attacks is that they’re so easily attributable. If some piece of hardware is exploited because some sort of backdoor had been planted in it (presumably in firmware), and if the hardware came from country X, then the US can immediately threaten to do all sorts of nasty stuff to country X, including of course military conflict. On the other hand, there’s no good way at all to determine where software was made (as opposed to who sells it) – in fact the question itself is close to meaningless. Almost all software nowadays is developed by teams collaborating around the world. As I've mentioned a couple times, Siemens has 21 R&D centers in China, that employ 5,000 people.

And WaPo knows this, too. They also know that the really serious supply chain threats come through software. Why do they say this? Because – as SolarWinds amply demonstrated, in a way that hasn’t been seen anywhere before this, to my knowledge – software supply chain attacks have a wonderful (for the attackers) ability to lead to breaches of many organizations at the same time. It’s now estimated that the Russians penetrated “only” about 250 organizations, even though 18,000 applied the infected patch and are presumably vulnerable. But even putting aside the fact that the 250 included the NSA, DHS, DoE, etc., just the fact that 250 organizations were compromised as the result of one attack shows the huge power of software supply chain attacks.

What do we need to do to prevent software supply chain attacks like SolarWinds? The article lists five mitigations, one or two of which I’ve never seen in the press before. All are very sensible, and none of them require throwing hundreds of millions of dollars into some digital Maginot Line like Project Einstein.

1.     There need to be security standards for developers of software that is used in high-value organizations like DoE and DHS. Even if the software is just used for football pools, the supplier needs to be vetted. Obviously, we don’t need any more suppliers who didn’t even hire a CISO until 2017 and have easily-guessed passwords on important servers. 

2.     Software needs to be tested to make sure it is secure. This is a good step in principle, but it needs to be conducted using a risk-based approach (in fact, all supply chain security requirements should be risk-based). If we tested every piece of software the government uses, for every conceivable risk, we’d probably have to devote close to the entire GNP to software testing.

3.     Simply asking suppliers about their practices would be a huge improvement over not doing anything. This is a point I’ve been making for a while, and I was quite glad to hear WaPo – or anybody, for that matter – agree with me. The article says “Merely requiring that firms responsible for critical functions certify to undertake certain cybersecurity practices before supplying to the government could prove transformative.” People often think the only possible way to get suppliers to follow good security practices is to beat them over the head with contract language, and threaten to terminate them at the slightest deviation from what they agreed to. However, just asking them some good questions about their practices and requiring them – using just moral force, if that’s the only route available – to mitigate any holes identified would be a huge improvement over the current situation.

4.     The article mentions code audits as the fourth mitigation. If this is done using a risk-based approach, it would be a good idea in high-risk cases. But this is another measure that could easily swallow the whole GNP, if we tried to do code audits on all software in use in the federal government, let alone private industry.

5.     We need to root out attackers. Of course, this isn’t just a supply chain security measure; this is a general cybersecurity measure. The fact that the Russians were inside some hugely important networks for months without being detected indicates that – ahem! – there’s lots of work to be done in this area.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] Transformers sometimes incorporate devices that do have microprocessors, especially load tap changers. But they are often made by a different company than the one that makes the transformer. They should be protected from cyberattacks, not the transformer itself.