Is there any doubt that software is the biggest source of supply chain cyber risk?

Sonatype came out with their annual State of the Software Supply Chain report last week. Who’s Sonatype, you ask? I must confess that, until I fell in with a bad crowd at NTIA last year, I hadn’t heard of Sonatype, either – or for that matter, a lot of companies that play an important role in securing our software supply chain. I first learned of them when their 2020 report came out a year ago, and I wrote this post about it.

What struck me most in last year’s report were

1.      The huge number of components found in the average software product (135 to be exact, although my guess is that number has gone up this year, since it’s been growing steadily. It was 73 in 2017. Of course many products have thousands of components);

2.      The large percentage of those that are open source (90%);  

3.      The fact that 11% of open source components have at least one vulnerability; and

4.      The fact that in 2020, a survey of 5,000 developers found that 21% had experienced an open source component-related breach in the past 12 months. This was down from 31% in 2018.

My big takeaway from last year’s report was that the biggest source of cyber risk in software is clearly the components that are included in it (as opposed to the small amount of code that the supplier of the software itself wrote), and the lion’s share of that risk is due to the open source components.

Does this mean you should tell your software supplier that they need to remove all open source components from their product, to reduce your risk? If they don’t laugh in your face (which would be dangerous at this time, given the Delta variant), they’ll probably say something like, “Sure we’d be glad to accommodate you. Let’s see…We already know that our product would be about 20-50 times more expensive if we had to replace the open source components with code we wrote ourselves – and if we even could write all of that code on our own.

“We invoiced you $100,000 when you bought our product recently. If you’d like our open source-free version when it’s ready in five years, that will cost you between $2 million and $5 million, which you’ll have to pay up front. Let’s say $5 million, to be safe. Will that be cash or credit card?”

You get the idea: we wouldn’t have anywhere near the volume of software we have today – and even then, we’d pay a much greater portion of our national income for software – if open source components weren’t available to effectively augment your supplier’s paid developers with an unseen worldwide army of unpaid developers. And software suppliers are becoming more and more dependent on open source components all the time.

However, the bad guys have also noticed that open source use is growing rapidly, so they’re continually finding new ways to take advantage of that growth. And this year’s report shows how successful they’ve been in that quest. In a section titled “Software Supply Chain Attacks Increase 650%” (page 10), the article points out that, in the 12 months starting May 2020, attacks increased from below 2,000 to around 12,000.

But this growth is even more amazing when you compare it to what came before: “From February 2015 to June 2019, 216 software supply chain attacks were recorded. Then, from July 2019 to May 2020, the number of attacks increased to 929 attacks.” So we went from 216 attacks in more than four years to 12,000 attacks in the last 12 months.

And what were these attacks? Little piddling attacks you never read about? Not at all. You can see the rogue’s gallery of attacks on page 12, but just in the last nine months, there were Solar Winds and Kaseya (which could count as 1500 attacks, since 1500 customers of MSPs that used Kaseya software ended up being compromised with malware. However, the article counts this as one attack).

How did these attacks increase so quickly? The article says (page 10):

Legacy software supply chain “exploits," such as the now infamous 2017 Struts incident at Equifax, prey on publicly disclosed open source vulnerabilities that are left unpatched in the wild. Next-generation software supply chain “attacks” are far more sinister, however, because bad actors are no longer waiting for public vulnerability disclosures to pursue an exploit. Instead, they are taking the initiative and injecting new vulnerabilities into open source projects that feed the global supply chain, and then exploiting those vulnerabilities before they are discovered. By shifting their attacks “upstream," bad actors can gain leverage and the crucial benefit of time that that enables malware to propagate throughout the supply chain, enabling far more scalable attacks on “downstream” users.

So it seems that the only thing growing faster than open source software use is open source software attacks. But there’s another really interesting trend. You might wonder if these attacks are happening because suppliers are getting sloppy about security. Au contraire. Later on in the paper (page 17), the authors discuss a metric called mean time to update (MTTU) – which is the mean time that a software supplier takes to update their open source components.

The reason this metric is so important, the paper shows, is that this metric is highly inversely predictive of the security level of the software product (i.e. relative absence of vulnerabilities). What’s really striking is how it’s changed:

⊲ 2011 average MTTU = 371 days

⊲ 2014 average MTTU = 302 days

⊲ 2018 average MTTU = 158 days

⊲ 2021 average MTTU (as of Aug 1) = 28 days

So at the same time that the number of attacks has gone through the roof, the developers have gotten security religion and improved this metric more than 10-fold since 2014 (302 days to 28 days). Why is this happening? Unfortunately, it seems that the bad guys are learning how to attack software even faster than the developers are learning how to protect it.

Now, do you doubt that software supply chain attacks are not only the most important source of supply chain cyber risk, but probably the most important source of cyber risk, period? I certainly think so.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the National Technology and Information Administration’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.