Software is developed everywhere. It's developed by everybody.

Many people – including the ones who wrote the recent Executive Order – think that an important component of software supply chain risk is due to provenance, meaning where the software came from. Of course, the idea behind that belief is certainly understandable: there has been a lot of concern about the idea of malicious actors in Russia, China, North Korea, Iran, Cuba, Venezuela – you name it – planting a backdoor or logic bomb in software used by government or private organizations in the US, causing havoc on the scale of the SolarWinds attacks.

This is why many people think it is important that critical infrastructure and government organizations in the US should inventory all of the software they use, as well as all of the components of the software they use (of course, you need SBOMs to do that!), and identify which of these originate in Bad Guy countries – or which might perhaps have been developed by an organization that is controlled by or under the influence of a Bad Guy country. Then they should at least consider removing any software they identify from their environment, or at least they should make sure they don’t buy any more of this Bad Guy software.

There’s only one problem with this way of thinking: with one exception, I know of no instance in which a Bad Guy nation-state has actually carried out a software supply chain attack on US interests, that could have been short-circuited by conducting such a provenance analysis (that one exception is of course Kaspersky, which at the time of the attack was located in Russia and whose founder had links to the Russian security services. They were alleged to be behind the attack on an NSA employee, who had stolen some of the NSA’s most potent malware - including the malware behind NotPetya - and stored it on his home computer, which ran Kaspersky’s antivirus software. Kaspersky swore up and down that this wasn’t the case, but I’m certainly willing to stipulate that the Russians had planted a backdoor in the Kaspersky software, which let them penetrate the NSA employee's computer and exfiltrate the NSA's malware weapons).

Of course, I’m sure there have been plenty of cases where a software company in a Bad Guy country has sold software that contains vulnerabilities to US customers. But every software company everywhere does that – there’s simply no way to ship vulnerability-free software. In fact, given that I’m sure most of the software used in the US is developed by American companies, it is quite likely that the biggest source of vulnerability-laden software to American organizations is…American software companies.

What about SolarWinds, you ask? They certainly shipped vulnerability-laden software to many companies and government organizations (18,000, in case you’re keeping score at home), in the US and abroad. And those vulnerabilities were planted by a Russian team of about 1,000 people (using Microsoft’s estimate) that deployed what may be the most sophisticated piece of malware ever. Surely this counts as a state-sponsored software supply chain attack that could have been prevented by provenance controls!

The problem with that narrative is that, the last time I checked, SolarWinds is a US company. Aha! But what about their suppliers? They used contract developers in Eastern Europe, for God’s sake. Surely they had a hand in this nefarious deed? That’s a good question, and one that I raised in one of my many posts after the SolarWinds attacks were discovered.

But as the article linked in the previous paragraph describes, those developers didn’t have their hands anywhere near this attack. It was carried out by servers located in the US that were controlled by the Russians, and the attack was on the SolarWinds Orion build environment, which was physically located in the US. The malware-laden Orion updates all were digitally signed by SolarWinds. How could provenance controls have prevented this attack?

There was another important Bad Guy connection that I read about in the New York Times in early January: SolarWinds was a big user – as are many other large US companies – of a development tool called JetBrains, that is developed and sold by a company headquartered in – get this – Moscow. In this post, I wondered (as the Times had) whether JetBrains might have been the vehicle for the attack on the SolarWinds build environment, although I stopped short of saying that was likely.

However, 12 days later – without any additional evidence – I said in another post “Of course, it was recently learned that the Russians did penetrate a very widely-used development tool called JetBrains. And one of JetBrains’ customers was in fact SolarWinds.” Three days later I received an email from a very polite public relations person for JetBrains in Moscow, asking me to please retract that statement, since JetBrains had just recently put out their own statement firmly denying any role in the attack. Of course I did that and apologized to JetBrains in a new post that day.

In that post, I pointed to a lack of care as the reason for my misstatement. However, four days – and more introspection – later, I confessed to the real reason: I was prejudiced against Russian companies because dontcha know they must all be captives of the Russian state, just like I believed Kaspersky was. And if I’d been in charge of those things, I might have banned all software sold by Russian companies from installation on important networks – which just goes to show that you shouldn’t put me in charge of those things.

The fact is that, if we start banning software from particular countries just because we think the governments of those countries are out to damage the US (and there isn’t a lot of doubt in my mind that the Russian government fits that description), we’ll end up damaging our own companies. JetBrains didn’t get its huge worldwide market share because it’s a so-so product; by all accounts (and I’m certainly not competent to judge this), it’s a very strategic tool for developers like Oracle and Microsoft (as well as SolarWinds, to be sure). Were we to prohibit US companies from buying JetBrains, we would be putting them at a competitive disadvantage vs. companies headquartered in other countries.

But most importantly, the whole idea that software “comes from” a particular country is now obsolete. Nowadays, software – other than perhaps software developed for DoD and the 3-letter agencies - is developed by teams of people from all over the world who collaborate online to develop the product. Sure, they might mostly (or even all) be employees of a company located in Country X, but a large percentage of them will almost undoubtedly not be citizens of X and probably not be located there, either.

Matt Wyckhouse of Finite State, in a presentation last fall, pointed out that Siemens – a huge German company that does a lot of business in the US, especially with critical infrastructure organizations – has 21 R&D hubs in China and employs about 5,000 Chinese staff there. Does this mean that Siemens software poses enough risk that you should consider removing it (along with whatever Siemens hardware it supports, of course) from your environment? After all, there’s a good chance that at least a portion of any Siemens software that you buy was developed in China.

If you’re DoD, the answer might be yes – i.e. DoD might decide (or have decided) not to accept that risk, however small it is – and to pay the undoubtedly high cost of finding and installing equally functional alternatives to whatever Siemens software they’re not buying. For almost any other company, the answer IMHO should be no. There’s simply no justification for subjecting your organization to the time and expense required to find an alternative to Siemens, given the very low provenance risk posed by their software.

So we all need to stop thinking of software supply chain risk as being somehow tied to where the software was developed, or the nationality of the people who developed it. If you’ll promise not to do that anymore, I’ll do the same. 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the National Technology and Information Administration’s Software Component Transparency Initiative, for which I volunteer. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.