What would have prevented the SolarWinds attack?

There has been no shortage of people who are glad to assert that this technology or that product would have “prevented” the SolarWinds™ attack. Of course, given how devastating the attack could have been (even if it might not have led to wholesale transfer of national secrets to Russia, as originally feared), it’s certainly permissible to speculate about how it could have been prevented, since there are undoubtedly many lessons to be learned from it.

However, before making any assertions about what could or couldn’t have prevented the attack, it’s important to remember that when we talk about the SolarWinds attack, we’re really talking about four distinct phases:

1.      The initial penetration by the Russians of the SolarWinds IT network in 2019. How the Russians did that is still unclear, but there are lots of ways they could have done it.

2.      The penetration of the SolarWinds software build environment by the Russians, after they penetrated the IT network. This resulted in their planting the Sunburst malware in at least seven updates to the SolarWinds Orion software platform. This is described in a great article by Crowdstrike, which I tried to summarize (and simplify, to the extent I could) in this post.

3.      Downloading one or more of the Sunburst-bearing updates by “fewer than” 18,000 SolarWinds customers (to quote the immortal words used in the SolarWinds SEC filing the day after the attack was announced. Of course, the country was greatly relieved to hear there had been only 17,999 victims, not 18,000); and

4.      Penetration by the Russians of perhaps 200 of those customers using the backdoor included in Sunburst, and exfiltration of an unknown quantity of information.

Of course, if any of these four phases could have been prevented, there would have been no serious damage from the attacks, and perhaps no damage at all. Which of them was the most “preventable”? It would have to be the one that had the least possible number of ways it could have been realized, and that’s attack number two. There were three stages in that phase. If the Russians had been prevented from accomplishing any one of these three stages, that would have been the end of the attack, period. Those stages are:

a)      Having previously penetrated the SolarWinds IT network, the Russians penetrate the software build environment.

b)     For about ten months, the Russians have access to that environment, although to avoid detection they operate mostly through the custom-created Sunspot malware, which had to operate completely autonomously. Sunspot plants Sunburst in at least seven Orion release updates.

c)      SolarWinds never discovers Sunburst in those seven updates, and provides them to their customers. The customers could never have discovered the problem on their own, since the binary files they received from SolarWinds were digitally signed by SolarWinds.

Here are possible means by which I believe each of these stages could have been prevented:

a) First, what could have prevented the Russians from penetrating the software build environment? It’s quite simple to describe: The software build environment would need to be protected in a similar fashion to how the Electronic Security Perimeter (ESP) is required to be protected by the NERC CIP standards – in other words, there should be no direct connection to the internet, and any connection to the IT network should be carefully circumscribed through measures like those required by CIP-005.

I’ll be the first to admit that this step would be very challenging to implement. But guess what? When electric utilities first had to create ESPs in some of their grid assets, they found this to be very challenging as well – and a lot of them paid fines for not getting it exactly right. Yet today I strongly doubt there’s any NERC entity that is subject to compliance with CIP-005, that hasn’t for the most part figured out how to do so. Were this to be required of some software suppliers, they would also figure it out, although some of them may also have to pay a few fines along the way.

b) Second, what could have led to the Russians being discovered as they were operating – for around ten months - inside the SolarWinds build environment? Actually, a better question to ask is how they could possibly not have been discovered. As the CrowdStrike article so aptly describes, if the Russians had made the slightest mistake at any point during this period, they would have been immediately noticed, not because SolarWinds was directly monitoring for intruders, but because just about anything they did wrong could have caused the build process to stop. The SolarWinds developers would immediately have found them when they investigated why that happened.

The article describes the great pains the Russians went to in order to ensure they weren’t discovered – which was much harder because they couldn’t operate inside the development environment in real time. Instead, they created what’s certainly the most amazing piece of malware since Stuxnet: Sunspot. Like Stuxnet, it had to operate completely autonomously. It communicated with the Russians about what it was doing through bogus VMWare™ log entries that the Russians could read outside of the build environment. Microsoft estimated there must have been at least 1,000 people involved in developing and testing Sunspot.

Given how difficult it was for the Russians to accomplish this stage, this is undoubtedly the most promising point at which the attack could have been prevented. And to their credit, SolarWinds has focused on this stage in the set of changes that CEO Sudhakar Ramakrishna says he has implemented since the attack, as described in a Dark Reading article. The article states:

To ensure that attackers cannot pull off a similar caper, SolarWinds is seeing if it can design its software-build systems and pipelines a bit differently.

Instead of a single build system in a single location, the company is looking at possibly running two or three parallel build systems through two or three parallel build chains. The goal is to see if SolarWinds can establish software integrity across multiple pipelines to avoid supply chain attacks of the kind it experienced a few months ago, Ramakrishna says. An attacker would have to be right three different times, identically, to be able to conduct an attack like the recent one with Orion. SolarWinds has also implemented a two-way hashing algorithm to further establish the integrity of its software.

Of course, building every release of every software product (or even just Orion) three times, not just one, will be very expensive for SolarWinds. On the other hand, given that failure to adequately address the causes of the attack could well spell the end of the company, this is certainly not too much to ask for. Given the amount of damage that the attack caused, SolarWinds is in no position to complain about having to spend a lot of money on this. And to their credit, they’re not complaining.

There’s another possible way in which this stage could have been prevented. In early February, Kevin Perry forwarded me a link to an interesting article about an open source (and therefore free) product called In-Toto. Development of this product was funded by the federal government for $2.2 million, which has to be something on the order of a thousandth (or less) of what the feds will ultimately pay to remediate the damage caused by the SolarWinds attack.

From the little I know about the product, I think it might well have prevented Stage 2 of Phase 2 of the SolarWinds attack from being accomplished. Unfortunately, the feds never pushed any software developers to use the product, although some did (including a SolarWinds competitor, as described in the article linked above). And at least a few large organizations are starting to require (or at least nudge) software suppliers to implement in-toto.

c) How about the third stage of phase 2 of the attack? Preventing that stage would require SolarWinds to be able to identify the Sunburst malware in the code they shipped. One of the first ideas I had about this was that having a software bill of materials (SBOM) could have alerted SolarWinds to the presence of Sunburst. I reasoned that, since Sunburst was effectively a component that had been added to the code, it should have been identified when the SBOM was generated.

I had this idea a few months ago and ran it by one of the mailing lists of the NTIA Software Transparency Initiative. There was a lot of discussion – which expanded to related questions, as those discussions often do. But the answer was clear: An SBOM is usually generated at the end of the software build process, but it reflects the components that were explicitly included in the build. Since the Russians had effectively substituted a component with Sunburst for a legitimate component (and that component was presumably included in whatever SBOM was generated), it would never have been identified on its own. I can’t think of any other way that the third stage had been prevented (given that the Russians had been successful in planting Sunburst in the Orion code without detection), but if you have an idea, I’d love to hear it.

(note that, while SBOMs wouldn't have explicitly prevented the SolarWinds attack, they're undoubtedly an essential component of a good software supply chain security program. This goes both for software users and software developers)

To summarize, I think Phase 2 of the four phases of the SolarWinds attack could have been short-circuited during either its first or second stages. The first stage could probably have been prevented had protections like those in NERC CIP-005 been in place. The second stage could probably have been prevented had SolarWinds implemented the redundant software build process they’re now instituting.

So what should a software supplier do, who wants to avoid being the inadvertent victim/perpetrator of a SolarWinds-type attack? Defense in depth requires they take measures to short-circuit both the first and second stages of Phase 2. That is, they should both isolate their development environment by taking steps somewhat like those required in NERC CIP-005, and they should also implement the redundant build process, in-toto, or some other means to short-circuit Stage 2.

Now we get to the interesting question: regulation. Should there be regulation of software companies in general, to prevent this from happening? Definitely not. How about regulation of all software suppliers to the federal government? Even that would be way overkill. There needs to be a definition of what types of software constitute critical infrastructure (CI) – whatever criterion is chosen, it needs to include SolarWinds and suppliers of other network management and security software (one suggestion I read somewhere was that software that must operate at a higher privilege level needs to be regulated. That was precisely why the SolarWinds attack was so devastating).

Suppliers of CI software to the federal government should be regulated, and a good start would be requiring them to take measures to a) isolate their development environment, and b) implement a system that will warn them if someone is trying to insert, or has already successfully inserted, malicious code into a software product they’re developing.

Of course, CI software suppliers should also be required to notify the federal government if they discover such a breach; Rep. Jim Langevin (D, RI) is proposing this idea, as described in this Wall Street Journal article. However, this wouldn’t have prevented the SolarWinds attack, since SolarWinds had no clue about any of this until FireEye reported the attack to the world.

I also think that cloud providers used by the feds should face mandatory cybersecurity rules (beyond what’s in FedRAMP, since that didn’t prevent the Capitol One or Cloudhopper breaches), as I discussed in this post in December.

It seems likely that the upcoming Executive Order may make a start at this sort of regulation, but of course doing it right will require a bill in Congress. As we all know, that won’t be easy. But it’s important that Congress (and the country) realize that some software and cloud services (and in some cases computing hardware as well) constitute critical infrastructure, just as much as a power grid control center or an oil refinery. We have Uncle Vlad to thank for that lesson!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.