There’s lots of good information coming out about the SolarWinds
attacks, as well as others that were very likely carried out by the Unnamed-threat-actor-that-sure-sounds-a-lot-like-it’s-Russia,
whoever that may be. All of this, along with information about other attacks (not
all by Russia) that was made public long ago, points to the need to start
regulating a bunch of tech companies that I hadn’t thought of as being part of
the country’s critical infrastructure – until SolarWinds caused the scales
to fall from my eyes. I’ll deal with SolarWinds in this post, and the other
companies in a post in the near future (although tomorrow's topic is going to be different).
There’s been a lot of discussion of the SUNBURST malware,
which was spread by SolarWinds updates and provided the backdoor that the
Russians exploited to penetrate a lot of important networks, especially US
government networks. Like most malware, SUNBURST used the “pray and spray” method
of cyberattacks: a) attempt to get the malware into as many organizations as possible
(which ended up being “only” 18,000, as SolarWinds said in their memorable SEC
filing the day after the attack was announced); b) wait until the phone home
beacons start appearing; c) choose the juiciest targets (hint: the National
Nuclear Safety Administration is juicier than Joe’s Heating and Cooling Supply);
and d) get to work reaping the fruits of their labors – which turned out to be
abundant in this case, of course.
In the SolarWinds attacks, the benefits the Russians were looking for seem to have been limited to national
security espionage. The Russians could easily have followed their normal pattern
of creating as much chaos and destruction as possible – as in the 2016 US
elections and especially NotPetya, which was a supply chain attack aimed at all
businesses in the Ukraine. NotPetya had no goal other than bricking as many machines
as possible as quickly as possible. It succeeded wildly not only in the Ukraine but
even more so in other countries like the US and Denmark, where it almost
sank (pun intended) Maersk, the largest ocean shipping company
in the world.[i]
Fortunately, the Russians – perhaps out of some sense of
pity for the American people, or more likely because they suspected a new sheriff
might be coming to town, who would be much less forgiving of little foibles
like SolarWinds or paying a bounty to kill American soldiers in Afghanistan,
than was the previous sheriff – decided not to take the destruction route.
Thank God for small favors, as my former boss used to say.
But there has been very little discussion of the SUNSPOT
malware, which the Russians wrote to attack only one company: SolarWinds. And
this malware wasn’t designed to do anything like what SUNBURST did: provide a
backdoor so the Russians could exfiltrate lots of data from SolarWinds. It was
designed to penetrate SolarWinds’ software build process and plant SUNBURST in
software updates, so that SolarWinds would then kindly distribute them to their
customers. And that attack succeeded brilliantly.
In fact, I nominate SUNSPOT as the second-most-sophisticated
piece of malware ever written – after Stuxnet, which was created by the US and
Israel to attack Iran’s uranium enrichment program, and also succeeded
brilliantly (and it might even be ahead of Stuxnet. I’m certainly not in a
position to judge between the two).
You can read the full story of SUNSPOT in this
article by CrowdStrike, which worked with SolarWinds to analyze the malware and
the attack. But here are some highlights, which relate to the subject of this
post:
1. The Russians first penetrated the SolarWinds software
development environment in September 2019 – i.e. 15 months before SolarWinds or
anyone else knew there was a problem.
2.
Like any good organization about to make a risky
investment, the Russians first executed a test of a “beta” version of SUNSPOT. They
wanted to see if they could in principle place a piece of software code in the SolarWinds
Orion platform, so that it would be included in subsequent updates shipped to
SolarWinds customers. They did this using a completely benign piece of code and the gambit worked perfectly. Just like later on with SUNBURST, SolarWinds had no clue
this was happening. In fact, as with SUNBURST, this test code was inserted in multiple
releases of the Orion platform, not just one.
3. With the concept proven, the Russians started to
build the final version of SUNSPOT, probably testing it in their own
environment. About four months later in February, the Russians compiled and
deployed this malware in the SolarWinds software build environment.
4. The most important feature of SUNSPOT was that,
just like Stuxnet, it was built assuming there was no possibility for any real-time
intervention by the Russians in the software build process. The software had to
act completely autonomously; yet it had to quickly adapt to any change in the
environment. This was even harder than it sounds since, while SolarWinds wasn’t
specifically looking for signs of an intruder inside the build process, the
slightest change in the environment – such as a hash value for one of the
Russians’ files (which were always named with names that matched files used in
the build process itself) not matching exactly the value for the same file the
previous day - would have set off alarms just in the normal build process. This
would likely have led to SUNSPOT being discovered and the end of this whole
affair (and we’d all have written about how astute SolarWinds was in heading
off at the pass what could have been a series of devastating attacks on important
government agencies).
5. I won’t go into all of the brilliant features in
SUNSPOT, but I’ll mention one: SUNSPOT was designed to surmount what may have been the biggest problem the Russians faced at SolarWinds: The software build process for the next Orion
release went on for months, but neither the Russians nor SolarWinds had any
idea exactly how long it would take. Moreover, the process was shut down and
restarted every morning, and there was no way to know in the morning whether
today might be the day that everything in the new build worked perfectly. If that
happened, the SolarWinds engineers would probably decide to release the code as
it stood at the end of that day. This meant that SUNSPOT needed to wipe away
all traces of its activity every evening, once it became clear that the build process
was about to shut down for the night without having produced the final product,
since the slightest discrepancy – for example in the size of a file – would have
alerted SolarWinds that something was wrong.
6. Yet the next morning, SUNSPOT had to recreate
everything that had been wiped away the previous evening, to prepare for the possibility
that today would be the day that the code was declared ready to ship. This
whole process is described in great detail in the CrowdStrike article. The
important point is that all of this had to be done by SUNSPOT alone, without
any prompting or guidance from the Russians - who had no real time visibility at all into the software build process. Just as the Americans and Israelis who created Stuxnet couldn't control it in any way once it was inside the Natanz uranium enrichment plant, which was completely air-gapped from the rest of the world (well, not quite. It turns out contractors were allowed to bring USB sticks in to aid their work, and that's how Stuxnet got in in the first place. It was also a classic supply chain attack, although different from SolarWinds).
7.
Despite these challenges, when SUNSPOT was
deployed last February it worked perfectly. The SUNBURST malware was planted in about seven releases of the Orion platform, before the Russians decided to
remove it from the build environment and cover all of their traces last June. I guess they figured that,
with 18,000 infected targets, they just didn’t have enough good cyber resources
available to exploit all of them, let alone any new ones that might be added.
An embarrassment of riches, it seems.
What’s the lesson of all this? SolarWinds really dropped the ball. Sure, this was an unprecedented attack that nobody saw coming. But it’s impossible to believe there was no way they could have detected the Russians in their network during the 15 months between when they first entered and when SolarWinds learned about it. Unforunately, SolarWinds only learned they had been penetrated when the rest of the world learned it: after FireEye discovered they’d been compromised through SolarWinds last month.
It’s also impossible to
believe there’s no way SolarWinds could have learned that their development
process was compromised, when there were about 5-6 months during which one or
the other of the two pieces of Russian code (the benign test code and SUNBURST)
was being inserted into every Orion update that left their shop.
Do I know how SolarWinds could have detected – and presumably stopped, since that would have been very easy had their presence been known – the Russians? No I don’t. But there’s one thing that I do know (although only with hindsight, I’ll readily admit): SolarWinds, in accepting license fees from huge companies and important government agencies like DoE, DHS and NSA, should have been paying a lot more attention to cybersecurity than they did. They were clearly fat, dumb and happy and on top of the world – until they weren’t.
Companies like SolarWinds are really critical infrastructure
organizations (again, I say this with hindsight. I wouldn’t have said it at all
two months ago). The entire public has a stake in their safe, successful operation;
that stake goes well beyond the license fees that SolarWinds earns. SolarWinds can’t
hide behind the “Well, if you don’t like our product, nobody’s forcing you to
buy it” rationale. Just as with public utilities, the public has a big stake in
SolarWinds’ safety and stability. The Northeast Blackout of 2003 showed that
protection of electric reliability is too important to leave up to the
utilities on their own, and the SolarWinds attacks show that protection of
network integrity in federal agencies is too important to leave entirely up to
providers like SolarWinds. There needs to be regulation to make sure SolarWinds
and their ilk do all they can to protect that network integrity, just like
there are regulations to make sure electric utilities do all they can to
protect electric reliability (including cybersecurity).
As an economics major at the University of
Chicago a fairly long (I’ll admit it!) time ago, I was fortunate enough to take
two courses on microeconomics[ii]
with Dr. Milton Friedman. While there was never a doubt that he was in
favor of capitalism and free markets, he always made it clear that in some
cases purely market mechanisms can’t adequately protect consumers against common
threats. The main case in point at the time was air and water pollution, since
the EPA had just been created by – believe it or not – Richard Nixon. Friedman
pointed out that there was no way that individual consumers, or even individual
companies, could use market power to protect themselves against most pollution
threats; there needed to be regulation.
The same reasoning applies today.
Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.
[i] BTW, NotPetya caused $10 billion in losses, but have the Russians ever been approached about paying a penny of that in damages to any of their victims? Certainly not that I know of. This might be surprising, unless you consider the case of flight MH17. Not only have the Russians not paid a penny for that either, but I know of no concerted effort by any government to force them to pay anything at all. Once it became clear the Russians were responsible for this, I would have banned Russian aircraft from all international airspace until the government had paid all just claims – both from individuals and governments – from this attack, as well as billions in punitive damages to some agreed-upon international organization. Of course, that might have meant Putin couldn’t build his $1.5 billion palace with an indoor hockey rink, that Mr. Navalny has graciously revealed to the Russian people. But with an estimated net worth of at least $25 billion (and maybe much more), Putin could have paid the entire damages out of his own pocket and still had money on hand to assure a comfortable retirement – although in a just world his retirement will be spent in a government-paid cell deep in some prison somewhere.
[ii] Which
at the time was called “price theory” at U of C. What other schools called
macroeconomics was “monetary theory”. I don’t know what terms U of C uses now.
No comments:
Post a Comment