Sunday, December 20, 2020

There are big cyberattacks, and then there’s SolarWinds


It was already becoming apparent to me that the scale of the damage that could be caused by the SolarWinds attacks is of an order of magnitude larger than the damage caused by any previous cyberattack, including NotPetya and Wannacry. Today I got more evidence that this is indeed the case.

Damage to SolarWinds itself

Today I had an email discussion with someone I often discuss cybersecurity issues with. I mentioned that all Orion users probably need to simply disconnect the system, as CISA has already ordered government users to do. I pointed out that it may take many months for SolarWinds to fix the problems that led to the attacks (i.e. the attacks on their customers, courtesy of the backdoor that SolarWinds unwittingly pushed out in a patch).

I also speculated that many customers won’t want to wait that long to be able to manage their networks, meaning they might abandon SW before they’re able to fix the problem. The company may suddenly find themselves in a severe cash flow problem and might have to seek bankruptcy protection – not even considering the lawsuits that will soon start to flow (and the revelation that an update server was accessible with a password of “solarwinds123” won’t help their position in lawsuits, even though that probably didn’t have an impact on the breach itself. However, it certainly speaks to their mindset regarding security).

My friend clearly thought I was exaggerating. He acknowledged that the attackers had probably taken advantage of the 6 months or so that they’ve lain in networks undiscovered to move laterally within the organizations affected (including the State Department, DoE, FERC and DHS) and plant all sorts of malware in other networks within the organization, as well as exfiltrated mountains of data.

But as for Orion itself, he thought the problem was like other software compromises: The supplier just needs to get all of their customers back running a known clean version, and make sure they keep their nose clean going forward. Of course, this alone would be a massive project, but at least it would be a bounded one. If SW needed to borrow money to fund this, they could probably do it, because the end would be clearly in sight, no matter how distant.

However, I pointed out to him that there’s no way SW could get their customers to install anything that they send out now: a “clean” version of the software, a patch, a chocolate chip cookie recipe, etc. Their software build process has been compromised, and evidently for a long time. Moreover, they clearly have no idea how it happened, since they haven’t been able to provide any good explanation (of course, this is the worst thing a company can do when they’re hacked).

In fact, I’m not sure I know of any official statement from them in the last week, other than their SEC filing that hoped to smooth things over by pointing out that “fewer than” 18,000 of their customers had applied the infected patch. Of course, it’s very reassuring that at most there are no more than 18,000 direct victims of this attack, rather than say 18 million.

So SolarWinds has to rip their entire build process apart and reconstruct it from scratch, make sure it’s working properly and is secure, and finally start providing code to their remaining customers (both of them) again. But even that won’t give their customers confidence, unless there are wholesale changes in management, and especially their internal security team. First, someone from outside will need to come in, figure out which of their management are part of the problem and which are part of the solution (and it might be hard to identify the latter). Then they will need to bring in their new team. And only then can the company be trusted to put in place a new development environment that is unlikely to be compromised.

The bottom line on this is it will be a loooong time before SolarWinds will be able to get back to business as usual. Whether they’ll still have any customers at all at that time is questionable.

Damage to everybody else

So SolarWinds has got a long road to travel before they can be at anywhere close to a “normal” state again. What about everyone else?

Slate.com put out an excellent article on Friday titled “The SolarWinds Hack Is Unlike Anything We Have Ever Seen Before”. The first paragraph contains these two sentences: “In the coming year, we won’t just be fighting about who was responsible or figuring out how this happened or assessing the fallout or repairing affected systems. That whole time, government and private sector systems will continue to actively be breached because of the malware that was surreptitiously included in updates to the SolarWinds Orion products.”

The article goes on to point out that in previous big attacks, like Equifax, Sony Pictures, and the Office of Personnel Management, there were clear targets and clear sources of the attacks. If you know the source of the attacks, you can trace, in any organization attacked, what that source did, including what data they accessed, what user accounts they compromised, what suspicious software they installed, etc. You can “be relatively confident the incident was confined to a particular department or target system and that wiping and restoring those systems would be sufficient to remove the intruders’ presence.” This would of course take a lot of work, but again it would be a bounded problem.

But since Orion servers have either direct or indirect access to everything on the network (direct in the case of switches, routers, firewalls, etc. but indirect in the case of servers and workstations – see this post for an explanation of the difference between these two cases), literally everything on the network could have been attacked – or could still be attacked.

The article turns the screw once again: “Even more worrisome is the fact that the attackers apparently made use of their initial access to targeted organizations, such as FireEye and Microsoft, to steal tools and code that would then enable them to compromise even more targets.” In other words, there’s no reason to believe the Russians haven’t penetrated – or still could penetrate - a lot more organizations than the “less than” 18,000 SolarWinds customers they have probably already penetrated.

And there’s another big group of organizations that may already be compromised, or could still be compromised: that’s the customers and business partners of all those other organizations: i.e. the Early 18,000, plus all of the companies compromised with the stolen FireEye and Microsoft tools. And perhaps the customers and business partners of those customers and business partners, etc. etc. The author states “So when I say the SolarWinds cyberespionage campaign will last years, I don’t just mean, as I usually do, that figuring out liability and settling costs and carrying out investigations will take years (though that is certainly true here). The actual, active theft of information from protected networks due to this breach will last years.”

Yet the author turns the screw again, revealing another whole dimension to the problem: “Another element adding to the challenge of trying to clean up this mess will be the thoroughness of the compromise of each individual system. Many cyberespionage activities begin with phishing campaigns or stolen credentials, which are then used to deliver malware to targeted systems. Those credentials, depending on whom they belong to and how much access that individual has, can be very effective ways to gain a toehold in a protected computer system, but they’re also very easy to change or reset when the compromise is discovered.”

However, “none of that matters with a breach like SolarWinds that granted intruders broad access to the entire network of every system it was installed on. Additionally, SolarWinds had apparently persuaded many of its customers that its Orion products needed to be exempt from existing antivirus and security restrictions on their computers because otherwise it might look like a threat or be unable to function properly[i].”

The author concludes by saying “So the access that the intruders had using the SolarWinds updates goes far beyond the access granted by many initial cyberespionage compromises, and the number of potential targets is enormous—and only growing every time we learn about the ways that each of those targets may have been leveraged to access new victims. As we continue to unravel all the different strands of this compromise, the federal government would do well to assume that its computer systems are still being actively infiltrated and not imagine that, simply having discovered this breach, they are anywhere close to reaching the end of it.”

Other than that, I’d say we’re in pretty good shape…

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.


[i] The author points out that SolarWinds wasn’t unique among network management and security vendors in requesting this be done.

No comments:

Post a Comment