Was SolarWinds even the Russians’ target?

I’m one by one pulling the threads of the two newspaper articles I first wrote about on Saturday, then Sunday, then yesterday. Yesterday, I raised the idea that the articles show SolarWinds may have first been breached by the Russians as part of an attack on an Office365 access reseller – so this was really a two-stage supply chain attack.

But early this morning – when my ideas come to me, if they’re going to come at all – I realized that, if this is true, it means SolarWinds wasn’t the Russians’ target at all. In fact the Russians were engaging in their normal “spray and pray” approach to supply chain attacks, which is simply to launch a cluster bomb in a crowded market and see who it hits.

The Times article said that 40 organizations had their credentials stolen in an initial attack on the reseller. In one case (CrowdStrike), someone was caught trying to use the credentials to penetrate the organization’s Office365 instance in the MS Azure cloud, but there was no word in the articles on who the other organizations were. If one of them was SolarWinds (and SolarWinds blamed an Office365 compromise for the original penetration of their organization), then it’s unlikely that the Russians were targeting them when they stole the credentials from the reseller.

When the attacks were first announced less than three weeks ago, people marveled (including on the SANS webinar) at what a brilliant choice it was when the Russians targeted SolarWinds. After all, they’re by far the leader in network management software, and that software can’t work unless it has the credentials for network devices. So if the Russians could penetrate SolarWinds’ customers through a software update, they would have an intelligence bonanza – which is of course what happened.

But it seems this might not have been a choice at all, just dumb luck. Some low level guy in Russian military intelligence was probably taking a lot of heat because his campaign to penetrate Azure access resellers hadn’t produced any usable results. Then one day he realized he could get into SolarWinds’ network and that could give them the keys to customers like the NSA and DHS – which is what happened.

The guy’s probably partying with Uncle Vlad as I write these words.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

Where did the SolarWinds attack start?

I’ve been thinking of the SolarWinds attack as a “classic” (if you can call anything classic that is such a recent concept) supply chain cyberattack, i.e.

1.      SolarWinds was somehow penetrated by the Russians;

2.      The Russians worked their way into the development network (which they shouldn’t have been able to do, of course, any more than they should be able to get into your electronic security perimeter);

3.      There, they loaded their malware onto a patch, which was of course immediately applied by “only” 18,000 of their customers; and

4.      The rest is history.

However, I realized as I was writing yesterday’s post that there’s a step before this – or it seems to be. I did this by putting together two facts:

1.      Soon after the attack became public, SolarWinds said they had been compromised through their hosted Office365 software.

2.      As I discussed in yesterday’s post and Saturday’s, Thursday articles in the Washington Post and New York Times said that about 40 customers of an Office365 access reseller were told last week that their credentials for access to Microsoft Azure had been stolen. One of those customers was CrowdStrike. Someone tried to get into Azure using their credentials, stolen from the reseller.

In best Sherlock Holmes fashion, I surmised from these two facts that SolarWinds might have been one of those unlucky 40 customers of the Office365 reseller. The attacker might have succeeded in getting into their Office365 instance on Azure (the CrowdStrike attacker was only caught because they tried to access the Office365 Outlook module, which CrowdStrike didn’t use), and from there found a way to penetrate the on prem SolarWinds network (of course, if you’re able to root around in all of the Office documents created by a company, as well as probably their email accounts, it’s likely it won’t take a pro too long to find a way to penetrate that company’s network as well. And these guys are pros!).

If this is correct – and it’s just a surmise, of course – this means the SolarWinds attack didn’t start with SolarWinds being penetrated. It started with the Office365 reseller being penetrated. Which means it was another two-level supply chain attack, like I described in my post yesterday.

So who’s to blame for all of this? Of course, SolarWinds still bears the majority of the blame, since they seem to have lacked some basic security controls that would have prevented the Russians from so easily penetrating their development network, as well as any ability to detect them once they were inside the network.

However – again, if my theory is correct – the Office365 reseller bears some blame, but I also put a good deal of blame on Microsoft. I honestly don’t know what kind of controls they put on their resellers, but clearly they aren’t enough. Here are two paragraphs that I added to yesterday’s post this morning (anyone who read the post in the email would have missed these, since the email went out last night), as well as the two paragraphs leading up to them. They give you an idea of what I’m thinking – still off the top of my head, of course – should happen:

But you can’t blame the Office365 reseller entirely for the fact that CrowdStrike and 40 of their other customers were compromised. In fact, I put most of the blame for this attack on Microsoft. What kind of vetting were they doing of these resellers? They need to make sure they follow certain basic security practices.

And more importantly, they need to make sure both their customers and their resellers understand that securing cloud- based networks isn’t the same thing as securing on-premises networks. This was, in my opinion, the big problem revealed by the Capital One breach, which turned out to be only one of close to 30 breaches of AWS customers by Paige Thompson, a former AWS employee. She had bragged online that she breached all of these customers through one particular AWS service – metadata - that the customers were charged with configuring, but which she said none of them understood.

I stated in a post in August 2019 that Amazon should offer free training to all customers, to make sure they understand what they need to do to secure their AWS environment. I'm told both AWS and Azure do that now, and probably did it before I wrote that post. But it's obviously not enough. I think the cloud providers should be required to identify say the top 20 security mistakes that customers and resellers make, and do external testing to determine whether any customer or reseller has made any of those mistakes. If they have, they should work with the customer/reseller to fix the problem - and if that party doesn't want to be helped, their access to the cloud should be discontinued.

Of course, the cloud providers also need to require resellers to follow more general security practices, like employee background checks and strong authentication. They’re presumably doing a lot of that already, but clearly what they’re doing now isn’t enough.[i] They should have the right to audit the reseller (of course, they can put that in their contract) and to exercise it whenever they suspect the reseller is falling down on security.

Again, there needs to be regulation to oversee this. I realize now that Microsoft, Amazon, SolarWinds, Facebook, Google, and probably a number of other companies – are really operators of critical infrastructure. The entire country has a stake in that infrastructure being run securely. All of these companies have demonstrated that – while I don’t attribute bad motives to any of them – they can use a little nudge to do the right thing.

Of course, if “the right thing” were clearly evident now, and had been clearly evident for a while, we wouldn’t be in the mess we’re in today. We’ve all been looking through a glass darkly in this matter, but maybe we can do better than that soon.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] And the courts don’t necessarily seem inclined to buy arguments from the cloud providers that their customers are responsible for breaches. The WaPo article said “When hackers stole more than 100 million credit card applications last year from a major bank’s cloud, which was provided by Amazon Web Services, customers sued the bank and AWS. In September, a federal judge denied Amazon’s motion to dismiss, saying its ‘negligent conduct’ probably ‘made the attack possible.’” In fact, this bank sounds a lot like Capital One.

In this post, Our Protagonist begins to see what the real problem is

Yesterday, I put up a post about the first of two articles in different news sources, that seemed to point to a similar problem: Attacks (probably by the Russians, according to the article) on third-party resellers of cloud services. The Washington Post article I wrote about yesterday described how at least 40 organizations, including CrowdStrike, had their Microsoft Azure cloud accounts penetrated, although at least in CrowdStrike’s case, the attack did not succeed in even reaching any sensitive data.

The attacks occurred through a third-party reseller of Azure services; the attackers were able to steal customer credentials to their Azure accounts. Note that the ultimate targets of these attacks were the customers using Azure services, not Azure or Microsoft itself. Of course, the attackers weren’t able to get into Azure directly, so they pulled off a supply chain attack to do so.

Note: While I didn’t realize it until I started writing this post, both articles are discussing the same incident, although WaPo goes into much more depth. They both mention CloudStrike as a victim, although the Times identifies the “third-party reseller” as a reseller of Office 365 services. However, I’m sure this same attack could have been launched against lots of other companies that resell cloud services, so I decided to leave the description general in this post, as the WaPo article did.

This means this was a two-level supply chain attack (which has always been theoretically possible, although I know of no example of one occurring in practice before this):

Just as in the SolarWinds attacks, the attackers wanted to get into data owned by organizations that might have information useful to the Russians; Azure was going to be their entry point to those organizations. It’s highly unlikely that the Russians were targeting any specific organizations in these attacks, just as it’s highly unlikely that the Russians attacked SolarWinds specifically in order to get into particular government agencies. They knew SolarWinds was used by all sorts of organizations, public and private, that probably stored a lot of important data. They just planted their malware in a patch under development and waited for the malware to start contacting them on their C&C server. I can only imagine the rejoicing in St. Petersburg and Moscow when they realized they had penetrated NSA, the Pentagon, DHS (including CISA), DoE, the National Nuclear Safety Administration, FERC, and other juicy plums.

However, when the Russians attacked the Azure reseller, their ultimate goal wasn’t Azure itself, but the data in the Azure accounts created by the reseller, for their customers. Had they succeeded (the article only says they didn’t succeed in the case of CrowdStrike, but it doesn’t say whether they had any success with the other “at least 40” customers of the reseller), they would have been able to get into whatever data those customers had stored on Azure through that reseller. That data might have then provided them with credentials to access data stored by the customer in other locations, such as on their own network or perhaps with other cloud providers.[i]

Why is it important that this was a two-level supply chain attack? It has to do with Microsoft’s role in the attack. As the WaPo article points out, two days after Microsoft had notified CrowdStrike that they’d been breached through the reseller, Microsoft’s president Brad Smith stated that they knew of no customers that had been attacked through Azure or Microsoft in general. But of course, he didn’t say anything about Microsoft resellers being a possible vector.

And here’s the problem:

SolarWinds was obviously the main actor at fault for the attacks that came through them. If they had properly protected their software development environment (air gapped it from the IT network with separate authentication, as well as other controls), it’s possible the Russians would never have been able to attack any of their customers – just SolarWinds itself. Obviously, the value of whatever data SolarWinds stored on its own network was miniscule compared to the value of the data that the Russians probably got from the federal government targets who were compromised.

But you can’t blame the Azure reseller entirely for the fact that CrowdStrike and 40 of their other customers were compromised. In fact, I put most of the blame for this attack on Microsoft. What kind of vetting were they doing of these resellers? They need to make sure they follow certain basic security practices.

And more importantly, they need to make sure both their customers and their resellers understand that securing cloud- based networks isn’t the same thing as securing on-premises networks. This was, in my opinion, the big problem revealed by the Capital One breach, which turned out to be only one of close to 30 breaches of AWS customers by Paige Thompson, a former AWS employee. She had bragged online that she breached all of these customers through one particular AWS service – metadata - that the customers were charged with configuring, but which she said none of them understood.

I stated in a post in August 2019 that Amazon should offer free training to all customers, to make sure they understand what they need to do to secure their AWS environment. I'm told they do that now, and Microsoft does it for Azure customers and resellers. But it's obviously not enough. I think the cloud providers should be required to identify say the top 20 security mistakes that customers and resellers make, and do external testing to determine whether any customer or reseller has made any of those mistakes. If they have, they should work with the customer/reseller to fix the problem - and if that party doesn't want to be helped, their access to the cloud should be discontinued.

Of course, the cloud providers also need to require resellers to follow more general security practices, like employee background checks and strong authentication. They’re presumably doing a lot of that already, but clearly what they’re doing now isn’t enough.[ii] They should have the right to audit the reseller (of course, they can put that in their contract) and to exercise it whenever they suspect the reseller is falling down on security.

Moreover, I think there needs to be regulation of cloud providers, as well as critical product and service providers like SolarWinds. As I said in this post last week, these vendors are really critical infrastructure providers, meaning they’re not simply selling a product or service of their own creation to users who are free not to buy it.

When US government agencies and large corporations choose to rely on SolarWinds or Microsoft to provide secure services and products to them, the American people aren’t given any choice in that decision, and in any case are in no position to determine for themselves whether or not these are trustworthy vendors. However, the American people ultimately pay all the bills when these huge breaches happen, and there’s little question that, when all the costs arising from the SolarWinds attacks are added up (if they ever are), every man, woman and child in the US will have a substantial sum to pay, in one form or another.

In the same way, residents of a city are not given any choice (except perhaps a very superficial one) in the decision of who transmits and delivers their electric power, gas or water. They rely on regulators to make sure the utilities are following appropriate safety and security guidelines.

However, I’m not in favor of direct regulation of these entities, which would of course require years of wrangling in Congress. Instead, this could be done following the same structure as the recently approved IoT Cybersecurity Improvement Act. That act calls on NIST to develop “standards and guidelines” for security of IoT devices, and on Federal agencies to require suppliers of any IoT devices that they buy to follow those standards and guidelines (with muscle from DHS behind this).

I see no reason that the same approach wouldn’t work in this case as well. No supplier would even think of dividing themselves in two, with the “safe” part just selling to the Feds and meeting all applicable safety and security requirements, and the “unsafe” part just selling to non-Federal entities and only following requirements from CPSC, OSHA, etc. They will definitely bring their entire organization up to whatever standards NIST develops.

Of course, the requirements placed on the “critical cyber infrastructure providers” (catchy phrase, no? And I like the acronym: CCIP) will be vastly different from those placed on IoT suppliers. Of course, they now need to comply with standards like FedRAMP and ISO 27001/2, which are almost exclusively aimed at protecting data in the possession of these organizations. In fact, I see no need to recreate these certifications in the CCIP standards. The certification standards can continue as they are.

Those standards are great at protecting data already in the possession of the cloud provider or service vendor. But they’re not so great at making sure the cloud provider goes beyond their own legal confines to ensure that their customers, and especially third-party service providers, also have good practices in place, and most importantly that they understand what’s really required for security.

As we have seen recently.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] Note that, if CrowdStrike or any of the other customers of the Azure reseller had other accounts on Azure, the Russians wouldn’t have been able to get into those accounts automatically, since they would of course had different credentials and protections than their accounts created by the reseller. 

[ii] And the courts don’t necessarily seem inclined to buy arguments from the cloud providers that their customers are responsible for breaches. The WaPo article said “When hackers stole more than 100 million credit card applications last year from a major bank’s cloud, which was provided by Amazon Web Services, customers sued the bank and AWS. In September, a federal judge denied Amazon’s motion to dismiss, saying its ‘negligent conduct’ probably ‘made the attack possible.’” In fact, this bank sounds a lot like Capital One.

It’s not just the SolarWinds attacks anymore

On Christmas Eve morning, I read two articles that showed me that the attacks on government agencies and private companies that came through SolarWinds just point to part of the problem. In other words, the Russians (or any other country or organization that wants to emulate them) don’t have to wait to find another network management company with poor internal security practices, in order to find other avenues by which to steal information and wreak havoc in the US. We’re providing them with plenty of other opportunities now, although neither of these articles points to any actual damages resulting from the attacks they describe. Dumb luck, I guess – but we can’t rely on that to carry us through the coming assaults.

The articles don’t seem to describe the same incidents. I will discuss the first article in this post, and the second one hopefully tomorrow.

This article was by Ellen Nakashima of the Washington Post, who has become something of a rock star among cybersecurity reporters. The article starts out, “Russian government hackers have compromised Microsoft cloud customers and stolen emails from at least one private-sector company, according to people familiar with the matter, a worrying development in Moscow’s ongoing cyberespionage campaign targeting numerous U.S. agencies and corporate computer networks.”

The article continues, “The intrusions appear to have occurred via a Microsoft corporate partner that handles cloud-access services, those familiar with the matter said.” Of course, this isn’t surprising. I would be quite surprised if the attackers had been able to succeed in a full-frontal assault on Microsoft’s cloud services themselves.

But the point is that the attackers didn’t have to do that. They followed the classic supply chain attack route: If you can’t beat down the front door of your target, find a vendor who doesn’t protect their front door nearly as well (and it seems few of them do) and penetrate them. Then search their premises to find a key to get you into your target (which can be many things, including simply the trust that customer employees place in employees of the vendor). In fact, I should have capitalized target, since Target was the target (OK, OK - no more targets in this sentence. I promise) of the first widely-reported supply chain attack, which came through an HVAC vendor who had access to the retailer’s IT network (which also held the point of sale systems that were the attackers’ ultimate goal – or target, if you will. Sorry, can’t help myself!).

So in this case, rather than waste their time trying to directly penetrate Azure (Microsoft’s cloud brand), the attackers seem to have compromised a company whose business model includes holding keys for access to Azure. The article points out that on December 15, “Microsoft notified the cybersecurity firm CrowdStrike of an issue with a third-party reseller that handles licensing for its Azure customers, according to a blog post CrowdStrike published Wednesday. In its (blog) post, CrowdStrike alerted customers that Microsoft had detected unusual behavior in CrowdStrike’s Azure account and that ‘there was an attempt to read email, which failed.’ CrowdStrike does not use Microsoft’s email service.”

In other words, someone was able to access CrowdStrike’s Azure account, but was discovered when they tried to access a nonexistent email service on the account. So it’s to Microsoft’s credit that they were able to detect this anomalous behavior. Is this actually good news?

I’m afraid not. The article points out “The troubling revelation comes several days after Microsoft’s president, Brad Smith, said the Fortune 500 company had not seen any customers breached through its services, including the vaunted Azure cloud platform used by governments, major corporations and universities worldwide. ‘I think we can give you a blanket answer that affirmatively states, no, we are not aware of any customers being attacked through Microsoft’s cloud services or any of our other services, for that matter, by this hacker,’ Smith told The Washington Post on Dec. 17.”

Note that Mr. Smith made that statement two days after Microsoft had notified CrowdStrike of the attempted breach. So while he was technically correct that no customers had been attacked through Azure itself, he conveniently left out the fact that at least one customer had been attacked through their business partner. Moreover,

In a blog post last week, Microsoft stated it was notifying “more than 40 customers” that they had been breached. Some of them were compromised through the third party, people familiar with the matter said.

Specifically, the adversary hacked the reseller, stealing credentials that can be used to gain broad access to its customers’ Azure accounts. Once inside a particular customer’s account, the adversary had the ability to read — and steal — emails, among other information.

In other words, at least 40 Azure customers were penetrated through the same Azure reseller. The fact that some of them were compromised through a breach of an Azure business partner, and not through a breach of Azure itself, is cold comfort for those customers.

Here’s the problem: An attacker who was more careful, and who had access to CrowdStrike’s network, might easily have been able to access whatever data or applications CrowdStrike has on Azure. The breach could have been just as damaging as the breaches of federal agencies through SolarWinds seem to have been – and remember, federal agencies, including the military and intelligence agencies, are already heavy users of cloud services.

In other words, this has been a two-level supply chain attack. The attacker’s targets were users of Azure, but they couldn’t just attack Azure directly, as they did with SolarWinds. So they had to attack a vendor to Azure. And the fact that they were able to penetrate at least 40 organizations' Azure accounts shows that they achieved their goal of attacking an Azure customer, even if they had to do this indirectly. It’s unfortunate that Brad Smith doesn’t understand that.

PS: Kevin Perry just forwarded me this link to a new post by Dragos, which provides a lot of good information on responding to this incident in ICS environments. It includes a discussion of NERC CIP considerations. 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

In case you can’t read everything available about SolarWinds…

Yesterday, Fortress Information Security posted an excellent, and quite readable, white paper on the Solar Winds attacks. Because Fortress’ primary focus is supply chain security for the power industry, they are especially qualified to write about these attacks, since these were (and are – they’re still ongoing) classic software supply chain attacks, although far more devastating than any previous such attacks. You can download the document here.

The article offers a good summary of what’s known about the attacks as of probably two days ago, as well as what organizations using SolarWinds should do to protect themselves, before they’re actually compromised through the infamous backdoor. However, it adds to that a set of recommendations to protect SCADA systems, as well as best practices for implementing a data security policy in general – since right now exfiltration of critical data should be the biggest concern of all SolarWinds customers, whether or not they think they have been actually compromised yet.

Fortress says they will post updates to this document as new information becomes available. I hope that at some point, once it’s known exactly how the attackers were able to plant malware in the SolarWinds update, they'll provide a comprehensive set of measures that organizations (not just electric utilities) should take to make sure their software suppliers are secure. These will mostly be recommendations for requirements that can be included in supplier questionnaires and contract language (I listed three examples in this post).

As I’ve said multiple times in this blog (e.g. in this post), I believe software (and firmware) supply chain security is the most important area of supply chain cybersecurity risk. However, I’ll admit I never imagined that so much damage could be caused by a software supply chain attack. As it is, lately – especially in the Executive Order – there has been excessive focus on hardware supply chain attacks by nation-states, which are not impossible but are far less likely. Software is where the true risk is! But maybe you already knew that...

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

SolarWinds is (was?) critical infrastructure

I’m working hard nowadays to finish my book on supply chain cybersecurity for critical infrastructure. Yesterday I was working on a chapter where I was discussing the difference between critical infrastructure industries and other industries. In doing so, I came to the realization that the essence of critical infrastructure is that owners of critical infrastructure should never have the final say on how it is used and especially how it is secured. This is because, by definition, the general public has a “stake” in critical infrastructure, even if it’s owned entirely by private parties.

Of course, everyone in the electric power industry is already well aware of this fact. If they do something wrong and there’s an outage, that isn’t just an internal problem. It needs to be reported to NERC, the federal, state and local governments, etc. Depending on what happened, there may be press reports, a fine or other public shaming.

Probably the most important “gift” that comes with owning critical infrastructure is the regulation that comes with it. Electric utilities are regulated by NERC, FERC, the PUCs and other regulatory bodies. Even though some of the regulations are overdone, all of this can be thought of as the public taking steps to guard its stake in critical infrastructure.

Of course, electric utilities aren’t unique in this regard. Oil refineries, natural gas pipelines, water treatment systems, chemical plants, transportation systems, banking, telecom and other industries all operate critical infrastructure. And they almost all are regulated in one way or the other, including for cybersecurity.

As I was writing this, I was also thinking about SolarWinds, and how it looks like some very bad security practices on their part may have led to what could be the greatest disaster – outside of wars and pandemics – ever experienced by the American people. And then I realized: SolarWinds is as much a critical infrastructure operator as any electric utility. Their software was used to manage the networks of many important agencies of the federal government (in hindsight, far too many!). Moreover, by definition network management software has to hold the “keys to the kingdom”: credentials for network infrastructure devices like routers, switches and firewalls, as well as an open pathway to control servers and workstations through those devices.

Of course, two weeks ago I would never have dreamed of calling SolarWinds a critical infrastructure operator. Even if someone had told me they had bad security, I would have said that’s basically their problem. I would have said this, even though I certainly knew that their poor security could lead to customers getting breached (e.g. as happened to Delta Airlines, who lost close to a million credit card numbers when their web site was penetrated by attackers, who used a backdoor that had been planted in recently purchased chatbot software they had deployed on their web site).

Obviously, there are lots of other companies, like software developers and other service providers, who have been breached – and those breaches have resulted in customers and other third parties being breached as well. One good example of this is Equifax, who lost control of the personal information of about half the adults in the US in 2017. When something like that happens, there’s lots of speculation about what will happen to the company’s stock price, or whether they’ll even survive as a going concern (some companies have gone under because of a cybersecurity breach). There’s even more speculation about the size of the settlement the company will make with its customers – with the presumption that once that settlement is agreed to, the incident is over and the company can begin to heal and grow again.

Because SolarWinds software managed the networks of so many important government agencies, and because the Russians had 6-9 months to root around in those networks, exfiltrate data and plant malware, it is likely the total cost of these breaches will never be known; but it will definitely be in the billions of dollars.

So there’s no question that this is no longer SolarWinds’ problem; this is the US taxpayer’s problem. Maybe there will be a settlement that will allow SolarWinds to stay in business; maybe there won’t, and SolarWinds may have to go through bankruptcy, be sold for a song, or liquidated. But none of those alternatives will come close to making taxpayers whole.

Clearly, SolarWinds was in fact a critical infrastructure operator, since the operation of a lot of the federal government depended on their good governance – and that proved elusive. What other service providers and software companies are in their position? I’m not sure – probably not many. Unfortunately, the Russians seem to have identified perhaps the single most critical of all critical infrastructure providers – long before we did.

There needs to be cybersecurity regulation of all critical infrastructure organizations, not just the ones who produce and transport natural gas or electric power.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

There are big cyberattacks, and then there’s SolarWinds

It was already becoming apparent to me that the scale of the damage that could be caused by the SolarWinds attacks is of an order of magnitude larger than the damage caused by any previous cyberattack, including NotPetya and Wannacry. Today I got more evidence that this is indeed the case.

Damage to SolarWinds itself

Today I had an email discussion with someone I often discuss cybersecurity issues with. I mentioned that all Orion users probably need to simply disconnect the system, as CISA has already ordered government users to do. I pointed out that it may take many months for SolarWinds to fix the problems that led to the attacks (i.e. the attacks on their customers, courtesy of the backdoor that SolarWinds unwittingly pushed out in a patch).

I also speculated that many customers won’t want to wait that long to be able to manage their networks, meaning they might abandon SW before they’re able to fix the problem. The company may suddenly find themselves in a severe cash flow problem and might have to seek bankruptcy protection – not even considering the lawsuits that will soon start to flow (and the revelation that an update server was accessible with a password of “solarwinds123” won’t help their position in lawsuits, even though that probably didn’t have an impact on the breach itself. However, it certainly speaks to their mindset regarding security).

My friend clearly thought I was exaggerating. He acknowledged that the attackers had probably taken advantage of the 6 months or so that they’ve lain in networks undiscovered to move laterally within the organizations affected (including the State Department, DoE, FERC and DHS) and plant all sorts of malware in other networks within the organization, as well as exfiltrated mountains of data.

But as for Orion itself, he thought the problem was like other software compromises: The supplier just needs to get all of their customers back running a known clean version, and make sure they keep their nose clean going forward. Of course, this alone would be a massive project, but at least it would be a bounded one. If SW needed to borrow money to fund this, they could probably do it, because the end would be clearly in sight, no matter how distant.

However, I pointed out to him that there’s no way SW could get their customers to install anything that they send out now: a “clean” version of the software, a patch, a chocolate chip cookie recipe, etc. Their software build process has been compromised, and evidently for a long time. Moreover, they clearly have no idea how it happened, since they haven’t been able to provide any good explanation (of course, this is the worst thing a company can do when they’re hacked).

In fact, I’m not sure I know of any official statement from them in the last week, other than their SEC filing that hoped to smooth things over by pointing out that “fewer than” 18,000 of their customers had applied the infected patch. Of course, it’s very reassuring that at most there are no more than 18,000 direct victims of this attack, rather than say 18 million.

So SolarWinds has to rip their entire build process apart and reconstruct it from scratch, make sure it’s working properly and is secure, and finally start providing code to their remaining customers (both of them) again. But even that won’t give their customers confidence, unless there are wholesale changes in management, and especially their internal security team. First, someone from outside will need to come in, figure out which of their management are part of the problem and which are part of the solution (and it might be hard to identify the latter). Then they will need to bring in their new team. And only then can the company be trusted to put in place a new development environment that is unlikely to be compromised.

The bottom line on this is it will be a loooong time before SolarWinds will be able to get back to business as usual. Whether they’ll still have any customers at all at that time is questionable.

Damage to everybody else

So SolarWinds has got a long road to travel before they can be at anywhere close to a “normal” state again. What about everyone else?

Slate.com put out an excellent article on Friday titled “The SolarWinds Hack Is Unlike Anything We Have Ever Seen Before”. The first paragraph contains these two sentences: “In the coming year, we won’t just be fighting about who was responsible or figuring out how this happened or assessing the fallout or repairing affected systems. That whole time, government and private sector systems will continue to actively be breached because of the malware that was surreptitiously included in updates to the SolarWinds Orion products.”

The article goes on to point out that in previous big attacks, like Equifax, Sony Pictures, and the Office of Personnel Management, there were clear targets and clear sources of the attacks. If you know the source of the attacks, you can trace, in any organization attacked, what that source did, including what data they accessed, what user accounts they compromised, what suspicious software they installed, etc. You can “be relatively confident the incident was confined to a particular department or target system and that wiping and restoring those systems would be sufficient to remove the intruders’ presence.” This would of course take a lot of work, but again it would be a bounded problem.

But since Orion servers have either direct or indirect access to everything on the network (direct in the case of switches, routers, firewalls, etc. but indirect in the case of servers and workstations – see this post for an explanation of the difference between these two cases), literally everything on the network could have been attacked – or could still be attacked.

The article turns the screw once again: “Even more worrisome is the fact that the attackers apparently made use of their initial access to targeted organizations, such as FireEye and Microsoft, to steal tools and code that would then enable them to compromise even more targets.” In other words, there’s no reason to believe the Russians haven’t penetrated – or still could penetrate - a lot more organizations than the “less than” 18,000 SolarWinds customers they have probably already penetrated.

And there’s another big group of organizations that may already be compromised, or could still be compromised: that’s the customers and business partners of all those other organizations: i.e. the Early 18,000, plus all of the companies compromised with the stolen FireEye and Microsoft tools. And perhaps the customers and business partners of those customers and business partners, etc. etc. The author states “So when I say the SolarWinds cyberespionage campaign will last years, I don’t just mean, as I usually do, that figuring out liability and settling costs and carrying out investigations will take years (though that is certainly true here). The actual, active theft of information from protected networks due to this breach will last years.”

Yet the author turns the screw again, revealing another whole dimension to the problem: “Another element adding to the challenge of trying to clean up this mess will be the thoroughness of the compromise of each individual system. Many cyberespionage activities begin with phishing campaigns or stolen credentials, which are then used to deliver malware to targeted systems. Those credentials, depending on whom they belong to and how much access that individual has, can be very effective ways to gain a toehold in a protected computer system, but they’re also very easy to change or reset when the compromise is discovered.”

However, “none of that matters with a breach like SolarWinds that granted intruders broad access to the entire network of every system it was installed on. Additionally, SolarWinds had apparently persuaded many of its customers that its Orion products needed to be exempt from existing antivirus and security restrictions on their computers because otherwise it might look like a threat or be unable to function properly[i].”

The author concludes by saying “So the access that the intruders had using the SolarWinds updates goes far beyond the access granted by many initial cyberespionage compromises, and the number of potential targets is enormous—and only growing every time we learn about the ways that each of those targets may have been leveraged to access new victims. As we continue to unravel all the different strands of this compromise, the federal government would do well to assume that its computer systems are still being actively infiltrated and not imagine that, simply having discovered this breach, they are anywhere close to reaching the end of it.”

Other than that, I’d say we’re in pretty good shape…

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] The author points out that SolarWinds wasn’t unique among network management and security vendors in requesting this be done.

Why are we “struggling” to respond?

Yesterday, the Wall Street Journal published an article titled “Broad Cyberattack Linked to Russia Leaves U.S. Struggling for Response”.[i] Since non-subscribers won’t be able to read it (unless you want to sign up for a free trial), I will summarize its contents in my discussion. The article starts by saying:

1.      The attacks just damaged data – unlike for example NotPetya, which of course was also a Russian operation. NotPetya rendered computer hard drives forever unreadable. It inflicted damage to many organizations worldwide, including Mondelez, Merck and FedEx in the US. And it almost sank (pun intended) Maersk, the world’s largest container shipper.

2.      As such, they were not different in principle from spying, which governments have been doing ever since there were governments, although this was far bigger in scale.

3.      The article says “U.S. intelligence agencies engage in cyberspying all the time, although U.S. officials say they don’t generally conduct destructive attacks or steal intellectual property. Because traditional cyber espionage is typically considered fair intelligence activity by most countries—even, sometimes, among allies—retaliation or public condemnation isn’t usually an option that is considered.”

4.      The article continues “Others said the sheer breadth of the SolarWinds hack makes it different from traditional cyberspying. ‘The fact that this took place on such a massive scale sort of puts it in a different category,’ said John Dermody, counsel at the O’Melveny law firm and former deputy legal adviser at the National Security Council. The economic costs could be enormous, as companies scour their networks to determine whether the perpetrators installed additional malware, he said.”

But I think the above statements miss the real nature of this problem: The fact that the Russians have been in the affected networks for so long (at least since last June, and in some cases since March) means they have had plenty of time to exfiltrate massive amounts of data, but also to cover their tracks so that it will be quite hard, and in many cases simply impossible, to discover every nook and cranny where they’re hiding.

This in turn means that employees of the agencies that are known to have been penetrated, including DoE - some parts of DoD, DHS, the State Department, the National Nuclear Security Administration, FERC, at least a couple National Labs, and certainly more to be discovered - have to assume until further notice that Vladimir Putin is BCC’d on every email they send, and that he will be the first one to read any new document they create.

Some networks – especially the most sensitive ones – will need to be rebuilt from the ground up, but this is a tremendously expensive exercise. Other networks will have to be cleansed as much as possible and employees will just have to get used to their new world. What will be the effects of this new situation? It’s hard to say at this point, but it’s safe to say that probably a lot less work will get done, since a large percentage of federal government employees – traditionally risk averse in the best of situations - will no longer be willing to take any risks at all; this means that a lot of government processes will simply grind to a halt. In other words, even say five years from now we will have a much less functional government than we did before this week, to say nothing of one that holds very few secrets that the Russians don’t already know.

What to do about this? The article concludes:

James Lewis, a cybersecurity expert at the Center for Strategic and International Studies think tank, said that Washington should exact a penalty for the SolarWinds hack. He cited the Cyber Command’s Task Force ARES, which in 2016 disrupted Islamic State’s ability to communicate, spread propaganda and recruit for the terrorist network, one of the first instances of U.S. offensive cyberwarfare.

“You interfere with the opponents’ ability to conduct operations. You sit on their networks,” Mr. Lewis said. “We really have to take a look at taking some kind of action against the Russians.”

Thomas Bossert, whose article in the NY Times I discussed in my post on Thursday, said in today’s Washington Post:

“The United States can now direct its focus and unite the world against this outrage.” He said the Russian government is holding American networks at risk. “We must impose a cost on the Russians,” he said. “Until we start defending digital infrastructure as if commercial and government operations depended on it, we will remain rudderless.”

Well guess what? Commercial and government operations depend on it. So what do we do now?

I would say there are four possible types of response:

1.      Military. This would obviously be quite dangerous, since the US and Russia are both nuclear powers. It’s highly unlikely that we would take military action in response to a cyberattack, and rightly so.

2.      Kinetic cyberattack. For example, an attack on the Russian grid that would put a lot of people in the dark and might well result in a few deaths. There are some people who seem to think that we should respond to a kinetic cyberattack on our grid with one of our own – and it’s highly likely we’ve already planted malware in Russia’s grid, just like they’ve planted it in ours.[ii] However, I discussed in this post why that would be a very bad idea. Briefly, especially if we end up killing people in our attack, it’s very possible that Russia might respond militarily. And once we’re in the military realm, a single mistake might literally result in the end of Western civilization (and most of us with it!), as almost happened during the Cuban Missile Crisis of 1962 (see the post just referenced for the hair-raising story of that incident). But most importantly, the SolarWinds attack wasn’t a kinetic one.

3.      Non-kinetic cyberattack. This is an attack that affects information – most likely by exfiltrating it, but perhaps by destroying it. This is possibly “all” that happened in the SolarWinds attacks, although up until now there have been no reports of information being destroyed (and important information is probably backed up so many times and in so many places that it would be impossible to destroy it anyway). But if we had found a way to massively exfiltrate Russian data, why haven’t we done it by now? We wouldn’t wait for them to launch a huge cyberattack on us, since spying is something that governments have done on each other ever since there were governments.

4.      Something else.

What could that be? Individual sanctions have been applied to Russia before, but unless they target the people directly responsible for these attacks, the deterrence message will not be clear. However, further financial sanctions on Russia, beyond those already in place because of what he’s done in the Ukraine, especially ones that would bite his oligarch friends and their businesses, might inflict damage where it’s most needed.

In 2014, the US is widely believed to have caused a total collapse of North Korea’s internet access. Something along those lines – that would be widespread and perhaps cause a lot of Russians to wonder why they support this guy – would also be great. It doesn’t have to be immediate. Let’s plan it carefully before it happens.

And I can think of one way the US could cause serious pain to the individual most responsible for these attacks, one V. Putin. We can do that by releasing whatever information we have on his personal misdeeds. This includes:

1.      His financial assets. He’s estimated to be worth up to $200 billion, but it’s unlikely the CIA has documentation for all of that. Whatever evidence we have of any large assets – such as the approximately $1 billion mansion widely reported to have been built for him – should be laid out for the citizens of Russia to see.

2.      The bombings of five apartment buildings in Moscow in 1999. Putin was then prime minister, and his effective and deadly response to the bombings – which he blamed on Chechnya and which led to the Second Chechnya War, that pretty much leveled a lot of that country – led to his becoming president a few months later. And he’s been president in fact, if not always in title, ever since. There have been many accusations that Putin himself was behind the bombings, and at least a few people who investigated them were assassinated in subsequent years. Does the CIA have any information that hasn’t already been made public on these bombings? If so, this would be a great time to release it.

Putin is worried about his popularity in Russia, given that he didn’t handle the coronavirus well and there are huge questions about the vaccine that he rushed into production without adequate testing. Whatever we can do to make this problem worse would be great.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] The headline on the electronic copy is different from this. This is the headline on the physical copy in this morning’s paper. The article is mostly the same, though.

[ii] This was what the FBI, CIA and ONI said in the 2019 Worldwide Threat Assessment. There has been no reported investigation of that statement, whether classified or unclassified. And the administration avoided the whole embarrassing issue this year by not putting out a WTA at all. This is of course very reasonable: It’s much better not to let the American people know something about the threats they face, than to possibly say some bad things about Vladimir Putin. Of course, that’s the same dynamic going on in the current case.

This is much bigger than anybody thought

If you want to read something really depressing – or if you don’t really want to, but you know you have to – then read these two articles about the SolarWinds attacks. The first is by Christian Vasquez of E&E News, discussing the impact on the energy industry. This came out before news that DoE, Los Alamos and Sandia National Labs, and the National Nuclear Safety Administration were attacked.

And FERC was attacked as well. Politico says “The hackers have been able to do more damage at FERC than the other agencies, and officials there have evidence of highly malicious activity, the officials said, but did not elaborate.”

The second article is an opinion piece in the NY Times, written by Thomas Bossert, former homeland security advisor to President Trump. One of its most important points is that the Russians have been inside some very important networks for many months. They have had the chance to take out lots of data and insert a lot of new malware (not the Sunburst malware, of course. That’s what got them in the door, but they have known all along that once this was be discovered, it would immediately be blocked). 

With so many networks now untrusted, a number of critical networks (all of them?) will need to be completely rebuilt, with of course lots of care to make sure that any compromises on the current networks are just transferred to the new ones.

I’d say the most depressing part of this article is these three paragraphs:

President Trump is on the verge of leaving behind a federal government, and perhaps a large number of major industries, compromised by the Russian government. He must use whatever leverage he can muster to protect the United States and severely punish the Russians.

President-elect Joe Biden must begin his planning to take charge of this crisis. He has to assume that communications about this matter are being read by Russia, and assume that any government data or email could be falsified.

At this moment, the two teams must find a way to cooperate.

Will that happen?

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

Further supply chain security lessons from SolarWinds

I’ve come to realize that there are a lot of supply chain cybersecurity lessons to be learned from the SolarWinds attack, beyond what I discussed in my post yesterday. Here are a few more. Note that two of these are risks from the list I’ve compiled with my clients for CIP-013 compliance purposes, but they’re very relevant in this context. Of course, you don’t need to be subject to CIP-013 compliance to use them!

First, you should contact your suppliers and find out if they run SolarWinds – and demand they disconnect it if they do.

Second, you should ask your supplier whether they have a configuration management system installed in their development environment. There’s no way to know for sure, but that might have caught the attack by the Russians in the first place.

Third, ask the supplier whether they have a process in place to investigate whether computer viruses or malware are present in any software or patches before providing such software or patches (note: this is one of the questions in the NATF questionnaire). Since the SUNBURST malware used by the Russians is based on a zero-day vulnerability, it wouldn’t have been picked up in SolarWinds’ case. On the other hand, if a supplier isn’t taking this step, their patches might have non-zero-day malware as well.    

Second, in an article titled “Hey, only as many as 18,000 customers installed backdoored software linked to US govt hacks”, The Register points out that there’s evidence that SolarWinds’ Office365 account “was hacked and its emails accessed, possibly leading to the dodgy downloads”. Of course, this isn’t proven, but it points to something that I’ve wondered about for a while: A software supplier’s development network(s) should be treated like an electric utility treats its Electronic Security Perimeter. It should be isolated from the IT networks, with no direct communication between systems on the two networks. Of course, separate authentication should be required for the two networks.

It sounds like that might not have been the case with SolarWinds, since otherwise a compromise of email and other office productivity systems would be dismissed out of hand as a possible cause of penetration of the development environment. My guess is a lot of other software suppliers may be in the same position. You definitely need to ask your suppliers questions about this (examples are in yesterday’s post).

You might wonder how much good it does to ask your supplier a question. I have two answers for that:

1.      It doesn’t do any good if you don’t follow up with the supplier whenever an answer indicates the likelihood of the risk being present is more than low. And you not only need to follow up with them, but you also need to try to get them to implement mitigation(s) for this risk.

2.      Remember that just the fact that you asked about a particular risk sends a message. For example, in item 2 above, that question sends the message: “Configuration management is very important. We are concerned that if you don’t manage configurations in your development environment, you might be penetrated by the Russians or some other malicious actors – just like SolarWinds was.”

3.      Even if the supplier doesn’t have config management but lies and says they do, right after they send you their answers to the questionnaire, the person filling it out might pick up the phone and say “Hey Joe. You remember that proposal for configuration management we’ve had on our desk for months? Let’s get it approved.” This will be a win for you, even though you will probably never know it happened. 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

Supply chain security lessons from the SolarWinds attacks

The SolarWinds attacks have been written about a lot because they have already had a huge impact, with more certainly to come. I don’t have anything technical to add to what you may have read, except to point out that the SANS Emergency Webinar on Monday was very good, as well as the E&E News article with an energy flavor. However, I'm very focused on supply chain cybersecurity nowadays, and I want to highlight lessons to be learned for that area in particular. I'm sure more and more of them will come out over the coming days and weeks.

This was one of the first true nation-state software supply chain attacks (other than the first and Mother of Them All). My opinion has been that there are so many vulnerabilities to be found in almost any software (or firmware), that it would be a waste of time for a nation-state to go through the trouble of a supply chain attack. This would require planting a vulnerability – meaning a backdoor – in a software product while it was being developed.  Why go to all that trouble when you can probably find a few vulnerabilities in a target just by scanning it?

However, if you’re trying to reach a number of networks at once and you’re willing to put a lot of work into the effort (and obviously, there was a huge amount of work that went into this attack!), there’s no better way to do it than a software supply chain attack. The attackers (very likely Russian) were able to penetrate the software development process at SolarWinds and planted a backdoor in one or more software updates for their flagship network management system, Orion.

The choice of a network management system to host the backdoor was inspired. As was pointed out in the SANS webinar, an NMS has to have credentials to access the devices it monitors; once you’ve penetrated the NMS at a site, you probably have access to most or all of the network devices at the site – switches, routers, firewalls, etc.

Jake Williams, who presented the SANS webinar, noted that some people have said to him “We don’t store any of our data on our switches; it’s on our servers. The NMS doesn’t have access to the servers.” However, Jake pointed out that the NMS has the ability to reconfigure network devices so that it can access servers. As I said, this was an inspired choice by the attackers.

Jake noted that SolarWinds has 300,000 customers but said that “only” 18,000 of them were affected by these attacks. Of course, that’s absolutely huge. Think of the Paige Thompson attacks on AWS customers, including Capital One. She penetrated at least 37 customers (by her admission) – and that was an exceptionally large number for a single set of attacks.

Think of it this way. If the Russians (there, I said it!) had devoted the same time and effort to attack a bunch of government agencies individually, they would probably have gotten nowhere, since they would have been noticed and blocked very early in the game. If they had succeeded in penetrating even five large organizations (governmental or non), that would have been big news. But they only had to penetrate one organization – SolarWinds – in order to penetrate 18,000 organizations. Talk about efficiency!

Even though as of now, fewer than ten organizations have confirmed they were attacked (i.e. data was exfiltrated, or at least that was attempted), there are certain to be many more who will find out later that they were. But even if it turns out that only say 50 organizations were really attacked, this is probably only because the Russians haven’t had time to get to any more than that. They presumably went after the cream of the crop, meaning the Federal agencies (and national labs, it seems). At a certain point, everybody who was penetrated but not yet attacked will have taken the steps necessary to prevent an actual attack on their data, but that might not be for a while.

How could this attack have been prevented? Some people will no doubt think of CIP-013-1 R1.2.5, which requires verification of software integrity and authenticity. The problem with that is it wouldn’t have detected this attack at all. The patch was developed (with malware included) on SolarWinds’ own servers and was signed with their certificate. Of course, the hash was computed from the final product, which included the malware. Everything would have looked fine from this point of view.

And given that the malware, Sunburst, was a zero-day, there was no way it could have been detected, even if a user had scanned the patch before installing it. In fact, Jake pointed out that he would very likely have fallen victim to this attack if he had been a SolarWinds user.

The attack could only have been prevented by good software development practices on the part of SolarWinds. And how can you ensure that your supplier follows good practices? Currently, the best that a normal software customer can do – unless they’re part of the military or the nuclear power industry – is ask questions about those processes, such as

·        Which of the following security controls have you implemented to protect your development environment? a) separation of IT, development and test networks; b) separate authentication for access to development and test networks; c) prohibition of removable media in the development network; d) logging all access to code; e) maintaining all code on private servers (not on GitHub and not in the cloud); f) regular code audits.

·        Do you require separate authentication for access to your software development network and/or hardware manufacturing network? Yes or No.

·        Do you have a documented program for secure product development, including applying security controls and secure coding techniques, within the system development life cycle? Yes or No.[i]

And what will you do if the supplier says no to questions like this? That’s where you need to get creative. These questions all correspond to real risks. We don’t yet know exactly how the Russians penetrated SolarWinds, but it presumably had something to do with improper development practices or improper protection of their development network. They really should protect the latter a lot like you protect your ESP.

You need to try to get a commitment from the supplier to address these deficiencies within a certain amount of time. If they don’t fix the problems in that time, you need to be firmer and set another date. And if they blow you off again, or they refuse even to commit to doing anything in the first place, that’s when you may bring your legal department in to pressure them.

Is this going to give you definite protection? No. As you perhaps have guessed, there ain’t nothing guaranteed in the security business. But just the fact that you’re asking the supplier questions like this will make them much more likely to try to improve than if you and their other customers never even show any interest in their development environment.

This might change if the new NIST standard being mandated by the IoT Security Improvement Act comes down strongly on the side of good development security. See this post.

I think software security is the biggest area of supply chain risk. In most cases, you’re not worried about a deliberately planted vulnerability, which is the definition of a backdoor. You’re worried about poor practices on the part of your supplier. But in either case, the result could be the same: a very bad day, like a lot of people in government had yesterday.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] Note that I always word my questions as multiple choice or Yes/No. See this post for more discussion of that.