The SolarWinds attacks have been written about a lot because they have already had a huge impact, with more certainly to come. I don’t have anything technical to add to what you may have read, except to point out that the SANS Emergency Webinar on Monday was very good, as well as the E&E News article with an energy flavor. However, I'm very focused on supply chain cybersecurity nowadays, and I want to highlight lessons to be learned for that area in particular. I'm sure more and more of them will come out over the coming days and weeks.
This was one of the first true nation-state software supply chain attacks (other than the first and Mother of Them All). My opinion has been that there are so many vulnerabilities to be found in almost any software (or firmware), that it would be a waste of time for a nation-state to go through the trouble of a supply chain attack. This would require planting a vulnerability – meaning a backdoor – in a software product while it was being developed. Why go to all that trouble when you can probably find a few vulnerabilities in a target just by scanning it?
However, if you’re trying to reach a number of networks at once and you’re willing to put a lot of work into the effort (and obviously, there was a huge amount of work that went into this attack!), there’s no better way to do it than a software supply chain attack. The attackers (very likely Russian) were able to penetrate the software development process at SolarWinds and planted a backdoor in one or more software updates for their flagship network management system, Orion.
The choice of a network management system to host the backdoor was inspired. As was pointed out in the SANS webinar, an NMS has to have credentials to access the devices it monitors; once you’ve penetrated the NMS at a site, you probably have access to most or all of the network devices at the site – switches, routers, firewalls, etc.
Jake Williams, who presented the SANS webinar, noted that some people have said to him “We don’t store any of our data on our switches; it’s on our servers. The NMS doesn’t have access to the servers.” However, Jake pointed out that the NMS has the ability to reconfigure network devices so that it can access servers. As I said, this was an inspired choice by the attackers.
Jake noted that SolarWinds has 300,000 customers but said that “only” 18,000 of them were affected by these attacks. Of course, that’s absolutely huge. Think of the Paige Thompson attacks on AWS customers, including Capital One. She penetrated at least 37 customers (by her admission) – and that was an exceptionally large number for a single set of attacks.
Think of it this way. If the Russians (there, I said it!) had devoted the same time and effort to attack a bunch of government agencies individually, they would probably have gotten nowhere, since they would have been noticed and blocked very early in the game. If they had succeeded in penetrating even five large organizations (governmental or non), that would have been big news. But they only had to penetrate one organization – SolarWinds – in order to penetrate 18,000 organizations. Talk about efficiency!
Even though as of now, fewer than ten organizations have confirmed they were attacked (i.e. data was exfiltrated, or at least that was attempted), there are certain to be many more who will find out later that they were. But even if it turns out that only say 50 organizations were really attacked, this is probably only because the Russians haven’t had time to get to any more than that. They presumably went after the cream of the crop, meaning the Federal agencies (and national labs, it seems). At a certain point, everybody who was penetrated but not yet attacked will have taken the steps necessary to prevent an actual attack on their data, but that might not be for a while.
How could this attack have been prevented? Some people will no doubt think of CIP-013-1 R1.2.5, which requires verification of software integrity and authenticity. The problem with that is it wouldn’t have detected this attack at all. The patch was developed (with malware included) on SolarWinds’ own servers and was signed with their certificate. Of course, the hash was computed from the final product, which included the malware. Everything would have looked fine from this point of view.
And given that the malware, Sunburst, was a zero-day, there was no way it could have been detected, even if a user had scanned the patch before installing it. In fact, Jake pointed out that he would very likely have fallen victim to this attack if he had been a SolarWinds user.
The attack could only have been prevented by good software development practices on the part of SolarWinds. And how can you ensure that your supplier follows good practices? Currently, the best that a normal software customer can do – unless they’re part of the military or the nuclear power industry – is ask questions about those processes, such as
· Which of the following security controls have you implemented to protect your development environment? a) separation of IT, development and test networks; b) separate authentication for access to development and test networks; c) prohibition of removable media in the development network; d) logging all access to code; e) maintaining all code on private servers (not on GitHub and not in the cloud); f) regular code audits.
· Do you require separate authentication for access to your software development network and/or hardware manufacturing network? Yes or No.
· Do you have a documented program for secure product development, including applying security controls and secure coding techniques, within the system development life cycle? Yes or No.[i]
And what will you do if the supplier says no to questions like this? That’s where you need to get creative. These questions all correspond to real risks. We don’t yet know exactly how the Russians penetrated SolarWinds, but it presumably had something to do with improper development practices or improper protection of their development network. They really should protect the latter a lot like you protect your ESP.
You need to try to get a commitment from the supplier to address these deficiencies within a certain amount of time. If they don’t fix the problems in that time, you need to be firmer and set another date. And if they blow you off again, or they refuse even to commit to doing anything in the first place, that’s when you may bring your legal department in to pressure them.
Is this going to give you definite protection? No. As you perhaps have guessed, there ain’t nothing guaranteed in the security business. But just the fact that you’re asking the supplier questions like this will make them much more likely to try to improve than if you and their other customers never even show any interest in their development environment.
This might change if the new NIST standard being mandated by the IoT Security Improvement Act comes down strongly on the side of good development security. See this post.
I think software security is the biggest area of supply chain risk. In most cases, you’re not worried about a deliberately planted vulnerability, which is the definition of a backdoor. You’re worried about poor practices on the part of your supplier. But in either case, the result could be the same: a very bad day, like a lot of people in government had yesterday.
Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.
No comments:
Post a Comment