In one of Energy Central’s emails today, I saw a post by Joe Weiss that looked interesting; it was entitled “SolarWinds Orion: The Weaponization of a Network Management System”. For those who are not EC members, here’s the link to the same post on Joe’s blog (BTW, for about 4 or 5 months I’ve been putting almost all of my posts on EC, as well as in this blog. They’re the same in both venues. I’m quite happy with the level of attention my posts have received on EC).
I confess that I’ve only written a
few posts about something Joe wrote, and none of them have been positive. So I’m
happy to say now that I completely agree with everything Joe says in the post,
which points to a mistake sometimes made with network management systems (NMS),
and more often with the devices that are controlled by NMS (including UPS, battery
management systems, building control systems and power distribution units):
they are placed directly on the internet, not even behind a firewall. Of course,
this makes them ripe for attack and compromise (especially given the weaknesses
of the SNMP protocol used for network monitoring).
However, Joe admits that the SolarWinds
NMS that were compromised by the attacks announced in December were almost all
(or probably all) behind a firewall. So how could these attacks have been
prevented?
If you read any of the 15 or so
posts I’ve written about the SolarWinds’ attacks since they were announced in
mid-December – or you read some of the huge number of articles and posts that
have been written about this subject by others – you’ll probably know the
answer to this question: There’s literally nothing a SolarWinds customer could
have done to prevent the attack from happening to them in the first place,
although they could have lessened the degree of compromise through various
measures.
This is because these were pure
supply chain attacks. SolarWinds’ development environment(s) was compromised by
Russian attackers, who placed an exquisitely designed piece of malware[i] into their software build
process. That malware then placed the Sunburst malware into the code of the
updates themselves. Sunburst contained a deliberately planted zero-day vulnerability (which is
called a backdoor. A backdoor is just a deliberately-planted vulnerability, as
opposed to one that finds its way into a software product because of poor security practices – or just plain bad luck –
This means that, when customers
loaded one of the tainted updates (it looks like there were about seven such
updates), they loaded Sunburst at the same time. The Russians then took advantage
of the backdoor to penetrate the customer’s network and do nasty deeds. But don’t
worry: Those customers were mostly unimportant ones – the NSA, DHS, DoE, the
National Nuclear Safety Agency, FERC, etc. Fortunately, the Russians didn’t get
into the White House football pool server.
Because the Russians had placed the
Sunburst malware into SolarWinds updates while they were being built, the
updates were signed by SolarWinds. This means that signature verification – or comparison
of hash values - didn’t raise any red flags about the updates. And since Sunburst
used a zero-day vulnerability, it wasn’t picked up by any malware scanners in
antivirus software. There is literally nothing an organization could have done to
detect these tainted updates, and thus prevent them from being installed.
So what could have at least
mitigated the SolarWinds attacks? Joe mentions one measure – not placing the
NMS directly on the internet – that I suspect just about every SolarWinds
customer already practices. What else could have been done?
Of course, there’s a lot written
about that issue (and Fortress Information Security is conducting a webinar
on the topic on Thursday, which will most likely be quite interesting). I would
just say that in general, network and server monitoring must have really fallen
down – or just not been in place in the first place – in many of the SolarWinds
customers who were actually attacked (i.e. they not only applied the tainted
update, but the Russians exploited the malware to exfiltrate files or in general
do bad things on the network. It seems there were “only” 2-300 of those and
perhaps fewer than that – vs. the 18,000 who downloaded one of the tainted
updates).
I say this because the Russians
stopped planting Sunburst in Orion updates in June, meaning it’s likely they
were inside every compromised network for a number of months. Of course, they
were certainly very careful, but they finally slipped up and were detected because
someone who worked for FireEye noticed an unknown login to their account. Did
they make zero mistakes between (at least) June and December with every other
customer besides FireEye? That’s not very likely. It seems all of those other
customers weren’t looking very hard for evidence of attacks or compromise.
What could have actually prevented
the SolarWinds attacks in the first place? Clearly, it has to do with SolarWinds’
controls (or more likely, the lack thereof) over their development network(s). There
are two components to this. The first is the technical controls that should
have been applied to the development network(s) themselves. Those controls are
familiar to most power industry networking people, since they’re very similar
to the ones required by the NERC CIP standards to protect the electronic
security perimeter and the devices within it (including BES Cyber Systems, of
course).
Perhaps the most important of
those controls are found in CIP-005-6. These include a) complete separation between
the IT network and the ESP/development network, including separate
authentication; b) tight control over open ports and services on the ESP
firewall, as well as on the network devices themselves; and c) requiring all
outside access to devices inside the ESP to be via encrypted VPN, which is
terminated at an Intermediate System located in the DMZ between the IT and OT
(ESP) networks. Maybe in some cases all of these controls aren’t needed of
every software developer (e.g. when the developer produces games). However, in
hindsight it’s clear that SolarWinds should have done much more to protect its
development networks than it did.
There should also have been
controls like those in CIP-004-6, CIP-007-6, and CIP-010-3, including background
checks on employees with access to the development network, training for them
on appropriate security procedures, strict configuration management, logging on
all devices on the network, and perhaps multifactor authentication to important
devices.
The second component of controls
is SolarWinds’ controls on access to their network in general. Even assuming the Russians penetrated the SolarWinds IT network first, how did they do that? We can’t say today what would have prevented the Russians from penetrating that network, since
we don’t know how the network was penetrated. However, at least three
possibilities have been raised:
1.
It might have been a
supply chain attack through a Microsoft Office 365 reseller, as discussed in this
post. In this case, this would be the first documented (that I know of) multi-level
supply chain attack, where a supply chain attack was used to penetrate a
supplier, and from there another supply chain attack was executed against the
customers of the supplier.
2.
It also might have had
something to do with the fact that SolarWinds had outsourced
a lot of their software development work to organizations in Poland, the Czech
Republic and Belarus (what could possibly go wrong with that?). A rogue
developer could have placed the Sunburst malware in the update code being
developed (although this idea goes against the fact that the Russians developed
and deployed a very sophisticated piece of malware called SUNSPOT that did
everything that was needed remotely; moreover, SUNSPOT painstakingly covered up
what it did. In fact, this was almost certainly better than using a human
being to plant the Sunburst malware, since they would have inevitably made a
mistake and been detected. The SUNSPOT malware was never detected by Solar
Winds until it was too late).
3.
Finally, the Russians
could have penetrated a software development tool (presumably by planting
malware in the tool developer’s network, which would have played the same role
that SUNSPOT did with SolarWinds). Then, if SolarWinds used that tool, the
Russians wouldn’t have to penetrate SolarWinds’ development network - they
would have already been there! This might be the ultimate supply chain attack, for
reasons described in this
post.
But how could users force
SolarWinds and similar software suppliers to implement these controls? Let’s be
clear: The only way to force them to do anything is with some kind of regulation.
That may well be in order, since I think it’s clear (in retrospect, of course)
that SolarWinds is as much of a critical infrastructure provider as any
electric utility. The same consideration applies to other organizations like
cloud providers. I believe that ultimately there will need to be mandatory
controls on these organizations, perhaps structured something like what’s
required by the recently approved IoT
Cybersecurity Improvement Act (which requires NIST to develop a framework
for IoT suppliers, rather than specifying specific controls. Moreover, the Act directly
applies only to federal purchasing, although there’s a high likelihood that it
will in fact serve as a standard for all IoT devices).
So barring regulation, what can we do to get software developers in general to improve their level of development security? The same thing we do regarding anything else we want a supplier to do: nudge them along the path of righteousness. This can include questionnaires; use of contract language where possible; other means of asking them to commit to doing something, like – gasp, shudder! – calling them on the phone and asking them point blank; RFPs and other means. Are you guaranteed to get results using any of these means? Of course not. But what is guaranteed is that you won’t get any results at all if you don’t try.
Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.
[i] I
hope to write a post about that malware soon. There are lots of lessons to be
learned from it!
No comments:
Post a Comment