The scariest supply chain attack yet

I didn’t mean for this to happen, but it turns out that this is my third post this week about cyberattacks that were revealed to be supply chain ones. Sunday’s post described a likely attack during the 2016 election, while Monday’s was about an event at the London Stock Exchange last August that’s now suspected to be a cyberattack that – of course – came through the supply chain.

My Supply Chain Attack of the Day (I hope I don’t start doing these every day!) was called to my attention by Kevin Perry, who now that he is (semi-)retired seems to have a lot more time to delve into news articles that I either don’t see in the first place or don’t bother to read because they don’t seem interesting.

And in my defense, this article at first glance certainly seems to be one for the dustbin (or the bit bucket, to be more accurate). It describes a ransomware attack on a school district in Michigan. I’m sure that, if I even saw this article in the first place, I would never have read it, since a) it’s about a ransomware attack, and we all know that the only question every day is how many new ransomware attacks will be reported that day; and b) it’s on a school district, when it seems almost all ransomware attacks nowadays are on school districts, municipalities, mosquito abatement districts, etc.

When Kevin sent it to me, he pointed out that it was a supply chain attack, yet I didn’t even pick that up in my first reading. It was only when I went back over the article that I found this sentence in the second paragraph: “The malware…was found to have entered systems through a network connection with the district’s heating and cooling service provider.” That’s the only word about the supply chain connection.

And even then I didn’t see this as a big deal at first. After all, HVAC vendors seem to be a tried-and-trusted vector for supply chain attacks, with the most notable example being the Target attack of 2013. But then it struck me that this was the first supply chain ransomware attack that I’d heard of. And then I started thinking of what this means.

Let’s say your organization, having seen the devastation that ransomware can unleash, has spent lots of time and money in doing all the right things – especially awareness training – to prevent ransomware attacks. But then some yahoo at one of your vendors clicks on a link promising untold riches and BAM! – the dreaded screen appears, telling him he needs to pay $10,000 (or whatever) in Bitcoin to a particular address (and even providing a number to call for support, since the ransomware industry seems to lead all industries in providing excellent customer support. I wouldn’t be surprised if they’ve received some awards for this).

However, the worst part is this screen starts popping up on other computers in the vendor’s office that are networked with the yahoo’s computer, including a computer of someone who is at the time remotely connected to your network (alas, without an Intermediate System on your side to prevent infection!). And the next thing you know, computers on your own network start to be infected, despite all of your best efforts to prevent this from happening.

Hopefully, for your organization this won’t turn into one of those huge ransomware disasters that befell the cities of Baltimore and Atlanta. It turns out the school district in Michigan had the right procedures in place – and had tested them – so that the damage was remarkably limited (and they didn’t pay any ransom).  But the point remains: Ransomware is without much doubt the most serious cyber threat today. The idea that, despite all of your organization’s diligence in trying to keep it out, it could arrive directly from a vendor and cause just as much damage as if the initial breach had happened on your network, is something I find pretty scary.

So since this blog is for electric utilities, not K-12 school districts, what are the lessons to be learned? First, since the ransomware would have been blocked by an Intermediate System like what’s required by CIP-005 R2, it wouldn’t have directly penetrated any ESP (of course, this applies just to Medium and High impact BES assets. Lows don’t have ESPs and therefore aren’t required to have Intermediate Systems to protect interactive remote acess). But if your utility allowed vendor access into a system on your IT network, and didn’t have an IS in place there, you might have had an IT network infection. And obviously, that’s not very pretty. So the first lesson is you should strongly consider protecting remote access to your IT network to the same degree as to your OT network.

But there are a couple other lessons that are much more concerning:

First, as you probably know, an Intermediate System is just for Interactive Remote Access, not for machine-to-machine access. Let’s say a vendor’s system is connected into your OT network (which might be an ESP, or might not – but which is critical in either case) at the same time that the ransomware spreads on that vendor’s network. Guess what? The vendor’s system may itself get infected and spread the ransomware to your OT network. And then there might be a huge problem. I have never heard of a ransomware attack on an electric utility’s OT network, but it wouldn’t be pretty at all if it happened. Even worse, I have no idea how you could prevent this, other than banning all machine-to-machine access to your OT networks.

The second lesson is also quite scary, all the more so because it seems it may already have happened, and at a major utility. In a recent year, a major utility reported a malware attack on its IT network which required a big effort to eradicate from the network, but which was ultimately successful. The utility stressed that there hadn’t been any impact on electric operations.

However, a friend of mine who used to work for a small utility in the big utility’s control area recently told me this statement wasn’t entirely true. It’s true that the infection – which sounds like it was ransomware, not ordinary malware – didn’t actually spread to any operational networks; this includes the control centers, which are often managed by IT, since the devices in a control center are IT ones, not OT ones (i.e. they’re Intel architecture and Windows or Linux O/S, for the most part).

But the IT network was thoroughly infected, and the IT people fighting the infection decided that every single system (over 10,000) needed to be wiped clean and rebuilt from the previous day’s backup (of course, there was no question that good backups were available). But then they went further: Even though there was no evidence the Control Centers had been infected, IT decided it would be far too risky not to wipe and rebuild all of the systems in the Control Centers. If just one of them turned out to be infected, the ransomware could have spread back into the IT network, once it was brought back up.

The upshot was every Control Center server and workstation had to be brought down and rebuilt from backup. This means for a period of up to 12 hours, some (and during a part of that time, all) of the Control Center’s operations had to be conducted the old-fashioned way: by phone. Of course, this is something Control Centers rehearse all the time, and there doesn’t seem to have been any direct power system impact.

You might wonder (as did I) whether the utility identified this to be a Reportable Cyber Security Incident and reported it to the E-ISAC. I’d say that’s one for the lawyers. Monitoring and Control are BES reliability functions, so if those were really lost, then it should have been reported. But it sounds like the Control Center staff were able to keep things running – and there’s no reason at all to believe that the lights went off anywhere because of this. So in the end, I’m not particularly concerned whether this was reported or not.

But here’s the lesson: Often, people in the industry (especially if they’re involved with cybersecurity and NERC CIP) tend to think that the wall between IT and OT is impenetrable, at least when the OT network is a Medium or High impact BES asset and subject to all of the CIP requirements. That is, an infection on the IT network will literally never get through to the OT network, and OT doesn’t have to concern itself too much with what goes on on the other side of the wall. No matter what happens over there, we’ll be just fine here, and the ratepayers will never see any problem.

Well, guess what? The OT network in this case (the Control Centers) was never penetrated, yet it ended up being effectively down for many hours! The results would probably have been the same if the ransomware had actually penetrated the Control Centers. The lesson is that OT needs to pay attention to what’s going on on the IT network, because a disaster over there will eventually come over here. And someday when the CIP standards are all written as risk-based ones, I think they need to include IT neworks (not individual IT systems) in some way, to address exactly this sort of situation.

As the great poet John Dunne wrote, “Ask not for whom the bell tolls. It tolls for thee.”

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. My offer of a free webinar on CIP-013, specifically for your organization, remains open to NERC entities and vendors of hardware or software components for BES Cyber Systems. To discuss this, you can email me at the same address.