Another important lesson from Colonial Pipeline

I honestly think that everyone – who isn’t a user of gasoline or jet fuel that lives on the East Coast – should put together a letter of thanks to Colonial Pipeline. The successful ransomware attack on them has produced a wealth of lessons for the country. I discussed an important lesson in my post on Saturday. But I’ve now read a number of news stories and opinion pieces, that point out more lessons to be learned (although these usually aren’t specifically called out as lessons learned in the stories). Here is one important lesson, and I’ll hope to discuss one or two more in other posts – coming soon to a blog near you.

My Saturday post made the point that it was almost inevitable that Colonial would have to shut down their OT network when they suffered a ransomware attack on their IT network. I based my reasoning on the 2018 case of a very large electric utility, who had to shut down their control centers and run the grid in a multistate area from cell phones for somewhere between 8 and 24 hours, during which time they wiped and rebuilt over 10,000 systems on their IT network and an unknown (to me) number in their control centers. See this post for more on this story.

However, they didn’t shut down their OT network because the ransomware had penetrated it and caused havoc there – there were no reports of any OT systems being affected. Rather, the utility felt they had no choice but to rebuild all of the OT systems along with the IT systems, since a single infected OT system might have somehow re-infected the IT network, once they’d rebuilt all of the IT systems and brought them back up. And that would have led to a very bad day, since they would have had to wipe and rebuild all of their IT systems again. The technical name for this situation is an “Oh, s_t moment”.

I thought that Colonial might have followed the same line of reasoning when they decided to bring their OT network down, even though it hadn’t been infected. That might have been the case, but WaPo pointed out an even simpler reason why Colonial might have done this in an editorial today: If they had kept their pipelines operating while the IT network was down, they wouldn’t have been able to invoice their customers. And it’s safe to say that Colonial doesn’t feel that it should deliver gasoline through their pipeline solely as a charity. And lest you think that this indicates some sort of moral turpitude on the part of Colonial’s owners (it’s a private company, owned in part by Koch Industries), I’ll point out that there is just about no for-profit company in the US that would have behaved any differently.

But there was another problem: Ransomware attackers are no longer content with just encrypting a bunch of systems and demanding a ransom to unlock them. They first root around in the network to find valuable information that the organization they’re attacking might not want to be made public (for all sorts of reasons). Then, before they encrypt any systems, they exfiltrate this information.

When they send the ransom note to the organization, they also point out the documents that they’ve exfiltrated, and state – ever so nicely, of course – that if the organization doesn’t pay the ransom they’re demanding (and the days of $10,000 and $20,000 ransoms are over, it seems. We’re talking now about ransoms around $1 million or more), they’ll publish them. The organization will not only have to deal with a bunch of encrypted systems, but they’ll also have to deal with the consequences of whatever information was in those documents becoming publicly available. It seems that Colonial was faced with that choice, although it’s not yet known whether they paid any ransom.

Yet there’s an even more serious problem that might have motivated Colonial to bring down the OT network: Some of the documents that were exfiltrated might not only have contained embarrassing information, but they might also have contained information that could allow the attackers to take over OT systems. I hinted at this in a quote published in this article in Utility Dive today

"You can't have a ransomware attack on your IT network and not have it affect the OT network unless it's like one machine," Alrich said. "In theory," Colonial could have shut down the IT network and left its OT operating, "but in practice that's a very bad idea," he said.

OT networks often need some information from the IT side, so there can be operational impacts [of an IT attack], Alrich said. While the risk of malware migrating from IT to OT may be minimal, if it were to happen, the effects could be devastating.

Anybody involved in NERC CIP compliance understands what I’m talking about here. Not only do the CIP standards protect OT systems used to operate the power gird, but they also protect information that might be used to attack those systems – e.g. network diagrams of the OT network, lists of devices with IP addresses, configuration information for OT devices, physical maps of control centers or generating plants, etc. The CIP standards (specifically, CIP-011 Requirement 1) don’t require that this information be stored on the OT network, but they do require that access to that information be restricted to those who need it.

So if there was information like this stored on Colonial’s IT network and it was exfiltrated by the attackers before the actual ransomware attack, this might well have been another reason for Colonial’s decision to shut down the OT network. If they had left the OT network running, the attackers might have decided to use the information they’d found and attack OT systems – leading to bad things happening, including perhaps an explosion or leak somewhere on the pipeline. And that could have turned a really bad day into a disaster.

What’s the moral of this story? It’s that critical infrastructure organizations (and I include pipeline companies in that category) need to protect not just the OT network, but the information about it. this is because loss of that information could lead to consequences every bit as serious as would a direct compromise of an OT network.

P.S. Kevin Perry emailed me later on Tuesday about this. I thought what he has to say is quite interesting. Here it is: "I don’t believe for a second that Colonial Pipeline’s OT was not compromised by the ransomware.  I cannot believe that if the OT were intact, they couldn’t disconnect from the corporate IT networks and bring their OT back online.  Maybe they would need to look at indicators of compromise first, but it shouldn’t take this long if the OT were fine."

Kevin is saying that the delay in bringing the pipeline itself back is probably because the OT network has been compromised by the ransomware, as well as the IT network. However, it's also possible that Colonial decided they couldn't take the chance that the OT network wasn't compromised, even though they didn't see any evidence of it - so they decided to wipe and restore all of the OT systems as well, as the big utility decided in the 2018 case I discussed near the beginning of this post.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.