Wednesday, July 21, 2021

How could a billing system attack shut down an OT network?


Yesterday, I attended an excellent webinar on a topic I’ve been waiting to have someone explain to me, “Consequence Driven Cyber Informed Engineering (CCE) – Resilience Strategies”. It was sponsored by Midwest Reliability Organization (MRO), and featured two longtime friends of mine: Jodi Jensen of WAPA and Sam Chanoski of INL. Since a recording will be available on MRO’s website soon, I won’t try to reiterate what was said in the webinar, other than saying it’s worth your while to listen to the recording.

What inspired me to write this post was Jodi’s statement, regarding the Colonial Pipeline ransomware attack, that Colonial had said that they shut down the actual pipeline (i.e. their OT network) because of the loss of their billing system (which was on the IT network). Of course, the IT network was compromised, so it had to be shut down and the machines rebuilt.

Colonial insisted that their OT network wasn’t affected by the ransomware, but they had to shut it down anyway due to the loss of their billing system. Jodi wondered why the billing system was essential to operations. In other words, couldn’t they have continued shipping petroleum products through the pipeline and worried about billing later?

I wrote three posts after the Colonial incident: Here, here, and here (in that order). In all three of them, I discussed possible reasons why the OT network (and pipeline) had to be shut down, even though the ransomware didn’t penetrate it. I also linked to a post I wrote last October, describing an incident in 2018 in which a major utility – a BA for a multi-state area – had to shut down their Control Centers (i.e. an important part of their OT network) for up to 24 hours and run the grid from cell phones, when their IT network was hit by a ransomware attack that required rebuilding 12,000 computers from scratch.

Just like in the case of Colonial, the utility swore the ransomware never penetrated their OT network (and I have no reason not to believe them), but they couldn’t take the chance that just one machine in the Control Center had been compromised. If that had happened, that one machine might have then compromised all of the IT network when it was restarted, requiring another huge shutdown and rebuild (and I’m told that this becomes much less fun the second time around, to say nothing of the third or fourth time). Which is why they shut down and rebuilt all the systems in the Control Centers as well.

I brought up that incident because this might have been another reason why Colonial shut down their pipeline. And after I wrote the second post, one of the most prolific commenters on my posts, a person named Unknown, wrote in once again to say

Like you, I also believe that Colonial shut down because they could not accurately bill customers or track their customers' assets (i.e. refined petroleum products).

Pipelines are like banks and oil in the pipeline is like cash in the bank. If a bank loses its ability to track who gave them cash (or who they loaned it to), then there is no point opening the doors, even if they can safely store the money in the vault.

Unknown wrote this because I had pointed out in the post that the Washington Post had said in an editorial (which I paraphrase), “If they had kept their pipelines operating while the IT network was down, they wouldn’t have been able to invoice their customers.” I added, “And it’s safe to say that Colonial doesn’t feel that it should deliver gasoline through their pipeline solely as a charity.”

Unknown was pointing out that it was more than the wish to avoid operating as a charity that motivated Colonial to shut down. They don’t own the gasoline they ship in their pipeline, any more than a trucking company owns the furniture they ship or a bank owns the money in its vaults. If either one loses track of what’s been entrusted to them, the trucking company or bank has to repay the entire amount (and certainly with consequential damages) to whoever shipped the furniture or deposited the money.

In other words, this isn’t like an electric distribution utility, which – at least for a brief period of time – owns the electric power they’re distributing to their customers (I’ll omit discussion of Retail Choice here). That utility has to keep the lights on, no matter what it costs them, and if they can’t bill during an emergency, they can usually bill later (the meters needed for billing are all on the OT network, so presumably an IT network shutdown wouldn’t affect them anyway). Colonial isn’t obligated to keep the cars in Georgia full of gas (nor are they paid to do that, of course). They obviously can’t keep shipping gasoline if it’s likely they’ll end up having to pay the full cost of the gas to the shippers.

I concluded my third post on Colonial by articulating the first law of nature that I’ve ever identified. Tom’s First Law of OT Networks says that “an “operations-focused” company – as opposed to an information-focused company like an insurance company or a consulting firm – will be forced to bring their OT network down if their IT network falls victim to a ransomware attack.”

I’ve been told that this can’t be considered as a new law of nature because there are already enough of those. How about Newton’s Laws of Motion? They’ve been around since the 1600s, and Einstein showed they’re not applicable in extreme conditions. Why not drop one of them, and put my law in its place? Seems sensible to me…

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the National Technology and Information Administration’s Software Component Transparency Initiative, for which I volunteer. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

No comments:

Post a Comment