Is this the cloud’s Tylenol™ moment?

My last post was prompted by a Wall Street Journal article on the Capital One breach that pointed out that Paige Thompson – according to prosecutors – stole “multiple terabytes” of data from 30 companies. It also pointed out that Ms. Thompson “displayed a high level of technical knowledge of the inner workings of Amazon’s cloud”. And I had already pointed out (based on the first WSJ article on her, which was the main inspiration for this previous post), that she stated in an online forum that lots of Amazon customers make mistakes in configuring firewalls, and specifically pointed to misunderstanding Amazon’s “metadata” service as a root cause of most of those mistakes.

My last post ended on a pretty pessimistic note, suggesting that probably the only way that Amazon could completely prevent technical staff who leave on bad terms from taking out their frustration on Amazon’s customers (and Capital One is now the poster child for this) is to kill them all (i.e. “terminate with extreme prejudice, as the CIA agent told Martin Sheen at the beginning of Apocalypse Now). As we all know, this is frowned on as a poor practice in most business schools, and also could land Amazon in some legal hot water - so this really isn’t a good option. I came to the conclusion that there is no other good option based on the following considerations:

1. This isn’t an insider problem, of course. Ms. Thompson was no longer an insider when she perpetrated her attacks, having been fired by Amazon in 2016. I’m sure her credentials had been thoroughly revoked, etc. It wasn’t for lack of following normal best practices that Amazon allowed this to happen.
2. In attacking these 30 Amazon customers (and I’ll admit that the more recent WSJ article didn’t say these were all AWS customers, probably because the prosecutors didn’t say so, either), she exploited the inside knowledge she gained while working at Amazon (that was definitely true for the Capitol One breach, as well as at least the two others mentioned in the first WSJ article).
3. You might be tempted to say “Well, IT staff are fired from companies all the time, and they often take with them insider knowledge of those companies’ security controls or lack thereof. Some of them probably do use that knowledge to hack into their former employers, but this doesn’t seem to be an unmanageable problem. We’ve been living with it for years.”
4. That’s true, but what’s different when you’re talking about someone who was fired from a cloud provider is that the knowledge they have can be used to attack many customers of that provider, since they all have to configure their defenses in the provider’s environment, and they may not understand the environment well enough to do that safely. If Paige Thompson had worked for Capital One and been fired, whatever knowledge she had about C1’s defenses would be highly unlikely to be useful in attacking any other reasonably well-defended company, let alone 30 of them. That’s what makes the fact that Amazon is a cloud provider so significant.
5. And even if Ms. Thompson gets thrown into the deepest dungeon in Seattle for life plus 50 years as a deterrent to other cloud ex-insiders who would like to follow in her footsteps, that doesn’t stop those ex-insiders from monetizing their insider knowledge by selling it. I’m sure the various hacking groups (some government-sponsored) in places like Iran, North Korea, Russia and China would pay good money for that knowledge (and the price they would pay has now gone way up, given that Ms. Thompson has shown how it could be applied to reap big rewards – although she didn’t choose to go that route).

In other words, this is a potential problem on a scale not seen before in the cybersecurity world. Compared to this, ransomware hackers who make maybe a few thousand (or a few hundred thousand if they’re lucky) on each score are playing penny-ante poker. Ms. Thompson seems to have been uninterested in monetizing the data she had stolen, so it’s possible the impact of her exploits may be fairly limited. However, others who come after her will be much more interested in the filthy lucre side of things, and therefore much harder to detect (presumably they won’t brag about what they’re doing on online forums) and stop. How can this problem at least be controlled, if not solved?

The reason there were at least 30 Amazon customers (and definitely many more who Paige just never had the time to get around to hacking, there being only so many hours in the day for a busy hacker) who were such easy prey for Ms. Thompson isn’t of course that their security staff members were without exception complete idiots, but that the Amazon network environment is clearly far less understandable than it should be, for people with reasonably good security chops who are assigned the task of configuring their organization’s firewalls in the Amazon cloud. And as I pointed out in my last post, Amazon – after first putting the entire blame for the Capitol One breach on C1 – now seems to be admitting that they have to educate their customers better, as well as make changes to their “cloud subsystems” that will make these breaches less likely to happen.

Which brings me to Tylenol. I’ll forgive those of you who weren’t avidly reading the papers (and especially in the Chicago area) in 1982 if you don’t know much about this crime, in which someone spiked a number of bottles of Tylenol capsules in stores in the Chicago area with cyanide, killing seven people, but I can assure you it was a very big deal back then. Of course, the perpetrator of these murders was a piker compared to present-day mass murderers, but hey – he just didn’t have the tools that guys (and they’re all guys. I have yet to hear of a female mass murderer in the US, although there have been a number of female suicide bombers in other countries) have nowadays.

However, the bigger story wasn’t the crime itself, but Johnson and Johnson’s reaction to it. Even though it was clear that there was nothing they’d done wrong, other than not anticipating that someone would do such a thing, J&J responded in a way that has made it a textbook example of how to respond to a security threat that could potentially send your whole business down the tubes. They halted all production and advertising of Tylenol and recalled all 31 million bottles that were in circulation. Then they advertised by media and loudspeakers (in Chicago) that nobody should consume any Tylenol products they had in their possession, and they would replace any capsules with solid Tylenol pills. J&J didn’t start selling the product again until they had developed tamper-resistant bottles like the ones we’re all familiar with today.

In other words, J&J decided to treat this as the maximum problem that it could have been, even though another manufacturer might have moved heaven and earth to show this was just a local problem (only eight bottles were ever discovered that had been tampered with, all in the Chicago area. Several other people were killed in copycat attacks). Those manufacturers would probably have told people outside of Chicago that the products they currently had were safe (as they indeed were, with 99.9% certainty), and they would have probably moved with much less alacrity to provide tamper-proof bottles.

J&J realized that, while the alternative approach might have worked, and would certainly have been far less costly, there would probably always have been a cloud (no pun intended – OK, not intended until half a minute ago, anyway) hanging over Tylenol. By taking this action, they ensured that Tylenol remained the huge seller that it is today. In fact, they probably ultimately increased their sales, because there was widespread public approval of J&J and admiration for their actions.

OK, by now you’re saying “Yeah, we get it. Amazon needs to act like J&J. But they can’t put tamper-proof packaging on the cloud, so what can they do to emulate what J&J did?”

Here’s what I think Amazon should do. This is really far less radical than what J&J did, but it might possibly be enough to decisively address this problem:

1. Even though I’m sure they’ve always offered an option for customers to pay them to handle security, they obviously didn’t warn them sufficiently of the dangers they might face if they handled security for themselves. They now need to come clean and admit that. What they did was something like what Boeing did – although on a very different scale, of course - when they assumed, in designing their MCAS system for the 737 MAX, that pilots would always know exactly what they had been trained to do in every emergency, even when a bunch of different emergency lights and sirens were going off at the same time. Amazon has the chance to keep this problem contained to Capital One, and that should be their goal.
2. They should offer free security consulting to every one of their customers – going over the configuration of their firewalls and other security measures, to make sure they are properly configured.
3. They should provide training – for free, of course – to all of their customers, covering the intricacies of their infrastructure (or “cloud subsystems”) - and especially the metadata system - inasmuch as that can have an effect on the customers’ security measures.
4. Most importantly, they need to re-engineer those infrastructure systems with the objective that, as much as possible, securing customer cloud networks won’t be very much different from securing networks installed at their customers’ own data centers. And any differences should be thoroughly documented, rather than being left to the Paige Thompsons of the world to smirk about online, then use to exploit the very customers they once were charged with protecting.

Of course, this isn’t a prescription for Boeing. A course of action something like this won’t fix Boeing’s problems. I can’t begin to prescribe what they should do, although I know that what they’re doing now won’t be enough. Their problems are just beginning, because they continue to follow the “too little, too late” playbook (for example, where’s the chairman’s resignation? How could somebody possibly have presided over such a debacle and remain at the helm?). Amazon could possibly end their problems in the near future, or at least get them to the point where they’re manageable, as J&J’s problems ultimately were. But they need to do a lot more than they’ve done so far.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. To discuss this, you can email me at the same address.