Is the Okta supply chain attack worse than SolarWinds? The response sure was…

I admit that I didn’t pay a lot of attention when the Okta hack was first announced. On March 23, the Washington Post said:

In a detailed blog post (in January, when the attack was discovered), (David) Bradbury (Okta CSO) said Okta identified a potential security compromise in January. After an investigation, it found that a hacker had obtained remote access to a contractor’s computer. The hacker appears to have had access for five days.

“So while the attacker never gained access to the Okta service via account takeover, a machine that was logged into Okta was compromised and they were able to obtain screenshots and control the machine through the (remote desktop protocol) session,” Bradbury wrote.

Customer service contractors cannot download customer databases or access source code, he wrote.

Not too bad, right? A help desk contractor’s machine was compromised while it was remotely logged into Okta’s network. But a help desk contractor would presumably just have access to a trouble ticketing system and maybe some support documentation – and the CSO himself says that customer service contractors can’t access customer databases. No harm, no foul?

That would be a reasonable conclusion to draw, even though Okta isn’t your run-of-the-mill say chicken processor. They provide third party identity and access management services to thousands of companies. Obviously, a company like that holds “keys to the kingdom” to their customers’ networks, just like SolarWinds did (and still does). A real breach might allow the attackers to penetrate lots of other customers. The SolarWinds attackers (aka the Russian government, who showed great competence in that cyberattack. However, it turns out those are the only types of attacks they’re good at, which is good news for the rest of the world, but not such good news for my Uncle Vlad)

But in January, Okta said there hadn’t been any adverse consequences other than somebody being able to see screens that a run-of-the-mill help desk contractor would see. The world moved on (if it had even paused in the first place).

That is, until March 21, when, according to a great article in Wired,

ON MONDAY EVENING, the Lapsus$ digital extortion gang published a series of increasingly shocking posts in its Telegram channel. First, the group dumped what it claims is extensive source code from Microsoft's Bing search engine, Bing Maps, and Cortana virtual assistant software. A potential breach of an organization as big and security-conscious as Microsoft would be significant in itself, but the group followed the post with something even more alarming: screenshots apparently taken on January 21 that seem to show Lapsus$ in control of an Okta administrative or “super user” account.

So now it seems this was a little more than a compromise of some single-shingle help desk contractor somewhere, where the prize was just some screenshots. This was a compromise of Okta itself, which included access to a super user account; and the compromise started with Sitel, a large company that provides customer support services to other organizations. Since a super user can pretty much do what they want on the network, including presumably getting into all customer accounts, perhaps this explains how Lapsus$ was able to pull off some fantastic heists since December, as the Wired article goes on to say:

Lapsus$ has been on a tear since it emerged in December, stealing source code and other valuable data from increasingly prominent companies, including Nvidia, Samsung, and Ubisoft, and leaking it in apparent extortion attempts. But researchers had only found broadly that the attackers seemed to be using phishing to compromise their victims. It wasn't clear how a previously unknown and seemingly amateur group had pulled off such monumental data heists. Now it seems possible that some of those high-profile breaches stemmed from the group's Okta compromise.

However, even last week, Okta was trying to downplay what happened. The same Wired article says:

“In late January 2022, Okta detected an attempt to compromise the account of a third-party customer support engineer working for one of our subprocessors. The matter was investigated and contained by the subprocessor,” Okta CEO Todd McKinnon said in a statement (presumably made on March 21). “We believe the screenshots shared online are connected to this January event. Based on our investigation to date, there is no evidence of ongoing malicious activity beyond the activity detected in January.”

But Mr. McKinnon probably already regrets that statement. Another great article appeared in Wired yesterday. It shows that on March 17, Okta received a Mandiant report on the Sitel breach, which made it clear the attackers had super user access. Yet Okta sat on the report for four days, and only mentioned it when Lapsus$ put up the screen shots showing the super user access the following Monday.

Of course, this is a wonderful way to treat your customers – don’t even let them know there’s a strong possibility that some bad guys were able to access lots of accounts on their networks, since they had access to every one of the usernames/passwords those customers had entrusted to them. Of course, maybe they just forgot...

The bottom line of this story – at the moment – is that:

·        Okta’s “subprocessor”, Sitel, had terrible security, as revealed by the Mandiant report. Did Okta even bother to verify their security? One guesses not.

·        Perhaps Okta didn’t verify Sitel’s security because theirs wasn’t a lot better. After all, if a help desk contractor somehow can obtain access to a super user account (which obviously wasn’t protected by multifactor authentication. Was that too much to ask, Okta?), that doesn’t indicate stellar security practices.

·        By Okta’s own admission, up to 366 of their customers’ credentials might have been compromised. In way of comparison, even though 18,000 SolarWinds customers downloaded the compromised Orion updates, only 1-200 of them were actually attacked by the Russians.

·        This is a true multi-level supply chain attack. Lapsus$ compromised Sitel, and from there compromised Okta. Even with Okta’s poor security practices, it would probably have been hard to compromise them with a direct frontal assault. Once in Sitel, Lapsus$ was then able to obtain the credentials it needed to compromise their customers. So this is worse than SolarWinds, which was a single-level supply chain attack; it’s more like Kaseya, which was also a multilevel attack.

·        Okta should have paid attention to SolarWinds’ response to their breach, which was to come out right away with full information, broadcast for everyone to see (and a the time, they got a lot of pushback from the FBI for doing that). Given the extent of damage that the SolarWinds attack may have caused, it’s quite possible they would have gone under, had they not taken that approach. Okta seems to have chosen a different path regarding disclosure; it remains to be seen whether that will ultimately be seen as a smart move or a disaster.

Once again, in hindsight it’s clear that Okta really is a critical infrastructure provider, just like SolarWinds is. They both need to be regulated as such.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

Here’s an idea: Let’s investigate the threats we know are real. We can leave the highly unlikely ones for another day.

On Wednesday, I was quoted in a good article by Robert Walton in Utility Dive, about an FBI bulletin announcing “abnormal scanning” of five electric utilities and 18 other critical infrastructure organizations from Russian IP addresses. Cue the scary music.

My primary reaction to this was, does the FBI think this is news? Since just about every big utility in the country is scanned probably thousands of times an hour (and I’m sure a lot of those scans come from Russia), the fact that five of them are now getting a few extra scans from Russia doesn’t make me want to check my flashlight batteries and lay in a store of MREs for the coming dark days. And given that it would be nearly impossible for the Russians (or anybody else) to cause an outage through a direct attack from the internet, I’m not worried about what would happen, even in the unlikely event that the Russians did break through the firewalls.

But if FBI Director Christopher Wray is worried about the Russians attacking the grid, he might want to go back to something that he and Gina Haspel, then Director of the CIA, said in the Worldwide Threat Assessment briefing to Congress in January 2019: "Russia has the ability to execute cyberattacks in the United States that generate localized, temporary disruptive effects on critical infrastructure — such as disrupting an electrical distribution network for at least a few hours.”

In other words, in 2019, Director Wray implied that the Russians had planted malware in the grid and could cause outages anytime they want to. Yet it seems neither the FBI nor anyone else in the federal government ever investigated these statements, since there were never any reports or briefings (classified or unclassified) to the power sector of any kind (whereas after the first Ukraine grid attack in 2015, there were classified and unclassified briefings across the country, as well as some very good reports).

Which leads me to believe that, if the malware was there in January 2019 (and the former Deputy Director of the NSA told a similar story in May 2019, although with much larger numbers), it’s still there now. Why would the Russians want to knock themselves out trying to break through the firewalls of large utilities, if they can just activate their malware and cause some big outages?

And Director Wray can’t blame the lack of an investigation on his predecessor!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

Reminder – tomorrow’s Tech Talk

I want to remind you that I’ll be speaking on “Better living through SBOMs” at a Tech Talk run by the RF region of NERC tomorrow (no registration required). I also want to remind you that, a) if you’re a NERC entity that’s part of a region other than RF, or b) if you have no association with the electric power industry other than you know how to turn on a light switch, you’re welcome at the meeting. My presentation is aimed at all industries, since there literally is no difference (that I have seen so far) between using SBOMs in an electric utility and using them in – say – an insurance company.

The Tech Talk won’t be recorded, although my slides will be posted on RF’s website afterwards.

 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

A Russian cyberattack can’t take down the grid. But what can it do?

Last week,  I was contacted by one of the five reporters who were working on this CNN story that appeared yesterday. I’ve never had as much interaction with a reporter for one story – we emailed and talked several times, and I wrote a small position paper for her, mainly so I could get my ideas straight and not give her a jumbled mess of statements.

It was interesting to see her evolution (which I believe was the evolution of the other reporters as well). When she first contacted me, what she really wanted to know about was how the Russians could take down the grid – not whether it could be done at all. As I argued in this post last year, it’s physically impossible for any kind of disturbance – other than an EMP attack, which is by far the worst threat the US faces, other than a full nuclear war, of course – to take down all three Interconnections that make up the US grid.

Even taking down one Interconnect would be close to impossible. That is, except for ERCOT, the small Interconnect that covers most of Texas, which almost succeeded in taking itself down early in the morning of February 15, 2021. But you can’t blame the Russians for that one!

Her main thrust then became that the Russians could cause a lot of small outages by attacking small utilities (especially distribution ones) that aren’t covered by the NERC CIP cybersecurity standards. But I pointed out that there’s never been a cyberattack that’s caused an outage, no matter how small, in North America. How is Russia going to cause a bunch of small attacks? And more importantly, how would you distinguish these from the daily attacks caused by the two most important causes of power outages: squirrels and copper theft?

But then what do I think are the biggest cyber threats to the grid? As I said in the article (near the end), there are really two. First, a ransomware attack that takes down the IT network results in the OT network having to be brought down as well, even though it wasn’t directly compromised. This isn’t a theoretical risk. This is what happened with Colonial Pipeline.

And more relevant for the article (although the reporter didn’t mention this), a devastating 2018 ransomware attack on the IT network of a very large electric utility resulted in the utility having to bring down a thousand or so systems in their control centers and re-image them. They lost the ability to monitor the grid in real time and also lost their VOIP phones. What saved the grid in the 3-5 states that were dependent on those control centers? Pure luck and a bunch of dedicated control center staff members who pulled out their cell phones and kept things on an even keel. For 24 hours.

Fortunately, there were no serious incidents that could have caused a sudden cascading outage, but without real-time monitoring, it’s very unlikely they would have even known there was a problem, if such an incident had occurred.

The second threat I pointed to was a software supply chain attack. I honestly can’t think of a likely scenario where such an attack would cause a widespread grid outage. But, given that software supply chain attacks increased by something like 10,000% in the last decade (and are still doubling or tripling every year. Read the story of the dependency confusion attacks in 2021, in which a researcher’s demonstration of a successful attack he conducted – without damage, of course – led within days to thousands of copycat attacks), it’s hard to rule anything out with software supply chain attacks.

So there’s plenty to worry about in grid attacks, even though there’s no chance that the whole country will go dark….Other than an EMP attack, of course. Or full-scale nuclear war. But then the lack of power will be the least of our problems.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

SBOMs and NERC CIP-013 compliance

Next Monday, I’ve been invited to discuss using SBOMs (and VEXes) during a monthly Tech Talk sponsored by RF, one of the six NERC Regional Entities that works with electric utilities to comply with the NERC standards (including the NERC CIP standards). The Tech Talk will run from 2:00 to 3:30 PM Eastern Time on March 21, and will be available at this URL. No pre-registration is required. The Tech Talk won’t be recorded.

My talk will probably start about 10-15 minutes into the program and will run (with Q&A) for about 40 minutes after that. If you’re with a NERC entity, you might want to stay around after I’m finished to listen to three very knowledgeable people from the North American Transmission Forum (NATF). They will “provide an update on supply chain risk management efforts and their proposed Implementation Guidance.”

Even though my stated topic is how SBOMs can help an electric utility comply with the NERC CIP-013 supply chain cybersecurity risk management standard, I want to point out that there will be literally nothing in my presentation that won’t be of interest to any organization (in any industry or government) that is concerned about software supply chain cybersecurity risks - and would like to know how SBOMs and VEXes can be used to mitigate those risks.

Here is alternative access information:

Meeting Number/Access Code:  2313 701 2627
Meeting Password:  0123456789
Join by Phone:  1-650-479-3207

Please join us on Slido.com using #TechTalkRF as the event code.

I’ll hope to see you there!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

You don’t need SBOMs! You don’t need VEXes!

There has been much written about software bills of materials and how they will do wonderful things for users – everything from curing asthma to finding lost car keys. However, a great deal of what has been written, including (truth be told) some of my posts on SBOMs, has focused mostly on the ultimate benefits of having SBOMs:

·        You’ll be able to find and remediate every instance of the next log4j or Ripple20 vulnerabilities that comes down the pike.

·        When you’re purchasing new software or intelligent devices, you’ll be able to instantly determine which of the products you’re considering are “safe” and which aren’t.

·        As you use software, you’ll be able to learn immediately of any critical new vulnerability in one of the components of a product you use, so that you can immediately bug your supplier to get it patched.

I can promise you’ll be able to do all of these things and more…within the next 5-10 years. But what about the near term? And if you work for a federal agency, what will you be able to do next August, with all of the SBOMs that your suppliers will be required to start pushing your way? After all, an SBOM just lists components. By itself, it doesn’t give you any information about the risks that apply to those components, the most important being the risks due to vulnerabilities found in the components.

Of course, when you have the SBOMs, you’ll then need to go to the National Vulnerability Database (NVD) and find all of the vulnerabilities that apply to components. Without a low-cost or open source tool or service to do this for you, you’ll find this involves a huge amount of drudge work, since you will often have to search for the CPE names of the components and since the average product has 135 components and lots of products have thousands of components. Plus, you should really repeat this exercise at least once a week and hopefully more often, for every software product or intelligent device on your network.

Fortunately, there is one open source tool that will do this for you now: Dependency-Track. In addition, there will be at least one or two low-cost services available in the next few months, to do what that tool does. And if open source isn’t your cup of tea, there are commercial tools that will do this, ranging from not-too-expensive to quite expensive.

However, there’s another thing that’s needed: Because the great majority of component vulnerabilities aren’t exploitable in the software product itself, you’ll waste a lot of time (and you’ll waste the supplier’s time) trying to find component vulnerabilities that aren’t really there, even though the NVD swears up and down that they’re in the components. You need to learn which component vulnerabilities aren’t in fact exploitable in the full product.

This is what VEX documents will do. They will be available by this summer, but you’ll need to have a tool or service that will be able to ingest and interpret them (and don’t even think about processing VEXes manually. You’ll go mad), then prune the list of component vulnerabilities in the products that you use down to the 5-10% that are exploitable.

This list is what you really need. How many tools (whether open source or commercial) or low-cost services will do all of this for you today? Zero, although I know there will be at least one service and at least one open source tool (Dependency-Track, again) that will do that by the summer, when SBOMs should start being more available, thanks to Executive Order 14028.

What’s the moral of this story? The big need now isn’t for SBOMs. Software suppliers are already producing them in large quantities for their own component risk management purposes (the suppliers know how valuable they are!). But at the moment, they’re not sharing them with their customers, because the customers aren’t asking for them. And why aren’t the customers asking for them? It’s probably because they don’t have tools or services available to make use of them.

So the SBOMs and VEXes themselves do you no good. They’re merely steppingstones to what you really need: a list of exploitable component vulnerabilities in the software you use. Both the services and the tools will get you that list, without your even having to see an SBOM or VEX, let alone manipulate it in some way.

Now, I’ll be the first to admit that it will be years before you have a complete, daily-updated list of all the exploitable component vulnerabilities in all of the software you use. On the other hand, the list you do have – say - six months or a year from now will be a lot more useful than the empty list you currently have.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

Putin’s endgame

It causes me intense regret to say that Vladimir Putin has entered what appears to be his endgame – the disastrous end of his invasion of Ukraine, the end of his regime (since a major military setback always seems to result in regime change in Russia, the most notable being when the Czars were overthrown by the Bolsheviks after major losses in World War I), and perhaps the end of his personal freedom (and maybe even his life – although that’s perhaps a little too much to hope for now).

How will this happen? A couple weeks ago, I would have said the oligarchs would push him out, but they are – of course – in the end just creations of Putin. They have no base of power on their own. And the people aren’t going to push him out, either. The majority still seem to support him, although they’re still in the initial stages where they’ve heard the thunderclap, but they haven’t felt the rain yet. Or they’re like the guy who jumped off the top of the Sears Tower and called out “So far, so good!” as he passed the 50th floor.

No, I think the military will push him out. Obviously, the invasion of Ukraine has been a disaster, and Putin will seek to blame his generals for that – but I doubt anybody in the military lifts a finger without Putin’s permission, and when he tells them to jump, their only response is, “How high?” The military's biggest concern will be the consequences of Putin's actions, now that he realizes he can’t win a straight-out military victory:

1.      He might try to pursue his Chechnya strategy – i.e. leveling the capital city and deliberately killing thousands of civilians. But that’s not going to work. At a certain point, the Western powers will decide it’s time to call Putin’s nuclear bluff and send in their own planes – including attacking the 20-mile convoy that the Russians very helpfully set up to be bombed into oblivion in an evening’s work. The West won’t stand for a major civilian slaughter in Ukraine, nukes or no nukes.

2.      Speaking of nuclear, Putin is deliberately risking nuclear disasters at Chernobyl and Zaporizhzhia (say that five times fast!), where it’s never a very good strategy to force the staff at gunpoint to keep running the plant safely, when none of the attackers know a thing about nuclear power. You might as well just point the gun at your own head. And remember that the prevailing westerly winds will blow the radioactive cloud from a disaster at either plant right into Russia. Hmm, what’s wrong with this picture?

3.      Perhaps most frightening for the military will be the idea that, once Putin realizes how much trouble he’s in, he’ll resort to using a “tactical” nuclear weapon, calculating that any retaliation by the West will also be “tactical” and won’t take out a big city like Moscow. That’s probably correct, but it would probably take out a big military base, which means lots of soldiers and also their families. Not too appetizing a prospect for the military, no?

It's hard to see how any replacement for Putin won’t be an improvement.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

A not-to-miss webinar on SBOMs

Earlier this week, I received – with no knowledge at all that it was coming - an emailed invitation from the London chapter of OWASP to a broadcast of their meeting next Thursday, that I wouldn’t miss for the world (well, maybe a portion of the world…). It features two of the most authoritative figures in SBOMs: Steve Springett, leader of the CycloneDX and Dependency-Track OWASP projects, and Jeff Williams, co-founder of OWASP and creator of the OWASP Top 10.

Their discussions won’t be the quick 10-15-minute wonders we’re used to in a 45-minute webinar. The entire meeting will be 2 ½ hours, with 15 minutes of OWASP business up front, followed by Jeff’s talk, a Q&A, and finally Steve’s talk.

I’ve written a lot about Steve in my posts lately, so I don’t think he needs an introduction. I’ve never met Jeff, but I first heard him speak a couple times during a 1 ½ - day NIST webinar on software security last September. I was very impressed with him. He really is a very creative person, as well as being very knowledgeable.

Note that, even though the meeting will be streamed in real time from London, it takes place at an easily accessible time for those of us who are across the ocean in the Colonies: It’s from 1:30 to 4:00 PM Eastern Time. I’ll see you there!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

The myth of “deterrence”

Last Friday, Politico ran a well-researched article about the Russian threat to the US grid. I was quoted saying that it would be extremely difficult for the Russians to cause a widespread grid outage, in large part because of NERC’s standards. And here I meant not just the NERC CIP cybersecurity standards, but the other NERC standards that focus on measures needed to keep the grid reliable. So events like the ones that led to the 2003 Northeast Blackout, whether or not induced by a cyberattack, simply wouldn’t be able to cause an event anything like that one, if those same events occurred today.

I also pointed the reporter to the 2019 Worldwide Threat Assessment, and made the same point to her that I made in this recent post: that if the US is really worried about Putin attacking our grid, it would be a good idea to thoroughly investigate these statements. If it turns out they’re completely wrong, then great…let’s hear about it. But if it turns out they’re right and there is Russian malware planted in grid control centers (since those would be the best points from which to cause outages. Attacking a substation – even taking one out completely – is highly unlikely to cause any outage at all, and certainly not a cascading one. The 2013 Metcalf substation attack demonstrated this).

But I was concerned about the article’s discussion of possible US “deterrence” of a Russian cyberattack on the grid by threatening to respond in kind, presumably with a cyberattack on our own (and the article repeats previous reports that the US has planted plenty of malware in the Russian grid. I don’t doubt that those reports are true, although I don’t have a way of knowing that). This idea is based on flawed logic:

1.      The Russians attack the US grid and cause serious outages that lead to some loss of civilian lives.

2.      We turn around and cause an even more serious outage in Russia, with presumably even more loss of civilian lives.

The fact is that this scenario will never happen. The US is never going to launch any sort of attack directly targeting civilians in another country, unless we’re actually at war with the country. No matter what Russia is doing in Ukraine, we’re simply not going to “launch a devastating cyberattack” on their grid, no matter what sort of cyberattack they launch on ours. Instead, we’ll impose other non-lethal punishment on the country – beyond what we’ve already imposed. Sure, impoverish the people. But don’t kill them.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.