Wednesday, July 25, 2018

What lessons can we learn from the Russia hacks?

Note from Tom 2/19/2020: I just saw that the WSJ now has made the article referred to in this post fully available at https://www.wsj.com/articles/russian-hackers-reach-u-s-utility-control-rooms-homeland-security-officials-say-1532388110 In rereading it, I was struck by the huge contrast between the scale of attacks the described by Jonathan Homer of DHS in the briefing that Ms. Smith reported on in this article, and the fact that DHS subsequently - in a meeting attended by the US Vice President and the Secretaries of DHS and Energy - tried to say later that the whole thing was a misunderstanding, and only two wind turbines were attacked.

In this post two weeks later, I wrote about my amazement that Ms. Smith had written another article that didn't take back what she wrote below, even though DHS had just issued the walkback about the wind turbines. It was only later - in good part as a result of a long phone conversation with Ms. Smith - that I began to believe that DHS' walk backs were the problem, not Ms. Smith's stories (there were a couple other walkbacks as well, all multually incompatible, as well as completely incompatible with Mr. Homer's statements. And by the way, Mr. Homer gave substantially the same presentation three times after DHS' first attempt to walk back what he said. Plus as of a year ago, I was told he remained an employee in good standing at DHS, despite the fact that DHS had been going out of their way to effectively say he was a liar, and despite the fact that he has seemingly been prevented from making any statements that might clarify the huge discrepancy between what he said and DHS' walkbacks).

In this post, I articulated what is more or less my position today: I no longer can say I have any clear idea whether or not the Russians have penetrated control centers of major US utilities. But Mr. Homer's briefings - and DHS' Keystone Kops-like attempts to try to walk back what he said, at the same time that they were letting him continue to make those statements - mean there has to be an investigation to find out what is the real story. The fact that in January 2019 the FBI and CIA essentially confirmed what Mr. Homer said only confirms that there needs to be an investigation.

So today, 13 months after the CIA/FBI report came out and 20 months after Mr. Homer's briefings, has there been any investigation of any kind? No, there hasn't. My opinion on this situation remains what it's been for a while: It's a disgrace that the US government and the US power industry seem afraid to investigate these reports, especially since the same players couldn't move fast enough to investigate the Ukraine attacks. Of course, those attacks caused short outages, and the Russians haven't to our knowledge caused any in the US yet. But the fact is that the former attacks were in the Ukraine, while these attacks are in the US. That we won't even investigate to see if the reports are true or false is a national disgrace, and proves that Putin has won once again.

Yesterday, the Wall Street Journal published an article titled “Russia hacks its way into U.S. utilities”.[i] Based on a Department of Homeland Security briefing, it says that Russian hackers “claimed hundreds of victims last year” in a campaign that “put them inside the control rooms of U.S. electric utilities..” Of course, DHS has said before that Russian hackers are targeting US electric utilities, but the scale of the attacks, and the fact that many “control rooms” were seemingly penetrated, hadn’t previously been disclosed. The article was based on a DHS briefing on Monday, which was repeated today (I attended it, and found it very good). It will be repeated next Monday and Wednesday; information should be available on the ICS-CERT web site.

Clearly, there are lessons to be learned from this, both for power industry cyber security in general and for the NERC CIP standards in particular, although some of these lessons will be contingent on getting further information from DHS. But there is one lesson that can be stated unequivocally: The attacks uncovered by DHS came through suppliers – often “smaller companies without big budgets for cybersecurity”.

So there you have it: If anybody had any doubt that supply chain security is the number one cyber security issue for the electric power industry today – as well as for probably most other industries as well – there is now a smoking gun. I wish I could say that CIP-013 is coming just when it’s needed most, and that it will go a long way to solving this problem, but I’m afraid that would be a big overstatement. CIP-013 will certainly contribute to solving the problem, but will accomplish nowhere near what it could if it were written differently, or even if NERC took a different approach to enforcing it than the approach that it appears they’re taking (I realize I am speaking darkly here. I hope to have a post on this question out in the very near future).

There is a good reason why supply chain security is so important: The bad guys seem to have figured out that their prime targets – bigger organizations, and utilities in particular – are doing a pretty good job of securing themselves. So instead of battering their heads against the high barriers that have been set up to keep them out, they’re targeting the soft underbelly of these organizations – the supplier organizations that they trust. Now the “electronic security perimeter” needs to be extended – in some way – to any organization that interacts with your organization electronically.

But beyond this, it’s impossible to draw any further conclusions; this is because there are a few questions that DHS needs to answer first. In the rest of this post, I’ll state what those questions are. But, since I know that answers from DHS might not be forthcoming immediately, I’ll describe several “scenarios” based on assumptions of what those answers might be. And for each of those scenarios, I’ll point out the conclusions that I think can be drawn if that turns out to be the correct meaning of what DHS said. Here are my questions for DHS:

First, I’d like to know if the assets that were compromised[ii] were distribution or Bulk Electric System assets. If the former, it means that it would be very hard to cause a widespread outage (let alone a cascading outage), unless the “hundreds” of assets were in one concentrated region. And it also means the assets are under the jurisdiction of the state Public Utility Commissions; any sort of regulations or even guidance should probably come through the PUCs.[iii] If the assets were BES assets (or even if they were a mixture of the two types), it would be a much more serious problem, since – depending on the assets involved and their locations – a widespread outage could be the result of a coordinated attack, even a cascading one. In the rest of this discussion, I’ll assume we’re talking about BES assets.

Second, I’d like to know the true nature of the attack. In the second paragraph of the WSJ article, it says that state-sponsored groups “broke into supposedly secure, ‘air-gapped’ networks…with relative ease” by first penetrating supplier networks. This presumably means that the attackers took advantage of the fact that these suppliers had access into important cyber systems at the assets being attacked. There are two ways that this access could be facilitated at the asset end.

The first way is interactive remote access, meaning a human being (in this case a hacker with stolen credentials, so they’re impersonating a real user) logs into an OT system located at the targeted asset.  If the asset is subject to the NERC CIP standards as Medium or High impact, then it should have an Intermediate System set up to intercept the communications, authenticate the user using two-factor authentication, then proxy the user’s communications with the OT system. This means that, if Medium or High impact BES assets are being compromised, either the asset owners aren’t complying with CIP-005 R2 or the attackers have somehow bypassed the two-factor authentication (which of course would be big news in itself)[iv]. On the other hand, if the assets being compromised are Low impact (and if they’re BES, they have to be High, Medium or Low impact), it will be incumbent on NERC, FERC and the trade associations to look at either more regulation for Lows or perhaps some sort of strong guidelines.

The other way that remote access can be facilitated is machine-to-machine, in which a vendor system has direct access to an OT device at the asset. This isn’t currently covered by the CIP standards at all, but there will be controls required for machine-to-machine access (CIP-005 R2.4 and R2.5); they will be implemented when CIP-013 is implemented. During today’s webinar the main speaker made clear that all of the remote access conducted by the attackers was interactive.

Third, you say “utilities” have been attacked, yet you speak of “control rooms” being penetrated. A control room is usually found in a single generating plant or substation, since it’s defined by NERC as just controlling a single Bulk Electric System asset; the majority of generating plants aren’t owned by utilities but by independent power producers. A control center controls multiple BES assets; these are the real “heart” of the BES, and if even tens of control centers were compromised (let alone hundreds!), that would be a very big problem indeed. Do you really mean “utilities”? And if so, do you really mean “control rooms”? In that case, many of the generating plants that were attacked were undoubtedly owned by IPPs, not by utilities. On the other hand, if you mean “control centers”, about how many of those were penetrated? This is important for me to know, since if hundreds of utility control centers are compromised by the Russians (or even ten large control centers), I’m going to slaughter my chickens and book the next flight to New Zealand.

On the other hand, if you didn’t mean just utilities were being targeted, and you also meant it when you said control rooms, then this would most likely primarily be either an attack on generating plants. I pointed out at the end of this post that it would be close to impossible to actually cause a big outage just by attacking generation – you would have to attack a large number of small plants (or single units of larger plants) in one region simultaneously, and even then any outage would probably be quickly corrected by diverting power from other regions. But regardless of whether a big outage could be caused, it would be a matter of great concern if many BES generating plants had been penetrated.

If the generating plants being penetrated are Medium impact and haven’t been segmented so that they have no Medium BES Cyber Systems, then they are supposed to be complying with CIP-005 R2. Either they’re not actually complying or the attackers have figured out how to break or bypass two-factor authentication. And if the generating plants are Low impact (or segmented Medium-impact plants), once again it will be incumbent on NERC, FERC and the trade associations to look at either more regulation for Lows or some sort of strong guidelines.

What if a significant number of the assets being compromised are transmission substations? That’s a different story. If an attacker were to thoroughly “own” a single significant transmission substation (perhaps controlling multiple lines rated at 345 kV or higher), I believe they could cause a wide-scale outage.[v] And if they owned about 3 or 4 such substations – again in the same region – it could be a really serious outage.

Again, if the attackers broke into Medium impact transmission substations, then the owners/operators of those substations were in violation of CIP-005 R2 – since that requires two-factor authentication, and in the webinar today the presenters said that all systems penetrated used single-factor authentication (i.e. username and password). This means that, if the attackers broke into transmission substations at all, it must have been Lows. And since Lows don’t have to deploy an Intermediate Server for CIP compliance, once again it will be incumbent on NERC, FERC and the trade associations to look at either more regulation for Lows or some sort of strong guidelines.

What do I think is the most likely of all the above scenarios? I would guess the number of assets actually penetrated is no higher than 25. Further, I would guess that they were mostly generating plants (the webinar presenters showed an HMI screen – heavily redacted, of course – that was uploaded by the attackers from one “victim”. It was clearly a generation asset), and they were all either distribution assets or Low impact BES assets.

So if I’m right, the other main lesson to be learned from this briefing (besides the lesson about the critical importance of supply chain security) is that FERC, NERC and the trades should decide whether increased regulation is appropriate for Lows (and the at least partial extension of CIP-013 to Lows would probably be the right vehicle for that, since FERC is already thinking about that anyway. Of course, the extension wouldn’t be included in the first version of CIP-013, but FERC would order it for the next version, meaning it’s 5-6 years away now), or whether some heavily-guided “voluntary” standard would be more appropriate. Plus, the PUCs need to start thinking more seriously about how to get owners/operators of purely distribution assets more concerned about supply chain security.


Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. And if you’re a security vendor to the power industry, TALLC can help you by developing marketing materials, delivering webinars, etc. To discuss any of this, you can email me at the same address.       



[i] The WSJ’s online edition is behind a paywall, so I can’t provide a link to the article. I have it in hard copy, so if you want to drop me an email at the above address, I’ll send you a scan of it.

[ii] Even though the WSJ article stated there were “hundreds of victims”, in the DHS briefing today they made it clear that a “victim” wasn’t necessarily compromised, just targeted. There definitely were some grid assets that were compromised, but it’s not clear how many there were.

[iii] Unfortunately, it’s a great simplification to say that the utilities in each state are subject to that state’s PUCs, since in fact it’s only the investor-owned utilities that are. If DHS wanted to promulgate any mandatory standards for all distribution assets, they would probably have to first get authorization from Congress. On the other hand, I can certainly see the trade associations (especially APPA, NRECA, EEI and EPSA) getting together and writing “voluntary” standards for distribution assets, working with DHS.

[iv] In the webinar today, the DHS people said that the attackers always got in through single-factor-authenticated systems, meaning they didn’t bypass two-factor authentication. Yet they also mentioned that they (DHS) were urgently looking to see if the attackers had penetrated any two-factor-authenticated systems, and if so how. So it seems they’re saying that, just because the attackers didn’t succeed in bypassing (or hacking) 2FA systems, they might have at least tried to.

They pointed out that some of the assets that were penetrated had an intermediate system using 2FA, but those assets had also deployed some “ancillary” remote access systems that used 1FA. Of course, if any of these assets were Medium impact for CIP, they were in violation if they did that; under CIP-005 R2, all interactive remote access must go through an Intermediate System with 2FA.

[v] In 2008, a fault at a single transmission substation in Florida caused an outage that affected most of southern Florida, and led to $20 million in fines by NERC and FERC, plus $5 million in remediation. I’ve heard that the BES effects were felt within a second in Canada.

No comments:

Post a Comment