Note from Tom 2/19/2020: I just saw that the WSJ now has made the article referred to in this post fully available at https://www.wsj.com/articles/russian-hackers-reach-u-s-utility-control-rooms-homeland-security-officials-say-1532388110 In rereading it, I was struck by the huge contrast between the scale of attacks the described by Jonathan Homer of DHS in the briefing that Ms. Smith reported on in this article, and the fact that DHS subsequently - in a meeting attended by the US Vice President and the Secretaries of DHS and Energy - tried to say later that the whole thing was a misunderstanding, and only two wind turbines were attacked.
In this post two weeks later, I wrote about my amazement that Ms. Smith had written another article that didn't take back what she wrote below, even though DHS had just issued the walkback about the wind turbines. It was only later - in good part as a result of a long phone conversation with Ms. Smith - that I began to believe that DHS' walk backs were the problem, not Ms. Smith's stories (there were a couple other walkbacks as well, all multually incompatible, as well as completely incompatible with Mr. Homer's statements. And by the way, Mr. Homer gave substantially the same presentation three times after DHS' first attempt to walk back what he said. Plus as of a year ago, I was told he remained an employee in good standing at DHS, despite the fact that DHS had been going out of their way to effectively say he was a liar, and despite the fact that he has seemingly been prevented from making any statements that might clarify the huge discrepancy between what he said and DHS' walkbacks).
In this post, I articulated what is more or less my position today: I no longer can say I have any clear idea whether or not the Russians have penetrated control centers of major US utilities. But Mr. Homer's briefings - and DHS' Keystone Kops-like attempts to try to walk back what he said, at the same time that they were letting him continue to make those statements - mean there has to be an investigation to find out what is the real story. The fact that in January 2019 the FBI and CIA essentially confirmed what Mr. Homer said only confirms that there needs to be an investigation.
So today, 13 months after the CIA/FBI report came out and 20 months after Mr. Homer's briefings, has there been any investigation of any kind? No, there hasn't. My opinion on this situation remains what it's been for a while: It's a disgrace that the US government and the US power industry seem afraid to investigate these reports, especially since the same players couldn't move fast enough to investigate the Ukraine attacks. Of course, those attacks caused short outages, and the Russians haven't to our knowledge caused any in the US yet. But the fact is that the former attacks were in the Ukraine, while these attacks are in the US. That we won't even investigate to see if the reports are true or false is a national disgrace, and proves that Putin has won once again.
In this post two weeks later, I wrote about my amazement that Ms. Smith had written another article that didn't take back what she wrote below, even though DHS had just issued the walkback about the wind turbines. It was only later - in good part as a result of a long phone conversation with Ms. Smith - that I began to believe that DHS' walk backs were the problem, not Ms. Smith's stories (there were a couple other walkbacks as well, all multually incompatible, as well as completely incompatible with Mr. Homer's statements. And by the way, Mr. Homer gave substantially the same presentation three times after DHS' first attempt to walk back what he said. Plus as of a year ago, I was told he remained an employee in good standing at DHS, despite the fact that DHS had been going out of their way to effectively say he was a liar, and despite the fact that he has seemingly been prevented from making any statements that might clarify the huge discrepancy between what he said and DHS' walkbacks).
In this post, I articulated what is more or less my position today: I no longer can say I have any clear idea whether or not the Russians have penetrated control centers of major US utilities. But Mr. Homer's briefings - and DHS' Keystone Kops-like attempts to try to walk back what he said, at the same time that they were letting him continue to make those statements - mean there has to be an investigation to find out what is the real story. The fact that in January 2019 the FBI and CIA essentially confirmed what Mr. Homer said only confirms that there needs to be an investigation.
So today, 13 months after the CIA/FBI report came out and 20 months after Mr. Homer's briefings, has there been any investigation of any kind? No, there hasn't. My opinion on this situation remains what it's been for a while: It's a disgrace that the US government and the US power industry seem afraid to investigate these reports, especially since the same players couldn't move fast enough to investigate the Ukraine attacks. Of course, those attacks caused short outages, and the Russians haven't to our knowledge caused any in the US yet. But the fact is that the former attacks were in the Ukraine, while these attacks are in the US. That we won't even investigate to see if the reports are true or false is a national disgrace, and proves that Putin has won once again.
Yesterday,
the Wall Street Journal published an article titled “Russia hacks its
way into U.S. utilities”.[i] Based on
a Department of Homeland Security briefing, it says that Russian hackers
“claimed hundreds of victims last year” in a campaign that “put them inside the
control rooms of U.S. electric utilities..” Of course, DHS has said before that
Russian hackers are targeting US electric utilities, but the scale of the
attacks, and the fact that many “control rooms” were seemingly penetrated, hadn’t
previously been disclosed. The article was based on a DHS briefing on Monday,
which was repeated today (I attended it, and found it very good). It will be
repeated next Monday and Wednesday; information should be available on the
ICS-CERT web site.
Clearly,
there are lessons to be learned from this, both for power industry cyber
security in general and for the NERC CIP standards in particular, although some
of these lessons will be contingent on getting further information from DHS.
But there is one lesson that can be stated unequivocally: The attacks uncovered
by DHS came through suppliers – often “smaller companies without big budgets
for cybersecurity”.
So there you
have it: If anybody had any doubt that supply chain security is the number one
cyber security issue for the electric power industry today – as well as for
probably most other industries as well – there is now a smoking gun. I wish I
could say that CIP-013 is coming just when it’s needed most, and that it will
go a long way to solving this problem, but I’m afraid that would be a big
overstatement. CIP-013 will certainly contribute to solving the problem, but will
accomplish nowhere near what it could if it were written differently, or even
if NERC took a different approach to enforcing it than the approach that it
appears they’re taking (I realize I am speaking darkly here. I hope to have a
post on this question out in the very near future).
There is a
good reason why supply chain security is so important: The bad guys seem to
have figured out that their prime targets – bigger organizations, and utilities
in particular – are doing a pretty good job of securing themselves. So instead
of battering their heads against the high barriers that have been set up to
keep them out, they’re targeting the soft underbelly of these organizations –
the supplier organizations that they trust. Now the “electronic security
perimeter” needs to be extended – in some way – to any organization that
interacts with your organization electronically.
But beyond
this, it’s impossible to draw any further conclusions; this is because there
are a few questions that DHS needs to answer first. In the rest of this post,
I’ll state what those questions are. But, since I know that answers from DHS
might not be forthcoming immediately, I’ll describe several “scenarios” based
on assumptions of what those answers might be. And for each of those scenarios,
I’ll point out the conclusions that I think can be drawn if that turns out to
be the correct meaning of what DHS said. Here are my questions for DHS:
First, I’d
like to know if the assets that were compromised[ii] were
distribution or Bulk Electric System assets. If the former, it means that it
would be very hard to cause a widespread outage (let alone a cascading outage),
unless the “hundreds” of assets were in one concentrated region. And it also
means the assets are under the jurisdiction of the state Public Utility
Commissions; any sort of regulations or even guidance should probably come through
the PUCs.[iii] If the
assets were BES assets (or even if they were a mixture of the two types), it
would be a much more serious problem, since – depending on the assets involved
and their locations – a widespread outage could be the result of a coordinated
attack, even a cascading one. In the rest of this discussion, I’ll assume we’re
talking about BES assets.
Second, I’d
like to know the true nature of the attack. In the second paragraph of the WSJ
article, it says that state-sponsored groups “broke into supposedly secure,
‘air-gapped’ networks…with relative ease” by first penetrating supplier
networks. This presumably means that the attackers took advantage of the fact
that these suppliers had access into important cyber systems at the assets
being attacked. There are two ways that this access could be facilitated at the
asset end.
The first
way is interactive remote access, meaning a human being (in this case a hacker
with stolen credentials, so they’re impersonating a real user) logs into an OT
system located at the targeted asset. If
the asset is subject to the NERC CIP standards as Medium or High impact, then
it should have an Intermediate System set up to intercept the communications,
authenticate the user using two-factor authentication, then proxy the user’s
communications with the OT system. This means that, if Medium or High impact
BES assets are being compromised, either the asset owners aren’t complying with
CIP-005 R2 or the attackers have somehow bypassed the two-factor authentication
(which of course would be big news in itself)[iv]. On the
other hand, if the assets being compromised are Low impact (and if they’re BES,
they have to be High, Medium or Low impact), it will be incumbent on NERC, FERC
and the trade associations to look at either more regulation for Lows or perhaps
some sort of strong guidelines.
The other
way that remote access can be facilitated is machine-to-machine, in which a
vendor system has direct access to an OT device at the asset. This isn’t
currently covered by the CIP standards at all, but there will be controls
required for machine-to-machine access (CIP-005 R2.4 and R2.5); they will be
implemented when CIP-013 is implemented. During today’s webinar the main
speaker made clear that all of the remote access conducted by the attackers was
interactive.
Third, you
say “utilities” have been attacked, yet you speak of “control rooms” being
penetrated. A control room is usually found in a single generating plant or
substation, since it’s defined by NERC as just controlling a single Bulk
Electric System asset; the majority of generating plants aren’t owned by
utilities but by independent power producers. A control center controls
multiple BES assets; these are the real “heart” of the BES, and if even tens of
control centers were compromised (let alone hundreds!), that would be a very
big problem indeed. Do you really mean “utilities”? And if so, do you really
mean “control rooms”? In that case, many of the generating plants that were
attacked were undoubtedly owned by IPPs, not by utilities. On the other hand,
if you mean “control centers”, about how many of those were penetrated? This is
important for me to know, since if hundreds of utility control centers are
compromised by the Russians (or even ten large control centers), I’m going to
slaughter my chickens and book the next flight to New Zealand.
On the other
hand, if you didn’t mean just
utilities were being targeted, and you also meant it when you said control
rooms, then this would most likely primarily be either an attack on generating plants.
I pointed out at the end of this
post that it would be close to impossible to actually cause a big outage
just by attacking generation – you would have to attack a large number of small
plants (or single units of larger plants) in one region simultaneously, and
even then any outage would probably be quickly corrected by diverting power
from other regions. But regardless of whether a big outage could be caused, it
would be a matter of great concern if many BES generating plants had been
penetrated.
If the
generating plants being penetrated are Medium impact and haven’t been segmented
so that they have no Medium BES Cyber Systems, then they are supposed to be
complying with CIP-005 R2. Either they’re not actually complying or the
attackers have figured out how to break or bypass two-factor authentication.
And if the generating plants are Low impact (or segmented Medium-impact plants),
once again it will be incumbent on NERC, FERC and the trade associations to
look at either more regulation for Lows or some sort of strong guidelines.
What if a
significant number of the assets being compromised are transmission
substations? That’s a different story. If an attacker were to thoroughly “own”
a single significant transmission substation (perhaps controlling multiple
lines rated at 345 kV or higher), I believe they could cause a wide-scale
outage.[v] And if
they owned about 3 or 4 such substations – again in the same region – it could
be a really serious outage.
Again, if
the attackers broke into Medium impact transmission substations, then the
owners/operators of those substations were in violation of CIP-005 R2 – since
that requires two-factor authentication, and in the webinar today the
presenters said that all systems penetrated used single-factor authentication
(i.e. username and password). This means that, if the attackers broke into
transmission substations at all, it must have been Lows. And since Lows don’t
have to deploy an Intermediate Server for CIP compliance, once again it will be
incumbent on NERC, FERC and the trade associations to look at either more
regulation for Lows or some sort of strong guidelines.
What do I
think is the most likely of all the above scenarios? I would guess the number
of assets actually penetrated is no higher than 25. Further, I would guess that
they were mostly generating plants (the webinar presenters showed an HMI screen
– heavily redacted, of course – that was uploaded by the attackers from one
“victim”. It was clearly a generation asset), and they were all either
distribution assets or Low impact BES assets.
So if I’m
right, the other main lesson to be learned from this briefing (besides the
lesson about the critical importance of supply chain security) is that FERC,
NERC and the trades should decide whether increased regulation is appropriate
for Lows (and the at least partial extension of CIP-013 to Lows would probably
be the right vehicle for that, since FERC is already thinking about that anyway.
Of course, the extension wouldn’t be included in the first version of CIP-013,
but FERC would order it for the next version, meaning it’s 5-6 years away now),
or whether some heavily-guided “voluntary” standard would be more appropriate.
Plus, the PUCs need to start thinking more seriously about how to get
owners/operators of purely distribution assets more concerned about supply
chain security.
Any opinions expressed in this blog post are strictly mine
and are not necessarily shared by any of the clients of Tom Alrich LLC.
If you would like to comment on what you have read here, I
would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that
if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or
challenges like what is discussed in this post – especially on compliance with
CIP-013. And if you’re a security vendor to the power industry, TALLC can help
you by developing marketing materials, delivering webinars, etc. To discuss any
of this, you can email me at the same address.
[i]
The WSJ’s online edition is behind a paywall, so I can’t provide a link to the
article. I have it in hard copy, so if you want to drop me an email at the
above address, I’ll send you a scan of it.
[ii]
Even though the WSJ article stated there were “hundreds of victims”, in the DHS
briefing today they made it clear that a “victim” wasn’t necessarily
compromised, just targeted. There definitely were some grid assets that were
compromised, but it’s not clear how many there were.
[iii]
Unfortunately, it’s a great simplification to say that the utilities in each
state are subject to that state’s PUCs, since in fact it’s only the investor-owned
utilities that are. If DHS wanted to promulgate any mandatory standards for all
distribution assets, they would probably have to first get authorization from
Congress. On the other hand, I can certainly see the trade associations
(especially APPA, NRECA, EEI and EPSA) getting together and writing “voluntary”
standards for distribution assets, working with DHS.
[iv]
In the webinar today, the DHS people said that the attackers always got in
through single-factor-authenticated systems, meaning they didn’t bypass
two-factor authentication. Yet they also mentioned that they (DHS) were
urgently looking to see if the attackers had penetrated any
two-factor-authenticated systems, and if so how. So it seems they’re saying
that, just because the attackers didn’t succeed in bypassing (or hacking) 2FA
systems, they might have at least tried to.
They pointed out that some of the assets that were
penetrated had an intermediate system using 2FA, but those assets had also
deployed some “ancillary” remote access systems that used 1FA. Of course, if
any of these assets were Medium impact for CIP, they were in violation if they
did that; under CIP-005 R2, all interactive remote access must go through an
Intermediate System with 2FA.
No comments:
Post a Comment