I have been
impressed (if that’s the right word) for a long time by the fact that many
commentators in the popular and trade press seem to agree on one key point: the
electric utility industry is rife with security vulnerabilities and is
ready to fall over tomorrow, with just one small push by some 11-year-old in
Estonia. This in spite of the fact that
I don’t know of a single directed cyber attack on a power facility (or facilities)
that has led to an outage for even one household for one minute. And while there definitely was a very serious physical attack on a substation in California last year (which didn't cause an outage, but well could have had the stars been aligned a little differently), the industry very quickly stepped up on their own to address problems, even before FERC ordered a mandatory standard.
This isn’t
to say that the electric power industry doesn’t need to do more for cyber security,
or that it doesn’t occupy a very special position that requires a lot of
scrutiny; even a one-minute outage for a whole city would be devastating. But it is to say that it’s really
unproductive for people who should know better (and if they don’t know better,
they shouldn’t be standing up as experts) to be making these statements with no
facts to back them up.
I also think
I know why these people don’t get called out more often. It’s because the electric power industry is mostly
regulated, and the entities that comprise it don’t want to stir up any
controversy, even if this means sitting back and taking some pretty unfair
hits. Here are three examples of what I’m
talking about:
Exhibit A is
this article
on the Smart Grid News website. I need to first say that I have a lot of
respect for what Jesse Berst has done for the smart grid; it’s no exaggeration
to say he’s a huge reason the smart grid is as successful as it is today. But this doesn’t excuse sloppy reporting and
commentary, which is the case here[i].
The article
is about NERC’s approval of CIP-014, the new physical security standard for
substations ordered
by FERC in early March. The article says
nothing good about the standard and makes three arguments why it is virtually
worthless:
- “On the plus side, the proposed rules gained 82% approval. On the downside, most of the votes were from the very utilities who (sic) would be subject to these new regulations.” This is a frightening observation – it seems the very rules that are needed to protect our way of life are being written by the thoughtless (evil?) utilities that have to follow them! What could be worse?
This ignores the small point that all NERC regulations are drafted by the utilities
subject to them, and have been since NERC was founded in 1967. But this is even more shocking! We should tell Congress! However, Congress already knows this, since
they wrote the Energy Policy Act of 2005.
This was the law that put in place the current structure in which FERC
is the regulator while NERC drafts standards, submits them for FERC approval
(or rejection), and audits the member entities for compliance.
Congress didn’t pass this law on a whim
late one night. It was very well vetted
with lots of hearings, testimony, etc.
And there are very good reasons why this structure was put in place, one
of the most important being that neither FERC nor any other organization has
the ability to understand all the nuances (many extremely local and particular)
of the electric power industry. It is
best that the people most involved day-to-day draft the standards, while FERC
makes their own judgment regarding their adequacy (and they often push
back. In fact, it seems that is happening
now with the new CIP requirements for Low impact assets – more on that in a
later post).
- “Proponents of much stiffer security safeguards wanted things like blast barriers to protect transformers and other critical equipment. NERC's proposed rules, however, allow utilities to determine on their own whether or not their substations are critical...” This is of course also very shocking – it seems the fox is not only guarding the henhouse, but is also judge and jury for any case that might be brought against him for violating the new standard. Because all the standard says is that utilities need to decide on their own what it applies to! Of course, we all know that these sinister utilities will choose to apply it to few if any substations.
Again, I hate to introduce these messy
things called facts into the discussion, but I do wish to point out that this
is exactly what FERC ordered NERC to do.
They realized there was so much variability in substations and their environments
that it would be impossible to draw up hard and fast prescriptive rules for
deciding which were critical and which weren’t.
Not only that, but there was no way NERC could even develop such rules
in the 90 days FERC was giving it. Instead,
FERC recommended that the standard require NERC entities to conduct a risk assessment to identify critical substations, then have that assessment reviewed by a neutral third party; they will also be audited on how well they did
this. Smart Grid News could have easily found this out by reading FERC’s Order
from March, or my post
that followed hard upon the Order.
Note from Tom on 2/20/18: It is quite interesting to point out that, not only did electric utilities not under-identify the substations subject to CIP-014, they over-identified them beyond NERC's wildest dreams. I believe there have been over 1,000 substations identified in North America as critical facilities under CIP-014. In my opinion, this is far more than should have been identified. It's simply a testament to the fact that the power industry wants to do the right thing, even if it costs them a lot of money (which this will, of course).
Note from Tom on 2/20/18: It is quite interesting to point out that, not only did electric utilities not under-identify the substations subject to CIP-014, they over-identified them beyond NERC's wildest dreams. I believe there have been over 1,000 substations identified in North America as critical facilities under CIP-014. In my opinion, this is far more than should have been identified. It's simply a testament to the fact that the power industry wants to do the right thing, even if it costs them a lot of money (which this will, of course).
- “...and then to decide on their own what measures they should take to protect the facilities, if any.” This is also shocking on the face of it. It seems that, even for substations (and control centers) that the utility deems critical, they can decide all on their own what needs to be done to protect them. And knowing utilities, they’ll decide it’s sufficient to just hang a sign that says “Do Not Attack” on every critical substation and call it a day.
Again, this was what FERC ordered, and
for the same reasons as above: there is way too much variability in substations
for any sort of prescriptive rules to work.
More importantly, though, the utilities don’t “decide on their own” what
measures they should take. FERC ordered,
and NERC included in the standard, that each NERC entity should have a third-party
review of their physical security plan, as well as of the risk assessment they
used to designate their critical substations in the first place. And NERC has set criteria for what third
parties can conduct this review.
Now let’s go
to Exhibit B. This isn’t any one
particular article or post, but a number of comments I’ve seen on LinkedIn and
other forums over a number of years about why the PCI standards for credit card
security are so superior to NERC CIP.
These are usually posted by consultants from the IT security industry
who have decided to make the switch to the Dark Side of control system
security, and are anxious to prove their manhood (I haven’t seen any woman do
this) by – what else? – bashing the utilities for writing wimpy standards that
just don’t cut it.
I don’t know
whether the PCI standards are technically more rigorous or well-written than
CIP. However, I do know there’s a big
difference in the cyber security records of the two industries. We could start with Target, then go back
through a whole host of massive thefts of credit card data, all the way back to
TJX (TJ Maxx). I’m sure the total losses
to American business from these breaches have been in the billions (and it
seems most costs are borne by the banks that have to reissue cards, not the retailers
themselves). If the electric power
industry had the same record, we’d all be sitting at home in the dark (and I obviously
wouldn’t be writing this post, which some might say would be a good thing).
I will point
out a couple really egregious things about PCI.
First, it may be very good at protecting what it protects, but it seems
all of the big breaches have been through another channel that was left unprotected
by PCI. The best case in point is
Target, where it seems there was no real separation between the corporate
systems and those that processed credit card data. Someone broke into the account of an HVAC
contractor (who clearly didn’t need access to credit card data and should have been excluded from it if some sort of least-privilege analysis had been applied), and was able
to traverse the network to get to the real-time transaction network – which allowed
them to plant their malware on the point-of-sale systems.
Of course,
this would be like having a Balancing Authority’s control center on the same
network as the corporate systems, so that the same HVAC contractor’s account
could be used to bring down the entire control area. Were this to be possible, the entity would
have been in gross violation of just about every CIP requirement for a long
time; and at a million dollars a day per requirement violated, that would be
one hefty fine. But Target was and is
completely PCI compliant! Obviously,
whatever technical merits PCI may have don’t make up for the fact that the
standard is fairly narrowly focused on the systems that store and process
credit card data. CIP is similarly focused
on control networks, but at least it is written from the standpoint that the
control network needs to be protected from the corporate network – and the fact
that this is the case is probably a big reason why there haven’t been any
outages caused by cyber attack.
Another
thing about PCI that’s always amazed me is that the entity being audited pays
the auditor! These auditors, called
QSAs, are security firms that have passed a rigorous certification; I’m sure
they’re quite qualified to do the audits.
But there is such an inherent conflict of interest in this process, since
every auditor must have it in the back of their head that they’d love to be
given the job of fixing all the problems they find. You might think this would incent them to find
lots of problems, but that’s not the way psychology works. If someone tells me – in a report that has to
be provided to the proper authorities – that I have all sorts of problems, do
you think I’ll love them for doing that?
Or am I more likely to love someone who gives me a pass on some of the
worst problems – yet still reserves a few meaty ones that he can help me fix?
Now we come
to Exhibit C, which is my favorite. It’s
a speech
that Mike McConnell, former NSA Director and now Vice Chairman of Booz Allen
Hamilton, gave in early March of this year.
In it, he said “In my mind, there is 100% certainty that cyber attacks
will occur.” And of course, he pointed
to the power industry[ii] as the
likely venue for this attack. As he
said, "Just imagine being in New York City in the middle of the summer
with no power."
Sounds
pretty scary, huh? 100% certain! Of course, that’s an exaggeration, since the
only events that are 100% certain are those that occurred in the past. And I’d say one of the most costly and
destructive cyber attacks in the past was that perpetrated by contractor Edward
Snowden against Mr. McConnell’s former employer, the NSA. And who was the firm that placed Snowden at
the NSA? Why, Booz Allen! Hmmm....
July 1: I just posted an update, discussing a new case in which Smart Grid News attacked (by implication) PG&E for not reporting the Metcalf attack until a year later, without bothering to ascertain whether the allegation was true or not. I certainly hope I don't have to post more of these, but I will if necessary.
July 1: I just posted an update, discussing a new case in which Smart Grid News attacked (by implication) PG&E for not reporting the Metcalf attack until a year later, without bothering to ascertain whether the allegation was true or not. I certainly hope I don't have to post more of these, but I will if necessary.
All opinions expressed herein are mine, not
necessarily those of Honeywell
International, Inc.
This comment has been removed by the author.
ReplyDeleteA few points:
ReplyDelete- CIP & electric sector cybersecurity proponents do a terrible job of stating the case that the security posture of the BES has improved due to their efforts. The WSJ letter to the editor is a prime example. This needs to be part of the sector's efforts. Saying we are writing standards to regulate ourselves (we know best), isn't a great defense.
- the argument that the appropriate amount of cyber security is in place because there has not been an outage due to a malicious attack can boomerang quickly. I know that was a bit of a throw away line in the article, but I don't think that is how we should be measuring cyber security in the critical infrastructure.
- the self-regulating point that you railed against has some validity. There is an incentive for utilities to write the standards to avoid regulatory risk. The burden on the BES is to prove they can be trusted to regulate themselves in the cyber security realm. They have not done a good job of this, per my first point.
- I think it is fair to say that many of the utilities (a majority?) have been unwillingly dragged into cyber security, and they have taken proactive efforts in crafting the standards and their responses to avoid assets and cyber assets falling under CIP. Correct me if I'm wrong on this. I know CIP V5 is designed to address this, but it has taken a long time to get there and let's see how that turns out. I contend that NERC works where most of the members agree something needs to be done, and they want to insure the laggards do it as well. This has not been the case in cyber security in my view.
BTW, your blog is my go to source for the latest and greatest on the CIP standards.
Dale Peterson
www.digitalbond.com
Thanks, Dale. What WSJ letter to the editor are you referring to?
ReplyDeleteI wasn't trying to make the point that utilities are doing a wonderful job on cyber security, but that verbal attacks on them should be based on some sort of facts. All three examples I gave seemed to me to ignore the facts, and simply take advantage of some overall perception that the grid is defenseless against cyber attacks; that is simply not true.
A case can be made that NERC CIP was (and even is) inadequate, that utilities are trying to get around it, that they're not doing enough outside of pure compliance, etc. The people attacking the utilities (and I don't think you're one of them) need to do some homework and make this case, not just say they're 100% certain that we'll all be living in the Stone Age in a few months and expect everyone to move to New Zealand tomorrow.
You can see the WSJ letter at http://online.wsj.com/news/articles/SB10001424052702304518704579524121645572470
ReplyDeleteI think we are looking at the same issue from different sides.
You are saying critics of NERC CIP need to factual about the flaws and risks to the BES. I agree there is a lot of hyperbole. A positive from S4x14 was researchers were starting to tie the cyber & engineering together for both offensive and defensive ICS security efforts.
I'm saying that defenders of NERC CIP need to be more factual and persuasive that the standards are improving the security posture of the BES. Simply stating there are standards that go through three levels of approvals says little about how effective the standards are, especially given the inherent conflict of interest.
Thanks, Dale. The WSJ letter may point to the CIP standards as evidence that utilities are protected from cyber attacks, but I've never said that. I've asked various NERC entities how much of every dollar they spend on CIP compliance actually goes to improving security; the answer is usually around 25-35 cents. Unfortunately, the majority of CIP spending is for compliance paperwork that probably wouldn't be needed in the absence of CIP.
ReplyDeleteSo the real question is, is that 25-35 cents on the dollar actually buying the utilities something, and is that enough? The answers to these questions are yes and no. In general, since the amount the industry spends on CIP compliance is so much higher than what other industries spend on cyber security in general, it is very possible that the power industry is getting more protection from what they spend on CIP than other industries are getting from their total cyber spending - just because of the huge difference in the numbers.
But it's clearly not enough, if that's your main question.
To continue the above conversation, I have always said that, if you want to impose cyber security regulation on the electric power industry, doing it through NERC standards is absolutely the wrong way to do it. Those standards are designed to completely prevent single incidents that could cause widespread havoc such as the 2003 Northeast blackout. As such, they require a focus on avoiding every possible violation, and being able to prove that every possible risk was addressed.
ReplyDeleteBut there is simply no way you can completely prevent any sort of cyber attack - there are literally innumerable ways one could happen, most of which we don't even know about yet. The best you can do is design a strategy that mitigates as much risk as possible with the available resources. This is completely different from the way NERC standards work.
The result of using NERC standards for cyber security enforcement has been what I said above: most of the effort and expense required to comply with CIP goes into activities that don't directly enhance security, but that are required because of the NERC framework itself. To solve this problem, some other organization would need to take over cyber security regulation from NERC, and make it more risk-based.
Of course, the big question is what organization that would be, and what framework would be used. My guess is DHS would be the leading candidate, but there would certainly be others. As far as framework goes, that's the big question. I don't think CFATS - which DHS does implement - has been a success. I'm not sure there is currently an example of a successful risk-based regulatory framework for cyber security (there are many voluntary frameworks which are good, but they couldn't be made mandatory without wholesale changes).
Dale (and anyone else), I'd be very interested in what you think might be a solution to this problem. The idea would be that, if we could have a framework where close to 100% of what the utilities spend on cyber security compliance actually goes to cyber security, there would be much less anxiety about a cyber attack on the grid.