Wednesday, May 7, 2014

Bashing Utilities for Fun and Profit

I have been impressed (if that’s the right word) for a long time by the fact that many commentators in the popular and trade press seem to agree on one key point: the electric utility industry is rife with security vulnerabilities and is ready to fall over tomorrow, with just one small push by some 11-year-old in Estonia.  This in spite of the fact that I don’t know of a single directed cyber attack on a power facility (or facilities) that has led to an outage for even one household for one minute.  And while there definitely was a very serious physical attack on a substation in California last year (which didn't cause an outage, but well could have had the stars been aligned a little differently), the industry very quickly stepped up on their own to address problems, even before FERC ordered a mandatory standard.

This isn’t to say that the electric power industry doesn’t need to do more for cyber security, or that it doesn’t occupy a very special position that requires a lot of scrutiny; even a one-minute outage for a whole city would be devastating.  But it is to say that it’s really unproductive for people who should know better (and if they don’t know better, they shouldn’t be standing up as experts) to be making these statements with no facts to back them up.

I also think I know why these people don’t get called out more often.  It’s because the electric power industry is mostly regulated, and the entities that comprise it don’t want to stir up any controversy, even if this means sitting back and taking some pretty unfair hits.   Here are three examples of what I’m talking about:

Exhibit A is this article on the Smart Grid News website.  I need to first say that I have a lot of respect for what Jesse Berst has done for the smart grid; it’s no exaggeration to say he’s a huge reason the smart grid is as successful as it is today.  But this doesn’t excuse sloppy reporting and commentary, which is the case here[i].

The article is about NERC’s approval of CIP-014, the new physical security standard for substations ordered by FERC in early March.  The article says nothing good about the standard and makes three arguments why it is virtually worthless:

  1. On the plus side, the proposed rules gained 82% approval. On the downside, most of the votes were from the very utilities who (sic) would be subject to these new regulations.”  This is a frightening observation – it seems the very rules that are needed to protect our way of life are being written by the thoughtless (evil?) utilities that have to follow them!  What could be worse? 
This ignores the small point that all NERC regulations are drafted by the utilities subject to them, and have been since NERC was founded in 1967.  But this is even more shocking!  We should tell Congress!  However, Congress already knows this, since they wrote the Energy Policy Act of 2005.  This was the law that put in place the current structure in which FERC is the regulator while NERC drafts standards, submits them for FERC approval (or rejection), and audits the member entities for compliance.

Congress didn’t pass this law on a whim late one night.  It was very well vetted with lots of hearings, testimony, etc.  And there are very good reasons why this structure was put in place, one of the most important being that neither FERC nor any other organization has the ability to understand all the nuances (many extremely local and particular) of the electric power industry.  It is best that the people most involved day-to-day draft the standards, while FERC makes their own judgment regarding their adequacy (and they often push back.  In fact, it seems that is happening now with the new CIP requirements for Low impact assets – more on that in a later post).

  1. Proponents of much stiffer security safeguards wanted things like blast barriers to protect transformers and other critical equipment. NERC's proposed rules, however, allow utilities to determine on their own whether or not their substations are critical...”  This is of course also very shocking – it seems the fox is not only guarding the henhouse, but is also judge and jury for any case that might be brought against him for violating the new standard. Because all the standard says is that utilities need to decide on their own what it applies to!  Of course, we all know that these sinister utilities will choose to apply it to few if any substations. 
Again, I hate to introduce these messy things called facts into the discussion, but I do wish to point out that this is exactly what FERC ordered NERC to do.  They realized there was so much variability in substations and their environments that it would be impossible to draw up hard and fast prescriptive rules for deciding which were critical and which weren’t.  Not only that, but there was no way NERC could even develop such rules in the 90 days FERC was giving it.  Instead, FERC recommended that the standard require NERC entities to conduct a risk assessment to identify critical substations, then have that assessment reviewed by a neutral third party;  they will also be audited on how well they did this.  Smart Grid News could have easily found this out by reading FERC’s Order from March, or my post that followed hard upon the Order.

Note from Tom on 2/20/18: It is quite interesting to point out that, not only did electric utilities not under-identify the substations subject to CIP-014, they over-identified them beyond NERC's wildest dreams. I believe there have been over 1,000 substations identified in North America as critical facilities under CIP-014. In my opinion, this is far more than should have been identified. It's simply a testament to the fact that the power industry wants to do the right thing, even if it costs them a lot of money (which this will, of course).

  1. “...and then to decide on their own what measures they should take to protect the facilities, if any.”  This is also shocking on the face of it.  It seems that, even for substations (and control centers) that the utility deems critical, they can decide all on their own what needs to be done to protect them.  And knowing utilities, they’ll decide it’s sufficient to just hang a sign that says “Do Not Attack” on every critical substation and call it a day. 
Again, this was what FERC ordered, and for the same reasons as above: there is way too much variability in substations for any sort of prescriptive rules to work.  More importantly, though, the utilities don’t “decide on their own” what measures they should take.  FERC ordered, and NERC included in the standard, that each NERC entity should have a third-party review of their physical security plan, as well as of the risk assessment they used to designate their critical substations in the first place.  And NERC has set criteria for what third parties can conduct this review.

Now let’s go to Exhibit B.  This isn’t any one particular article or post, but a number of comments I’ve seen on LinkedIn and other forums over a number of years about why the PCI standards for credit card security are so superior to NERC CIP.  These are usually posted by consultants from the IT security industry who have decided to make the switch to the Dark Side of control system security, and are anxious to prove their manhood (I haven’t seen any woman do this) by – what else? – bashing the utilities for writing wimpy standards that just don’t cut it.

I don’t know whether the PCI standards are technically more rigorous or well-written than CIP.  However, I do know there’s a big difference in the cyber security records of the two industries.  We could start with Target, then go back through a whole host of massive thefts of credit card data, all the way back to TJX (TJ Maxx).  I’m sure the total losses to American business from these breaches have been in the billions (and it seems most costs are borne by the banks that have to reissue cards, not the retailers themselves).  If the electric power industry had the same record, we’d all be sitting at home in the dark (and I obviously wouldn’t be writing this post, which some might say would be a good thing).

I will point out a couple really egregious things about PCI.  First, it may be very good at protecting what it protects, but it seems all of the big breaches have been through another channel that was left unprotected by PCI.  The best case in point is Target, where it seems there was no real separation between the corporate systems and those that processed credit card data.  Someone broke into the account of an HVAC contractor (who clearly didn’t need access to credit card data and should have been excluded from it if some sort of least-privilege analysis had been applied), and was able to traverse the network to get to the real-time transaction network – which allowed them to plant their malware on the point-of-sale systems. 

Of course, this would be like having a Balancing Authority’s control center on the same network as the corporate systems, so that the same HVAC contractor’s account could be used to bring down the entire control area.  Were this to be possible, the entity would have been in gross violation of just about every CIP requirement for a long time; and at a million dollars a day per requirement violated, that would be one hefty fine.  But Target was and is completely PCI compliant!  Obviously, whatever technical merits PCI may have don’t make up for the fact that the standard is fairly narrowly focused on the systems that store and process credit card data.  CIP is similarly focused on control networks, but at least it is written from the standpoint that the control network needs to be protected from the corporate network – and the fact that this is the case is probably a big reason why there haven’t been any outages caused by cyber attack.

Another thing about PCI that’s always amazed me is that the entity being audited pays the auditor!  These auditors, called QSAs, are security firms that have passed a rigorous certification; I’m sure they’re quite qualified to do the audits.  But there is such an inherent conflict of interest in this process, since every auditor must have it in the back of their head that they’d love to be given the job of fixing all the problems they find.  You might think this would incent them to find lots of problems, but that’s not the way psychology works.  If someone tells me – in a report that has to be provided to the proper authorities – that I have all sorts of problems, do you think I’ll love them for doing that?  Or am I more likely to love someone who gives me a pass on some of the worst problems – yet still reserves a few meaty ones that he can help me fix?

Now we come to Exhibit C, which is my favorite.  It’s a speech that Mike McConnell, former NSA Director and now Vice Chairman of Booz Allen Hamilton, gave in early March of this year.  In it, he said “In my mind, there is 100% certainty that cyber attacks will occur.”  And of course, he pointed to the power industry[ii] as the likely venue for this attack.  As he said, "Just imagine being in New York City in the middle of the summer with no power." 

Sounds pretty scary, huh?  100% certain!  Of course, that’s an exaggeration, since the only events that are 100% certain are those that occurred in the past.  And I’d say one of the most costly and destructive cyber attacks in the past was that perpetrated by contractor Edward Snowden against Mr. McConnell’s former employer, the NSA.  And who was the firm that placed Snowden at the NSA?  Why, Booz Allen!  Hmmm....

July 1: I just posted an update, discussing a new case in which Smart Grid News attacked (by implication) PG&E for not reporting the Metcalf attack until a year later, without bothering to ascertain whether the allegation was true or not.  I certainly hope I don't have to post more of these, but I will if necessary.

All opinions expressed herein are mine, not necessarily those of Honeywell International, Inc.

[i] I did submit a comment on the article, but it was never posted.

[ii] Also the banking industry.


  1. This comment has been removed by the author.

  2. A few points:

    - CIP & electric sector cybersecurity proponents do a terrible job of stating the case that the security posture of the BES has improved due to their efforts. The WSJ letter to the editor is a prime example. This needs to be part of the sector's efforts. Saying we are writing standards to regulate ourselves (we know best), isn't a great defense.

    - the argument that the appropriate amount of cyber security is in place because there has not been an outage due to a malicious attack can boomerang quickly. I know that was a bit of a throw away line in the article, but I don't think that is how we should be measuring cyber security in the critical infrastructure.

    - the self-regulating point that you railed against has some validity. There is an incentive for utilities to write the standards to avoid regulatory risk. The burden on the BES is to prove they can be trusted to regulate themselves in the cyber security realm. They have not done a good job of this, per my first point.

    - I think it is fair to say that many of the utilities (a majority?) have been unwillingly dragged into cyber security, and they have taken proactive efforts in crafting the standards and their responses to avoid assets and cyber assets falling under CIP. Correct me if I'm wrong on this. I know CIP V5 is designed to address this, but it has taken a long time to get there and let's see how that turns out. I contend that NERC works where most of the members agree something needs to be done, and they want to insure the laggards do it as well. This has not been the case in cyber security in my view.

    BTW, your blog is my go to source for the latest and greatest on the CIP standards.

    Dale Peterson

  3. Thanks, Dale. What WSJ letter to the editor are you referring to?
    I wasn't trying to make the point that utilities are doing a wonderful job on cyber security, but that verbal attacks on them should be based on some sort of facts. All three examples I gave seemed to me to ignore the facts, and simply take advantage of some overall perception that the grid is defenseless against cyber attacks; that is simply not true.
    A case can be made that NERC CIP was (and even is) inadequate, that utilities are trying to get around it, that they're not doing enough outside of pure compliance, etc. The people attacking the utilities (and I don't think you're one of them) need to do some homework and make this case, not just say they're 100% certain that we'll all be living in the Stone Age in a few months and expect everyone to move to New Zealand tomorrow.

  4. You can see the WSJ letter at

    I think we are looking at the same issue from different sides.

    You are saying critics of NERC CIP need to factual about the flaws and risks to the BES. I agree there is a lot of hyperbole. A positive from S4x14 was researchers were starting to tie the cyber & engineering together for both offensive and defensive ICS security efforts.

    I'm saying that defenders of NERC CIP need to be more factual and persuasive that the standards are improving the security posture of the BES. Simply stating there are standards that go through three levels of approvals says little about how effective the standards are, especially given the inherent conflict of interest.

  5. Thanks, Dale. The WSJ letter may point to the CIP standards as evidence that utilities are protected from cyber attacks, but I've never said that. I've asked various NERC entities how much of every dollar they spend on CIP compliance actually goes to improving security; the answer is usually around 25-35 cents. Unfortunately, the majority of CIP spending is for compliance paperwork that probably wouldn't be needed in the absence of CIP.
    So the real question is, is that 25-35 cents on the dollar actually buying the utilities something, and is that enough? The answers to these questions are yes and no. In general, since the amount the industry spends on CIP compliance is so much higher than what other industries spend on cyber security in general, it is very possible that the power industry is getting more protection from what they spend on CIP than other industries are getting from their total cyber spending - just because of the huge difference in the numbers.
    But it's clearly not enough, if that's your main question.

  6. To continue the above conversation, I have always said that, if you want to impose cyber security regulation on the electric power industry, doing it through NERC standards is absolutely the wrong way to do it. Those standards are designed to completely prevent single incidents that could cause widespread havoc such as the 2003 Northeast blackout. As such, they require a focus on avoiding every possible violation, and being able to prove that every possible risk was addressed.

    But there is simply no way you can completely prevent any sort of cyber attack - there are literally innumerable ways one could happen, most of which we don't even know about yet. The best you can do is design a strategy that mitigates as much risk as possible with the available resources. This is completely different from the way NERC standards work.

    The result of using NERC standards for cyber security enforcement has been what I said above: most of the effort and expense required to comply with CIP goes into activities that don't directly enhance security, but that are required because of the NERC framework itself. To solve this problem, some other organization would need to take over cyber security regulation from NERC, and make it more risk-based.

    Of course, the big question is what organization that would be, and what framework would be used. My guess is DHS would be the leading candidate, but there would certainly be others. As far as framework goes, that's the big question. I don't think CFATS - which DHS does implement - has been a success. I'm not sure there is currently an example of a successful risk-based regulatory framework for cyber security (there are many voluntary frameworks which are good, but they couldn't be made mandatory without wholesale changes).

    Dale (and anyone else), I'd be very interested in what you think might be a solution to this problem. The idea would be that, if we could have a framework where close to 100% of what the utilities spend on cyber security compliance actually goes to cyber security, there would be much less anxiety about a cyber attack on the grid.