Bashing Utilities for Fun and Profit

I have been impressed (if that’s the right word) for a long time by the fact that many commentators in the popular and trade press seem to agree on one key point: the electric utility industry is rife with security vulnerabilities and is ready to fall over tomorrow, with just one small push by some 11-year-old in Estonia.  This in spite of the fact that I don’t know of a single directed cyber attack on a power facility (or facilities) that has led to an outage for even one household for one minute.  And while there definitely was a very serious physical attack on a substation in California last year (which didn't cause an outage, but well could have had the stars been aligned a little differently), the industry very quickly stepped up on their own to address problems, even before FERC ordered a mandatory standard.

This isn’t to say that the electric power industry doesn’t need to do more for cyber security, or that it doesn’t occupy a very special position that requires a lot of scrutiny; even a one-minute outage for a whole city would be devastating.  But it is to say that it’s really unproductive for people who should know better (and if they don’t know better, they shouldn’t be standing up as experts) to be making these statements with no facts to back them up.

I also think I know why these people don’t get called out more often.  It’s because the electric power industry is mostly regulated, and the entities that comprise it don’t want to stir up any controversy, even if this means sitting back and taking some pretty unfair hits.   Here are three examples of what I’m talking about:

Exhibit A is this article on the Smart Grid News website.  I need to first say that I have a lot of respect for what Jesse Berst has done for the smart grid; it’s no exaggeration to say he’s a huge reason the smart grid is as successful as it is today.  But this doesn’t excuse sloppy reporting and commentary, which is the case here[i].

The article is about NERC’s approval of CIP-014, the new physical security standard for substations ordered by FERC in early March.  The article says nothing good about the standard and makes three arguments why it is virtually worthless:

1. “On the plus side, the proposed rules gained 82% approval. On the downside, most of the votes were from the very utilities who (sic) would be subject to these new regulations.”  This is a frightening observation – it seems the very rules that are needed to protect our way of life are being written by the thoughtless (evil?) utilities that have to follow them!  What could be worse? 

This ignores the small point that all NERC regulations are drafted by the utilities subject to them, and have been since NERC was founded in 1967.  But this is even more shocking!  We should tell Congress!  However, Congress already knows this, since they wrote the Energy Policy Act of 2005.  This was the law that put in place the current structure in which FERC is the regulator while NERC drafts standards, submits them for FERC approval (or rejection), and audits the member entities for compliance.

Congress didn’t pass this law on a whim late one night.  It was very well vetted with lots of hearings, testimony, etc.  And there are very good reasons why this structure was put in place, one of the most important being that neither FERC nor any other organization has the ability to understand all the nuances (many extremely local and particular) of the electric power industry.  It is best that the people most involved day-to-day draft the standards, while FERC makes their own judgment regarding their adequacy (and they often push back.  In fact, it seems that is happening now with the new CIP requirements for Low impact assets – more on that in a later post).

2. “Proponents of much stiffer security safeguards wanted things like blast barriers to protect transformers and other critical equipment. NERC's proposed rules, however, allow utilities to determine on their own whether or not their substations are critical...”  This is of course also very shocking – it seems the fox is not only guarding the henhouse, but is also judge and jury for any case that might be brought against him for violating the new standard. Because all the standard says is that utilities need to decide on their own what it applies to!  Of course, we all know that these sinister utilities will choose to apply it to few if any substations. 

Again, I hate to introduce these messy things called facts into the discussion, but I do wish to point out that this is exactly what FERC ordered NERC to do.  They realized there was so much variability in substations and their environments that it would be impossible to draw up hard and fast prescriptive rules for deciding which were critical and which weren’t.  Not only that, but there was no way NERC could even develop such rules in the 90 days FERC was giving it.  Instead, FERC recommended that the standard require NERC entities to conduct a risk assessment to identify critical substations, then have that assessment reviewed by a neutral third party;  they will also be audited on how well they did this.  Smart Grid News could have easily found this out by reading FERC’s Order from March, or my post that followed hard upon the Order.

Note from Tom on 2/20/18: It is quite interesting to point out that, not only did electric utilities not under-identify the substations subject to CIP-014, they over-identified them beyond NERC's wildest dreams. I believe there have been over 1,000 substations identified in North America as critical facilities under CIP-014. In my opinion, this is far more than should have been identified. It's simply a testament to the fact that the power industry wants to do the right thing, even if it costs them a lot of money (which this will, of course).

3. “...and then to decide on their own what measures they should take to protect the facilities, if any.”  This is also shocking on the face of it.  It seems that, even for substations (and control centers) that the utility deems critical, they can decide all on their own what needs to be done to protect them.  And knowing utilities, they’ll decide it’s sufficient to just hang a sign that says “Do Not Attack” on every critical substation and call it a day. 

Again, this was what FERC ordered, and for the same reasons as above: there is way too much variability in substations for any sort of prescriptive rules to work.  More importantly, though, the utilities don’t “decide on their own” what measures they should take.  FERC ordered, and NERC included in the standard, that each NERC entity should have a third-party review of their physical security plan, as well as of the risk assessment they used to designate their critical substations in the first place.  And NERC has set criteria for what third parties can conduct this review.

Now let’s go to Exhibit B.  This isn’t any one particular article or post, but a number of comments I’ve seen on LinkedIn and other forums over a number of years about why the PCI standards for credit card security are so superior to NERC CIP.  These are usually posted by consultants from the IT security industry who have decided to make the switch to the Dark Side of control system security, and are anxious to prove their manhood (I haven’t seen any woman do this) by – what else? – bashing the utilities for writing wimpy standards that just don’t cut it.

I don’t know whether the PCI standards are technically more rigorous or well-written than CIP.  However, I do know there’s a big difference in the cyber security records of the two industries.  We could start with Target, then go back through a whole host of massive thefts of credit card data, all the way back to TJX (TJ Maxx).  I’m sure the total losses to American business from these breaches have been in the billions (and it seems most costs are borne by the banks that have to reissue cards, not the retailers themselves).  If the electric power industry had the same record, we’d all be sitting at home in the dark (and I obviously wouldn’t be writing this post, which some might say would be a good thing).

I will point out a couple really egregious things about PCI.  First, it may be very good at protecting what it protects, but it seems all of the big breaches have been through another channel that was left unprotected by PCI.  The best case in point is Target, where it seems there was no real separation between the corporate systems and those that processed credit card data.  Someone broke into the account of an HVAC contractor (who clearly didn’t need access to credit card data and should have been excluded from it if some sort of least-privilege analysis had been applied), and was able to traverse the network to get to the real-time transaction network – which allowed them to plant their malware on the point-of-sale systems. 

Of course, this would be like having a Balancing Authority’s control center on the same network as the corporate systems, so that the same HVAC contractor’s account could be used to bring down the entire control area.  Were this to be possible, the entity would have been in gross violation of just about every CIP requirement for a long time; and at a million dollars a day per requirement violated, that would be one hefty fine.  But Target was and is completely PCI compliant!  Obviously, whatever technical merits PCI may have don’t make up for the fact that the standard is fairly narrowly focused on the systems that store and process credit card data.  CIP is similarly focused on control networks, but at least it is written from the standpoint that the control network needs to be protected from the corporate network – and the fact that this is the case is probably a big reason why there haven’t been any outages caused by cyber attack.

Another thing about PCI that’s always amazed me is that the entity being audited pays the auditor!  These auditors, called QSAs, are security firms that have passed a rigorous certification; I’m sure they’re quite qualified to do the audits.  But there is such an inherent conflict of interest in this process, since every auditor must have it in the back of their head that they’d love to be given the job of fixing all the problems they find.  You might think this would incent them to find lots of problems, but that’s not the way psychology works.  If someone tells me – in a report that has to be provided to the proper authorities – that I have all sorts of problems, do you think I’ll love them for doing that?  Or am I more likely to love someone who gives me a pass on some of the worst problems – yet still reserves a few meaty ones that he can help me fix?

  

Now we come to Exhibit C, which is my favorite.  It’s a speech that Mike McConnell, former NSA Director and now Vice Chairman of Booz Allen Hamilton, gave in early March of this year.  In it, he said “In my mind, there is 100% certainty that cyber attacks will occur.”  And of course, he pointed to the power industry[ii] as the likely venue for this attack.  As he said, "Just imagine being in New York City in the middle of the summer with no power." 

Sounds pretty scary, huh?  100% certain!  Of course, that’s an exaggeration, since the only events that are 100% certain are those that occurred in the past.  And I’d say one of the most costly and destructive cyber attacks in the past was that perpetrated by contractor Edward Snowden against Mr. McConnell’s former employer, the NSA.  And who was the firm that placed Snowden at the NSA?  Why, Booz Allen!  Hmmm....

July 1: I just posted an update, discussing a new case in which Smart Grid News attacked (by implication) PG&E for not reporting the Metcalf attack until a year later, without bothering to ascertain whether the allegation was true or not.  I certainly hope I don't have to post more of these, but I will if necessary.

All opinions expressed herein are mine, not necessarily those of Honeywell International, Inc.

---

[i] I did submit a comment on the article, but it was never posted.

[ii] Also the banking industry.