Wednesday, January 30, 2019

A new record!



I think a lot of my readers will already know this, but if you don’t – NERC just announced the largest-ever CIP fine, which adds another decimal place to the previous largest fine: $10 million even (in fact, I imagine this figure, being the smallest possible eight-digit amount, was deliberately chosen for its ability to strike terror into the hearts of utility compliance folks nationwide). It’s all outlined in a voluminous four-part Notice of Penalty totaling over 700 pages. I’ve only seen the first part, available here, and that alone is 250 pages! Naturally, I’ve only skimmed through it, and I’m not sure when I’ll read the whole part 1, let alone all four parts.

Of course, the name of the entity (or really entities. In fact, the organization is always referred to as “The Companies”) isn’t provided. Beyond that, NERC has redacted all information that might refer to a particular NERC Region (although it’s clear there were at least two or three Regions involved); NERC clearly believes it would constitute a big threat to the BES to provide any information that might lead to identification of the entity.

However, I’m much more interested in what the violations were, and what overall lessons can be learned by other utilities. There are 127 violations, covering all currently-enforced CIP standards including CIP-014. The details of those violations are up to you to read, but I call your attention to pages 10-13, which discuss a) Facts common to the violations (i.e. common causes); b) Risks common to the violations; and c) Mitigations common to the violations.

Since the PDF is high security, I can’t copy any text to paste it here, but I’ll summarize. First, the common causes they point to are:
  • Lack of management engagement and support for the CIP program;
  • Program deficiencies, including deficient documents, training, and implementation;
  • Lack of communication between management levels in the company; and
  • Lack of communication between business units on who is responsible for which tasks.

The entity committed to:
  • Increasing senior leadership and oversight;
  • Centralized CIP oversight department;
  • Conducting industry surveys and benchmarking regarding best compliance practices (I admit I have a hard time understanding this one. I have never yet seen any sort of comprehensive industry survey of compliance practices – mainly because for a utility to provide that information, it will almost always require providing BES Cyber System Information at the same time);
  • Continuing to develop an in-house CIP program and talent development program;
  • Investing in enterprise-wide tools (configuration management, etc.);
  • Adding security and compliance resources;
  • Instituting annual compliance drills (that’s an interesting idea; I hadn’t heard of that before); and
  • Creating three levels of security and compliance training.

These are the common mitigation actions the entity committed to:
  • Revising their corporate IT compliance program so that it meets the requirements of all stakeholders;
  • Requiring each business units to revise their procedures and controls so that they follow the corporate IT program;
  • Each business unit will document and track its controls for CIP compliance; and
  • Documenting how each non-compliance listed in the settlement agreement was mitigated, and how this will prevent recurrence of the violation (of course, that document will be about three times the length of the NOP. There’ll be a whole lotta writin’ going on!).

I must say that I have yet to hear of any utility that couldn’t also benefit from at least a few of these same practices. Go thou and do likewise.


Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013; we also work with security product or service vendors that need help articulating their message to the power industry. To discuss this, you can email me at the same address.

4 comments:

  1. I have heard two different rumors as to who it is. The WSJ has published one of them.

    I did a quick scan of the requirements violated. I saw no violations that were High-Impact only requirements (maybe I am wrong and missed one). This leads me to conclude the org(s) are all small to medium-sized with no High-Impact Assets. If so and these org(s) are small to mmedoum-sized (like one that has 40K customers), $10M is a pretty significant fine.

    If the WSJ is correct, then $10M is pocket change, especially for the amount of time covered and requirements violated.

    ReplyDelete
  2. I don't doubt it's Duke. Your observations are quite good, Jason. I'm writing a post, that should be up Sunday, that follows up on this - in clarification of a quote from me that appeared in today's Wall Street Journal. I'm sure Duke has High impact Control Centers, so if there aren't any High requirement violations, this comports with NERC's statement that the risk to the BES is more due to the collective impact of all the violations, than any one or two which in themselves could have enabled a serious cyber attack.

    ReplyDelete
  3. By the way, my quote got cut out of the WSJ's print edition, so it's only online. But I just realized that the whole article is available, even though generally they're all behind a paywall (I assume this was because of WSJ's feelings about the importance of the subject): https://www.wsj.com/articles/duke-energy-broke-rules-designed-to-keep-electric-grid-safe-11549056238?mod=hp_major_pos13

    ReplyDelete
  4. Regarding Jason R's comment, a CIP compliance person wrote in to point out to me that at least one of the requirement parts Duke violated is High-only: CIP-010 R3.3. So Jason R's comment needs to be corrected (Jason R has privately confirmed to me that he might have missed one or more High-only requirements in the NoP, since as he pointed out, this observation was based on a "quick scan").

    So obviously there was one High Control Center in violation of at least one requirement part. But I'm sure Duke has a number of High Control Centers. Were all of them in violation of this part? Not likely, but because of the redaction, we simply can't answer questions like this.

    And since I'm a Larger Lessons kind of guy, here's a larger lesson: I understand that NERC wants to protect information that might in any way enable an attack on the grid. However, there's a cost to this, which is that it is very hard to draw concrete lessons from NoPs, meaning it is hard to learn from the mistakes of others.

    ReplyDelete