Saturday, December 17, 2016

A Lesson Still Unlearned


I attended the NERC CIPC meeting in Atlanta this week. There wasn’t a lot of discussion of the CIP standards, since they are just one of the topics discussed at those meetings, and we aren’t currently in the throes of any momentous developments in CIP. But I was struck by one common theme I heard from a couple of the speakers, and realized that a lesson I thought the NERC community had learned over the past four or five years still hasn’t been learned.

The lesson is simple: If there are gaps or inconsistencies in a NERC standard that can’t be fixed with a straightforward reading of the text, the only permanent remedy is to write a SAR (Standards Authorization Request) and draft a new or revised standard that fixes the problem. There is no other remedy allowed by NERC’s Rules of Procedure.

NERC has run into this immutable fact several times in the past (at least regarding CIP), and has spent a lot of time and effort trying to get around it. Does anybody remember the 2012 CANs (Compliance Application Notices) and CARs (Compliance Analysis Reports) that were intended to fix inconsistencies in interpretation of the CIP v3 requirements? They caused a lot of weeping, wailing and gnashing of teeth in the NERC community, and they were ultimately withdrawn.

Now let’s go to CIP v5. After FERC approved v5 in late 2013 and entities took a serious look at the requirements in 2014, they began to realize there were a lot of problems with the wording – ambiguities, inconsistencies and just plain holes. NERC acknowledged there were some problems and vowed to address them in time for entities to be fully compliant by the faraway date of April 1, 2016.

They promised a large and shifting array of documents to fix the problems. There may have been others, but I remember NERC pointing at various times to the V5 Implementation Study, the RSAWs, Section 11 Supporting Documents, the Lessons Learned and finally the infamous Memoranda (which created a huge firestorm, as a result of which all of them were revoked and removed from the NERC web site).

All of these documents fell into one of two types: serious attempts to address gaps or inconsistencies; or “helpful hints” regarding CIP compliance that didn’t address wording problems. Documents of the first type (for example, the Lesson Learned on “Programmable Electronic Devices”, which seemed to many of us to be a sensible way to fix the problem caused by the fact that the word “Programmable” in the Cyber Asset definition wasn’t itself defined) were without exception ultimately withdrawn and removed from NERC’s website – when it became clear they weren’t just providing implementation guidance but actually going beyond the strict wording. Documents of the second type were left in place, and remain there to this day (note that I’m not complaining about the second type of documents. These certainly did fulfill an important need. But they didn’t clear up gaps or inconsistencies in the wording of the standards, as they were initially touted as doing).

Late in 2015, NERC seemed to finally acknowledge the hard truth that I’ve already stated: The only way to change a NERC standard or definition is to go through the standards drafting process, which of course usually takes years. They bit the bullet and called for a new CIP drafting team to fix some problems in CIP v5 (which were identified by the CIP v5 Transition Advisory Group). When FERC approved CIP v6 in January of this year, they ordered NERC to make three changes: clarify the LERC definition, develop requirements for Transient Electronic Devices at Low impact assets, and protect all Control Center-to-Control Center communications. These items were added to the new SDT’s SAR. I supported the new SDT, but pointed out that the really fundamental problems with CIP v5/v6 weren’t even mentioned in the SAR. In fact, I later published a list of important problems that aren’t included in the SAR.

To be honest, at that point (around April of this year) I assumed that it had finally become apparent to the NERC community, and especially to NERC itself, that there was no point in talking any more about ingenious “solutions” to CIP wording problems, that didn’t involve a new SAR[i]. That is why I was surprised by three statements that were made at this week’s CIPC meeting:

1)      Tobias Whitney of NERC always leads a discussion of current developments in the CIP standards at CIPC meetings. He discussed the recent Technical Conference that included a day of discussion on the problem of how entities can utilize the cloud while staying compliant with CIP[ii]. He said NERC was working on this issue and would put out a paper soon. He implied that this would settle the matter of how the cloud could be utilized in a CIP environment.

I pointed out in the meeting that this paper will certainly be valuable, but it isn’t suddenly going to change the fact that a strict reading of the CIP standards wouldn’t allow an entity to utilize the cloud. Tobias didn’t seem to disagree with this, but he also didn’t state its implication: Unless NERC plans to write a SAR for a new CIP version to incorporate the cloud (and NERC has made no moves to do that), the only way that a NERC entity can safely utilize the cloud is to reach an accommodation with their Regional Entity to allow this. I happen to think that most if not all of the RE’s will be open to entities that want to utilize the cloud (as they have been to virtualization); but I’m also sure a lot of entities will hesitate to move in this direction, until the cloud is officially recognized in the CIP standards. And I won’t even venture a guess as to how many years from now that will be!

2)      The second statement was made during a discussion of the VOIP issue. There, it was again implied that a rigorous, thoughtful analysis of the wording of the requirements and the BES Cyber Asset definition would resolve the problem. I’ve got news for you, guys: This has been a serious concern for at least a couple of years, and has been debated endlessly. At least 7 of the 8 NERC regions have decided they do not consider VOIP systems to be automatically BES Cyber Assets; and I believe the eighth region now takes this position as well. But the problem can’t be fixed permanently until the BCA definition is changed, which – ta da! – requires a SAR (and while the current CIP SDT is charged with changing the BCA definition to address two other problems, they are not charged with addressing this one. Moreover, it is highly unlikely they will decide to add this to their already-overfull agenda).

3)      At one point, Tobias brought up something I’d forgotten about: that somehow a number of industry organizations, including the trade associations, have become empowered to write up “guidance” on CIP compliance questions. I had heard this before, but couldn’t understand what it meant – and I still can’t. Any organization has always been empowered to write guidance for its members (and any others who wish to follow it) on how to comply with any standard – whether a NERC standard or not. But there is no way that this can be considered some sort of “official” guidance, which NERC will endorse as something the regional auditors should follow. And if that’s the case, why even imply that allowing the organizations to issue guidance is in some way a mitigation of the wording problems with CIP v5/v6? It isn’t.
                                                            
You may have noticed one common theme to all three of the points above: In the end, the only “interpreters” of the CIP standards that matter are the regions. If the auditors in a region think a requirement should be interpreted in a particular way, the entities in that region would be well served to make sure they understand their thinking, since they are likely to be audited based on that. And guess what? I doubt there’s any NERC entity in the US who won’t assign the utmost importance to their own region’s interpretation of a CIP requirement.

But there is a problem with this: Since the Rules of Procedure say nothing about the regions having any authority to interpret the standards, no region will ever commit an interpretation to writing, even in an email. I have heard from a lot of entities that you have to call up an auditor and ask his or her opinion on an interpretation question. They might not tell you, of course, but if they do they will only do it on the phone. Of course, this means that if, three years from now, a different auditor issues a PV because their interpretation was different from that of the auditor you talked to, you won’t have any documentation of what the original auditor told you. So this is obviously not any sort of permanent solution.

The irony of this is that, after the CAN/CAR debacle with CIP v3, NERC pointed to CIP v5 as the version that would finally fix the problem of regional variability in interpretation of requirements. Not only has v5 (and v6) not fixed that problem, it has made it worse. Because of the much larger number of gaps and inconsistencies in the wording of the v5/v6 requirements and definitions, the regions have now become essential to understanding how to comply. I have said it a number of times – but never in this blog – that the CIP v5/v6 requirements only “mean” what your region says they mean. Nothing more, nothing less. So keep your eye on your region.[iii]

Before I let you go, repeat after me: There is no way a problem with the wording of a NERC standard can be permanently fixed, other than by writing a SAR and drafting a new standard. Ignore anyone who tells you something different.

But before you say, “Well, then we just need to draft a bunch of SARs and get some new SDTs to work on fixing the wording problems in CIP v5/v6”, I recommend you read my next post. This will prove (using advanced mathematics) that we have reached the limit of changes that can be made in the current prescriptive CIP standards. If we want to incorporate new areas like virtualization and the cloud, we will have to move to a non-prescriptive, outcomes-based approach. And that will be the only way to permanently “fix” the problems with the current wording (most of which problems will become moot in a non-prescriptive format).

This will require a complete rewrite of CIP, as well as changes in how the standards are audited and managed. But the alternative is to have what we have now: a set of standards whose interpretation is ultimately dependent on the auditors. If this is your idea of how mandatory standards with million-dollar-a-day penalties should be interpreted, then you should be happy as a clam now. If you aren’t comfortable with this, then I suggest you start asking what can be done to change the situation.


The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte Advisory.


[i] When I say that a SAR is the only way to change a standard, you may be asking “How about RFIs (Requests for Interpretation)? They are also permitted in the NERC Rules of Procedure. An Interpretation is voted on by the NERC ballot body and approved by FERC, just like a new standard is.” But Interpretations aren’t meant to clarify problems with wording; they are only to explain wording that is already in place. So they can’t help with the problems I’m talking about. Those problems can only be addressed by changes in the wording of the standards or definitions.

[ii] There are a number of CIP problems that come up with the cloud. One of the most important has to do with CIP-004 R3 – R5. For example, if a NERC entity puts data from an ESP in the cloud, it will most likely reside in one or more huge server rooms, to which hundreds of technicians may have access. Since any one of those technicians might in theory be able to walk up to a server with that data and look at it, a strict reading of the standard would require that each one of them be vetted by the entity for access to their data before being allowed in that server room, and that the cloud provider notify the entity when any of those people leaves their employ. No cloud provider in the world will ever agree to do this. The provider would need to make sure that all servers that hold the entity’s data be housed in a locked room, with access granted only to technicians that have been vetted by the entity. This would effectively destroy any cloud provider’s business model.

[iii] I just noticed that I had predicted this situation in a post a month after FERC approved v5 in 2013. In a footnote, I said “I’m guessing that, if CIP-002-5 isn’t changed, the way this problem will be finally dealt with (not solved) is through the Regional Entities taking it upon themselves to develop their own interpretations.  These won’t have any more force than a NERC interpretation, but since the RE’s are the ones who do the auditing, it is far more likely the registered entities will follow the lead of their region.” Of course, I was specifically discussing problems with CIP-002 here, but the same can be said for any of the v5/v6 standards: the Regional Entities are the only real arbiters of interpretation questions. This situation will remain until the standards are revised or rewritten entirely (which is my preference, in case you haven’t noticed that yet).

No comments:

Post a Comment