Sunday, January 21, 2024

Once again, an ad hoc fix to a serious problem with a NERC CIP requirement

On New Year’ s Day, I wrote a post that described what could be a showstopper problem with the wording in the new CIP-004-7 R6, which took effect that day. This new requirement was the most important part of the changes that took effect that day. The changes were nominally made to open the way for BES Cyber System Information (BCSI) to be “stored” in the cloud, but in fact just storing BCSI in the cloud was only a small part of the reason why the changes were needed.

The main reason why these changes were needed was that the fact that BCSI couldn’t be stored in the cloud meant that use of SaaS (software as a service) was effectively prohibited in OT environments with medium and/or high impact BES Cyber Systems – since a SaaS app would never be able to utilize data that meets the BCSI definition.

Not being able to use SaaS would have been a huge disappointment for NERC entities with medium and/or high impact BES Cyber Systems. In fact, it would be almost as bad as the disappointment of not being able to use EACMS (Electronic Access Control or Monitoring Systems) in the cloud, which has been the case for many years. That problem effectively prevents the use of most outsourced security monitoring services in medium and high impact CIP environments. Unfortunately, fixing the EACMS problem will require almost as much change as will fixing the problem with BES Cyber Systems in the cloud, so that fix is still at least 2-4 years off, absent divine intervention.

When I wrote my New Year’s Day post, I thought the only way out of the SaaS problem would be for CIP-004-7 R6 to be revised, but I didn’t see any movement to do that. However, it turns out that help has arrived from a source I didn’t expect: a paper called “Usage of Cloud Solutions for BES Cyber System Information”.

The details of this are, as usual with questions regarding NERC CIP compliance, too complex to lay out here. However, suffice it to say that this paper was prepared by a committee of NERC entities (the “RSTC”) convened by NERC; they developed the paper to be guidelines for themselves and other NERC members on how to use BCSI in the cloud.

Besides “guidelines”, NERC also recognizes another type of document called “implementation guidance”; this provides guidance on implementation of a standard or standards. However, it is not an Interpretation of a standard. Developing an Interpretation of a NERC standard requires going through a process very similar to the process of developing or amending a CIP standard: constituting a drafting team to draft the Interpretation, having multiple NERC ballots with comment periods until the draft Interpretation meets the stringent criteria for approval, getting approval by the NERC Board of Trustees, and finally getting approval by FERC. This is easily a 1-2 year process and isn’t for the faint hearted. Moreover, in one notorious case more than ten years ago, two CIP interpretations went through this whole process, then were unexpectedly rejected by FERC in the end. That certainly dampened enthusiasm for Interpretations of CIP, if there was ever “enthusiasm” in the first place.

Is there a difference in form between guidelines and implementation guidance? None at all – what makes a document implementation guidance is that NERC (technically, the “NERC ERO”) has “endorsed” it to be such. However, once NERC has endorsed a document as implementation guidance, NERC auditors are required to “give deference” to it when they are auditing compliance with the standard(s) that the guidance refers to. In other words, this doesn’t make it a binding interpretation of a standard, but it’s about as close as you’ll get to that in the NERC world, unless you’re ready to go through a 1-3 year process to modify the standard or develop an Interpretation and get it approved.

Thus, even though the RSTC developed their document to be guidelines, they really wanted it to be implementation guidance; they then submitted it to NERC for approval as such. When the document was originally published last summer I thought it was good, but that wasn’t a universal opinion in the NERC community. I don’t think many people expected that NERC would endorse it as implementation guidance.

But that is exactly what NERC did last month, and I just learned a couple of weeks ago that the document, now that it is official implementation guidance, may prove to be the knight in shining armor that slays the dragon let loose by the problem with the wording of CIP-004-7 R6. Thus, this wording problem may in fact prove not to be a barrier to SaaS – although NERC entities who are especially cautious might want to wait a month or two before signing any contracts for using SaaS in the cloud. Thus, it looks like the NERC community might have been saved from what would have been a very unfortunate outcome. That’s the good news.

So, what’s the bad news? The bad news is this is hardly the first time something like this has happened. These unexpected wording glitches (which have nothing to do with security concerns and everything to do with Standards Drafting Teams being composed of human beings who aren’t blessed with omniscience - although maybe that should be a requirement for SDT membership from now on!) have occurred many times in the past.

I lived through, and wrote about, a number of these controversies regarding wording during the period between when FERC announced they would approve CIP version 5 in April 2013 and July 1, 2016, when the CIP version 5 standards came into effect. CIP v5 was a fundamental rewriting of CIP and is essentially what we’re living with today, along with the standards that came into effect later (CIP-012 through CIP-014). As such, it introduced a whole set of new concepts, like Cyber Asset, BES Cyber Asset, BES Cyber System, EACMS, external routable connectivity, Intermediate System, BCSI and more.

Most of those new concepts had their own definition, although some were undefined, which also led to big problems. But the ambiguities that are almost unavoidable when you’re dealing with cybersecurity regulations often had huge consequences that would cost some group of NERC entities a lot of money, even though that result was never intended.

Thus, there were some terrific battles regarding topics like:

·        The meaning of “programmable” in the definition of “Cyber Asset”;

·        The meaning of “external routable connectivity (ERC)”; and

·        The meaning of “serial” (related to the ERC definition).

There were many more of these battles. In fact, there was a meeting between NERC and the trade associations during that period regarding the meaning of “programmable”. I of course couldn’t attend it and I wouldn’t have wanted to. However, from what I heard at the time, tensions were so high during the meeting that it’s a wonder nobody was killed.

My point in bringing all of this up is that these controversies keep appearing, whenever there’s any change to a CIP standard, or introduction of a new standard. What is most interesting, and depressing, is that almost none of these controversies have been resolved in the intended fashion: by drafting a new or revised definition, a new or revised standard, or an Interpretation. This is because nobody wants to wait for a cumbersome, multiyear process to run its course, before there can be a real answer to one of these questions.

Instead, I believe these controversies have literally all been resolved in the same way the problem with CIP-004-7 R6 has been “resolved”: by some process that has no firm basis in the Rules of Procedure or the CIP standards, but is very much ad hoc. In one serious controversy regarding “transfer trip” relays and the interpretation of Criterion 2.5 in CIP-002 Attachment 1, the “resolution” consisted of a NERC official uttering two magic words at a NERC CIP Committee meeting, without even any documentation – other than in my blog – that this had happened. Yet every NERC entity that was struggling with this issue immediately heard about the statement, as did all the auditors. I never heard about this problem again afterwards. It was no longer an issue, even though nothing at all had officially changed.

Folks, we can’t go on like this. We (i.e., the NERC community) need to determine why these completely unnecessary problems with wording in the CIP standards keep recurring, and figure out a better way to deal with them, than either a multi-year process or some sheriff shooting the bad guy down in the middle of the street, without any due process at all.

This is especially important as the community begins the slow (of course!) effort to amend the CIP standards  to allow full use of the cloud by NERC entities with medium or high impact CIP environments. This may require even more radical changes than those introduced by CIP version 5. Moreover, along with changes to the CIP standards, there will likely need to be changes to the Rules of Procedure that will finally put an end to these debilitating glitches (my favorite idea during the CIP v5 Wars was a Supreme Court of CIP, which would settle every issue once and for all. Of course, that’s not realistic, but I’m not sure any other solution is realistic, either).

Making the required changes to the Rules of Procedure will be quite difficult, I agree. But after they’re made, we hopefully won’t need any more knights in shining armor to save us.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

I lead the OWASP SBOM Forum. If you would like to join or contribute to our group, please go here, or email me with any questions.

 

No comments:

Post a Comment