Hold the (VOIP) Phones!

In April of 2015, I was trying to write some “Tom’s Lessons Learned” to address CIP v5 “interpretation” concerns that NERC was clearly not going to address itself – mostly ambiguities in the standards. There are many of these. As of today, I count 4,357, but there are still a few hours left in the day and that number might increase.

My third LL had to do with VOIP phone systems. This became a big issue in one of the NERC regions, where the region was saying that VOIP systems that served Control Centers need to be evaluated as BES Cyber Assets. This in itself wouldn’t have caused a lot of consternation if the region hadn’t also said – or at least a number of entities in the region believed they said it – that the burden of proof would be on the entity to show why the VOIP system should not be a BCA.

There is no dispute that phones play a crucial role at Control Centers. Take the example of a Control Center that controls a crucial peaker plant. On a hot day in the summer, if the RTO or ISO called to order that the plant be brought up and couldn’t get through to anybody because the phones were down, there could conceivably be a very serious BES impact.

Let me say at the outset that, if the VOIP system is networked with BES Cyber Systems within the ESP (which I would call a deplorable security practice, if that word hadn’t been overused lately), this whole discussion is moot. It is a PCA and therefore subject to almost all of the requirements that BCS are. The rest of this post assumes this is not the case.

Of course, the question whether a Cyber Asset is a BCA depends on whether it meets the BCA definition, the heart of which reads “A Cyber Asset that if rendered unavailable, degraded, or misused would, within 15 minutes of its required operation, misoperation, or non-operation, adversely impact one or more Facilities, systems, or equipment, which, if destroyed, degraded, or otherwise rendered unavailable when needed, would affect the reliable operation of the Bulk Electric System. Redundancy of affected Facilities, systems, and equipment shall not be considered when determining adverse impact.”

Let’s concede right away that the above example shows that a VOIP system failure in a Control Center could indeed have a very serious BES impact within 15 minutes. So if this is indeed what the BCA definition says, there is no question the VOIP system would have to be a BCA. But note the word “could” here. Does that appear in the definition? No, it doesn’t; instead the word “would” appears, which I translate loosely as meaning “inevitably”. In other words, I interpret the BCA definition to require that there would inevitably be a BES impact if the Cyber Asset were to fail, be misused, etc.

Now, I freely admit that the word “inevitably” is not in the BCA definition. But neither is the word “could” or “possibly”, so it can’t be said that the meaning is that there only has to be a possibility of BES damage for the Cyber Asset to meet the definition. I interpret “would” to mean inevitably, but others may not. This is just one of the 4,357 ambiguities in CIP v5. And, like all the other ambiguities, it can’t be removed by a closer reading of the words. It is just there, period.

If you read “inevitably” into the BCA definition, IMHO, there should be no question that VOIP systems are not BCAs. Sure, the system may go down on a hot day and the initial call from the RTO won’t go through. But will the RTO just give up and say, “Oh well, I guess we’ll just have to accept a cascading outage”? Of course not. They will be prepared for this eventuality, and there will be a number of other ways they can get through. They can call another phone at the same organization (perhaps the Finance department) and ask that they bring the message to the Control Center immediately. Or they will have stored some cell phone numbers of Control Center personnel that they can call in just this situation. Or they can get in via a satellite phone. Or they can use smoke signals or carrier pigeons. The message will get through, one way or the other.

So it seemed clear to me that there would not inevitably be a BES impact if the VOIP system in our hypothetical Control Center went down; given that I interpreted “would” to imply inevitability, it seemed reasonable to say that VOIP systems could never be BCAs. But in response to assertions about alternative communications pathways, I am told (and I heard this once with my own ears) that the region would say something to the effect of “Aha, but as the BCA definition says, the fact that there is some sort of redundant method of getting through can’t be used to ‘excuse’ the VOIP system from being a BCA.”

In other words, the region seemed to be arguing that the fact that there were so many alternative communications pathways couldn’t be used to excuse the VOIP system from being a BCA. This assumed that the fact that the cellular communications systems, satellite systems, etc. effectively back up the VOIP system in the Control Center fell under what was meant by “Redundancy” in the BCA definition.

If this were true, then my insertion of the word “inevitably” in the BCA definition has no effect since, if all of those alternative systems can’t be considered, then the VOIP being down in the Control Center will inevitably have a huge impact on the BES; therefore, the VOIP system will be a BCA. But the fact is that there are all of these alternative methods of communication, and there assuredly won’t be a BES impact.

This always struck me as not at all what the CIP v5 drafting team had in mind when they talked about redundancy.[i] It seemed that they meant identical systems, configured identically, deployed in such a manner that if one failed, the other would immediately pick up where the other left off – think redundant servers. But the local cellular system and the Control Center’s VOIP system are hardly identical. If the VOIP goes down, the cellular system (or another like the satellite phone system) will indeed provide an alternative communications vehicle, but the converse isn’t true: If the cellular system goes down in say a city, the utility’s VOIP won’t instantly back it up. So there is no symmetry here.

While my original post on VOIP didn’t mention it, there is another good argument why the word “inevitably” should be inserted in the BCA definition: If a Cyber Asset should be declared a BCA even if its loss, misuse, etc. wouldn’t inevitably impact the BES, then other systems at BES assets – especially including HVAC and lighting – need to be treated the same. If the heat goes off in a power plant in northern Ontario in January, there might be a need to shut it down – but it certainly isn’t inevitable, given that people might be able to just put on their coats, hats and gloves and keep working. So if the magic word “inevitable” isn’t inserted in the BCA definition, all of these systems would have to be declared BCAs as well.

At this point, you might bring up NERC’s FAQ document from early 2015, which I wrote about in this  post. That document stated that “support systems” should be excluded from consideration as BES Cyber Assets; the two examples provided were HVAC and lighting systems, although I know some of the NERC regions consider VOIP systems to fall in the same category. Because of this, I believe almost all of the regions haven’t required their entities even to consider whether VOIP systems are BCAs.

I’m certainly not objecting to this practice, since it produces the same result as what I’ve been advocating. However, what is disturbing about this FAQ statement is that there is absolutely no basis for it in CIP v5. Where is the definition of “support system”, and where in the BCA definition does it say that support systems are excluded from consideration? Obviously, the answer is “nowhere”.

But what about the region that had (according to some entities in the region) originally stated that VOIP systems[ii] had to be declared as BCAs? Did they change their position? Since I stopped hearing cries of anguish from that region, I assumed they had. However, at a recent compliance meeting for that region, one of their auditors clearly implied they had not.

He did this when he replied to a question whether, if an entity had in place the Alternative Interpersonal Communications system (AIC) mandated by the NERC COM-001-3 standard (essentially, a backup phone system for control centers), this would negate the need to declare a VOIP system as a BCA. The auditor placed some conditions on his statement, but agreed that this was the case.

Of course, since the result of this statement conforms to what I’ve been advocating, I’m not going to scream and rant about this (there are plenty of other opportunities to do that!). However, I want to point out that the implication of the auditor’s answer is that no, the region has not changed its basic position. Effectively, all they are now saying is that there is one particular type of alternative communications system that does actually lead to the VOIP system not having an inevitable impact on the BES. But the implication of this is that all of the other alternatives I mentioned earlier – cell phones, satellite phones, etc. – do not make the VOIP system’s impact on the BES any less inevitable, and do not make the system any less of a BCA. This was the region’s original position (again, judging by what I was told by some entities, although I myself heard this stated at a compliance meeting for that region earlier this year).

However, the auditor introduced an interesting new word in his reply: “dissimilar”. He used it in referring to the AIC system, meaning it was dissimilar to the VOIP system (which indeed it is). I believe the reason he used this word is he is implicitly advocating that it be inserted in the second sentence of the BCA definition, so that this sentence would now read something like “Redundancy of affected Facilities, systems, and equipment shall not be considered when determining adverse impact, in the case where such Facilities, systems, and equipment[iii] are similar[iv] to the Cyber Asset under consideration.”

This is terrible language from a legal point of view, but it illustrates the idea. If the entity is considering whether a Cyber Asset is a BES Cyber Asset, and if there is a redundant system (e.g. a backup server) that is in a similar configuration, the mere presence of that redundant system does not remove the inevitability of the Cyber Asset’s impact on the BES if lost, misused, etc. This is partly because the same attack that disabled the original Cyber Asset may very well attack the similarly-configured redundanct system. On the other hand, if there are multiple alternatives that remove the inevitability from the impact (in the case of VOIP these are the cell systems, smoke signals, etc), then that redundancy can be considered as removing the inevitability.[v]

And if this is what the auditor is implying, then I am in total agreement with him. In fact, I believe the ambiguity regarding the status of VOIP, HVAC and lighting systems, with respect to their status as BCAs, could be completely eliminated if two words (actually, one word and one phrase) were added to the BCA definition: “inevitably” and “similar”. I would advocate that the current CIP v7 SDT add this to their agenda, were that agenda not already overwhelming (more on this point in a post coming soon).

So why did I write this post? After all, there clearly is no danger that entities will be forced to declare their VOIP systems as BCAs. And even though I’ve just stated how I think the ambiguity regarding VOIP can be cleared up, I’m not even forcefully advocating that this needs to happen.

Well, I’ll tell you why I’m writing this post. I’m writing it because it illustrates an unfortunate fact of life regarding the CIP v5 and v6 standards (and almost certainly the v7 ones as well): Sometimes, the only way to effectively comply with and audit a requirement is to interpret a requirement or definition as if it had been written differently. In this case, both the entity and the auditor need to insert two words into a definition. In other cases (and I will illustrate one in another post that is coming soon), both the entity and the auditor need to ignore some of the wording in the requirements.

I used to get very worked up over the fact that, in my personal opinion, the need to do this makes the CIP v5 and v6 standards unenforceable in the strict sense that a fine that is appealed to the courts by an entity would almost surely be thrown out, due to the ambiguity in key areas (almost all having to do with CIP-002 R1 and asset identification). However, I’ve more lately come to the conclusion that this is not worth worrying about. I say this not because my advancing age makes me more mellow (although it does in my case), but because I’ve found so many other examples where requirements or definitions need to be reinterpreted in order to be followed or audited, and where there is about zero probability that these will be fixed in any of our lifetimes. What’s another one or two ambiguities?

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte Advisory.

---

[i] I had to think long and hard before I used the words “what (the SDT) had in mind”. This is because I have written at least one post pointing out that there is no possible way to verify what the SDT had in mind – or more specifically what they “intended” – when they used or didn’t use any particular words. I excuse myself here because I’m really referring more to common English usage (and specifically IT and engineering usage), rather than conducting a hypothetical inquiry into the state of mind of the SDT members when the BCA definition was approved.

[ii] Actually, this whole discussion applies to a lot more than VOIP systems. Any PBX that relies on Cyber Assets to operate would have to be considered as well, not just those that rely on VOIP. However, this issue has always been discussed as relating to VOIP, so I’ll continue to call it that.

[iii] I would much prefer that “Facilities” and “equipment” be removed from this phrase (as well as from Section 4.2 of all of the CIP standards), so that it just referred to “systems”. I have always considered the use of  this phrase to be a huge mistake.

[iv] In an email discussion on this issue with an auditor from another region, he pointed out to me that, if there is only one dissimilar system that backs up a Cyber Asset, then that should not be considered as removing the inevitability of the impact of the loss of the Cyber Asset on the BES. I agree with this, and thank the auditor for pointing it out to me (I’m not going to add this to my proposed new definition, though. If the v7 SDT asks me to draft the complete language of my proposed changes to the BCA definition, I’d be happy to oblige. I would like to see a number of other changes as well, that have nothing to do with the subject of this post). Of course, VOIP, HVAC etc. all have multiple “systems” backing them up, as I have illustrated above. In other words, if there were really only one other way – say a particular vendor’s cell phone system in that area - to get through to the Control Center if their VOIP system were down, then the message really would not get through, assuming both the VOIP and the cell system were down. But it is very hard to imagine a case where there would be no other way at all to get the message to the control center in 15 minutes.

[v] Of course, the entity should not be able to blithely state that there are a lot of alternatives, without documenting that there are any. In the case of VOIP, this shouldn’t be too hard an exercise.