What’s a High-Impact PACS like You Doing in a Low-Impact Substation like This?

I very recently engaged in a good email discussion that I’d like to share with you. I think it’s worth reading because:

1. It can be important to certain entities that might be in a similar situation;
2. The discussion illustrates how this kind of CIP-002 question can be addressed; and
3. As an afterthought, a good – and much less expensive – solution to the problem came up. It’s important always to be looking for these.

The discussion began when a consultant that I have known for a long time – and engaged in a number of CIP discussions with, some of which have made it into this blog – emailed me about my recent post on the question of “far-end” relays. He stated that he had a client who has a PACS system in a Low impact substation, which is controlled by the PACS server that also protects their High impact Control Center. 

He had determined that, even though the PACS was in a Low substation, the fact that it was associated with the High impact PACS meant it was, itself, High impact. The client at first resisted this finding (understandably, given the higher compliance costs it would entail), but when he pointed out that an attacker could use the substation PACS workstation to alter the configuration of the PACS server in the Control Center, the client came around. The consultant wondered whether I agreed with his position.[i]

My answer was a very safe “It depends”.[ii] If the question were whether a BES Cyber System at a substation could be High impact because it’s associated with a BCS at a High Control Center, my answer would be “Absolutely not”. High BCS are limited to those “used by and located at” High assets (which are all Control Centers, of course). Since the device in question isn’t located at the Control Center, it clearly can’t be a High BCS.

But since we’re talking about a PACS system, not a BCS, the question is different. “Used by and located at” doesn’t apply in this case. What does apply is the word “associated”. Normally, when I talk about this word – which is a lot – I’m referring to the preamble to Section 2 of CIP-002-5.1 Attachment 1, which says that Medium BCS are those that are “associated with” the assets or Facilities that are the subject of the 2.X criteria. 

However, “associated” is used in another location in v5 (actually, many locations): that is the Applicability column of all of the CIP v5 standards (except for CIP-002 and -003). Just about every requirement applies to some variation of Medium and/or High impact BCS and “their associated…PACS”. The question now becomes, “Is the PACS workstation in the Low substation associated with the BCS in the High Control Center?” If so, then many (but not all) of the High impact requirements will apply to it.

Note the question isn’t whether the workstation is associated with the High impact PACS server itself; since it uses the server, the answer to that is clearly yes. But is the workstation in the Low impact substation really associated with the different BES Cyber Systems – ICCP servers, etc. – in the Control Center?

Two things are clear: First, the fact that the PACS workstation in the Low substation could be used to attack the PACS server in the Control Center is completely irrelevant here. Any computer in the world that’s connected to the Internet could, in theory, be used to attack the PACS server. Obviously, you want to take good precautions to protect the PACS workstation in any case, but it won’t be “High impact” for this reason.

Second, the “high water mark” concept has nothing to do with this situation. The fact that the workstation in the substation forms part of the overall PACS system – which includes the “High impact” PACS server – doesn’t mean that all of the components of the PACS system suddenly become “High impact”. Unless, of course, they’re all on a single network in a single ESP. But I really don’t recommend having an ESP that spans locations.[iii]

An implicit assumption so far is that the PACS server in the High Control Center is protecting the High BCS in the Control Center. If the server were really controlling just substations and/or generating stations, but was simply located at the Control Center for convenience, then I contend it would be a) Medium impact if one or more of the assets for which it controlled access were Medium, or b) Low if it controlled only Low assets.[iv]

At this point in the discussion (actually earlier – I’m condensing some), I brought in an auditor friend, with whom I often have conversations about different CIP v5 issues. He said he thought the PACS workstation in the Low impact substation was actually “associated” with the BCS in the High Control Center, and therefore would itself be subject to many of the High requirements - by the Applicability references in the v5 requirements that apply to High BCS. I believe he bases this interpretation on the idea that the workstation could actually change the security settings of the High BCS. This is because the workstation presumably would have access to the entire database on the server, not just the settings for the users with access to the substation it’s located at.[v] This means it would be associated with High BCS.

The auditor elaborated by saying:

“There are multiple components to a PACS and they all have different roles and types of access.  Starting at the top, you have the PACS server that contains the database of access rights for all access points being controlled by the PACS.  So, if you have multiple sites being controlled, the PACS server is associated with each and every one of them, and by extension, the BCS at those sites for which the PACS is controlling access.

“Then you have the door control panels, which is where the local site control takes place.  The PACS server pushes the access rights for the access points controlled by the door control panel to the panel itself.  The panel then operates autonomously, opening (or not) the doors and logging all transactions.  It will ship its logged transactions to server if the server is available, waiting if necessary until communications with the server is reestablished.  But, it can and will operate in the absence of the server using its latest download.  So, the door control panel is associated with whatever is behind the door(s) it is controlling.

“And, of course, you have the badge readers, door strikes/mag locks, and door open/closed sensors located at each access point.  They interface with the door controller.  The badge reader sends the information from the badge just presented (along with the scanned finger print, entered PIN, or whatever in a multi-factor access) to the door controller and the door controller sends the signal to unlock the door if access is to be granted.  These PACS components are explicitly excluded from the CIP standards.

“The odd man out is the workstation.  The workstation is how you interface with the servers to see the logs and/or configure/revoke a badge’s access rights.  A number of entities have tried to argue that the workstation is not part of the PACS because the workstation is not “controlling” access.  The Regions have argued that the workstation is part of the PACS because granting access is more than reading a badge and opening the door.  Granting access includes assigning access rights and that is done with the workstation.[vi]  So, the workstation is associated with every access point for which it can manage access rights; usually the entire PACS database.  And, by extension, it is therefore associated with the BCS sitting behind the door(s).  If you have multiple impact levels, then the workstation takes on the highest impact level of what is behind the doors the workstation can affect.[vii]

“Most, but not all, entities segregate their PACS network from their operational networks.  The workstations will either be on the PACS network, or they will be on the corporate network, with the server bridging the workstation and door controller networks.  Some entities have put their PACS inside their ESP; we usually recommend they rethink that decision.”

I’m stopping here. This particular auditor says the PACS workstation in the Low substation is associated with High impact BCS, in the case presented to him. I don’t agree with him, because I believe this is stretching the meaning of the word “associated”. It would be nice if NERC were to develop a Lesson Learned on this topic, but I doubt that will happen any time before the compliance date next April, and perhaps not even after that. I’m afraid this has to go on top of the already big pile of questions for which the entity – that finds itself in this situation – needs to consider all available guidance, then “roll their own” approach.

I also want to call your attention to footnote v, which lays out a technical control that might by itself prevent this situation from happening. It will probably be much easier to simply implement that control than have to apply a bunch of High impact requirements to the workstation in the Low substation.

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte Advisory.

---

[i] Of course, this question is tied to the far-end relay question by the fact that both involve the issue of a High or Medium impact cyber asset being found in a Low impact substation – which can potentially be expensive, from the compliance point of view.

[ii] It was also followed by my standard disclaimer that the ultimate authority on any question of CIP v5 for any NERC entity is the applicable Regional Entity. In the end, it matters naught what I say, what NERC says, or what the other regions say. None of us will audit the entity.

[iii] An auditor friend – the same one quoted below in this post – pointed out that if an ESP encompasses multiple assets (locations), the communications equipment that connects the two assets will no longer be exempt from CIP v5, per Section 4.2.3.2 of each of the Standards. This is because that exemption applies to devices that are between ESPs; if there’s just one ESP, all the devices are potentially BES Cyber Assets.

[iv] The PACS server wouldn’t be subject to High requirements because in this case it wouldn’t be “associated with” any High BCS. If it controlled access at any Medium substations or plants, it would be subject to some Medium requirements, because then it would be “associated” with Medium BCS. If it only controlled access at Low assets, it would only be subject to CIP-003-6 R2.

[v] As I wrote these words, it occurred to me that there could be a way to restrict any user of the substation workstation from changing any security settings on the PACS server, other than those that apply to that particular substation. I discussed this with the auditor, who confirmed that, if an account on the server were linked to only that workstation – through say a digital certificate, as well as the usual user name/password – and if that account were restricted to only being able to change the settings for access to the Low substation, then that could possibly mean the workstation would no longer be considered “associated with” High BCS. Thus, it wouldn’t have to comply with any High impact requirements. This really seems to be the best solution to this problem, rather than making the PACS in the Low substation subject to a number of the High requirements.

Another assumption we made is that the PACS server isn’t networked with the High BCS in the Control Center. If it were, that might conceivably strengthen the case that the workstation in the Low substation was “associated” with the High BCS. But I don’t think having the PACS server on the same network as the BCS is a good practice in any case – for one thing, it increases the compliance burden since the PACS server now also becomes a Protected Cyber Asset.

[vi] The auditor made this note:

“(Here is) the definition of PACS from the NERC Glossary:

Physical Access Control Systems (PACS): Cyber Assets that control, alert, or log access to the Physical Security Perimeter(s), exclusive of locally mounted hardware or devices at the Physical Security Perimeter such as motion sensors, electronic lock control mechanisms, and badge readers.

“It could be a bit more clear for sure, but the workstation is needed for the control function (the act of granting and revoking access rights) and the alerting function (alarm display).  That is why we have, since Day 1, viewed the workstation as part of the PACS.”

[vii] I think this is a good principle for entities to follow. In practice, it means that, when there are Medium and Low BCS at a single substation or generating station, the PACS will be Medium impact. This follows pretty clearly from the requirements.