Tuesday, July 16, 2019

An (ex-)Auditor comments on CIP-013 for Low impact assets

Kevin Perry, the former chief CIP auditor of SPP Regional Entity, wrote in today to comment on my Sunday post, the second (and last) in a series on NERC’s draft Data Request on Supply Chain Risk Assessment - which is currently out for comment by NERC entities. Specifically, Kevin wanted to comment on my suggestion that NERC proactively propose to include a basic supply chain security requirement for Low impact assets in CIP-013, rather than wait for FERC to order that Lows be made to comply with all of CIP-013. That would be much more burdensome and expensive.

Specifically, I suggested that NERC propose something like Requirement R5 in the first draft of CIP-013 (from January 2017), which was definitely a very basic requirement but also managed to address two of the biggest BES supply chain risks faced by all NERC entities: compromised patches and vendor remote access. That requirement read:

R5. Each Responsible Entity with at least one asset identified in CIP-002 containing low impact BES Cyber Systems shall have one or more documented cyber security policies, which shall be reviewed and approved by the CIP Senior Manager or delegate at least once every 15 calendar months, that address the following topics for its low impact BES Cyber Systems:

5.1. Integrity and authenticity of software and firmware and any patches, updates, and upgrades to software and firmware; and 
5.2. Controlling vendor-initiated remote access, including system-to-system remote access with vendor(s). 

Kevin thought this was a good idea because it would reduce a lot of supply chain risk, but also because it wouldn’t put a big compliance burden on Low-only entities (and probably zero burden on entities with Mediums and Highs that also have Lows, which is just about all of them. Those entities wouldn’t have to comply with these two new requirement parts, which are just for Lows. But they already have to comply with two parts that are very similar, CIP-013-1 R1.2.5 and R1.2.6).

Specifically, Kevin said (with my comments in italics):

I could see implementing your (of course, these aren’t mine! They were written by the CIP 13 drafting team) suggested 5.1 and 5.2 with relatively little effort.  Both can be implemented holistically.

1.       5.1 is vendor-centric and would apply to all Low Impact BCS, whether one or thousands, because the focus is on the vendor and not the Cyber Asset. Kevin is of course referring to the fact that the vendor is really the entity that has to “comply” with this requirement part. They will need to figure out a way to provide their electric utility customers with a way to verify integrity (e.g. checksum) and authenticity (e.g. digital signature) of all patches and other software they send to them. Once they’ve solved this problem for just one utility (whether they have High, Medium or Low assets), they’ve solved it for them all. It’s unlikely that many if any vendors that sell software that goes on BES Cyber Systems will balk at doing this.

2.       With respect to 5.2, putting in a gateway would be appropriate. This could be as simple as enabling/disabling a VPN account that the vendor uses to get to the supported entity’s network.  Access permissions then control where the vendor can go.  Best practice, you install the equivalent of an Intermediate System to maintain a breakpoint between the vendor and the supported systems. This requirement part will definitely be a little harder to comply with, because it does require some configuration at each Low asset. But my guess is that most entities with Low assets are already doing something like this. And if they aren’t, there are a lot of inexpensive remote access gateways for sale.

Note that Kevin isn’t saying that the only good way to comply with this part is to install an Intermediate System (as Mediums and Highs do, per CIP-005 R5.2), which would definitely be a good deal more expensive and time consuming, especially for entities with a lot of Low assets (one large entity told me a few years ago that they have over a thousand Low impact substations! Putting an IS in each one of those would be…well it would be pretty burdensome).

Kevin later emailed back to say “One more thought...  there is an added benefit.  If the entity is also a Distribution Provider, the Low impact BCS controls will also apply to the same relays in the distribution subs - no extra cost.” Such a deal!

July 18: Kevin emailed me yesterday to emphasize the point that compliance with R5.2 wouldn't be very burdensome on Lows (although I'd like to hear from anyone who thinks differently):

If the entity remotely manages its LBCS, such as over a TCP/IP network or by using a centrally managed “dial up” system to authenticate the user and then connect the user to the supported Cyber Asset, that is where you would place a central gateway to control vendor access to the company network and resources required to access the LBCS.  If the entity has a gateway device in the substation, then it is mostly a matter of adding the vendor to the gateway access list and, of course, enabling the vendor to get to the gateway, most likely through a central gateway that authenticates the vendor onto the entity network.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. To discuss this, you can email me at the same address.

Sunday, July 14, 2019

NERC’s draft Data Request on supply chain risks for Lows, part II

This is the second (and last) of my two posts on NERC’s draft Data Request on supply chain risks for Low impact assets. The first one is here (it includes a link to the draft DR itself). I actually ended up writing two more posts last week based on comments and questions I received about the issue of how to define external routable connectivity for Lows; but they weren’t about the DR itself, which is why I say this is part II (I expect to have another post or two – or maybe more – on the topic of “erc” in the near future).

There are two main parts to the DR. The first part is the long section entitled “BES Cyber Systems”. In the first post, I discussed at length the many wording problems in that section, and expressed the opinion that it could be cleaned up and actually sent out. But I also said that without changes I think it’s just going to cause a lot of confusion and – most importantly – it’s very unlikely to yield any usable data.

There are two issues I want to discuss in this post. The first issue has to do with the second part of the DR, but the second is strategic: I think NERC is pursuing exactly the wrong strategy in dealing with what is evidently very strong pressure from Congress to increase requirements for Lows. I think the DR isn’t going to satisfy these Congressional hawks at all, and will probably make the industry’s position with Congress worse, not better.

To start with the first issue, let’s go to the second part of the DR, which is much shorter than the first part. You can find it on pages 7 and 8 of the DR, under the heading “CIP-013 Cost of Implementation”, but since it’s so short I’ll reproduce the whole section here:

“Stakeholders, regulators and legislator’s decisions on mitigating and preventing supply chain risk depend on the costs and benefits associated with those decisions. While utilities would want and share this information, it is not currently available. Therefore, subject matter experts believe it is premature for CIP-013 registered entities to determine, or estimate costs or benefits associated with the implementation of the standard.

  • The standard is new and there is no historic precedence (note: this should be ‘precedent’) for registered entities to pre‐determine costs based on furthering relationships with existing and new vendors. 
  • These costs and benefits are intangible and depend on a spectrum of actions, from internal process refinement costs to extensive costs associated with replacement of blacklisted vendors.
  • The cost of compliance is currently unknown as this is a new standard.
  • Many utilities are experiencing push back from vendors for CIP‐013 compliance that could require vendor change or increase in cost from such vendors. 
Consequently, CIP-013 is causing, and will necessitate many changes for complying utilities from now until the July 1, 2020 implementation date. Therefore, currently providing any credible cost or benefit information is premature.

      6. Do you agree with the above SME assessment – Yes or No?    

Please provide CIP‐013 cost or benefit amounts should you answer “no” to the above question:”

In case you’re puzzled by this (and if you’re not, I don’t think you’re reading it closely), this section essentially says “We’re not stupid. We know there’s no way that a Low impact entity can estimate their costs for compliance with CIP-013. Do you agree with that statement? But if you don’t, could you give us your estimate anyway?”

Here’s a little history: The first draft of the DR that the SCWG received from NERC asked entities to estimate the cost for complying with CIP-013 for Low impact assets. At our first meeting with NERC, we pointed out that even Medium and High entities still can’t give a good estimate of their cost for complying with CIP-013, so how could Low-only entities possibly do this (although, based on the fact that I’ve been working on nothing but CIP-013 compliance with three entities of different sizes since January 1, I have come to realize my initial estimates of the total cost were too high. I don’t think people understand how important it is that this is a risk-based standard, not a prescriptive one. That makes for a huge difference in costs)?

The NERC people didn’t dispute this, but they pointed out that the Board of Trustees had ordered them to ask this question, so they felt they had to do it. I suggested (at a later meeting) that the SCWG might actually be able to come up with a cost estimate for Lows (since almost all of the NERC entities that are part of the SCWG have Low impact assets, as well as Highs and/or Mediums). I said it would be much better to have us estimate costs than the Low-only entities, who have no experience at all with CIP-013, and probably haven’t thought about it at all yet.

But no, NERC said they had to ask a question since the Board had ordered it. So some people in the SCWG came up with the above “question”. I analogize it this way:

  1. The Board has asked NERC staff to ask entities how to design a perpetual motion machine.
  2. Instead of simply telling the Board that there’s no way to design a perpetual motion machine, the staff members have decided to ask entities this ‘question’: “We know there’s no way to design a perpetual motion machine. Do you agree with this statement? But if you don’t agree, please give us your diagrams.”

My main objection to this “question” is that it makes NERC and the SCWG look ridiculous for even asking it. And my other objection is that any data that come out of it are guaranteed to be garbage. Anyone who answers this is simply going to take a guess, and they will all understand that if the cost estimates come in low, this will probably lead to CIP-013 being applied to Lows. So they’ll just estimate as high as they think is credible. I think this whole question needs to be removed.

Now I will discuss my strategic objection to this whole Data Request. It is based on what the SCWG members were told by NERC staff about the reason for this DR (with a little inference on my part, to fill in some gaps):

  1. Congress and FERC have been leaning very heavily on NERC in recent months to increase CIP requirements on Low assets in general.
  2. Since the question of Lows and CIP-013 was already on the table, that pressure is now focused on requiring Lows to comply with CIP-013.
  3. NERC staff (and maybe the Board) are worried that FERC is simply going to put out an Order saying that Lows need to be included in CIP-013. To stave off that event (or at least push it back), NERC will do this Data Request, which – due to all the steps required to approve it, gather answers and then analyze them – will probably take six months to complete.
  4. My guess is that part of the Board’s (and NERC staff’s) concern here is showing Low entities that they’re trying to fight this off any way they can – and when and if FERC issues their Order, NERC will be able to tell the Lows “Well, we tried…”

This wouldn’t be a terrible strategy if there were any conceivable way it could work, but I don’t see one. The main problem is that, even if the first question is cleaned up and sent out, it’s not going to ask for the data that Congress wants. During one of the SCWG meetings, I asked Lonnie Ratliff of NERC (whose title is Senior Manager, Cyber and Physical Assurance) what exactly the Congressional staffers had been asking him about Lows.

He said they wanted to know how many Critical Cyber Assets were at Lows. Remember that CCAs were a single device, so these Congressional staffers were obviously trying to find out how many computing devices were at Lows. Lonnie told them that term had been replaced by BES Cyber System, and of course that’s why the first draft of the DR that NERC showed to the SCWG asked for information about Low BCS. As I mentioned in the first post, the idea of asking about BCS was finally dropped when, at one of the SCWG meetings, I pointed out that there were some entities who have over a thousand BES Cyber Assets in a single BCS, whereas there are others who are classifying every BCA as a BCS . So asking about BCS isn’t going to yield any meaningful data about BES Cyber Assets, which is the closest current term to Critical Cyber Asset – which always referred to an individual device.

So if NERC were really going to satisfy Congress, they would need to ask about BES Cyber Assets in the DR. But to even ask this question would require owners of Low assets to have to go through a huge effort to identify all Cyber Assets in every Low asset, then consider each one as to whether it meets the BCA definition. They would rebel if NERC even asked a question about Low BCS, but there would probably be blood in the streets if NERC asked about Low BCAs. Even if NERC just asked Lows to estimate the number of BCAs, they would still have to go through a lot of effort.

So in question 1 of the draft DR, NERC is just asking about Low assets – primarily substations, generating plants, and Control Centers. It would certainly be interesting to get this data (although a lot of this has already been given to the Regions, since every Low entity has to state their number of Low assets of each type). But I see no way this is going to satisfy Congress. They want to measure the degree of cyber risk posed by Lows based on the number of computing devices installed. They’re going to be very disappointed when NERC hands them a list of assets, and when they’re asked how many devices this covers, the NERC representative will say “Oh, we can’t get that information. Sorry.”

But here’s an idea, NERC: Instead of making your entities go through a lot of work to respond to a DR, then six months later give Congress and FERC a list that isn’t what they wanted, why don’t you get ahead of this issue and say “You know, you people are right. CIP-013 should be applied to Lows in some way, although we’re sure you’ll agree there’s no need to make Lows go through everything that Highs and Mediums need to do. We’ll draft a Low-only requirement and add that into version 2 of CIP-013, since we’re just now putting together a team to develop that version.”

And what should this Low-only requirement be? I think NERC would be well advised to go back to the first draft of CIP-013, which had this requirement part R5:

"R5. Each Responsible Entity with at least one asset identified in CIP-002 containing low impact BES Cyber Systems shall have one or more documented cyber security policies, which shall be reviewed and approved by the CIP Senior Manager or delegate at least once every 15 calendar months, that address the following topics for its low impact BES Cyber Systems: 
5.1. Integrity and authenticity of software and firmware and any patches, updates, and upgrades to software and firmware; and 
5.2. Controlling vendor-initiated remote access, including system-to-system remote access with vendor(s)."

CIP-013 affandicios will recognize 5.1 as being pretty close to CIP-013-1 R1.2.5, while 5.2 is close to R1.2.6. So what would Lows have to do to comply with these two requirement parts? They would have to develop a policy for Low BCS that includes these two items (and I would recommend that the new SDT not only require Lows to have a policy, but to implement it. CIP-003-5 R2 just required Lows to have four policies, but said nothing about implementing them. FERC then told NERC to go beyond policies and add “specific requirements” for Lows in CIP v6. If the v5 SDT had included “implement” in the v5 requirement in the first place, FERC would probably have said that was good enough and they wouldn’t have required any more. Instead, there has been all sorts of anguish over CIP-003-7 R2 , since FERC didn’t like the CIP-003-6 R2 that NERC came up with. This anguish was reflected in the emails I received last week about my two erc posts, and it will increase as the compliance date approaches).

And if CIP-013-2 includes this requirement, what would Lows not have to do, compared with their Medium and High brethren? The most important thing they wouldn’t have to do is comply with R1.1, which requires the entity to consider all of its supply chain cyber risks and mitigate the most important ones. While – as I mentioned above – I no longer think this will require of Mediums and Highs the degree of effort that I estimated originally, it will still be a large amount of work, and will require that people in supply chain and legal be heavily involved, which wouldn’t be the case if the Low-only requirement were implemented as described above.

The other thing Lows wouldn’t have to do is comply with R1.2.1 through R1.2.4. I just looked at how these parts differ from R1.2.5 and R1.2.6, and it seems the big difference is that the latter two parts aren’t going to require a lot of heavy lifting with vendors, while R1.2.1-R1.2.4 will be more difficult to get vendor agreement on. This may be why the CIP-013 SDT only required that Lows “comply” with R1.2.5 and R1.2.6 in the first draft of the standard (which went down to defeat with only a 9% positive vote, due in part to the fact that the Low requirement was there. There will definitely be opposition if NERC moves to put a Low requirement in CIP-013-2, but I think it will be much more defensible if NERC points to what may be the alternative – having FERC require that Lows be included in scope for all CIP-013 requirements).

Will this suggested requirement mean that Lows have to keep an inventory of BES Cyber Systems? Definitely not. Lows will have to show auditors that they have these policies and have implemented them, but that doesn’t require having evidence that they were applied in every instance and for every component of a BCS.

What will happen if NERC persists in their current course and sends out this Data Request, then turns the results over to FERC and Congress late this year or early in 2020? As I’ve said, the data that will be gathered aren’t what Congress is looking for, so they’re unlikely to be satisfied. Will they all just give up and focus on getting elected again? I really doubt it. The cyber security of the grid has become a big concern of Congress, and both parties agree that electric utilities should be doing more about this. I think they will pressure FERC to go ahead and tell NERC to simply include Lows in CIP-013, which probably means having all of the current requirements apply to Lows. I think that, if NERC made a proactive move like what I’ve suggested above, they’d very possibly be able to preclude this much more drastic step.

And there’s another reason why I think this would be a good move. In case you haven’t noticed, electric power isn’t the most popular industry nowadays (not that it ever was, of course), and many in Congress feel the industry just isn’t stepping up to the plate enough to combat cyber threats. At the same time, they wonder why such an important industry should have the power to write its own cyber security standards.

If NERC is perceived as reflexively fighting all cyber regulation, Congress will soon feel (and probably are already) that they need to think about taking cyber regulation of the power industry away from NERC (the O&P standards are a different deal, of course) and giving it directly to DoE or DHS (or maybe a new Department of Cybersecurity).

So sometimes it’s better to acknowledge the other party’s concerns and say “We agree. We do need to do more about this. Here’s our proposal…”, rather than try to simply stonewall them. Especially when that other party is much more powerful than you are.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. To discuss this, you can email me at the same address.

Saturday, July 13, 2019


As I was writing my most recent post on Thursday, I noticed it would be my 500th post, since I started the blog in January 2013. This comes on top of my 500,000th “page view” and the fifth anniversary of my blog, both of which occurred last year.

Is there any connection among all of these fives? Not that I know of. I just find this interesting.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. To discuss this, you can email me at the same address.

Thursday, July 11, 2019

How do you define external routable connectivity for Lows?

I want to follow up on my last post, which was itself a follow-up to my Part I post on NERC’s draft Data Request regarding supply-chain risks for Low impact assets, which is now out for comment until July 22. And of course, what the DR is really about (in case you’re new to the world of NERC-speak, which requires decades of experience to learn to decipher) is “Should CIP-013 be applied to Low impact assets?”

You may think this question was already addressed by NERC in their recent “Cyber Security Supply Chain Risks” report (which they filed with FERC a few weeks ago), where they said there was no need to move further on the idea at this time, but more study is needed. However, it seems that NERC is now getting a lot of pressure from FERC and from Congress on this issue, and they’re accelerating this DR from what might have otherwise been a more leisurely schedule.

I’m not calling this post and the previous one parts II and III of the DR post – instead, I’m sticking to my original idea of doing two posts, with the second likely to come out at the beginning of next week, God willing and the creek don’t rise. But I raised a question in the first post about how the equivalent of External Routable Connectivity can be defined for Low BES Cyber Systems, given that the term (which is of course defined in the NERC Glossary) includes the words “Electronic Security Perimeter”, and ESP only applies to High and Medium BCS (really only to Medium BCS, since all High impact assets are Control Centers, and trying to find a Control Center without ERC would be like trying to find out if there has ever been a Jewish Pope – the whole purpose of a CC is connectivity). Since I’ve received some interesting answers to that question from devoted readers, I’m devoting these two posts to them.

My last post was devoted to a suggestion by two people (one a current CIP auditor) that simply de-capitalizing the words “Electronic Security Perimeter” in the ERC definition would work fine. But I received two comments that I’d like to throw out there, because of light they shed on the difficult – and still not settled, four years after it was a very hot topic and probably the subject of fist fights – question of what ERC (or erc for that matter) really means.

The first comment (really a question) I received after my last post – and this was from more than one person – was “Where in the ESP definition does it say it’s restricted to High and Medium BCS?” Of course, the answer to that is “Nowhere”. So why do I say this is the case? Because an ESP is defined as essentially a logical “line” that includes all of the BES Cyber Systems within an asset (or perhaps just one part of an asset). If you haven’t included all the BCS, then you haven’t properly drawn your ESP and you might be in for some fines.

However, the CIP standards make clear in a couple of places that an inventory of Low impact BCS isn’t required. This means that “ESP” officially has no meaning for Lows (this issue has been fought over many times, and nobody wants to resurrect it anytime soon). But as I stated in the last post, just decapitalizing ERC should eliminate that problem, since this is just a Data Request, not an audit.

The other comment I received was from Kevin Perry, who has been involved with NERC CIP since approximately the Korean War (OK, that’s cruel. He and I are the same age, I believe!), and was for many years the Chief CIP Auditor of SPP Regional Entity. He is allegedly retired now, although I haven’t seen a lot of evidence of that.

Kevin pointed to the former definition of LERC (Low impact External Routable Connectivity) – which I had mentioned in my Part I post as a possibly usable definition, but rejected it because LERC was part of CIP-003-6, and that version of CIP-003 never came into effect, having been replaced by CIP-003-7 (which comes into effect Jan. 1, 2020). He said “Rather than decapitalizing ESP, why not go back to the original intent of LERC - the ability of a Cyber Asset to access a Low Impact BCS from beyond the border (e.g., fence line) of the asset (e.g., the substation) in which the Low Impact BCS resides?”

I must admit this is also a good definition. It was carefully crafted by the CIP v6 drafting team to get around the problem of not having an ESP in Low assets. The ESP was replaced with the idea of the “border” – a term that wasn’t defined, but which I thought was something that people could probably agree on without a lot of disputation. As it is, LERC wasn’t retired because people were fighting over what a border was, but because FERC was concerned about the word “direct” in the LERC definition. Of course, there’s a big back story behind that concern (as there always is with anything relating to NERC or FERC). And if you want to read all 499 of my previous posts (yes, I just realized this will be my 500th post!), I’m sure you’ll understand it.

In any case, LERC now sleeps with the fishes, but I don’t see why it couldn’t also be used as a definition of ERC that applies to Lows, for the purposes of this Data Request. That is, unless you consider the fact that I think this whole DR is misguided and the result of a terrible strategy, which might result in the death of NERC and the end of Life As We Know It. But that discussion is for my post early next week.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. To discuss this, you can email me at the same address.

Tuesday, July 9, 2019

A definition of external routable connectivity

In last week’s post on NERC’s draft Data Request on supply chain security for Low impact assets, one of the objections I made to NERC’s initial draft of the DR had to do with how you define External Routable Connectivity at a Low asset. This is important because, in its original form, the DR requested NERC entities to state their number of Low impact assets with ERC (i.e. capitalized). The official NERC definition of ERC, applicable to Medium and High impact BES Cyber Systems, is “The ability to access a BES Cyber System from a Cyber Asset that is outside of its associated Electronic Security Perimeter via a bi-directional routable protocol connection.” Of course, since Low impact assets don’t have ESPs, this definition doesn’t directly apply to them.

In the post, I pointed out that the “solution” to this problem that ended up in the final draft sent out for comment last week (which was actually something suggested by the team from the Supply Chain Working Group that reviewed and revised the draft) was probably worse than NERC’s original suggestion, since it simply pointed to CIP-003-7 and stated that the discussion of “external routable connectivity” (i.e. ERC for Lows) in there could be the guide to how a NERC entity responds to this question. I pointed out that very few people in the NERC community – or even at NERC itself – could correctly state how CIP-003-7 “defines” erc. I opined that it would be pretty easy for most entities to come up with their own definition, but then – very helpfully – I didn’t state a definition of erc I thought might be good!

Fortunately, two people – one a current CIP auditor who wants to remain anonymous, and the other my longtime friend and grizzled veteran of the CIP Wars, Joe Garmon – pointed out an obvious solution to this problem: Simply de-capitalize ESP in the ERC definition, so that it reads “The ability to access a BES Cyber System from a Cyber Asset that is outside of its associated electronic security perimeter via a bi-directional routable protocol connection.” I don’t think anyone can go too wrong if they use this definition – after all, keep in mind that the DR has nothing to do with CIP compliance, even though it will come from NERC (although I know a lot of people will never believe this).

However, my opinion remains that NERC would be making a big error to release any DR at all, and they need to completely rethink how they’re responding to the pressure they’re evidently getting from Congress and FERC on this matter. I will make that point in part II of the DR post (this was part 1 ½, I guess), which should be coming within days to a blog near you. Please try to contain your excitement until it arrives!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. To discuss this, you can email me at the same address.

Monday, July 8, 2019

Mike Assante

I just can’t get Mike’s parting message to the ICS security community out of my mind. I can’t put aside the fact that, when he was obviously very near the end and probably very weak, as well as distracted with thoughts of the family he was leaving behind (including young children), his number 2 concern (after his family, I’m sure) was the unsung throngs of people working day by day to help make the grid more secure.

Notice he didn’t name any names, or ever say something like “…and I especially want to thank Joe Jones for his immeasurable contributions…” Instead, look at the first three sentences of his last paragraph, before the last sentence (which was a critical infrastructure joke, and quite timely): “Continue your hard work and be confident that it is noble to protect infrastructures and your organization’s value.  Your work is more important than most organizations understand, but we continue in it out of a sense of duty and passion.  You must carry on, keep sharing with each other, and use this forum to continue to help our community.”

Almost literally Mike’s last published thought was about the people who work for organizations that don’t bestow on those who labor in their security vineyards either the monetary or psychic rewards they deserve. His message: “Your work is important and noble. You must carry it on, but you’re not alone. There are many others doing the same thing. You can all support each other.”

Rest in peace, Mike.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Saturday, July 6, 2019

A farewell from Mike Assante

I just read on the SANS ICS Community perhaps the saddest post I've ever seen. It's by Mike Assante, former NERC CSO (in fact, I believe he was the first one), and someone who has been hugely important to the industrial cyber security field, and especially to grid security. To read about his accomplishments, you can go to his LinkedIn page.

He has been battling leukemia for a long time, and there have been ups and downs. This post is his farewell. When I first saw it, I thought he had written it because he had been told that the end was near. However, as you can see, he wrote it while he still could, with instructions that it not be posted until he had passed on. I hadn't heard that he had - and LinkedIn and the SANS website don't mention this at all - but I take this as notification of his passing.

Even though I didn't know him very well, I'm really going to miss him! As a final note, I'll point out that he concluded with a joke. Here's Mike's final post:

As you might have surmised this note is a goodbye to this ICS community.  I have been in a long, hard fight with Leukemia and it has come to an end.  I wanted to provide a final update to all my friends and the general ICS community upon my passing, as well as to provide my support in what all of you have chosen to do in your careers. 

I have had the honor to work alongside so many talented and passionate professionals throughout the years.  I need you to know that what you do and how you do it matters to communities, to companies, and to our nations.  I have worked hard alongside some of the greatest people to demonstrate the risks, provide strategies to manage it, and now to develop the workforce to enhance its security.  

I am especially proud to have collaborated with the communities from inside of Idaho National Laboratories (INL) and SANS.  I leave behind the people I loved to work with, especially as they are so focused on making a difference.  I want to thank all of you for what you do and what you have done, and I want to say farewell!

Continue your hard work and be confident that it is noble to protect infrastructures and your organizations value.  Your work is more important than most organizations understand, but we continue in it out of a sense of duty and passion.  You must carry-on, keep sharing with each other, and use this forum to continue to help our community.  I now get to take a whole new look at another infrastructure, but this time it’s in the sky (and I’m not talking about the 737 Max)!  

Sunday morning, July 7: For Robert M. Lee's very moving post on his good friend Mike, go here:

Tuesday, July 2, 2019

NERC’s draft Data Request on supply chain risks for Lows, Part I

Today, NERC put out a draft Data Request. NERC entities will have 3 weeks to make comments on it. The purpose of this DR is to help NERC staff determine “whether new information supports modifying the standards to include low impact BES Cyber Systems with External Routable Connectivity” (and by “the standards”, NERC means the Supply Chain Standards, CIP-013-1, CIP-005-6 and CIP-010-3).

I will point out to start that, if NERC is really just interested in whether Low impact assets with External Routable Connectivity should be included in CIP-013, I can give them a quick answer: There aren’t any Low impact assets with ERC, so it doesn’t matter if they’re included or not. ERC (capitalized, since it’s a NERC Glossary term) only applies to High and Medium impact assets. The definition refers to the Electronic Security Perimeter, and only Highs and Mediums have ESPs.

But I digress. This DR wasn’t a surprise to me, since I and a small subcommittee of the NERC CIPC’s Supply Chain Working Group (which is itself a subcommittee of the CIPC, making the small group a sub-sub-committee, for those keeping score at home) have been drafting this with Howard Gugel, NERC Director of Standards Development, for more than a month (Howard came to the group with a first draft, but what came out in the end was completely different). I’ll say at the outset that there were some very interesting discussions over that time, and this version was approved over my strenuous objections (not that there was a vote, but it seemed pretty clear that I was the only person still willing to voice objections at the end of our discussions, which was two weeks ago. What was sent out today seems to be close to what was approved then, but is somewhat improved – at least in the first section).

To spoil the ending for you, my objections to this draft are:

1) There are two overall sections in the DR. The first starts on page 5 under the heading “BES Cyber Systems”. I think this section is confusing and misuses terms, but it could be saved with some important changes (it was worse two weeks ago, but it was somewhat cleaned up after it went from the SCWG to NERC). As it stands now, I doubt it will provide much useful data.

2) The second section begins on page 7 under the heading “CIP-013 Cost of Implementation”. This can’t be saved, and needs to be put to sleep with the fishes, as they say in “The Godfather” (or, if you prefer, it needs to be “terminated with extreme prejudice”, as the CIA agent says to Martin Sheen in Saigon at the beginning of “Apocalypse Now”. I just realized that both of these great movies are from Francis Ford Coppola! No wonder they’re the two that seem relevant here).

3) But my biggest objection isn’t to this draft, but the whole idea that there is a need for a DR in the first place. I think this reflects very bad strategy on NERC’s part. What’s driving this DR – judging from what I’ve heard from people at NERC and elsewhere, and drawing a couple conclusions of my own from what I’ve heard – is that people in Congress are itching to see much tougher cyber regulation of the power industry, and they’ve keyed in on the idea that this should start with more controls on Lows (and CIP-013 would probably be the logical place to start with Lows).

The data generated by the DR won’t satisfy Congress (as I’ll show later on), and the fact that NERC will spend 4-7 months drafting the DR, getting comments, gathering responses, analyzing them and then generating some sort of report might very well be held against the indusry. This means that the hawks in Congress are still likely to require that Lows be included in CIP-013 (and FERC will go along with them), regardless of the DR results, meaning the DR is probably a big waste of everybody’s time. Even more importantly, if this all comes to pass a big blow will have been delivered to the idea that the electric power industry should be both developing and enforcing its own cybersecurity standards in the first place. Need I point out that that isn't good for NERC or the industry?

I had thought I could easily write about all three objections in one post, but when I got to page 4 of the draft and I was still nowhere near finished with the first objection, I decided to leave the other two objections for the second post. I expect I’ll have that up on Sunday evening, so your otherwise dreary Monday-after-a-long-July 4th weekend will be greatly enlivened by Part II of this post – or maybe not. In either case, I think I’ll have it up Sunday evening.

Now to my first objection, which consists of about 7 or so sub-objections on the first section of the DR. The first sub-objection is to the title of the section itself, which is “BES Cyber Systems”. Those of you who survived the CIP v5 and v6 rollouts probably have your antennae up when you hear that this is the title of the section, since a DR regarding Low impact assets shouldn’t be talking about Low BCS at all (reasons to follow).  

To give you some background on this section, the DR started off (in the first draft, that NERC provided to the SCWG) requesting rough numbers of Low impact BES Cyber Systems with and without ERC – and when a few of us pointed out that ERC doesn’t apply to Lows, this was changed to “external routable connectivity”.   

I and a few others on the sub-sub-committee pointed out next that, while some of the Regions have in fact requested lists of Low BCS from their member entities, since CIP v5 came into effect (where the Low/Medium/High impact levels were introduced) the standards have stated explicitly that an inventory of Low BCS isn’t required; the fact that NERC was just requesting rough numbers in the initial draft of the DR really didn’t change that at all – there would definitely be a lot of resistance to this question among NERC entities.

That objection didn’t seem to change things, but what finally did was when I pointed out in a later meeting that I knew one large NERC entity that stated in a Regional compliance meeting a few years ago that they had in the thousands of relays in their Medium impact substations, but they were grouped into only 2 or 3 (I believe) BES Cyber Systems (of course, this is perfectly legal). But I also know one even larger utility that has declared every Medium or High impact BES Cyber Asset to be its own BCS. So, if those two entities both have say 5,000 relays at Low assets, the first entity might report that it has say five Low BCS, while the other will say it has 5,000 Low BCS. If hypothetically they were the only two entities responding to the DR, NERC would dutifully say that the average large NERC entity has between 2502 and 2503 Low BCS! Obviously, such data are totally useless. That finally seemed to put the kibosh on requiring entities to estimate Low impact BCS numbers in the DR.

We then agreed that we should ask for numbers of Low assets with and without risk factors like external routable connectivity, dial-up access, etc. I suggested to keep it simple: Ask the entities to just estimate numbers of Low assets with erc, dial-up, and maybe a couple other risk factors – as well as those without any of these, of course (I also suggested breaking each of these down into Control Centers, substations and generating stations). As you can see, the first section of NERC’s draft DR goes beyond what I suggested, and breaks each type of asset down by three “Location Risk Scores” (so you have three types of Low Control Centers, three types of substations, etc. - and questions on the risk factors for each of these types).

I have no objection in principle to that change, but the problem is that it’s been worded very poorly. Were these problems to be fixed, I think this would be a usable question, but that’s only if the DR goes out in the first place (and for that issue, see the second post). However, I think the following problems first need to be fixed.

The first “question” starts off on page 5 by saying “In order for NERC to understand the data they receive from this Data Request, they need to understand the basis for each entity’s answer. This means that how your entity categorized your BES Cyber Systems can have a huge impact on these survey results.” OK, fair enough.

The rest of that paragraph is “In order to have Data Request results that can be used and compared, the common basis are the six locations called out in CIP‐002. This Data Request is focused on those locations and not how entities have designed their BES Cyber Systems.” OK, that also sounds reasonable. Let’s go to CIP-002-5.1a to determine what NERC means by “locations”. Hmmm, it seems that the word “locations” isn’t used in CIP-002-5.1a at all, except in a few of the bright-line criteria in Attachment 1 (e.g. Criterion 2.1), where it has a very specific use that has nothing to do with what we’re talking about here. So what does NERC mean by this term?

Two paragraphs below, NERC says “The term “location” refers to physical space associated with an asset.” And there is a footnote that says in part “For the purpose of this data request, the word “asset” is used in the same way as it is used in CIP‐002‐5.1a Requirement R1.” Since what they’re referring to is the six types of assets listed in R1.1, I’ll combine the three statements to yield what I think is the meaning of what NERC’s asking: “This Data Request is focused on the physical spaces associated with assets (i.e. the six asset types listed) in CIP-002-5.1a Requirement R1.” I assume “physical space” means the land on which the asset is built and the air for a certain distance above the land.

Isn’t this kind of odd? You’re being asked how many physical spaces you have that house Low impact assets; I guess that makes a little bit of sense. But look at the “Location Risk Score Table” on page 5. Let’s take the row that starts with “3.3” (but any row will do) and continues with “Generation resources”. It provides location risk scores based on megawatts at the location. Do the dirt and air in which a generating plant is built generate megawatts of power? No, the plant itself does, which is an “asset” in R1. Why not just ask about the asset in the first place?

And while we’re on page 5, move up to the table before this one. That asks for “Number of assets containing BES Cyber Systems”, broken down in four ways. That comports perfectly with R1, but it doesn’t with the rest of this section of the DR, which of course is after “locations” not assets. And a final confusion regarding “location” is found in the introduction to the table on page 7, which says “You may group assets with the same answers into a single line item.” However, of course, the table below it is asking about locations! The word “asset” is nowhere to be found there. It seems whoever wrote that really thought he or she was talking about assets anyway.

Again, why is “location” being asked for at all in the DR? I asked this of the person on the SCWG that drafted the version of the DR that’s the basis for what NERC sent out today. He said it had to do with the fact that SPS/RAS (Special Protection Systems/Remedial Action Schemes) can be at multiple locations, and it’s important to protect each of those locations. Presumably, NERC wants an entity to count each of the locations that contains part of an SPS or RAS as a separate location for the purpose of this DR. But nowhere in the DR does it say this! So why even ask for locations in the first place?

This is a classic case of the tail wagging the dog. For every Low SPS/RAS, how many Low impact Control Centers, substations and generating plants are there? My guess it’s in the high hundreds, if not over a thousand. So NERC is confusing the whole first question because of a need to get SPS/RAS right, when they are a very small percentage of the total Low assets. This could be handled much better by replacing all the references to “location” with “asset”, and simply putting a footnote on the last line of the table on page 6, explaining that the entity should count each location of a part of an SPS or RAS as if it were its own asset, or something like that.

And while we’re on the subject of misrepresenting CIP-002-5.1a R1, let’s go to footnote 6 on page 6. There we read “If your entity has performed generation segmentation and created multiple low impact BES Cyber Systems, please account for them as individual low impact BESCS as per your CIP‐002.”

I don’t want to go into detail on the reason this is a misrepresentation of what Attachment 1 Criterion 2.1 says (misunderstanding would be a better word); if you would like to research this fascinating subject, I refer you to at least four or five posts I wrote about this criterion alone in 2014 and 2015, when people were trying – mostly unsuccessfully - to figure out CIP version 5 (but don’t ask me which posts they were. Who wants to read all that stuff?).

I’ll just point out that the footnote sounds as if the entity is being asked to enter numbers of Low BCS (or BESCS) in this table, when of course the idea is that it’s just Low assets – or locations or whatever. This may be explainable by the fact that I think this table was originally drawn up when the DR was actually asking about Low BCS; it was probably never changed. I did point out to the person that drafted the final SCWG version of the DR that the wording of this footnote wasn’t really what he wanted to ask – and I suggested wording that I thought would work. But this change wasn’t made, as well as all of the other changes I suggested at that time (basically, everything that’s in these two posts was rejected by the sub-sub-committee. But don’t worry, I can take rejection. In fact, the fact that nobody listens to what I say saved my life earlier this year, as well as won me a prestigious Russian medal – which I’m still waiting for, but the Russians assure me it must have been lost in the mail and will reach me eventually).

And two footnotes above footnote 6 (that’s footnote 4, for those who struggled with math in second grade) we find this sentence: ‘“external routable connectivity” refers to the requirements under CIP‐003‐7, Attachment 1, Section 3’. When I and a couple others on the SCWG pointed out, early in the DR process, that the capitalized ERC was just for Mediums and Highs, the person drafting this for the SCWG obviously decided to find an official “definition” of erc (lower case). 

If the term Low impact External Routable Connectivity (LERC) were still in force, that definition would have been perfect here. However, that definition was part of the late, lamented CIP-003-6 R2 Attachment 1 Section 3, which never was implemented, having been replaced – at FERC’s insistence - by CIP-003-7 R2 Attachment 1 Section 3, which will come into effect January 1, 2020. What has replaced it is an operational “definition” which is effectively 16 pages long (occupying pages 32-48 of the Guidance and Technical Basis for CIP-003-7).

I think this is actually a really great way of “defining” what external routable connectivity means at a Low impact asset (since, of course, there is no ESP to make it easy to write the definition). In fact, the post where I raved about the CIP Modifications SDT meeting that developed this definition (written on the 4th of July, 2016. Must have been a rainy day in Chicago) became a sudden hit in the last couple of weeks, yet I have no idea why - since I haven’t referred to it in a post for a long time. My guess/hope is somebody realized it can be a guide to CIP-003-7 compliance, or more specifically to documentation of that compliance. But I hope to write a post in the not-too-distant future to actually spell out why I say this.

However, great as this “definition” is, I really doubt there are 50 people in the NERC community who really understand how that replaced LERC – and I doubt half of them could clearly articulate this to you. So for NERC to suggest here that, if you want to understand what external routable connectivity means, you just have to understand CIP-003-7, Attachment 1 Section 3 is somewhat like saying that if you want to understand why your car keeps rolling after you turn off the gas but leave it in neutral, you just need to develop a thorough understanding of Einstein’s theory of General Relativity, with maybe Quantum Mechanics and String Theory thrown in for good measure.

So while it may be technically correct to refer people to CIP-003-7 to understand erc, NERC should instead provide an unofficial definition that people can understand without undertaking a huge research project. I suggested a pretty simple explanation to the SCWG team at one point, but that didn’t gain traction either. Maybe it will now, although it doesn’t require me to write it. Anybody experienced in the black arts of NERC CIP could write a simple explanation of erc in one sentence (that doesn’t use the phrase Electronic Security Perimeter). Just try it.

I have just one more issue with the first “question”, and then I’ll stop writing ‘til the second post, where I’ll address my second and third objections (I heard loud cheers from across North America when I said I’d stop writing. Don’t you know that’s mean?). This has to do with the rows labeled f. and g. in the table on page 7. These ask for “Number of locations with constant monitoring of remote connectivity to a BES Cyber System” and “Number of locations participating in industry programs” respectively (a footnote clarifies that “Industry programs include CRISP, CYOTE and/or Neighborhood Keeper”).

What’s my objection to these questions? I take you back to page iv of the DR, where (in the second paragraph of the Background section) it states “…the Board directed NERC to study the nature and complexity of cyber security supply chain risks, including those associated with low impact assets..” (my emphasis). Returning to the table on page 7, I can certainly understand how rows b. through e. are asking about potential sources of risk. But f. and g. seem to me to be mitigations of risk. Why is NERC asking about those here? It doesn’t seem to comport with the purpose of issuing the DR in the first place.

I can think of two possible answers to this question. The first is that NERC thinks both constant monitoring and “industry programs” might be good things to require – or at least strongly advise – NERC entities to deploy at BES assets (and presumably not just at Lows). So they decided to piggyback this separate question onto this DR. If so, I can understand that, but they should state this up front, rather than slip the questions in and hope nobody will notice.

The other possible answer is that NERC isn’t looking at just risks to the BES from Low impact assets, but possible mitigations of those risks. So if they find that a large number of Low assets participate in these programs, they can tell Congress “I know we have a lot of Low assets and they all pose some risk, but on the other hand these two mitigations are widely deployed, so a lot of that risk is already mitigated.”

This is again a perfectly legitimate objective, but my question is, why stop with these two mitigations? There are a number of other mitigations which I’m pretty sure are much more widely deployed at Lows, including regular patching (say every two months), daily A/V signature updates, background checks, etc. Why not ask questions about all of these as well?

Have a good Fourth!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. To discuss this, you can email me at the same address.