NERC’s draft Data Request on supply chain risks for Lows, Part I

Today, NERC put out a draft Data Request. NERC entities will have 3 weeks to make comments on it. The purpose of this DR is to help NERC staff determine “whether new information supports modifying the standards to include low impact BES Cyber Systems with External Routable Connectivity” (and by “the standards”, NERC means the Supply Chain Standards, CIP-013-1, CIP-005-6 and CIP-010-3).

I will point out to start that, if NERC is really just interested in whether Low impact assets with External Routable Connectivity should be included in CIP-013, I can give them a quick answer: There aren’t any Low impact assets with ERC, so it doesn’t matter if they’re included or not. ERC (capitalized, since it’s a NERC Glossary term) only applies to High and Medium impact assets. The definition refers to the Electronic Security Perimeter, and only Highs and Mediums have ESPs.

But I digress. This DR wasn’t a surprise to me, since I and a small subcommittee of the NERC CIPC’s Supply Chain Working Group (which is itself a subcommittee of the CIPC, making the small group a sub-sub-committee, for those keeping score at home) have been drafting this with Howard Gugel, NERC Director of Standards Development, for more than a month (Howard came to the group with a first draft, but what came out in the end was completely different). I’ll say at the outset that there were some very interesting discussions over that time, and this version was approved over my strenuous objections (not that there was a vote, but it seemed pretty clear that I was the only person still willing to voice objections at the end of our discussions, which was two weeks ago. What was sent out today seems to be close to what was approved then, but is somewhat improved – at least in the first section).

To spoil the ending for you, my objections to this draft are:

1) There are two overall sections in the DR. The first starts on page 5 under the heading “BES Cyber Systems”. I think this section is confusing and misuses terms, but it could be saved with some important changes (it was worse two weeks ago, but it was somewhat cleaned up after it went from the SCWG to NERC). As it stands now, I doubt it will provide much useful data.

2) The second section begins on page 7 under the heading “CIP-013 Cost of Implementation”. This can’t be saved, and needs to be put to sleep with the fishes, as they say in “The Godfather” (or, if you prefer, it needs to be “terminated with extreme prejudice”, as the CIA agent says to Martin Sheen in Saigon at the beginning of “Apocalypse Now”. I just realized that both of these great movies are from Francis Ford Coppola! No wonder they’re the two that seem relevant here).

3) But my biggest objection isn’t to this draft, but the whole idea that there is a need for a DR in the first place. I think this reflects very bad strategy on NERC’s part. What’s driving this DR – judging from what I’ve heard from people at NERC and elsewhere, and drawing a couple conclusions of my own from what I’ve heard – is that people in Congress are itching to see much tougher cyber regulation of the power industry, and they’ve keyed in on the idea that this should start with more controls on Lows (and CIP-013 would probably be the logical place to start with Lows).

The data generated by the DR won’t satisfy Congress (as I’ll show later on), and the fact that NERC will spend 4-7 months drafting the DR, getting comments, gathering responses, analyzing them and then generating some sort of report might very well be held against the indusry. This means that the hawks in Congress are still likely to require that Lows be included in CIP-013 (and FERC will go along with them), regardless of the DR results, meaning the DR is probably a big waste of everybody’s time. Even more importantly, if this all comes to pass a big blow will have been delivered to the idea that the electric power industry should be both developing and enforcing its own cybersecurity standards in the first place. Need I point out that that isn't good for NERC or the industry?

I had thought I could easily write about all three objections in one post, but when I got to page 4 of the draft and I was still nowhere near finished with the first objection, I decided to leave the other two objections for the second post. I expect I’ll have that up on Sunday evening, so your otherwise dreary Monday-after-a-long-July 4^th weekend will be greatly enlivened by Part II of this post – or maybe not. In either case, I think I’ll have it up Sunday evening.

Now to my first objection, which consists of about 7 or so sub-objections on the first section of the DR. The first sub-objection is to the title of the section itself, which is “BES Cyber Systems”. Those of you who survived the CIP v5 and v6 rollouts probably have your antennae up when you hear that this is the title of the section, since a DR regarding Low impact assets shouldn’t be talking about Low BCS at all (reasons to follow).  

To give you some background on this section, the DR started off (in the first draft, that NERC provided to the SCWG) requesting rough numbers of Low impact BES Cyber Systems with and without ERC – and when a few of us pointed out that ERC doesn’t apply to Lows, this was changed to “external routable connectivity”.   

I and a few others on the sub-sub-committee pointed out next that, while some of the Regions have in fact requested lists of Low BCS from their member entities, since CIP v5 came into effect (where the Low/Medium/High impact levels were introduced) the standards have stated explicitly that an inventory of Low BCS isn’t required; the fact that NERC was just requesting rough numbers in the initial draft of the DR really didn’t change that at all – there would definitely be a lot of resistance to this question among NERC entities.

That objection didn’t seem to change things, but what finally did was when I pointed out in a later meeting that I knew one large NERC entity that stated in a Regional compliance meeting a few years ago that they had in the thousands of relays in their Medium impact substations, but they were grouped into only 2 or 3 (I believe) BES Cyber Systems (of course, this is perfectly legal). But I also know one even larger utility that has declared every Medium or High impact BES Cyber Asset to be its own BCS. So, if those two entities both have say 5,000 relays at Low assets, the first entity might report that it has say five Low BCS, while the other will say it has 5,000 Low BCS. If hypothetically they were the only two entities responding to the DR, NERC would dutifully say that the average large NERC entity has between 2502 and 2503 Low BCS! Obviously, such data are totally useless. That finally seemed to put the kibosh on requiring entities to estimate Low impact BCS numbers in the DR.

We then agreed that we should ask for numbers of Low assets with and without risk factors like external routable connectivity, dial-up access, etc. I suggested to keep it simple: Ask the entities to just estimate numbers of Low assets with erc, dial-up, and maybe a couple other risk factors – as well as those without any of these, of course (I also suggested breaking each of these down into Control Centers, substations and generating stations). As you can see, the first section of NERC’s draft DR goes beyond what I suggested, and breaks each type of asset down by three “Location Risk Scores” (so you have three types of Low Control Centers, three types of substations, etc. - and questions on the risk factors for each of these types).

I have no objection in principle to that change, but the problem is that it’s been worded very poorly. Were these problems to be fixed, I think this would be a usable question, but that’s only if the DR goes out in the first place (and for that issue, see the second post). However, I think the following problems first need to be fixed.

The first “question” starts off on page 5 by saying “In order for NERC to understand the data they receive from this Data Request, they need to understand the basis for each entity’s answer. This means that how your entity categorized your BES Cyber Systems can have a huge impact on these survey results.” OK, fair enough.

The rest of that paragraph is “In order to have Data Request results that can be used and compared, the common basis are the six locations called out in CIP‐002. This Data Request is focused on those locations and not how entities have designed their BES Cyber Systems.” OK, that also sounds reasonable. Let’s go to CIP-002-5.1a to determine what NERC means by “locations”. Hmmm, it seems that the word “locations” isn’t used in CIP-002-5.1a at all, except in a few of the bright-line criteria in Attachment 1 (e.g. Criterion 2.1), where it has a very specific use that has nothing to do with what we’re talking about here. So what does NERC mean by this term?

Two paragraphs below, NERC says “The term “location” refers to physical space associated with an asset.” And there is a footnote that says in part “For the purpose of this data request, the word “asset” is used in the same way as it is used in CIP‐002‐5.1a Requirement R1.” Since what they’re referring to is the six types of assets listed in R1.1, I’ll combine the three statements to yield what I think is the meaning of what NERC’s asking: “This Data Request is focused on the physical spaces associated with assets (i.e. the six asset types listed) in CIP-002-5.1a Requirement R1.” I assume “physical space” means the land on which the asset is built and the air for a certain distance above the land.

Isn’t this kind of odd? You’re being asked how many physical spaces you have that house Low impact assets; I guess that makes a little bit of sense. But look at the “Location Risk Score Table” on page 5. Let’s take the row that starts with “3.3” (but any row will do) and continues with “Generation resources”. It provides location risk scores based on megawatts at the location. Do the dirt and air in which a generating plant is built generate megawatts of power? No, the plant itself does, which is an “asset” in R1. Why not just ask about the asset in the first place?

And while we’re on page 5, move up to the table before this one. That asks for “Number of assets containing BES Cyber Systems”, broken down in four ways. That comports perfectly with R1, but it doesn’t with the rest of this section of the DR, which of course is after “locations” not assets. And a final confusion regarding “location” is found in the introduction to the table on page 7, which says “You may group assets with the same answers into a single line item.” However, of course, the table below it is asking about locations! The word “asset” is nowhere to be found there. It seems whoever wrote that really thought he or she was talking about assets anyway.

Again, why is “location” being asked for at all in the DR? I asked this of the person on the SCWG that drafted the version of the DR that’s the basis for what NERC sent out today. He said it had to do with the fact that SPS/RAS (Special Protection Systems/Remedial Action Schemes) can be at multiple locations, and it’s important to protect each of those locations. Presumably, NERC wants an entity to count each of the locations that contains part of an SPS or RAS as a separate location for the purpose of this DR. But nowhere in the DR does it say this! So why even ask for locations in the first place?

This is a classic case of the tail wagging the dog. For every Low SPS/RAS, how many Low impact Control Centers, substations and generating plants are there? My guess it’s in the high hundreds, if not over a thousand. So NERC is confusing the whole first question because of a need to get SPS/RAS right, when they are a very small percentage of the total Low assets. This could be handled much better by replacing all the references to “location” with “asset”, and simply putting a footnote on the last line of the table on page 6, explaining that the entity should count each location of a part of an SPS or RAS as if it were its own asset, or something like that.

And while we’re on the subject of misrepresenting CIP-002-5.1a R1, let’s go to footnote 6 on page 6. There we read “If your entity has performed generation segmentation and created multiple low impact BES Cyber Systems, please account for them as individual low impact BESCS as per your CIP‐002.”

I don’t want to go into detail on the reason this is a misrepresentation of what Attachment 1 Criterion 2.1 says (misunderstanding would be a better word); if you would like to research this fascinating subject, I refer you to at least four or five posts I wrote about this criterion alone in 2014 and 2015, when people were trying – mostly unsuccessfully - to figure out CIP version 5 (but don’t ask me which posts they were. Who wants to read all that stuff?).

I’ll just point out that the footnote sounds as if the entity is being asked to enter numbers of Low BCS (or BESCS) in this table, when of course the idea is that it’s just Low assets – or locations or whatever. This may be explainable by the fact that I think this table was originally drawn up when the DR was actually asking about Low BCS; it was probably never changed. I did point out to the person that drafted the final SCWG version of the DR that the wording of this footnote wasn’t really what he wanted to ask – and I suggested wording that I thought would work. But this change wasn’t made, as well as all of the other changes I suggested at that time (basically, everything that’s in these two posts was rejected by the sub-sub-committee. But don’t worry, I can take rejection. In fact, the fact that nobody listens to what I say saved my life earlier this year, as well as won me a prestigious Russian medal – which I’m still waiting for, but the Russians assure me it must have been lost in the mail and will reach me eventually).

And two footnotes above footnote 6 (that’s footnote 4, for those who struggled with math in second grade) we find this sentence: ‘“external routable connectivity” refers to the requirements under CIP‐003‐7, Attachment 1, Section 3’. When I and a couple others on the SCWG pointed out, early in the DR process, that the capitalized ERC was just for Mediums and Highs, the person drafting this for the SCWG obviously decided to find an official “definition” of erc (lower case). 

If the term Low impact External Routable Connectivity (LERC) were still in force, that definition would have been perfect here. However, that definition was part of the late, lamented CIP-003-6 R2 Attachment 1 Section 3, which never was implemented, having been replaced – at FERC’s insistence - by CIP-003-7 R2 Attachment 1 Section 3, which will come into effect January 1, 2020. What has replaced it is an operational “definition” which is effectively 16 pages long (occupying pages 32-48 of the Guidance and Technical Basis for CIP-003-7).

I think this is actually a really great way of “defining” what external routable connectivity means at a Low impact asset (since, of course, there is no ESP to make it easy to write the definition). In fact, the post where I raved about the CIP Modifications SDT meeting that developed this definition (written on the 4^th of July, 2016. Must have been a rainy day in Chicago) became a sudden hit in the last couple of weeks, yet I have no idea why - since I haven’t referred to it in a post for a long time. My guess/hope is somebody realized it can be a guide to CIP-003-7 compliance, or more specifically to documentation of that compliance. But I hope to write a post in the not-too-distant future to actually spell out why I say this.

However, great as this “definition” is, I really doubt there are 50 people in the NERC community who really understand how that replaced LERC – and I doubt half of them could clearly articulate this to you. So for NERC to suggest here that, if you want to understand what external routable connectivity means, you just have to understand CIP-003-7, Attachment 1 Section 3 is somewhat like saying that if you want to understand why your car keeps rolling after you turn off the gas but leave it in neutral, you just need to develop a thorough understanding of Einstein’s theory of General Relativity, with maybe Quantum Mechanics and String Theory thrown in for good measure.

So while it may be technically correct to refer people to CIP-003-7 to understand erc, NERC should instead provide an unofficial definition that people can understand without undertaking a huge research project. I suggested a pretty simple explanation to the SCWG team at one point, but that didn’t gain traction either. Maybe it will now, although it doesn’t require me to write it. Anybody experienced in the black arts of NERC CIP could write a simple explanation of erc in one sentence (that doesn’t use the phrase Electronic Security Perimeter). Just try it.

I have just one more issue with the first “question”, and then I’ll stop writing ‘til the second post, where I’ll address my second and third objections (I heard loud cheers from across North America when I said I’d stop writing. Don’t you know that’s mean?). This has to do with the rows labeled f. and g. in the table on page 7. These ask for “Number of locations with constant monitoring of remote connectivity to a BES Cyber System” and “Number of locations participating in industry programs” respectively (a footnote clarifies that “Industry programs include CRISP, CYOTE and/or Neighborhood Keeper”).

What’s my objection to these questions? I take you back to page iv of the DR, where (in the second paragraph of the Background section) it states “…the Board directed NERC to study the nature and complexity of cyber security supply chain risks, including those associated with low impact assets..” (my emphasis). Returning to the table on page 7, I can certainly understand how rows b. through e. are asking about potential sources of risk. But f. and g. seem to me to be mitigations of risk. Why is NERC asking about those here? It doesn’t seem to comport with the purpose of issuing the DR in the first place.

I can think of two possible answers to this question. The first is that NERC thinks both constant monitoring and “industry programs” might be good things to require – or at least strongly advise – NERC entities to deploy at BES assets (and presumably not just at Lows). So they decided to piggyback this separate question onto this DR. If so, I can understand that, but they should state this up front, rather than slip the questions in and hope nobody will notice.

The other possible answer is that NERC isn’t looking at just risks to the BES from Low impact assets, but possible mitigations of those risks. So if they find that a large number of Low assets participate in these programs, they can tell Congress “I know we have a lot of Low assets and they all pose some risk, but on the other hand these two mitigations are widely deployed, so a lot of that risk is already mitigated.”

This is again a perfectly legitimate objective, but my question is, why stop with these two mitigations? There are a number of other mitigations which I’m pretty sure are much more widely deployed at Lows, including regular patching (say every two months), daily A/V signature updates, background checks, etc. Why not ask questions about all of these as well?

Have a good Fourth!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. To discuss this, you can email me at the same address.