A note to my Russian friends (at least, I hope you’re friends!)

For the first time that I know of, this blog has had more hits in the previous seven days from a foreign country than from the US. Specifically, there have been 553 page views from Russia vs. 382 from the US, and smaller numbers from other countries (of course, this doesn’t include the close-to-700 subscribers to the email feed, who probably read my posts - or don’t read them - from the email without going to the site). But it is still remarkable that US residents are currently the minority of my non-subscriber readers, although I don’t think that will last very long.

I’ve had one big Russian spike in July and August, when I was writing[i] about the Russian supply-chain attacks on the power industry, and more specifically DHS’s wildly-exaggerated reporting of those attacks - although even then I don’t think Russians were ever responsible for more page views than Americans, over a period as long as a week. The reason why there might be a lot of interest in Russia in what I was writing was pretty obvious then, whereas I can’t think of anything I’ve written lately that even refers to Russia.

I’ve always had a decent contingent of non-North American readers[ii], but I always assumed that was because other countries are always considering the question whether they should impose mandatory cyber security regulations on their electric utilities. From that point of view, just about everything I write has some relevance for them, because so much of it has to do with problems with the NERC CIP standards – and some of it points to how I would rewrite CIP if given the chance.

In any case, to my new Russian readers, welcome! I hope you find information here that will help Russia design workable guidelines or regulations for your own power industry. But if you happen to be one of the small number of Russians actively engaged in trying to hack into the US power grid, you aren’t going to find anything in this blog that will help you in your job, so I suggest you find some more productive line of work. Your efforts so far have been a dismal failure, and I don’t want you to hope that this will change anytime soon, because it won’t. There can’t be a lot of job security in your current position, and a lot of downside – like being indicted and sanctioned by the US.

P.S. While I have your attention, I'd like you to relay a question to Mr. P, the next time you run into him at the grocery store: If - say - 10 or 15 years ago, he had made the decision to unleash the vast scientific and technical talents of the Russians on developing computer software and hardware that the rest of the world would want, rather than on trying to attack them (for little or no visible gain to the Russian people, regardless of a few feathers in Mr. P's cap), wouldn't your country be much better off? And wouldn't a lot of you be hugely better off personally, in Russia's own Silicon Valley? Just asking.

Note from Tom 4/1/19: I was right in stating in this post that the spike in views from Russia was probably temporary. The next week, presumably due to this post, the numbers went way down. However, at the same time there was a huge spikes in hits from "Unknown Region" - they went from the normal 10-20 range to around 250. I assumed this was because a lot of my supposed Russian friends wanted to keep reading, but used IP addresses not assigned to any region. This leads me to believe that at least a lot of the Russian spike consisted of people whose motives for reading me weren't pure - i.e. they may be looking for me to reveal weaknesses in the US grid. I  know of only one possible serious weakness that hasn't been widely discussed, and I turned that over to FERC last year to investigate. I will never write about that in this blog.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013; we also work with security product or service vendors that need help articulating their message to the power industry. To discuss this, you can email me at the same address.

---

[i] And I don’t want to leave the impression that I consider this story over, since I haven’t heard anybody from DHS explain to my satisfaction how the apocalyptic statements they made initially are consistent with the shifting explanations they have provided since then – or for that matter, with the much more measured statements they made when they first pointed out the issue, in far less detail, last March. I have another post on the topic in my to-do list, but that list keeps getting longer as more pressing issues like this one pop up.

[ii] And they have been distributed across a number of countries, although lately the Ukraine and Eastern Europe (Poland, Czechia and Hungary) have all had their spikes. I’ve been especially pleased with some recent hits from Vietnam, since my wife is Vietnamese and I’ve been there a number of times in recent years – a really great place to visit, by the way, with the friendliest people you’ll ever meet. I’ve always been surprised by how few Canadian hits I have, but I know there are a number of Canadians that subscribe to the email feed, since I often get email comments from them based on having read the feed.

An (ex-) auditor weighs in on “retropatches”

My post yesterday called attention to a post by Monta Elkins of FoxGuard Solutions on the problem of “retropatches” – patches that carry a date that is sometimes months before the actual release date of the patch. These can play havoc with a NERC entity’s audit of CIP-007 R2 compliance, since an auditor may well inquire why the entity didn’t apply a patch dated January until May. The post got a lot of attention, despite the slow holiday week.

One person who paid attention to the post was Kevin Perry, who recently retired after 21 years with SPP, including nine years as the Chief CIP Auditor of SPP RE. He sent me the following email, which NERC entities should find to be helpful:

If you use a patch service, then you can readily maintain records of when the patch actually showed up on the applicable, not installed list.  That solves the problem.  If you receive email or snail mail notifications of available patches, again no issue.  Just keep the notice.  If you manually check, you need to keep evidence that you checked anyhow, so make sure your date-stamped evidence shows no patch was available until it actually shows up.  Where you get into trouble is when your program documentation consists of an attestation that you checked and found nothing, but you have no supporting evidence.  You cannot prove you did not miss a backdated patch.

Most entities I dealt with used a patch service for most of their software and received email notifications for the rest.  Very few applications required a manual site visit.

The entities that eschew available technology and do everything manually are the ones with the greatest burden and risk.

By the way, there is another nuance.  You upgrade a system and all of a sudden a bunch of old patches are now applicable.  Again, good record keeping of the upgrade addresses that “discrepancy.”

Most auditors are reasonable as long as you can tell your story without the “deer in the headlights” look.  We really do understand these issues occur.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013; we also work with security product or service vendors that need help articulating their message to the power industry. To discuss this, you can email me at the same address.

A very interesting problem

Monta Elkins, Hacker-in-Chief[i] of FoxGuard Solutions, dropped me an email recently to point out a post he’d written for the company’s blog regarding a problem I’d never heard of: “retropatches”. This is a problem that wouldn’t arise at all were it not for a prescriptive patching requirement like CIP-007 R2 (and I doubt such a prescriptive patching requirement exists anywhere else in the known universe, although I can’t vouch for any parallel universes).

I’ll let Monte provide the details (the post is very easy to read, and provides very compelling evidence), but in brief, Foxguard (one of whose businesses is researching and providing available patches for ICS devices, meaning they can be your close-to-one-stop-shop patch source) has discovered that vendors sometimes release patches weeks or months after the official date of the patch[ii].

Of course, CIP-007 R2 requires that you, during an audit, be prepared to provide evidence that you checked for new security patches every 35 days, and obtained the ones that were applicable to software you have installed on devices in your ESP. This means it’s very possible that an auditor will notice there were some months where you documented that no new patch was available, yet when the auditor checks the vendor’s website, they see there is a patch that was supposedly available on the date you checked for it. The auditor might well ask why you didn’t download this patch when you checked for patches in the month after it was released. And you will reply, in best audit response mode, “Well, I…uh…hmm, I must have missed that.” Then you get ready to give your boss some very unwelcome news.

I recommend you read the post to make sure this doesn’t happen to you!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – and especially on compliance with CIP-013; we also work with security product or service vendors that need help articulating their message to the power industry. To discuss this, you can email me at the same address.

---

[i] This title is very appropriate. If you’ve never seen Monte demonstrate how to hack into an electric drill and get it to play the Darth Vader theme…well, I’ll just say it should be on your bucket list.

[ii] Monte points out that this doesn’t mean vendors are deliberately backdating patches, just that they’re probably dating them by when they start building the patch or something like that. He also points out that most vendors would never dream this would cause a problem for customers – since they never dreamed (or “nightmared”) that there would be a requirement which threatens a million-dollar-a-day fine for not downloading a single patch. This is certainly understandable. I wouldn’t believe it either, if I didn’t know it’s true.

Be careful how you use the Facilities

Last week I participated in a phone meeting of a subcommittee of the NERC CIPC, where we were marking up a document that will be submitted soon to NERC (not a draft standard, of course). At one point in the document, I noticed that one of the participants had deleted the word “assets” and replaced that with “Facilities” – which was capitalized, even though it wasn’t at the beginning of a sentence.

I asked why she had done this, and she pointed out that “asset” isn’t a NERC-defined term, even though it plays a very important role in the CIP standards, while “Facilities” is NERC-defined. She had substituted Facilities because she preferred to have something with a precise definition. Nothing wrong with that, of course. I’m all for precision, which is unfortunately lacking in many places in the CIP standards.

However, I pointed out to her that if she uses Facilities, she would radically change the meaning of what was written. This is because asset, while officially undefined, is in fact “defined” (as far as CIP is concerned) in CIP-002 R1.1, where there is a list of six “assets” that are in scope for CIP: transmission substations, Control Centers, etc. However, the NERC definition of Facility reads “A set of electrical equipment that operates as a single Bulk Electric System Element (e.g., a line, a generator, a shunt compensator, transformer, etc.).” So things like lines, generators, etc. are Facilities. This is very far from transmission substations, generating stations and Control Centers.

She readily agreed with me, and reversed her change. But I want to point out that a lot of the applicability of the CIP standards, from version 5 on, is based on a misuse of the word Facilities. This is because all of those standards – from v5 through CIP-003-7, which comes into effect 1/1/20 - contain Section 4, titled Applicability, near the beginning. This has two sub-sections: 4.1 Functional Entities and 4.2 Facilities. Clearly, 4.1 tells you what NERC entities have to comply with CIP, and 4.2 tells you which of their facilities – in the generic sense of “structures” or “locations of equipment” – are in scope. Clearly, substations, generating stations and Control Centers are all little-f facilities. Up to this point, one can assume that the word “Facilities” in 4.2 is capitalized only because it’s the first (and last) word of that title.  

However, in Section 4.2.1, discussing Distribution Providers, Facilities is capitalized, meaning that it refers to things that meet the NERC definition. However, what’s listed below it aren’t substations or Control Centers, but UVLS, UFLS, SPS, RAS, cranking paths, etc. – all items that probably do meet the NERC definition. No problem so far.

But now we get to Section 4.2.2, which is titled “Responsible Entities listed in 4.1 other than Distribution Providers”. These are of course close to 99% of the entities that have to comply with CIP in general (since only DP’s that own Facilities listed under 4.2.1 have to comply, and I believe that number is pretty small), and literally 100% of entities with High or Medium impact assets. And what are the facilities that need to be in compliance? “All BES Facilities”. No mistaking what this says: For every non-DP NERC entity that has to comply with CIP, their scope is what meets the Facilities definition: lines, generators, shunt compensators, etc. Nothing about…you know, stuff like Control Centers, substations and generating stations. So how do you reconcile 4.2.2 with the fact that CIP-002 R1.1 says that these latter three assets are what’s in scope – and in fact I’m sure that over 99% of the assets that are now under CIP compliance fall into one of these types?

For substations and generating stations, you can kind of bridge the gap by pointing out that these two asset types both contain lots of items – transformers, lines, buses, etc. – that actually meet the definition of Facility (or of Element, which is closely related). So it’s not too much of a stretch to say that 4.2.2 makes sense as far as those two asset types are concerned. In fact, Criteria 2.3 through 2.8 of Attachment 1 of CIP-002 all apply to Facilities, although they’re almost universally interpreted – by entities and auditors – as applying to generating stations (2.3) or substations (2.4-2.8).

But how about Control Centers? They clearly don’t meet the definition of Facility. Does a Control Center contain any Facilities – lines, generators, buses, etc? Of course not. Facilities are all high-voltage devices. Servers and workstations are all low-voltage, although they are obviously used to operate or monitor true Facilities at other locations. Thus, it seems that 4.2.2 makes it clear that Control Centers are out of scope for the CIP standards!

So I’m sure the many NERC entities that own Control Centers will be pleased to learn that they really don’t have to bother with CIP compliance anymore.  If those pesky auditors come around wanting to look at a bunch of compliance evidence, just tell them you haven’t collected any since you read this post, and stick Section 4.2.2 in front of their noses. Tell them that if “Facilities” weren’t capitalized in 4.2.2, you would be more than happy to entertain them and their nosy questions, but since it is capitalized, it’s clear that you were never meant to comply with CIP…and by the way don’t let the door hit you on the way out.

Of course, I’m not suggesting that you try this at home, kids. Because it’s quite clear from other wording in the CIP standards that Control Centers are very much in scope. Clearly, Facilities is capitalized in Section 4.2 because the drafting team for CIP v5 (these “preamble” sections in each CIP standard first made their appearance in v5) thought they were being nice and precise by using a defined term rather than “assets” in Section 4.2, when in fact they had just undercut the strict legal foundation of all of the CIP standards. Of course, CIP hasn’t fallen into disuse because of this, but it’s also true that in the 7 or 8 years since that language was first drafted, it has never been corrected (despite a few mentions in my blog early on, including in the section “Questions of Scope” from this post of May 2014).

Why am I bringing this up, since clearly no Control Center is going to stop complying with CIP? I’m trying to make the point – now and in other posts – that the supposed legal foundations of the NERC CIP standards are in fact only held in place because of a lot of “Don’t ask, don’t tell” understandings between NERC entities and auditors. And there are many, many other examples of this situation, including examples discussed in this post and this one, as well as just about every post I wrote in 2014 and 2015 (after which I gave up this quixotic quest in exhaustion and went on to write about other things). 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013; we also work with security product or service vendors that need help articulating their message to the power industry. To discuss this, you can email me at the same address.

Going for the whole thing

Last week’s NERC CIPC meeting in Atlanta was the first I attended this year (the last was a year ago in December, at the same hotel), and it was certainly one of the best I’ve attended. The highlight of the meeting, for me and many others, was the presentation from Dave Revill, vice-chair of the CIPC and (I believe) co-chair of the CIP Modifications Standards Drafting Team.

Dave’s topic was everything that the team is working on now, but his big focus – as I expected – was the modifications the team is proposing in order to allow the standards to finally take account of virtualization. The team recently posted these modifications, which include new definitions, retired definitions, and some significant changes to various requirements. The modifications were posted for comment (not ballot), but the team intends to post modifications for ballot early next year. The comment period expires tomorrow (Dec. 18), so I realize that what I’ll say in this post is too late for anybody to use in their comments – but that’s OK, since as you’ll see my main concern isn’t tweaking what the SDT posted. I think there needs to be a very different approach.

If you want to get the full picture of what the team is proposing, you should go to the team’s web site and go to the set of documents posted under “CIP Virtualization Updates”. I will very briefly summarize what they are proposing to do (this isn’t hugely different from what they first outlined in a white paper and webinar this June. I wrote three posts on what they were proposing, starting here):

1. The team wants to move CIP from being based on individual Cyber Assets to being based on systems (specifically, BES Cyber Systems). Of course, a systems-based approach is needed if you’re going to include virtualization, since a Cyber Asset is a “device”. There is no NERC-approved definition of this term, but my unofficial definition is “Something is a device if it will hurt you if dropped on your foot”. Even if you can figure out how to drop a virtual system on your foot, it won’t hurt it. So it isn’t a device. Q.E.D.
2. They are eliminating the definitions of both Cyber Asset and BES Cyber Asset, both of which were very problematic. Instead, they’ve transferred everything important from the BCA definition to the BES Cyber System definition (most notably, the provision about 15-minute BES impact). Entities will now start their BCS identification process by looking at all the functions performed at an asset (especially the BES Reliability Operating Services, although those still have no official status in the requirements) and identifying those that can have a 15-minute impact on the BES if misused, etc. Those systems are BCS, whether they’re entirely physical, entirely virtual or a mixture of the two.
3. Of course, since the BCS definition is now changed, the question becomes what changes will need to be made in the requirements in CIP-003 through CIP-011. The good news is that a lot of the current requirements (including all requirements in the current CIP-003, -004, -008, and -009) require exactly zero changes, since they will work fine with any system, physical or virtual. The somewhat-good news is that many of the other requirements don’t require any substantial changes.
4. But the bad news is that a few of the most prescriptive requirements, like CIP-005 R1, CIP-007 R2 and CIP-010 R1, require very substantial changes due to the new systems-based approach. Trying to make all these changes to those requirements would mire the SDT in an endless fight and probably ten years of NERC ballots - as I estimated a couple years ago, when the SDT was discussing a different approach which would also require a huge number of specific changes to individual requirements. Instead, the SDT is doing the smart thing and rewriting those requirements – or incorporating them into others – in a series of steps that I don’t want to go into here, since that isn’t my purpose in this post.

I agree with the SDT’s whole approach, which I think is absolutely the right one if they are to fulfill their mandate to incorporate virtualization into CIP – although there would need to be a lot of tweaks to what they wrote, which just about everybody that commented at the CIPC meeting agreed on (just about everyone started out by saying “I applaud the SDT since it seems they’re on the right course. But I see some big problems that need to be addressed…). But there’s one fundamental problem with what the SDT is proposing (and I’ll admit I didn’t realize this until after the CIPC session): I don’t think there’s any way the NERC ballot body will approve the virtualization changes, no matter how many tweaks the SDT makes to wording.

I will write a post soon on why I make this assertion, but I’ll say for the moment that there are two basic reasons:

1. There is no pressing need to officially incorporate virtualization technologies into CIP, because lots of entities have already done that. Many (even most?) are already using virtual machines in Control Centers, and the NERC Regions are all supporting them in that effort (in fact, NERC itself has acknowledged that virtualization is allowed, as long as no VM hosts mix ESP and non-ESP virtual machines). I know one entity that virtualized their Control Center in the CIP v1 or V2 days, and passed an audit with flying colors in 2010. Of course, I’m sure everyone will agree that it would be better to make it official that virtualization is allowed. And I’ll also grant that use of switch and storage virtualization is probably being held back some by the lack of official status for those two technologies. But I am also fairly sure that these considerations alone won’t be enough for NERC entities to accept the big changes the SDT is proposing, which brings me to the second reason.
2. Why won’t NERC entities accept the big changes the SDT is proposing? Is it just because utilities are very conservative and don’t like to change something that is working, however flawed it may be? I don’t think so. In fact, in a minute I’m going to discuss an even-more-radical change that I think the industry would accept in a heartbeat. The reason why I don’t see the SDT’s proposed changes succeeding, no matter how much they’re tweaked, is because the smell of the bad experience with the CIP version 5 rollout still hangs in the air in CIP circles.

As I will discuss in a post soon, the CIP v5 rollout was quite flawed (despite the words of a NERC official who spoke at the CIPC meeting and seemed to think it was a huge success), and has left a huge set of unanswered questions about what the requirements and definitions mean – that are only papered over by a tacit mutual agreement between the Regions and the entities that I call Don’t Ask, Don’t Tell. There is every expectation that, given the current NERC auditing program (and again I’ll have to wait until a new post to explain what I mean by this), exactly the same thing will happen with the new changes as happened with CIP v5 (one questioner raised exactly this point, suggesting that the many lingering uncertainties in the “v5” requirements and definitions should be fixed before we introduce a bunch more due to the virtualization changes). Given this situation and the fact that there isn’t a pressing need to change the CIP standards to allow virtualization, I believe NERC entities will choose the status quo over what the SDT is proposing.

However, rather than being a big defeat for CIP reform, I think this is actually a big opportunity. That’s because there is one very pressing concern that almost all – if not all – NERC entities share, and that could be addressed by changes to the CIP standards and definitions along the lines of what the SDT is proposing, except going further.

Of course, what I’m talking about here is use of the cloud in CIP environments. Currently, there are two big issues: a) BCS Information (BCSI) in the cloud, and b) actual BCS in the cloud. BCSI in the cloud is currently allowed by the standards, simply because there is no prohibition against having a third party hold BCSI, whether they’re a vendor or a cloud provider (in fact, I know that a number of entities with High and/or Medium impact BCS are using cloud-based configuration management software and have passed audits. There are probably also some entities using cloud-based security monitoring services. Both services store BCSI in the cloud). But the problem is that the third party will have to comply with a set of four or five requirement parts from CIP-004, which I discussed in this post.

Let’s consider just one of these requirement parts, CIP-004-6 R5.3; this requires the entity (whether the Registered Entity or a third party holding their BCSI) to remove physical access to storage locations for BCSI within a day of an employee being terminated. This shouldn’t be a problem for a regular vendor. All of your BCSI is probably stored on servers in their single data center. It’s no problem to just remove that terminated employee’s access to that data center in a day or even a few hours – in fact, they probably do that now.

But what about a cloud vendor? By the very nature of their business, different parts of your BCSI could be flitting among servers by the minute, and jumping from one data center to another across the country any moment. And just one of their data centers might have in the thousands of employees with physical access to systems in that center. How could they prove, for every individual employee that is terminated across all of their data centers, that the person’s physical access to all of their other data centers was removed within 24 hours?

I’m not saying it would be completely impossible for a cloud vendor to prove this, or to mitigate the risk by say keeping all of your BCSI on servers in one particular data center. But doing so would break their whole business model, and they simply aren’t going to do that for one customer, or even for all utilities acting together (and the utilities can’t band together to force these concessions for antitrust reasons anyway – even if they had the collective clout for it).

Of course, there’s little question that any big cloud provider probably has security practices that go far beyond what any utility could think of implementing on their own – and they have certifications like FedRAMP to prove this. But where in the Measures section of any of the CIP requirements will you find a phrase like “…or submit evidence of a current FedRAMP certification”? Obviously, the answer to that question is “Nowhere”.  I will point out that the CEWG, a subcommittee of the CIPC whose acronym – I believe - means “Compliance and Enforcement Working Group”, is working on the issue of BCSI in the cloud – in fact they held a meeting after the CIPC itself concluded, which I attended. I’ll hope to report on that in a post in the somewhat near future, although I’ll summarize by saying this group is doing great work, but don’t look for any official change to the Measures for quite some time.

So the problem with BCSI in the cloud isn’t that it’s not allowed by the CIP standards, but that there is currently no workable way for a cloud vendor to prove compliance with the requirement parts that apply to them. On the other hand, like server virtualization, there are definitely a number of entities with High and/or Medium impact BCS that are in fact storing BCSI in the cloud, with the tacit approval (remember: Don’t Ask, Don’t Tell!) of their Regions. In other words, if the SDT were to decide they should extend the modifications they’re now proposing for virtualization so that BCSI could be realistically stored in the cloud (which could be done, and would perhaps be easier than trying to get the Measures changed, as the CEWG is currently thinking), NERC entities would be pleased to hear this. However, as with the virtualization changes by themselves, I doubt including both virtualization and BCSI-in-the-cloud would be enough to overcome the industry’s reluctance to go through a wrenching change in the CIP standards like CIP v5 was.

But what about storing actual BES Cyber Systems in the cloud? There, the situation is very different, for two reasons. For one, while I can’t guarantee that no entity with Medium or High impact BCS is storing any of them in the cloud (and of course, outsourced SCADA would be the most tempting way to do so, for many smaller entities), I am pretty sure they’re not doing it with the tacit or explicit approval of their Region – meaning they could have a big problem come audit time, if they haven’t let their Region know they’re doing this.

The other reason is that having actual BCS in the cloud would require huge changes to a number of the CIP requirements, even after all of the changes that the SDT is proposing for virtualization (and BTW, all of the virtualization changes are also necessary for BCS-in-the-cloud to be allowed by CIP in the future. The problem is those changes alone aren’t anywhere near enough). Even if an entity were to get their Region’s agreement that they could store one or more BCS in the cloud, they would have to sit down with the Region and essentially rewrite most of the requirements – either the requirements themselves or at least the Measures.

This leads me to the point I’ve been hinting at: Were the SDT to decide to extend the changes they’re proposing so that BCS could be stored in the cloud (as well as to incorporate virtualization and BCSI in the cloud), I think this would be something that could actually pass the NERC ballot body (and by the way, at last December’s CIPC meeting, both NERC and even a senior FERC staff member – as always, speaking for himself, not the Commissioners – indicated they don’t have fundamental objections to BCS in the cloud).

Yes, this would require much more thorough changes to the CIP standards and definitions than even what the SDT is now proposing for virtualization. But I can already see a lot of utility financial types, hearing that BCS-in-the-cloud might actually be happening, jumping for joy and telling their NERC compliance people to do everything they can to help NERC bring this to fruition.

So I really think that we shouldn’t mess around with changing CIP just in order to officially incorporate virtualization. As I said, it’s not really solving any big problems, but will nevertheless consume at least a couple years of SDT time. Why put off even talking about addressing BCS-in-the-cloud until the hoped-for day 2-3 years (at least) from now when virtualization is finally written into the CIP standards? Why not do both of them at once, even though this will be a much bigger effort?

I’ll tell you right now the main reason the SDT won’t consider this now – because their Standards Authorization Request (SAR) only lists virtualization as something they need to address. It will be a big deal to go back to NERC for a new SAR, then have that drafted and balloted, since SARs also need to be approved by the NERC ballot body. In fact, I think just drafting the SAR as I’m proposing might require a lot of debate and meetings. Why go through all of that, when the SDT can just get virtualization passed (and a couple other things), then ride off into the sunset with the endless thanks of the NERC community?

The answer to that is: It’s not at all guaranteed that the SDT will ever be able to get the virtualization changes passed, even if they go back now and make a bunch of changes before they even go to the first ballot. It’s very possible – and I think likely – that they will spend 1-2 years on this effort and finally decide it’s time to “declare victory and go home” (as was suggested by Sen. George Aiken as a solution to the Vietnam War quagmire).

I can see that some of the current SDT members wouldn’t feel like gearing up now for a massive change in CIP (much more far-reaching than CIP v5 was), so it’s very likely that NERC would need to empanel a new SDT to deal with this new CIP version. And it’s very likely that, even if the work to draft the new SAR started today, whatever changes finally resulted wouldn’t be in place for at least five years and even longer than that.

But guess what? I think that there are a couple even further changes that could literally make this the last CIP version that would need to be drafted. I’ve actually outlined these – at least mentally – already, since I’ve been working on a book (now suspended because I’ve been too busy, but I still hope to get back to it in the future) on the problems of CIP and how they can be addressed. You might think it’s impossible for me to say (at least without fingers crossed) that there could be a set of CIP changes that would ensure no further changes would ever be needed. But this is what I believe, although I’ll admit this would require changes in how the CIP standards are enforced, not just in the standards themselves. That is beyond the capacity of any SDT acting on their own, but isn’t beyond the capacity of NERC, working in consensus with FERC – i.e. it wouldn’t require going back to Congress, which could literally make it impossible.

However, even if we don’t go even further and implement the changes that would make this the last time CIP has to be redrafted, I definitely think it’s worth going back and drawing up a new SAR so that both virtualization and the cloud can be officially included in CIP. In fact, since I don’t see the SDT’s virtualization changes passing without this, I’d say it’s the only way to proceed. Sometimes you have to aim high if you want to accomplish anything at all.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013; we also work with security product or service vendors that need help articulating their message to the power industry. To discuss this, you can email me at the same address.

RSA 2019

I recently received the good news that I’ll be participating in the RSA Conference again next year. As I was this year, I’ll be one of three panelists on a panel – our topic will be “Supply Chain Security for Critical Energy Infrastructure”. This year the conference is from March 4-8; as always, it’s at the Moscone Center in San Francisco. Our panel is on Wednesday March 6 from 8:00-8:45 in Moscone South 204 (it doesn’t appear on the conference website yet, but will soon).

This year’s session was well received, with a lot of good audience interaction – and that’s good, because the three panelists are all returning next year, although the moderator is different. The topic this year was “How can we regulate critical energy infrastructure”. However, based on audience questions, the session turned into a very good discussion about grid security in general - and there’s nothing wrong with that!

This year, the panelists, besides me, will be Marc Sachs, former NERC CSO and head of the E-ISAC, and Dr. Art Conklin of the University of Houston, noted author and speaker on ICS security for the energy industry. The moderator will be Sharla Artz, VP of Government Affairs, Policy and Cybersecurity for the Utilities Technology Council. Here is our description of the session:

The purpose of this panel is to have an interactive dialogue between panelists and audience members on some important questions regarding supply chain cyber security for critical energy infrastructure (CEI). We will pose a series of questions, and as each question is asked, both panelists and audience members will be able to respond. While it is unlikely that a definitive answer will be reached on any of these questions, it is important to hear as many different answers as possible!

The panelists will bring a diverse set of perspectives to this discussion, based on their backgrounds in electric power, natural gas, water, petroleum refining and transport, and chemicals. It is hoped that audience members will bring many other perspectives to the discussion, especially if they are from other industries – finance, insurance, retailing, etc. – in which supply chain security is as important as it is in critical energy infrastructure.

The session will open with examples from the panelists of supply chain risks to energy systems. After that, possible questions to discuss include:

• What are currently the primary vectors for supply chain cyber attacks?
• How can we put in place a program to manage supply chain cyber risk?
• How can CEI organizations gain assurance that vendors have good cyber security practices in place? Do most other organizations require assessment or certification by an outside party, or are there alternative means to gain this assurance?
• What usable controls frameworks are available to help my organization understand supply chain cyber security risks?
• What is the role of contract language? Is it a) always, b) sometimes or c) never advisable to insist that the vendor agree to certain contract terms?
• We will have to comply with NERC CIP-013, which requires that we develop a supply chain cyber security risk management plan. How does the plan we need to develop for CIP-013 compliance differ from the plan that we would develop if we were addressing supply chain cyber risk in the absence of regulation?

I hope to see you there!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013; we also work with security product or service vendors that need help articulating their message to the power industry. To discuss this, you can email me at the same address.

More on implicit requirements: Lew’s article

In my last post, I went back to a topic I’d only touched on briefly in a post a few years ago, but which is certainly an important concept to understand if you are working on NERC CIP compliance nowadays: implicit requirements. I mentioned that I first heard the term from Lew Folkerth of RF in a presentation in October 2015.

But I neglected to also mention that Lew wrote an article for the RF newsletter on this subject in December 2015[i]. I just went back and reread it. To my great surprise – and I just now took a quick look out the window to make sure there weren’t any dark clouds that might strike me down with a bolt of lightning – I actually find myself disagreeing with Lew.

This article is done in a Q and A style, although I suspect the Q’s and the A’s were both written by Mr. F (nothing wrong with that, of course). In his first A, Lew says that implicit requirements are due to the fact that CIP v5 was written as “results-based Standards”. He goes on to explain “In a results-based Standard, the desired end result is specified, with the method of achieving the result left unspecified. This provides great flexibility in how the result is achieved, but one effect is that some actions that are actually required are not explicitly stated in the Standard.” In other words, he’s saying that implicit requirements are actually a virtuous byproduct of the drafting team’s high-minded purpose of making all of CIP objectives-based.

First off, the CIP v5 standards aren’t results-based or anything-else-based. Some of the requirements are results-based (CIP-007 R3 and CIP-011 R1, to name two). But these aren’t the ones with a whole set of implicit requirements built into them. Implicit requirements are only a problem when the underlying requirement is prescriptive. In my last post, I provided several examples of prescriptive CIP requirements that contained lots of implicit requirements. The basic pattern that leads to a lot of implicit requirements is that the base requirement uses a number of words that each then lead to one or more implicit requirements.

To show what I mean, let’s turn to CIP-007 R2, my poster child for a prescriptive requirement. R2.2 says “At least once every 35 calendar days, evaluate security patches for applicability that have been released since the last evaluation from the source or sources identified in Part 2.1.” Let’s say you’ve never seen this requirement before, and you’re trying to understand what you will need to do to comply with it. You’ll ask a set of questions, and the answers in most cases will lead to an implicit requirement:

Q1: How do I know which security patches have been released since the last evaluation?

A1: You approach the patch sources you identified in Part 2.1. This leads to the first implicit requirement:

IR1: Identify patch sources to approach regarding new security patches.

Of course, each implicit requirement usually leads to a new question.

Q2: I have a lot of patch sources. Do I need to approach them all?

A2: No, you just have to approach the patch sources for the software and firmware packages that you currently have installed on systems within your ESP.

Q3: How do I know what those software and firmware packages are?

A3: You take an inventory of all software and firmware currently installed on systems in your ESP.

This answer leads to another prescriptive requirement:

IR2: Every 35 days, take an inventory of all software and firmware currently installed on systems in your ESP.

Q4: Do I need to inventory anything else?

A4: Yes, you need to list the version numbers. Of course, this leads to:

IR3: Include version numbers in your inventory.

Q5: Anything else?

A5: Yes, now that you mention it. If the software includes various third-party applets, you need to inventory those as well. This leads to:

IR4: Include all applets for any software package in your inventory.

Q6: OK, now that I know what I need, what’s my next step?

A6: You’re ready to go. Now you need to find out from each patch source if they have a new security patch this month. This leads to:

IR5: Ascertain from each patch source identified in Part 2.1 whether a new security patch has been released.

Q7: What do I do now?

A7: You evaluate each patch for applicability.

Q8: And what’s applicability?

A8: A patch is applicable if it can be applied.

Q9: Well, that’s really helpful! So what patches can be applied?

A9: There are a lot of factors that determine applicability. For one, a patch is applicable if it is for a software or firmware package that is installed on at least one system in your ESP, taking into account the version number and any possible applets. This leads to:

IR6: Discard as inapplicable any patch that doesn’t apply to a software or firmware package that is installed on at least one system in your ESP, taking into account the version number and possible applets.

Q10: Does anything else determine applicability?

A10: Well, yes. What if you’ve already installed the patch? And what if you’ve installed a different patch that also closed the vulnerability that the current patch addresses? In both cases you shouldn’t install the new patch……

Q11: I’m getting a headache…

A11: That isn’t a question. If you’re looking for headache remedies, I recommend name of product removed.

OK, that’s enough. We’ve listed six implicit requirements so far, and A10 will lead to at least two more. Most important, we’re just getting started on “applicability”, since there are all sorts of other considerations that would lead one to declare a patch to be applicable or not. So it turns out that just this one part of CIP-007 R2 (and the requirement has four parts) has well over eight implicit requirements. It’s not out of the question that each of the 40-odd CIP requirements, leaving out CIP-002 R1 for the moment, has between 5 and 50 implicit requirements, leading to somewhere between 200 and 2000 implicit requirements in CIP.

And I deliberately left out CIP-002 R1, since – as I mentioned in my last post – just the “bright-line” criteria in Attachment 1 could easily lead to almost an infinite number of implicit requirements. But I won’t try to justify this statement, since it’s like delivering an ocean instead of a lake, when all you had asked for in the first place was a cup of water.

So what does it mean – this discovery (and it’s almost as much a discovery for me as for you) that there are a huge number of implicit requirements in the CIP standards? For one thing, it means that the goal of coming up with a definitive list of implicit requirements (as I mentioned that Lew had initially talked about doing, in the last post) is certainly well-intentioned, but nothing that can be accomplished in a human lifetime. Face it: There is always going to be a lot of uncertainty around how you comply with the prescriptive CIP requirements.

Uncertainty isn’t good when you’re talking about mandatory requirements with big potential fines. What’s the solution? In the near term, the solution is the same one as for all of the other problems with CIP: It will be handled one-on-one between the auditors and the Responsible Entities, as all the other problems have been handled. I wrote over 50 posts on the problems with CIP-002 R1 after CIP v5 was approved. None of these problems was ever resolved, yet the CIP compliance process today works fairly smoothly. This is because all parties – NERC, the Regions (especially the auditors) and the entities – have a big stake in having things run smoothly. They have always figured out how to do it in the past, and they’ll continue to do that. I call this the “Don’t Ask, Don’t Tell” NERC CIP compliance approach. But the original DADT worked fairly well also, until the climate had shifted enough that it wasn’t needed anymore. Maybe that will happen with CIP as well.

What about the longer term? Fortunately, there the outlook is brighter. This is because it seems there’s widespread recognition that the prescriptive requirements in CIP are a dead end. Specifically, while the current CIP Modifications drafting team doesn’t have fixing the prescriptive requirements as an item in their SAR, they do have virtualization. And they’ve realized that a comprehensive solution to the problem of integrating virtualization into CIP requires fixing the most prescriptive requirements.

I wrote about the SDT’s initial work on this problem in this post (and two more in the series) in June. They’ve recently come up with what looks like a great set of follow-on documents, including suggested revisions to the standards and new definitions. I have to spend some quality time with these, which I hope to do soon, but my initial impression is that they seem to have some really good ideas. But I do think the SDT is greatly underestimating how difficult it will be to get all of this passed by the NERC ballot body (and they may find it hard to get approved by FERC as well). However, once these changes are in place, they will go a long way to fixing the problem of overly prescriptive CIP requirements.

But it still isn’t all puppies and roses on this front. While prescriptive requirements are a big problem, there’s an even bigger problem: prescriptive auditing. This is a much harder problem to solve, and will be hanging over the effort to get the new virtualization requirements approved. While the problems with prescriptive requirements are also hard to solve, the means to solve them are clear: revising the CIP requirements and definitions. The same can’t be said for prescriptive auditing. In fact, at the moment I don’t know how that will be solved, although ultimately I think it will be. More on this topic soon.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013; we also work with security product or service vendors that need help articulating their message to the power industry. To discuss this, you can email me at the same address.

---

[i] To find the article, go here and look for the Newsletters line near the bottom. Click on the + and then click on 2015. Click on the Nov-Dec newsletter. Go to the index on the left of the first page and click on The Lighthouse.