Thursday, August 15, 2019

The Cloud’s Achilles heel(s)


I’ve come to believe the Wall Street Journal has the best coverage of cyber security issues of any general news publication. Exhibit A is today’s follow-up, by Robert McMillan, to their article last Monday on the Capital One breach. Here are six important statements from the article:

  1. In a court filing on Tuesday, prosecutors said Paige Thompson had stolen “multiple terabytes” of data from more than 30 companies, educational institutions, and others.
  2. Ms. Thompson had “expressed frustration over her 2016 dismissal from Amazon…” in online discussion forums.
  3. Amazon had previously said that “..none of its services were the underlying cause of the break-in.”
  4. However, on Wednesday a company spokesman said the company is now “running checks and alerting customers if they have the kind of firewall mis-configuration that Ms. Thompson allegedly exploited.” Furthermore, “Amazon is also considering additional changes it can make to its cloud subsystems that will better protect its customers”, according to a letter dated Wednesday.
  5. Winning my 2019 Clueless in Seattle award hands down, an Amazon spokesman said “Other than Capital One, we haven’t yet heard from customers about a significant loss.” This is roughly the equivalent of the White Star Line, in their annual report for 1912, saying “Other than the loss of our ship the Titanic, we had a very good year.”
  6. The most significant statement in the article – from my point of view, anyway - is “Security experts who have viewed her posts said Ms. Thompson displayed a high level of technical knowledge of the inner workings of Amazon’s cloud.”

Here are my takeaways from the article:

a)      It seems very possible that all of the 30+ organizations that Ms. Thompson may have breached were Amazon customers. It wasn’t that she was pinging random IP addresses and just happened to find C1 and these other Amazon customers, who had poorly-configured firewalls. She was almost certainly just attacking Amazon customers.
b)      Ms. Thompson had been terminated by Amazon in 2016, and obviously felt she was treated unfairly. So there’s the motive (she doesn’t seem to have been trying to make money off all of the data she stole, although that’s not certain yet, of course).
c)       The sentiment last week – both Amazon’s and I believe the online community’s in general – was that this breach was mainly a result of Capital One’s poor management of their online firewalls. However, that position is a little hard to maintain, now that there are at least 30 other victims. Sure, they all probably had made mistakes on their firewalls. But if any system is so difficult to figure out that 30 companies don’t get it right (plus God knows how many other Amazon customers who weren’t lucky enough to be on Ms. Thompson’s target list), it seems to me (rude unlettered blogger that I am) that Amazon might look for a common cause for all of these errors, beyond pure stupidity on the customers’ part. After all, there’s another large American company that’s in a bit of a pickle nowadays because they have a system that, while it probably works as specified, requires airline pilots to figure out a previously unknown (to them) problem in the 2-3 minutes they have left before they and their passengers all die – not exactly a great design, IMHO. So far, nobody has been killed by an Amazon breach – that’s one thing they can be happy for in Seattle.
d)      Fortunately, it does sound like Amazon is finally acknowledging that they have to share the blame for these problems, as well as figure out a solution – better training for customers, hopefully redesign of the systems to make them more understandable, etc.

But I almost forgot – this blog is about securing the Bulk Electric System, as well as information about the systems that run it. What does the above mean for that?

I think today’s WSJ article shows there’s a big Achilles heel (in fact, two of them) to cloud security. The first is the one we already knew about last week: that making security totally the customer’s responsibility is inviting trouble. The CSP’s have to do a much better job of educating customers on the risks they face and how to mitigate them, as well as of providing them tools that are understandable. And the CSP’s probably do need to look over their customers’ shoulders as they configure their firewalls, even if their lawyers will probably get apoplectic about the liability this might cause (memo to Amazon’s lawyers: It’s a little late to be worrying about liability in this matter. If you think Amazon is going to get off scot free in court when the lawsuits from Ms. Thompson’s breaches are all settled, you’re grievously mistaken).

But the big Achilles heel is the fact that, while the CSP’s are probably doing an excellent job of protecting against insider threats, they’re not doing any job at all (at least one of them isn’t) in protecting their customers against a disgruntled employee who was fired three years ago and has lots of knowledge of how internal security works at the CSP – as well as of how that security is typically misconfigured by customers. How do they protect against this threat? I suggested last week that maybe there’s a machine that can suck from an employee’s brain all of the knowledge they’ve accumulated about the CSP’s security, once they’ve left the company. At the moment, that’s about the only practical solution I can think of – but I’m sure it’s not ready yet (perhaps it’s in beta). Or maybe they can incarcerate any employee they fire on a desert island somewhere for say five years – at which point any knowledge they may still have would probably be useless. Or maybe they can bring new meaning to the term “employee termination”.

So many possibilities.


Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. To discuss this, you can email me at the same address.

Friday, August 9, 2019

An ex-auditor provides a model for encrypting BCSI in the cloud


After my first post on BES Cyber System Information (BCSI) in the cloud on Monday, Kevin Perry, formerly Chief CIP Auditor for SPP RE, emailed me

“I believe the following model goes a long way to protect BCSI in the cloud:

  1. The cloud service provider furnishes the servers and network infrastructure necessary for the NERC entity to store and manage its BCSI. 
  2. The entity manages security for its own network and servers.
  3. The entity owns and manages the data itself.  BCSI should be protected by encrypting it. 
  4. The entity needs to control the encryption keys, as well as who has the ability to decrypt the data. 
  5. BCSI access is limited to the entity’s staff who are authorized to access it.
  6. The encryption tools should automatically encrypt the BCSI any time it’s copied or moved to the CSP. If this doesn’t happen, the staff moving the data to the cloud must ensure the data is encrypted prior to the move.
  7. The CSP’s staff has absolutely no access to or control over the encryption keys and the encryption/decryption process.”


I agree with Kevin that this is a good model.


Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. To discuss this, you can email me at the same address.

Thursday, August 8, 2019

How can CIP be amended to allow BCSI in the cloud?



After Monday’s post appeared, I had extended email conversations with three people who are very involved with NERC CIP at their organizations (one is a retired CIP auditor – but he’s still very involved with CIP!). The conversations all centered around the recommendations that I provided for changes to the current CIP standards to accommodate BES Cyber System Information (BCSI) in the cloud (I made it clear at the start of Monday’s post that the question of BES Cyber Systems themselves being allowed to be put in the cloud – e.g. outsourced SCADA – is very different).

On Monday, I stated that I thought the best way to make sure all important cloud risks are addressed by CIP was for the drafting team to do two things:

1.       Develop a risk-based standard or requirement like CIP-013 that requires the entity to develop a plan to identify, assess and mitigate[i] cloud risks.
2.       In the standard, provide a list of areas of risk that need to be addressed in the plan, while leaving it up to the entity to identify the particular threats in each area (a term I prefer to “risks” in this context, although risks will work almost as well) that apply to them (this is something that wasn’t done for CIP-013. This makes it flawed, but it’s still the closest thing to a model for all other CIP standards that I know of).
3.       For the four existing CIP requirement parts where this is needed, develop language for the Measures that would allow entities to cite a cloud provider’s certification under FedRAMP (and possibly other certifications) as evidence that they have complied.

I believed previously that FedRAMP would most likely address any possible threat that applies to data stored in the cloud, but – and this was the main point of my post – the Capital One breach points to a threat that clearly isn’t covered by FedRAMP. Since I never completely stated that threat, here it is now (or at least, what I understand of it), broken down into logical steps:

a)      Each organization’s level of security within the cloud service provider’s network is usually their responsibility, not the CSP’s.
b)      That level of security is usually dependent on the organization understanding the differences between – say – configuring a firewall in the cloud and configuring a firewall on a separate standalone network, like their own.
c)       It seems many organizations don’t completely understand those differences. In particular, according to the Wall Street Journal, understanding Amazon’s “metadata” service is particularly important.
d)      Paige Thompson, the Capital One hacker, mentioned in online forums that a lot of Amazon clients don’t understand this well, so they have misconfigurations.
e)      After she left Amazon, she exploited this knowledge to hack into C1 and – by her admission – at least two other firms.

So the threat is that a CSP employee will utilize his or her knowledge of likely customer misconfigurations to penetrate a customer’s systems at the CSP and obtain BCSI that way.

What are the mitigations for this threat? On the utility’s part, the most important mitigation is to make sure they understand all the nuances of the CSP’s network that could affect their security configuration, and if that is just too difficult for them, they should pay the CSP itself (I assume this is always an option) to configure their firewalls – and other cloud-based security devices – for them.

But the CSP also has a mitigation they need to implement: They need to do a much better job of explaining these nuances to their customers, since it seems that Amazon (at least) hasn’t done this well enough. However, if C1 is really the only AWS customer who has made these mistakes, then I’d put all of the blame for the breach on them. But until I know whether or not C1 was unique, I have to assign equal blame to Amazon and C1.

This is where things stood for me on Monday regarding required changes to CIP to accommodate BCSI in the cloud. However, on Tuesday (as I’ve mentioned), three individuals told me several things that caused me to change my position (since one of the three – the ex-auditor – focused on a different issue than the one in this post, I’ll put his short comments up tomorrow night).

The most important of these was pointed out to me by a longtime friend who is in charge of CIP compliance for a mid-sized NERC entity, and who has some knowledge of discussions on the BCSI-cloud question in the NERC community. He said that the main thrust of the discussion within NERC circles appears  for the time being to have shifted away from an administrative approach of allowing or accepting FedRAMP as the primary evidence of the vendor’s compliance (which of course had been my assumption. When I last participated in those discussions, that was definitely the idea) to a revised Standard language approch focused on making encryption of BCSI in the cloud (and perhaps in transit to and from the cloud) the main means of compliance with CIP-004 R4.1.3, CIP-004 R4.4, CIP-004 R5.3 and CIP-011 R2 (both parts). These are the requirements that cause all of the problems for BCSI in the current CIP standards. This could be done by changing the wording of these Requirement parts (as well as their Measures), so that encryption of the data is an option for compliance. In fact, my friend thinks that this is now being talked about as the only means of compliance – that is, the only thing that would be needed to show compliance is encryption; FedRAMP wouldn’t necessarily come into the picture at all.

To be honest, I find this idea pretty sub-optimal from a security point of view. Sure, encryption will solve almost every cloud risk I can think of (it definitely covers confidentiality and integrity. Availability would presumably be covered by having duplicate BCSI repositories elsewhere, either at the entity itself or at a different cloud provider). But I’m a firm believer in defense in depth, and putting all of your security eggs in the encryption basket seems to be like depending on the Maginot Line to protect you. Yes, it will protect you fine until something changes about your assumptions (the keys aren’t protected properly and are stolen, quantum computers become available that can make mincemeat of any conventional encryption, or something else), and then it doesn’t protect you at all.

But what if we do both? Along with the encryption requirement, what if the SDT also changed the Measures section for each of those four requirement parts, so that an entity could use their CSP’s FedRAMP certification (or encryption) to show they are compliant with that requirement part? That would help, but it’s not going to cover all cloud risks.

Here, I want to add something another friend of mine – Brent Sessions of  the Western Area Power Administration – pointed out to me yesterday: There’s another whole area of threats that I’ve been ignoring regarding BCSI in the cloud, which should have been clear as day after the C1 breach. These are threats due to the fact that the cloud customer usually is responsible for their own security and – believe it or not – even customers can make mistakes.

As I said, I consider Capital One to be half responsible for their breach, because they hadn’t properly configured their firewall or firewalls – which were virtualized on the AWS network; the fact that AWS probably didn’t provide C1 with enough of a tutorial on the metadata service doesn’t mean
 C1 is off the hook morally. At the least, they could have pressed AWS for more information until they did understand the service. Of course, C1 has taken responsibility on their part, but if lawsuits against C1 start flying around later on, I wouldn’t rule out the possibility that C1 might invite (ever so nicely, of course) AWS to share the defendant’s table with them.

I believe there need to be three major elements in a new requirement part, which should be part of CIP-011 R1 (this is of course the current CIP requirement for information protection, and it’s risk-based to boot): 1) encryption of BCSI (including key protection and encryption of BCSI in transit to or from the cloud), 2) certification of the CSP, and 3) mitigating risks caused by the fact that the entity is most likely responsible for its own cloud security, while at the same time it’s operating in a new environment that’s in many ways very different from that of standalone networks and standalone firewalls.

I suggest that the drafting team draft a new CIP-011 requirement part R1.3, which would contain wording something like this:

“Measures to protect BCSI stored with a Cloud Service, including:
  • Methods to encrypt data in storage
  • Methods to encrypt data in transit between the Responsible Entity and the CSP (I went back and forth with my friends on the question of data in transit. We all ended up agreeing that a) data in transit definitely poses a risk to using cloud services to store BCSI, and b) if the entity doesn’t want to mitigate that risk, then they should keep all of their BCSI inhouse).
  • Methods for protecting the encryption keys, so that only the Responsible Entity's full-time employees or contractors who have BCS access have access to them
  • Assurance of the quality of the CSP’s security practices, evidenced through certification under FedRAMP (or other certs like SOC II, ISO 27001/2, etc. Of course, there's a lot more to it than just having a "FedRAMP certificate", if one even exists. There would need to be an explanation of what actually constitutes acceptable certification) - or through another acceptable certification process (this would cover the case where the entity – especially a government one - has no choice except to use a private cloud operated by themselves or another organization. This cloud might possibly have some other certification. The entity would need to show evidence that this was of similar quality to FedRAMP).
  • Methods to secure the physical or virtual security devices that protect the Responsible Entity's data in the CSP's cloud environment, if those measures are by contract the responsibility of the RE, not the CSP.” (and the SDT should do a risk analysis, perhaps with one or two security experts who really understand the cloud environment. In fact, maybe Paige Thompson would be available! :)  )

Besides this, there would need to be changes to the Measures sections (and possibly the Requirements themselves) of CIP-004 R4.1.3, CIP-004 R4.4, CIP-004 R5.3 and CIP-011 R2 (both parts) that would allow encryption, CSP certification[ii], or both to be used as evidence for compliance.


Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. To discuss this, you can email me at the same address.


[i] Although I hope the SDT doesn’t make the same mistake the CIP-013 drafting team made, namely leaving the word “mitigate” out of R1.1, meaning the entity could legally call it a day if they just identified and assessed risks, but didn’t do anything about them! I actually had one person – from a software vendor to the power industry, although one that I doubt actually sells software for BCS so CIP-013 doesn’t apply to their customers – try to tell me this omission was intentional, so the entity really doesn’t need to mitigate supply chain risks once they’ve identified and assessed them. I pointed out to him that none of the guidance or anything else that’s been written on CIP-013 makes sense if he’s right. I don’t know if he still takes that point of view or not.

[ii] I learned today that FedRAMP has different levels of certification, and the medium level includes encryption. Since Capital One shows that at least some AWS customers don’t have encryption, this means that FedRAMP needs to be looked at as as much of a marketing concept as a certification. In particular, if the SDT decides that both encryption and FedRAMP certification will be required, NERC entities will need to make sure their CSP is FedRAMP certified and offers services at the medium level. Of course, fries are extra.

Monday, August 5, 2019

What does Capital One mean for NERC CIP and the cloud?



I’ve been intending to write two posts on CIP and the cloud, since that topic is much in the air nowadays. I want to do two separate posts because there are two very different questions: 1) what it will take to officially allow BES Cyber System Information (BCSI) to be put in the cloud, and 2) what it will take to allow NERC entities to put BES Cyber Systems themselves in the cloud. The Capital One incident announced last week provides a perfect excuse to write the first of these posts.

Here’s where we stand now on BCSI in the cloud:
  1. There is nothing in the current NERC CIP requirements themselves that prevents putting BCSI in the cloud for High, Medium and Low impact BCS; in fact a number of entities are doing that now, including many using a cloud-based configuration management service.
  2. I wrote a series of posts (starting with this one in early 2017) that described what needed to be done to comply. These were based in large part on input from an auditor, who I can finally reveal is Kevin Perry, former Chief CIP Auditor of SPP-RE, now retired (in fact, he was quoted in many of my posts, especially during the difficult years of the CIP v5 rollout). Kevin wants me to point out that “the comments (in the post) are a couple of years old and might not reflect current thinking on the part of NERC, the Regions, or the Standards Development Team.”
  3. The most important point was that the entity needs to include provision for cloud service providers in their CIP-011 R1 information protection plan – they need to write reasonable requirements for the CSP and make sure the CSP is following them; this shouldn’t be hard to do.
  4. But the entity also needs to comply with four requirement parts in CIP-004, all having to do with things the CSP needs to do. Of course, there’s no question that any CSP isn’t already doing all four of these things, but the problem is evidence. With the prescriptive NERC CIP requirements like those in CIP-004 (as opposed to objectives-based ones like CIP-007 R3), the NERC entity is required to have evidence that each of these things was done every time it was required. This means that the CSP will need to have evidence that any of their perhaps thousands of employees who had access to the room (which could be huge) where the servers that contained the entity’s BCSI were located, and was terminated, had their access to that room revoked within 24 hours. Asking the CSP to maintain this evidence would obviously be the same thing as asking them to give up their whole business model in order to please a few electric utilities. Good luck with that.
  5. Nevertheless, until recently I believed that all of the NERC Regions were OK with BCSI in the cloud, although I was told a couple weeks ago that one Region still says it isn’t allowed (despite the fact that I heard one of this Region’s most-respected auditors describe at a compliance workshop in 2016 how to comply with CIP for BCSI in the cloud! Of course, I was shocked – Shocked! – to learn there is inconsistency in statements by the NERC Regions, even within one Region. Who would have thought this was possible? J ).
  6. And speaking of consistency, NERC is now working on a project called Align, which will consolidate the standards enforcement process across all of the Regions. And guess where that system – including tons of data that is without doubt BCSI – will reside? I’ll give you three choices: 1) NERC headquarters. 2) My basement. 3) The cloud. If you guessed 3), you’re a winner! I sure hope NERC’s able to pass their CIP audit next time.

To make it clear that BCSI can be stored in the cloud, and to determine how NERC entities can provide evidence that it’s secure, a new Standards Drafting Team was created and has already started meeting. I haven’t been able to attend their meetings, although I hope to at least review drafts when they start producing them.

The SAR for this team makes it look like their job will be pretty simple (and up until last week I thought it was): Modify CIP-004 so that the focus is on the cloud provider’s controls for electronic and physical access to BCSI and the devices on which it is stored. In practice, everyone was anticipating that it was a certainty the team would identify one or two certifications – with FedRAMP the far more likely one – that would constitute evidence for compliance with the modestly revised CIP-004 requirements (although the SAR also points out the need to revisit CIP-011 R1. Since that’s a completely objectives-based and risk-based requirement, I can’t see too much need to revise it, although explicit mention of data services like cloud providers – rather than just referring to “third parties” – would be a help).

Of course, determining what FedRAMP “compliance” means (and how it relates to the CIP-004 requirements) isn’t simple at all, since different audits cover different things. And it might be hard to get AWS or MS Azure to just hand you their latest FedRAMP audit report in toto. There’s a lot to be negotiated here. But until last week, I – and others – was saying “There’s no question that no utility can have a level of security on their own network that remotely approaches the average cloud provider’s security level. This will work out fine. What could possibly go wrong?”

And now there’s the Capital One breach. I’m sure some will jump up and say “The breach was Capital One’s fault, and they’ve admitted it. Amazon provided a secure infrastructure like they promised. The fact that C1 didn’t configure their firewalls correctly is their problem – they had total control of them, even though they resided on Amazon’s network.”

From a legal point of view, that’s probably true; I wouldn’t sell all of your Amazon stock right now, if you were thinking of doing that. But the big fly in this ointment, from a pure security point of view, is that Paige Thompson had worked for AWS until recently, and seems to have exploited the knowledge she gained there to breach Capital One’s systems on AWS. In fact, she hinted at having penetrated a couple other AWS customers as well.

Since she wasn’t an employee at the time of the breach, this wasn’t an inside job – yet to pretend that this is something that any competent hacker anywhere could have done, and her former-employee status had absolutely nothing to do with the fact that she was able to hack C1, seems to me to be bending over backwards to make all of this go away. She had inside knowledge (including of likely mistakes that customers would make in configuring their firewalls on AWS), and that played a big part in her breaking into C1.

But let’s get back to the CIP issue: Amazon was FedRAMP certified, yet this happened. Suppose the SDT waits until they have good information on what happened and knows how Amazon could have prevented it, then writes that down as a procedure. We’ll call this Procedure X (it might be a new technology to erase from the person’s brain all memory of anything technical they’ve learned while working at AWS, something benign and non-intrusive like that). The team then adds some sort of provision in the Measures section of CIP-004 that says a FedRAMP certification, plus evidence that Procedure X is in place, is all that’s required to show that the entity is compliant with the four requirement parts in CIP-004. Would that be sufficient?

I shudder to think there are a few people who might think that is sufficient, so I’ll answer this question myself. Of course it wouldn’t be. People are endlessly creative. If they want to wreak havoc and they already have a good knowledge of cloud systems, they’ll figure out another way to do it if they can’t emulate Paige Thompson. And don’t say the answer lies in better training on cloud security for the NERC entity that’s storing information in the cloud. Yes, that’s required, but to say that’s the answer is to say that if only people were perfect and never made mistakes, we wouldn’t have any problems like this.

IMHO, the SDT needs to start looking at a risk-based approach to securing BCSI in the cloud. CIP already has at least four risk-based requirements (including CIP-010 R1, as well as CIP-007 R3, CIP-010 R4 and CIP-003-8 R2) and one risk-based standard (CIP-013)[i]. The requirement that I think best illustrates what this requirement should look like is CIP-010 R4, which mandates that the NERC entity develop a plan to manage the risks of Transient Cyber Assets and Removable Media. Granted, the words “manage the risks of” are nowhere to be found in R1, but effectively that’s what is required.

The plan’s objective is to protect BES assets from risks posed by TCAs and RM. While CIP-010 R1 Attachment 1 (which is part of the requirement) provides guidelines on areas of risk that should be addressed in the plan, there are no prescriptive requirements saying “You either do X within Y days or you get your head cut off.” When you’re making a plan to do something and you’re not told exactly what to do, you need to make trade-offs. You consider roughly the amount of spending (and staff time) that you can get away with in achieving the objective of the plan, and you try to allocate that so that you mitigate the most risk possible given your available resources.

Of course, CIP-013 uses the word “risk” freely; there’s no question it’s risk-based. However, the biggest problem with CIP-013 is that it doesn’t give you any guidance on the types of risks you should look at mitigating – e.g. risks of vendor-caused software vulnerabilities, risks due to provenance, etc. I like CIP-010 R4 Attachment 1 because it does list areas of risk to address: TCA Authorization, Software Vulnerability Mitigation, Introduction of Malicious Code Mitigation, etc.

So I think whatever the SDT develops should require the NERC entity to develop a plan to mitigate risks attendant on putting BCSI in the cloud; then it should suggest particular areas of risk that should be addressed. Of course, most of them will be areas of risk addressed now by FedRAMP, e.g. authentication, vulnerability management, etc. You shouldn’t have to do any extra documentation – and you certainly shouldn’t have to require Amazon to negotiate a separate contract with you if they want your business! – if the SDT decides that whatever FedRAMP requires is adequate to address these areas.

But clearly FedRAMP doesn’t address all areas of risk! So I think the SDT needs to put their thinking caps on and identify risks it doesn’t address. Here’s an idea: What about risks due to ex-employees utilizing their knowledge of likely vulnerabilities caused by customers not completely understanding the inner workings of AWS systems to hack into those customers? My guess is the SDT could come up with other risks like that through talking with experts and just thinking through these things logically.

I’m actually doing a lot of work like that right now. My CIP-013 clients – and all NERC entities who have to comply with CIP-013 – are facing the challenge that R1.1 requires them to “identify and assess” supply chain cyber security risks to the BES, yet it provides no further guidance on where to find those risks. So we’re looking through lots of NIST documents, guidance papers, procurement language from other utilities, etc. And we’re participating in the discussions of the NERC CIPC Supply Chain Working Group, which are excellent. The goal is to put together a list of what my customers consider the supply chain cyber threats that pose the highest level of risk to their BES assets, and figure out how to mitigate those risks within the utility’s own structure and limitations (especially its resource limitations!). The goal is to achieve the most mitigation of BES supply chain security risk possible, given the fact that these utilities don’t have unlimited budgets for cyber security (or anything else, of course).

But I came to realize early on in this process that, as far as risk identification and assessment go, there is just about nothing that’s specific to any particular utility. The areas of risk could and should be identified on a national level, with the entities then being free to decide how much emphasis to put on mitigating risks in each of those areas, depending on their particular environment. This is essentially the approach used in CIP-010 R4.

It would be even better if the new cloud SDT could actually produce a list of threats within each of those risk areas (which is what I’m doing with my clients for CIP-013). Using this list, the entity could then a) add other threats that the SDT didn’t identify, b) assign a risk score (presumably high/medium/low) to each threat, based on their own estimates of probability and impact, c) rank all of the threats by risk score and choose the highest-risk threats to mitigate, then d) develop plans to mitigate the selected risks.

But it would be a huge deal for the SDT to get to the level of identifying individual threats, so I’ll settle for just identifying the risk areas. This will itself be a big help for NERC entities, since they will at least have a starting point for risk identification. Even more importantly, it would give the auditors something to hang their hats on: They could go through each area of risk to make sure the entity understands a) the threats involved in that area, b) the vulnerabilities that enable those threats to be realized, and c) the mitigations for those vulnerabilities. If the entity doesn’t seem to understand something, they certainly won’t give them a Potential non-Compliance finding (those will be reserved for entities that just blow off this whole process), but they are likely to issue an Area of Concern, so that the entity will address the identified problems by their next audit.

My bottom line observation on BCSI in the cloud in CIP is that it’s a moderately hard problem to address, but certainly can be done. In one of my next posts (I no longer say “my next post”, since every time I do that it ends up being 3 or 4 posts later, and sometimes never), I’ll get into the question of BCS themselves in the cloud. That’s nowhere near such a nice picture.


Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. To discuss this, you can email me at the same address.


[i] Some people say that CIP-014 is a risk-based standard, but I still haven’t made up my mind on that. Whether it’s risk-based or not, it’s been sometimes audited as a prescriptive standard.

Tuesday, July 16, 2019

An (ex-)Auditor comments on CIP-013 for Low impact assets



Kevin Perry, the former chief CIP auditor of SPP Regional Entity, wrote in today to comment on my Sunday post, the second (and last) in a series on NERC’s draft Data Request on Supply Chain Risk Assessment - which is currently out for comment by NERC entities. Specifically, Kevin wanted to comment on my suggestion that NERC proactively propose to include a basic supply chain security requirement for Low impact assets in CIP-013, rather than wait for FERC to order that Lows be made to comply with all of CIP-013. That would be much more burdensome and expensive.

Specifically, I suggested that NERC propose something like Requirement R5 in the first draft of CIP-013 (from January 2017), which was definitely a very basic requirement but also managed to address two of the biggest BES supply chain risks faced by all NERC entities: compromised patches and vendor remote access. That requirement read:

R5. Each Responsible Entity with at least one asset identified in CIP-002 containing low impact BES Cyber Systems shall have one or more documented cyber security policies, which shall be reviewed and approved by the CIP Senior Manager or delegate at least once every 15 calendar months, that address the following topics for its low impact BES Cyber Systems:

5.1. Integrity and authenticity of software and firmware and any patches, updates, and upgrades to software and firmware; and 
5.2. Controlling vendor-initiated remote access, including system-to-system remote access with vendor(s). 

Kevin thought this was a good idea because it would reduce a lot of supply chain risk, but also because it wouldn’t put a big compliance burden on Low-only entities (and probably zero burden on entities with Mediums and Highs that also have Lows, which is just about all of them. Those entities wouldn’t have to comply with these two new requirement parts, which are just for Lows. But they already have to comply with two parts that are very similar, CIP-013-1 R1.2.5 and R1.2.6).

Specifically, Kevin said (with my comments in italics):

I could see implementing your (of course, these aren’t mine! They were written by the CIP 13 drafting team) suggested 5.1 and 5.2 with relatively little effort.  Both can be implemented holistically.

1.       5.1 is vendor-centric and would apply to all Low Impact BCS, whether one or thousands, because the focus is on the vendor and not the Cyber Asset. Kevin is of course referring to the fact that the vendor is really the entity that has to “comply” with this requirement part. They will need to figure out a way to provide their electric utility customers with a way to verify integrity (e.g. checksum) and authenticity (e.g. digital signature) of all patches and other software they send to them. Once they’ve solved this problem for just one utility (whether they have High, Medium or Low assets), they’ve solved it for them all. It’s unlikely that many if any vendors that sell software that goes on BES Cyber Systems will balk at doing this.

2.       With respect to 5.2, putting in a gateway would be appropriate. This could be as simple as enabling/disabling a VPN account that the vendor uses to get to the supported entity’s network.  Access permissions then control where the vendor can go.  Best practice, you install the equivalent of an Intermediate System to maintain a breakpoint between the vendor and the supported systems. This requirement part will definitely be a little harder to comply with, because it does require some configuration at each Low asset. But my guess is that most entities with Low assets are already doing something like this. And if they aren’t, there are a lot of inexpensive remote access gateways for sale.

Note that Kevin isn’t saying that the only good way to comply with this part is to install an Intermediate System (as Mediums and Highs do, per CIP-005 R5.2), which would definitely be a good deal more expensive and time consuming, especially for entities with a lot of Low assets (one large entity told me a few years ago that they have over a thousand Low impact substations! Putting an IS in each one of those would be…well it would be pretty burdensome).

Kevin later emailed back to say “One more thought...  there is an added benefit.  If the entity is also a Distribution Provider, the Low impact BCS controls will also apply to the same relays in the distribution subs - no extra cost.” Such a deal!

July 18: Kevin emailed me yesterday to emphasize the point that compliance with R5.2 wouldn't be very burdensome on Lows (although I'd like to hear from anyone who thinks differently):

If the entity remotely manages its LBCS, such as over a TCP/IP network or by using a centrally managed “dial up” system to authenticate the user and then connect the user to the supported Cyber Asset, that is where you would place a central gateway to control vendor access to the company network and resources required to access the LBCS.  If the entity has a gateway device in the substation, then it is mostly a matter of adding the vendor to the gateway access list and, of course, enabling the vendor to get to the gateway, most likely through a central gateway that authenticates the vendor onto the entity network.



Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. To discuss this, you can email me at the same address.

Sunday, July 14, 2019

NERC’s draft Data Request on supply chain risks for Lows, part II



This is the second (and last) of my two posts on NERC’s draft Data Request on supply chain risks for Low impact assets. The first one is here (it includes a link to the draft DR itself). I actually ended up writing two more posts last week based on comments and questions I received about the issue of how to define external routable connectivity for Lows; but they weren’t about the DR itself, which is why I say this is part II (I expect to have another post or two – or maybe more – on the topic of “erc” in the near future).

There are two main parts to the DR. The first part is the long section entitled “BES Cyber Systems”. In the first post, I discussed at length the many wording problems in that section, and expressed the opinion that it could be cleaned up and actually sent out. But I also said that without changes I think it’s just going to cause a lot of confusion and – most importantly – it’s very unlikely to yield any usable data.

There are two issues I want to discuss in this post. The first issue has to do with the second part of the DR, but the second is strategic: I think NERC is pursuing exactly the wrong strategy in dealing with what is evidently very strong pressure from Congress to increase requirements for Lows. I think the DR isn’t going to satisfy these Congressional hawks at all, and will probably make the industry’s position with Congress worse, not better.

To start with the first issue, let’s go to the second part of the DR, which is much shorter than the first part. You can find it on pages 7 and 8 of the DR, under the heading “CIP-013 Cost of Implementation”, but since it’s so short I’ll reproduce the whole section here:


“Stakeholders, regulators and legislator’s decisions on mitigating and preventing supply chain risk depend on the costs and benefits associated with those decisions. While utilities would want and share this information, it is not currently available. Therefore, subject matter experts believe it is premature for CIP-013 registered entities to determine, or estimate costs or benefits associated with the implementation of the standard.

  • The standard is new and there is no historic precedence (note: this should be ‘precedent’) for registered entities to pre‐determine costs based on furthering relationships with existing and new vendors. 
  • These costs and benefits are intangible and depend on a spectrum of actions, from internal process refinement costs to extensive costs associated with replacement of blacklisted vendors.
  • The cost of compliance is currently unknown as this is a new standard.
  • Many utilities are experiencing push back from vendors for CIP‐013 compliance that could require vendor change or increase in cost from such vendors. 
Consequently, CIP-013 is causing, and will necessitate many changes for complying utilities from now until the July 1, 2020 implementation date. Therefore, currently providing any credible cost or benefit information is premature.

      6. Do you agree with the above SME assessment – Yes or No?    

Please provide CIP‐013 cost or benefit amounts should you answer “no” to the above question:”


In case you’re puzzled by this (and if you’re not, I don’t think you’re reading it closely), this section essentially says “We’re not stupid. We know there’s no way that a Low impact entity can estimate their costs for compliance with CIP-013. Do you agree with that statement? But if you don’t, could you give us your estimate anyway?”

Here’s a little history: The first draft of the DR that the SCWG received from NERC asked entities to estimate the cost for complying with CIP-013 for Low impact assets. At our first meeting with NERC, we pointed out that even Medium and High entities still can’t give a good estimate of their cost for complying with CIP-013, so how could Low-only entities possibly do this (although, based on the fact that I’ve been working on nothing but CIP-013 compliance with three entities of different sizes since January 1, I have come to realize my initial estimates of the total cost were too high. I don’t think people understand how important it is that this is a risk-based standard, not a prescriptive one. That makes for a huge difference in costs)?

The NERC people didn’t dispute this, but they pointed out that the Board of Trustees had ordered them to ask this question, so they felt they had to do it. I suggested (at a later meeting) that the SCWG might actually be able to come up with a cost estimate for Lows (since almost all of the NERC entities that are part of the SCWG have Low impact assets, as well as Highs and/or Mediums). I said it would be much better to have us estimate costs than the Low-only entities, who have no experience at all with CIP-013, and probably haven’t thought about it at all yet.

But no, NERC said they had to ask a question since the Board had ordered it. So some people in the SCWG came up with the above “question”. I analogize it this way:

  1. The Board has asked NERC staff to ask entities how to design a perpetual motion machine.
  2. Instead of simply telling the Board that there’s no way to design a perpetual motion machine, the staff members have decided to ask entities this ‘question’: “We know there’s no way to design a perpetual motion machine. Do you agree with this statement? But if you don’t agree, please give us your diagrams.”

My main objection to this “question” is that it makes NERC and the SCWG look ridiculous for even asking it. And my other objection is that any data that come out of it are guaranteed to be garbage. Anyone who answers this is simply going to take a guess, and they will all understand that if the cost estimates come in low, this will probably lead to CIP-013 being applied to Lows. So they’ll just estimate as high as they think is credible. I think this whole question needs to be removed.

Now I will discuss my strategic objection to this whole Data Request. It is based on what the SCWG members were told by NERC staff about the reason for this DR (with a little inference on my part, to fill in some gaps):

  1. Congress and FERC have been leaning very heavily on NERC in recent months to increase CIP requirements on Low assets in general.
  2. Since the question of Lows and CIP-013 was already on the table, that pressure is now focused on requiring Lows to comply with CIP-013.
  3. NERC staff (and maybe the Board) are worried that FERC is simply going to put out an Order saying that Lows need to be included in CIP-013. To stave off that event (or at least push it back), NERC will do this Data Request, which – due to all the steps required to approve it, gather answers and then analyze them – will probably take six months to complete.
  4. My guess is that part of the Board’s (and NERC staff’s) concern here is showing Low entities that they’re trying to fight this off any way they can – and when and if FERC issues their Order, NERC will be able to tell the Lows “Well, we tried…”

This wouldn’t be a terrible strategy if there were any conceivable way it could work, but I don’t see one. The main problem is that, even if the first question is cleaned up and sent out, it’s not going to ask for the data that Congress wants. During one of the SCWG meetings, I asked Lonnie Ratliff of NERC (whose title is Senior Manager, Cyber and Physical Assurance) what exactly the Congressional staffers had been asking him about Lows.

He said they wanted to know how many Critical Cyber Assets were at Lows. Remember that CCAs were a single device, so these Congressional staffers were obviously trying to find out how many computing devices were at Lows. Lonnie told them that term had been replaced by BES Cyber System, and of course that’s why the first draft of the DR that NERC showed to the SCWG asked for information about Low BCS. As I mentioned in the first post, the idea of asking about BCS was finally dropped when, at one of the SCWG meetings, I pointed out that there were some entities who have over a thousand BES Cyber Assets in a single BCS, whereas there are others who are classifying every BCA as a BCS . So asking about BCS isn’t going to yield any meaningful data about BES Cyber Assets, which is the closest current term to Critical Cyber Asset – which always referred to an individual device.

So if NERC were really going to satisfy Congress, they would need to ask about BES Cyber Assets in the DR. But to even ask this question would require owners of Low assets to have to go through a huge effort to identify all Cyber Assets in every Low asset, then consider each one as to whether it meets the BCA definition. They would rebel if NERC even asked a question about Low BCS, but there would probably be blood in the streets if NERC asked about Low BCAs. Even if NERC just asked Lows to estimate the number of BCAs, they would still have to go through a lot of effort.

So in question 1 of the draft DR, NERC is just asking about Low assets – primarily substations, generating plants, and Control Centers. It would certainly be interesting to get this data (although a lot of this has already been given to the Regions, since every Low entity has to state their number of Low assets of each type). But I see no way this is going to satisfy Congress. They want to measure the degree of cyber risk posed by Lows based on the number of computing devices installed. They’re going to be very disappointed when NERC hands them a list of assets, and when they’re asked how many devices this covers, the NERC representative will say “Oh, we can’t get that information. Sorry.”

But here’s an idea, NERC: Instead of making your entities go through a lot of work to respond to a DR, then six months later give Congress and FERC a list that isn’t what they wanted, why don’t you get ahead of this issue and say “You know, you people are right. CIP-013 should be applied to Lows in some way, although we’re sure you’ll agree there’s no need to make Lows go through everything that Highs and Mediums need to do. We’ll draft a Low-only requirement and add that into version 2 of CIP-013, since we’re just now putting together a team to develop that version.”

And what should this Low-only requirement be? I think NERC would be well advised to go back to the first draft of CIP-013, which had this requirement part R5:


"R5. Each Responsible Entity with at least one asset identified in CIP-002 containing low impact BES Cyber Systems shall have one or more documented cyber security policies, which shall be reviewed and approved by the CIP Senior Manager or delegate at least once every 15 calendar months, that address the following topics for its low impact BES Cyber Systems: 
5.1. Integrity and authenticity of software and firmware and any patches, updates, and upgrades to software and firmware; and 
5.2. Controlling vendor-initiated remote access, including system-to-system remote access with vendor(s)."


CIP-013 affandicios will recognize 5.1 as being pretty close to CIP-013-1 R1.2.5, while 5.2 is close to R1.2.6. So what would Lows have to do to comply with these two requirement parts? They would have to develop a policy for Low BCS that includes these two items (and I would recommend that the new SDT not only require Lows to have a policy, but to implement it. CIP-003-5 R2 just required Lows to have four policies, but said nothing about implementing them. FERC then told NERC to go beyond policies and add “specific requirements” for Lows in CIP v6. If the v5 SDT had included “implement” in the v5 requirement in the first place, FERC would probably have said that was good enough and they wouldn’t have required any more. Instead, there has been all sorts of anguish over CIP-003-7 R2 , since FERC didn’t like the CIP-003-6 R2 that NERC came up with. This anguish was reflected in the emails I received last week about my two erc posts, and it will increase as the compliance date approaches).

And if CIP-013-2 includes this requirement, what would Lows not have to do, compared with their Medium and High brethren? The most important thing they wouldn’t have to do is comply with R1.1, which requires the entity to consider all of its supply chain cyber risks and mitigate the most important ones. While – as I mentioned above – I no longer think this will require of Mediums and Highs the degree of effort that I estimated originally, it will still be a large amount of work, and will require that people in supply chain and legal be heavily involved, which wouldn’t be the case if the Low-only requirement were implemented as described above.

The other thing Lows wouldn’t have to do is comply with R1.2.1 through R1.2.4. I just looked at how these parts differ from R1.2.5 and R1.2.6, and it seems the big difference is that the latter two parts aren’t going to require a lot of heavy lifting with vendors, while R1.2.1-R1.2.4 will be more difficult to get vendor agreement on. This may be why the CIP-013 SDT only required that Lows “comply” with R1.2.5 and R1.2.6 in the first draft of the standard (which went down to defeat with only a 9% positive vote, due in part to the fact that the Low requirement was there. There will definitely be opposition if NERC moves to put a Low requirement in CIP-013-2, but I think it will be much more defensible if NERC points to what may be the alternative – having FERC require that Lows be included in scope for all CIP-013 requirements).

Will this suggested requirement mean that Lows have to keep an inventory of BES Cyber Systems? Definitely not. Lows will have to show auditors that they have these policies and have implemented them, but that doesn’t require having evidence that they were applied in every instance and for every component of a BCS.

What will happen if NERC persists in their current course and sends out this Data Request, then turns the results over to FERC and Congress late this year or early in 2020? As I’ve said, the data that will be gathered aren’t what Congress is looking for, so they’re unlikely to be satisfied. Will they all just give up and focus on getting elected again? I really doubt it. The cyber security of the grid has become a big concern of Congress, and both parties agree that electric utilities should be doing more about this. I think they will pressure FERC to go ahead and tell NERC to simply include Lows in CIP-013, which probably means having all of the current requirements apply to Lows. I think that, if NERC made a proactive move like what I’ve suggested above, they’d very possibly be able to preclude this much more drastic step.

And there’s another reason why I think this would be a good move. In case you haven’t noticed, electric power isn’t the most popular industry nowadays (not that it ever was, of course), and many in Congress feel the industry just isn’t stepping up to the plate enough to combat cyber threats. At the same time, they wonder why such an important industry should have the power to write its own cyber security standards.

If NERC is perceived as reflexively fighting all cyber regulation, Congress will soon feel (and probably are already) that they need to think about taking cyber regulation of the power industry away from NERC (the O&P standards are a different deal, of course) and giving it directly to DoE or DHS (or maybe a new Department of Cybersecurity).

So sometimes it’s better to acknowledge the other party’s concerns and say “We agree. We do need to do more about this. Here’s our proposal…”, rather than try to simply stonewall them. Especially when that other party is much more powerful than you are.


Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. To discuss this, you can email me at the same address.