The DoE RFI, part II

If you're looking for my pandemic posts, go here.

This is the second part of my post describing my response to the DoE RFI, which I will send in to DoE soon. The first part covered my response to question A-3 and also reproduced my extended quotation in an article in E&E News on the subject of the RFI. This part will discuss my responses to three of the other questions (I decided I didn’t have anything to say about the remaining four  questions).

Question A-4 actually consists of about nine separate questions:

What information is available concerning the following: BPS electric equipment cyber vulnerability testing standards, analyses of vulnerabilities, and information on compromises of BPS electric equipment over the last five years, including results of independent BPS electric equipment testing and penetration testing of enterprise systems for vulnerabilities (including methodology for discovery and remediation)?

a. What process does the energy sector have to share information with utilities regarding vulnerabilities and vice versa? Are contingency plans in place?

How is the effectiveness of vulnerability testing and mitigation efforts monitored, tracked, and audited?

b. Is a record of an analysis of component vulnerabilities and any compromises of components and systems maintained for a specific period of time (e.g., five years)? If yes, are the results of independent component testing and penetration testing of enterprise systems for vulnerabilities (including timeline for discovery and remediation) also maintained?

c. How are the results of independent component testing and penetration testing of enterprise systems for vulnerabilities (including timeline for discovery and remediation) maintained?

d. How are vulnerabilities identified by external entities addressed? How is the distribution of information regarding patching security vulnerabilities in the supply chain facilitated?

e. What insecure by design/vulnerable communication protocols exist today that should be retired or cannot be disabled or mitigated from BPS electric equipment (examples of protocols include Distributed Network Protocol 3 [DNP3], File Transfer Protocol [FTP], Telnet, or Modbus)?

My basic answer to these questions is simple: Other than item a) (where the E-ISAC does a great job of letting the industry know about vulnerabilities and mitigations for those), there is very little public information available about any of these items. If DoE thinks all of these things are important to test, they will need to take the lead on getting it done. But the first thing DoE needs to do is focus their efforts on real, vs. imagined threats:

1.      Any supposed threat to a piece of equipment that isn’t controlled by a microprocessor (or other logic circuits) doesn’t need to be tested; devices with microprocessors meet the NERC definition of Cyber Asset. In this post, Kevin Perry and I pointed out something that should have been obvious to whoever wrote the EO: A cyberattack can only affect a Cyber Asset. Only about five of the 25-odd devices listed as targets of the EO are Cyber Assets.

2.      But as Kevin and I went on to point out in that post, what’s really important is that the device is performing a function that can impact the Bulk Electric System. These devices are defined by NERC as BES Cyber Assets. Kevin and I couldn’t identify any BES Cyber Assets that are sold by Chinese companies (and certainly none sold by companies headquartered in any of the five other countries on the list of “foreign adversaries” in the RFI).

3.      However, it’s beyond doubt that many of the components – especially chips – found in just about any BES Cyber Asset have some sort of Chinese connection. It’s unlikely that any of these could be used to cause damage to the BES, or even to steal important information and relay it back to China (given that all BES Control Centers and substations are required by the NERC CIP standards to prevent any outgoing communications traffic that doesn’t fulfill a known purpose). But if anything listed in the EO is worth investigating, the threat of compromised components is.

However, no electric utility of any size has the resources available to do the testing that’s needed to be assured that components are benign, and no government agency has stepped up to take the lead in doing this for the industry (I’m sure DoD does lots of this testing now, but I don’t know if they’re specifically testing power industry device components). It would be great if DoE could step up and do that.

Question A-5 reads “What governance of sub-tier vendors do energy sector asset owners and/or vendors have in place? Is contract language for Supply Chain Security included in procurement contracts? Are metrics for supply chain security, along with cost, schedule, and performance maintained? What specific guidance should be developed for Integrator/Installer/Maintenance Service provider activities?”

I summarize this as “What steps does the power industry take to assure itself that “third party” (also sometimes called “fourth party”) suppliers of software and hardware components of BES Cyber Systems, as well as subcontractors of entities that provide services for BCS, follow good cybersecurity practices, especially when they relate to provision of these products and services?”

To be honest, I don’t know exactly what the industry in general is doing currently about the issue of third party security, but it’s definitely an important one. What I can say is that this should be a big focus of NERC entities’ supply chain cybersecurity risk management plans required by CIP-013-1. I can also say that the only viable approach is for the NERC entity to make sure their suppliers of BCS hardware and software have good programs for managing their own suppliers’ security. The utility shouldn’t even try to reach out to third-party suppliers themselves unless a) the supplier has made it clear they can’t be bothered with making sure their own suppliers are secure, and b) the supplier is so strategic that they can’t be replaced.

There’s a great example of good practices in third party supply chain cybersecurity risk management to be found starting on page 4 of this NIST document about Schweitzer Engineering Labs, and also in this post of mine, mostly written by SEL CEO Dave Whitehead last year. My favorite two practices are on page 7 of the NIST document:

1.      For hardware, SEL tries to manufacture as much as it can in house and maintains strict controls on everything else.

2.      For software, SEL makes sure they own every line of code in their software. That is, if they’re including code from a third party, they not only license the code for use, they buy the code outright. This allows them to troubleshoot later problems and fix vulnerabilities themselves, rather than have to wait for the third party to address the problem.

Neither the NIST document nor my post discusses SEL’s approach to risks from open source software that they incorporate into their products, but if you’d like to get some flavor of that, read (or hopefully re-read) the Guideline document from the NERC Supply Chain Working Group on this topic (which can be found here). Development of that paper was led by George Masters of SEL (and very ably led, I might add. I can’t say I contributed a lot to the paper myself, but I certainly learned a lot from the meetings of the group that developed it!).

Question A-6 reads: “Can energy sector asset owners and/or vendors document the level of engagement in information sharing and testing programs that identify threats and vulnerabilities and incorporation of indicators of compromise (e.g., Information Sharing and Analysis Center, Information Sharing and Analysis Organization)? Does the energy sector participate in a community for sharing supply chain risks? Does the energy sector encourage security related information exchange with external entities, including the Federal government?”

The answers to the first and third questions seem obvious to me: They’re both yes. I’ll leave these to others to answer. However, I do want to answer the second question: There’s currently no vehicle for the energy sector to participate in a community to share supply chain risks. And there needs to be. I’ve been saying since soon after CIP 13 was approved that its biggest problem is that it doesn’t provide a listing of risks (or really categories of risk, not the risks themselves) that need to be considered in the R1.1 supply chain cybersecurity risk management plan.

If there were such a list of risk categories included in the requirement (and this is essentially what CIP-010-2 R4 Attachment 1 does. I consider this to be a risk-based requirement, although R4 doesn’t explicitly state that its goal is risk management), then NERC entities would at least have some guidelines for what they need to consider in R1.1 – and the auditors would have something to hang their hat on in audits. The entity would need to show they at least considered each area of risk (e.g. risk due to third party hardware and software components, as discussed above) in developing their plan.

There was no way the CIP-013 Standards Drafting Team could have included this list in the standard, since they had a very unrealistic one-year deadline from FERC to get the standard developed, completely approved by NERC, and on FERC’s desk for them to approve (at which point FERC waited 13 months to approve it, although for a part of that time they had no quorum and couldn’t approve anything, and they lost all of their members but one). And NERC won’t even try to develop a list like this, since that would probably get slapped down by their lawyers as a backdoor rewriting of the standard (which I don't agree with, of course).

So there’s no way a list of risks (more specifically, categories of risk) will be developed by NERC in any official capacity, and I see no inclination on the part of other organizations to develop such a list. On the other hand, I think there needs to be some written discussion of categories of supply chain cyber risks to the BES, as well as at least examples of some of those risks. For this reason, Steve Briggs and I have decided to address this need in our upcoming book on CIP-013 compliance, which we expect to have published later this year.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Are you hot at work – or should be – on getting ready for CIP-013-1 compliance on October 1? Here is my summary of what you need to do between now and then.