How can the power industry really collaborate with vendors?

On September 17, Robert Walton of Utility Dive published a well-researched article that asked the question why vendors weren’t invited for the upcoming GridEx exercises.  There were quotations for and against the idea of vendors participating. One of the “for” votes was from Nick Cappi of Hexagon PPM, who said "If we want transformational events in improving the security of our critical infrastructure, then we need to break down the silos of information and obstacles to collaborate."

I was one of the “against” votes, and was quoted as saying

"GridEx is an operational exercise for responding to a hypothetical attack, so only the operators of the [bulk electric system] can participate. When an attack happens, it's too late [for] most vendors to be able to help — the software has already been written, the control systems have already been developed, etc."

Of course, my point was that, when it comes to a situation where the grid is being defended against a serious attack, there’s very little that vendors can do – given the very short (in some cases, literally infinitesimal) timeline for possible action.[i]  If there’s a vulnerability in some software, the software supplier can’t patch it in real time. If a router, server or any hardware device is compromised, the best thing is usually to shut it down and remove it from the network, not leave it operating and try to block or remove the avenue of compromise.

But later I found myself thinking, “You know, there really isn’t a lot of collaboration between electric utilities and industry suppliers, to solve security problems they jointly face. It’s certainly not as exciting as a 2 or 3-day exercise like GridEx, in which the entire American way of life is assumed to be on the line. But it’s still very important, and it can be quite satisfying to collaborate to find a solution to a specific cybersecurity problem. Maybe the solution won’t make everybody happy, but if it leaves everybody better off and no significant group worse off, that’s something worth doing.”

I thought about the collaboration that goes on between vendors and asset owners today. It seems the biggest collaboration nowadays revolves around CIP-013 compliance. Of course, it’s the utilities who have to comply, not the vendors. But CIP-013 won’t work if the vendors refuse to cooperate in filling out questionnaires, improving their practices when needed, etc. (fortunately, they do seem to be cooperating). This is all good, of course, but collaboration on compliance, where the ultimate goal is to help utilities pass audits, isn’t the same as collaboration on security, where the goal is to find ways to make the power grid more secure.

I wondered, “What would be an example of real security collaboration between vendors and asset owners?” Then I realized that I’m involved in one right now: the Energy SBOM Proof of Concept. Here’s why we started this initiative:

1.      There isn’t a huge amount of disagreement in the cybersecurity community that one of the biggest sources of cyber risk is software supply chain risk.

2.      There also isn’t a huge amount of disagreement – although there is some, I’ll admit – that software bills of materials (SBOMs) are a crucial tool for mitigating software supply chain risk. This is because, numerically speaking, there are many more component vulnerabilities in the average software product, than there are vulnerabilities in the relatively small amount of code that was actually written by the company whose name is on the software.

3.      While the supplier bears responsibility for preventing those vulnerabilities from appearing in the first place, and fixing them when they do appear, the customer plays a crucial role, too. They need to let the supplier know (in contract language where possible and feasible, but that’s certainly not the only way to do it) that they want component vulnerabilities fixed as fast as the supplier (usually) now fixes vulnerabilities in their own code. Only an SBOM allows the customer to do that, since that’s the only way they can track the supplier’s progress (or perhaps lack thereof) in fixing component vulnerabilities.

4.      But if SBOMs are so obviously a good thing, and if software suppliers are using them now for their own risk management purposes (which they are, in large part), why aren’t they giving them to their customers?

5.      The reason is there’s still uncertainty about:

a.      The “rules of the road” for production, distribution and use of SBOMs (as well as VEXes. Just as SBOMs are crucial for enabling software supply chain cybersecurity, VEXes are crucial for enabling SBOMs – and it turns out they’re good for a lot more than that). These rules don’t need to be etched in stone and handed down at the summit of Mt. Sinai, but they do need to be agreed upon between suppliers and their customers. And not necessarily on a country-wide or world-wide basis; on the basis of a particular industry or say one part of one industry would be fine.

b.      Tools available to “ingest” SBOMs, so that organizations can use them to learn about vulnerabilities found in the components in the software they use.

c.      Services available to help software-using organizations “consume” SBOMs and use them for vulnerability management, the number one use case for SBOMs.

d.      Support for SBOMs from the vendors of tools for vulnerability and configuration management.

As I said, large numbers of software suppliers – including probably all of the bigger ones – are producing SBOMS for their products now, and using them to manage their own product risk. For example, tools that produce SBOMs will often search the NVD for vulnerabilities that apply to components in the product, giving the supplier a chance either to patch the vulnerabilities or replace a risky component with a less-risky one – before they ship the product. On the consumption side, there’s very little in the way of tools or services now. Fortunately, they’re on the way.

There are two goals of the energy SBOM proof of concept. The first is education. While it’s certainly not necessary for everyone who uses SBOMs to become an expert in them, it does help to have a general understanding of what they are and how they’re put together. That’s why the PoC is currently running “cooking classes”, in which very knowledgeable – but accessible – SBOM practitioners demonstrate how their tools work.

On September 22, Steve Springett, co-leader of the OWASP project that developed and supports the CycloneDX format, demonstrated production of an SBOM in that format, based on an open source project. It was a great presentation, and the video should be up soon on the web site (along with the videos of most of our meetings so far). On October 6, I believe Kate Stewart of the Linux Foundation, who has been the leader of the SPDX project for a number of years (SPDX and CycloneDX are the two “full-featured” SBOM formats. SPDX was just designated an ISO standard), will provide a similar presentation using her favorite format.

We’ll have other cooking classes – including on VEXes – after that. We currently have over 200 people on our mailing list for this track. You don’t have to be in the energy industry to attend the education meetings – you just have to be a user of electricity.

The other track of our project has been slower to start up, but we expect it will start in October. In this track (the meetings for the two tracks are held every other week on alternating weeks), individual asset owners will sign NDAs with individual suppliers. We currently have about six asset owners and six suppliers interested in doing this. The suppliers are all very large ones, whose products are widely used in the electric power industry. The suppliers produce both intelligent devices and separately-installed software.

Each supplier/asset owner pair will decide on one or more products for their exchange. The supplier will provide SBOMs and VEXes for those product(s), and the asset owner will consume them. As I mentioned, the tools and services available on the consumption side are much fewer than on the production side. However, both tools and services are available, and an asset owner won’t be required to create their own software tools to ingest and parse the SBOMs and VEXes.

So here’s where the cooperation comes in: These exchanges won’t work unless the suppliers and asset owners can agree on the rules for the road on SBOMs and VEXes: the fields included in the documents, how the information will be extracted from the SBOMs and VEXes, how that information will be integrated with current vulnerability management processes, etc. As issues come up, the group will discuss them together. And as we learn what works and what doesn’t, we’ll make that information available to the NTIA initiative and to the general public (since every document the NTIA produces is public).

And to be honest, this strikes me as not a bad model for supplier/vendor cooperation in cybersecurity in general. For example, it would be great if NERC put together a team of suppliers and asset owners to work on a document on good security practices for suppliers.

However, that idea has been floated a few times in the past few years, and it always runs into the same hurdle: Everyone at NERC is sure it will never be allowed by the lawyers (although I kind of doubt that they run to the lawyers every time questions like this come up). The reason why they’re sure it won’t be allowed is that it could be seen as providing compliance guidance on CIP-013. Providing guidance is a firm no-no for employees of NERC and the Regional Entities.

Of course RE employees, including auditors, provide compliance guidance all the time. But – except for one employee of one of the RE’s, whom I have quoted many times in these posts – the guidance is never written down and always delivered with the disclaimer that “These are my personal opinions, not those of NERC or my Region.” And ironically, nobody has ever been able to point out to me where in the NERC Rules of Procedure (or even GAGAS) the ban on compliance guidance can be found. Nevertheless, it’s firmly written in the mind of every NERC or Regional employee, even if they provide guidance anyway. And in cases where NERC has provided real guidance on NERC CIP – as in some of the Lessons Learned provided during the CIP v5 implementation period – those were slapped down by the lawyers, and the documents were withdrawn. The most egregious example of this had to do with the controversy over the meaning of the word “programmable”.

So don’t look for much help from NERC on the idea of a forum on supplier best practices. Who else might do this?

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the National Technology and Information Administration’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] I was referring to hardware and software vendors, though. As Ben Miller of Dragos pointed out in the article, service vendors do participate behind the scenes sometimes, since they can easily provide consultation to their utility clients who are directly participating in GridEx.