Solving the vendor problem (both of them!)

Introduction

I have been thinking about vendors a lot this fall, for three main reasons. First, the Russian attacks that were played up by DHS this summer (and I owe a new post on that subject – please don’t cancel your subscription. It will come soon) came entirely through vendors. It doesn’t seem like the Russians succeeded in their goal of implanting malware in the systems that control the grid, but they definitely were able to penetrate a lot of vendors. Second, I’ve been doing work for a vendor who is considering registering with NERC as a Generator Operator. And third, CIP-013 has a lot to do with vendors, in case you hadn’t noticed that.

It seems to me there are two big problems related to power industry vendors (at least those that sell control systems. However those are a pretty significant fraction of total product vendors to the industry, since it seems there are very few products – well, maybe insulation and wire – that don’t come with their own control systems, or at least the capability of being controlled). One is the industry’s problem, the other is the vendors’ problem. I will deal with these separately.

The industry’s problem

The industry’s problem is due to the current big concern with supply chain security, and with CIP-013 compliance in particular. The problem can be simply stated: On the one hand, having good supply chain security requires getting vendors to do a number of things, six of which are described in CIP-013 R1.2 (although these are far from being the only concerns that the industry has with vendors). On the other hand, at least for larger vendors, individual utilities don’t have much leverage to get them to do something they don’t want to do.

Moreover, the industry can’t gang up on a recalcitrant vendor and say “Mr. Vendor, either you take the actions on this list or you can say goodbye to any further power industry business.” This would be a clear violation of antitrust laws. Anyone who has attended even just a few NERC meetings probably can cite the antitrust disclaimer by heart. It specifically forbids any discussion of vendors at the meeting.

So what does the industry do about this? Unfortunately, some people – including, it seems, many at NERC – seem to think that the ultimate solution to this problem is getting vendors to agree to certain contract language. Their thinking seems to be that, as long as a few words are on some parchment somewhere, the “vendor problem” is more or less solved. In fact, I heard a senior NERC official summarize CIP-013 for a group of security professionals as “a standard for incorporating cyber security language int0 vendor contracts” – or words to that effect. This person was clearly saying that contract language was the whole purpose of CIP-013, although he backtracked when I questioned him on it afterwards.

I wrote a post on contract language early this year, and followed it up with a post summarizing an email exchange with an auditor, which was prompted by the original post. The point of both posts was that, while you do need to be able to demonstrate that you tried to get a vendor to do certain things to comply with CIP-013 R1, and while there needs to be some written documentation of that agreement, this in no way requires contract language itself. In fact, I think of contract language as the last resort, not the first one, for these reasons:

1. It is by far the most expensive way to get a vendor to commit to doing something, since it will require lots of lawyers’ time on both sides – and this is even assuming the vendor is willing to negotiate in the first place. If they won’t negotiate this, you have to start examining other vendors and determining what kind of contract language they’ll accept, then threaten to leave the original vendor if they don’t give in. None of that is easy or fun to do.
2. It is even more “expensive” in the political sense, since hammering your vendor with lawyers is a terrible way to achieve what should be your primary aim: having a close relationship, based on mutual trust, in which both parties work together to achieve the common goal of secure products and services.
3. You’re still going to have to provide some documentation that the vendor is actually doing what they promised to do in the contract. This is because, contrary to the NERC official I paraphrased earlier, CIP-013 is a standard for management and mitigation of supply chain security risks, not contract language – which is just one of many possible mitigation measures. Just getting a vendor to put language in a contract doesn’t in itself mitigate one iota of risk. You need to take steps to mitigate each risk, and if your first steps are contract language and they fail (either because the vendor wouldn’t agree to the language you wanted, or because they agreed to it but didn’t live up to the agreement), you still need to do something to mitigate the risk, either with or without the cooperation of your vendor.
4. Think of this: What if you get your vendor to agree to your contract language, but they don’t live up to the agreement? You then have your lawyers send them multiple letters, and they ignore all of them. Furthermore – and this is very commonly the case – suppose that switching to another vendor would be difficult, if not impossible. What’s your leverage at this point? I guess you can sue the vendor – and with a little luck, in three years you may get some money from them, or at least a promise to actually live up to the language they agreed to. How has all of this in any way made the Bulk Electric System more secure? All of this time, you’ve presumably been without protection against the risks that the vendor refuses to mitigate.
5. Perhaps most important, even if you get the vendor to agree to your language, you very well may not be allowed by your lawyers to show the auditors the main piece of evidence: the contract. Because of the special provisions in the requirement, the auditors won’t be able to demand you show them the contract, but – as an auditor pointed out to me in the second of the two posts linked above – they will be able to require that you show them some written evidence that the vendor agreed with the language. That might be a separate letter, or even an email. But in that case, what’s the point of the contract? The vendor is agreeing in a letter to do what you’re asking. Why not just go for the letter in the first place, and save thousands of dollars and maybe a couple months of delay by not worrying about the contract in the first place?

So contract language isn’t the magic bullet for getting vendors to take security seriously. Then what is? How about certification? There are various certifications that the vendor can achieve (e.g. IEC 62443), which will presumably demonstrate that their products are secure. Why not just require they get this certification? Well, for one thing, getting these certifications is very expensive. Unless a lot of other NERC entities are demanding the same thing, you’re unlikely to be able to force a vendor to do this, unless you’re negotiating a huge contract with them.

Well, how about getting a bunch of friends at other utilities together and all demanding the same certification of the vendor? Or maybe you can “name and shame” the vendor in industry forums or online? You know the answer to that: You (and your friends) will be violating antitrust laws. The fact is that there isn’t currently any way that one or more NERC entities can force a vendor to comply with a certain set of cyber security requirements, if the vendor really doesn’t want to do it and is prepared to sacrifice some industry market share because of that decision.

The vendors’ problem

Let’s turn now to the other problem: the one the vendors have. That problem can be stated very simply: Even before CIP-013 made its appearance, but certainly after that, vendors have been trying to figure out a way to head off demands for contract language at the pass. They know how expensive it is to negotiate contract language with just one customer. What do you do if every one of your customers starts coming at you with their own customized set of contract language (in some cases, simply downloaded from the internet, and having little or nothing to do with threats to control systems)? I know of one vendor that complained of exactly this situation in early 2018, and I’m sure they’re not alone. How can a vendor assure their power industry customers as a whole that they have good cyber security?

Now let’s go back to the second reason why I’ve been thinking about vendors lately: because I’ve been working for one. This vendor – who sells a product extensively used in power generation, and only in power generation – contacted me over the summer and said they wanted to register with NERC, so that they could be subject to compliance with the NERC CIP standards.

I naturally first thought they should see a psychiatrist, not me, but as I worked with them I began to see why they want to do this: They ultimately want to sell additional services to their customers that would require their having a secure environment to deliver those services. What better way to do that than to point out to your customers “Hey, we comply with NERC CIP, just as you do”? Complying with CIP is expensive, but negotiating a bunch of security contract language with every customer is probably even more expensive, and much less productive.

Does this solve the vendors’ problem? I think it does. While NERC entities might have differences on what would be an ideal standard to have vendors comply with, I think they will (almost) all agree that a vendor that complies with CIP will have good security, for the simple reason that the NERC entities themselves are CIP compliant and (hopefully) have good security – why shouldn’t that be the case with a vendor as well? In fact, NERC entities could simply put wording in their RFPs and contracts saying that the vendor must be registered with NERC and compliant with the CIP standards. Since they’re not ganging up with other NERC entities in doing so (and I’m sure some NERC entities wouldn’t want to have this language in their contracts – that’s fine), I don’t think there’s a question of running afoul of antitrust law.

But I suppose some lawyer might find a reason why even this violates antitrust laws. In that case, nobody needs to mention CIP compliance in their RFPs or contracts. I contend that many vendors will still feel just as compelled to become CIP compliant as they would if forced to by a contract – especially if a lot of their customers make clear to them that it would be really nice if they became CIP compliant.

Is this a solution to both problems?

And how about the industry’s problem discussed at the beginning of this post, which – to refresh your memory – is that they don’t have a way of enforcing standard cyber security practices on vendors? Will having a way for vendors to register with NERC and comply with CIP solve this problem? I believe it will, with one caveat: This can’t be a forced process. For one thing, NERC and FERC don’t have any jurisdiction over vendors, so they certainly can’t force them to register – as they can with power market participants.[i]

But this is an advantage, not a drawback, of this idea. If vendor registration is voluntary, there can be no question of antitrust violations, as there would inevitably be with any sort of forced registration. If a vendor doesn’t see the need to register and comply with CIP, fine. The NERC entities will be able to decide individually in what cases they will continue to be a customer of a non-CIP-compliant vendor. Some entities might want all their OT hardware and software vendors to be registered and CIP compliant; others might want just their most strategic vendors to be registered; while others (perhaps smaller entities with only Low impact assets, who don’t have to comply with CIP-013) would continue business as usual with their current vendors, CIP compliant or not.

Meanwhile, some vendors wouldn’t want to lose a single power industry customer, and will be the first to line up at NERC’s door, demanding to register and be made to comply with CIP. Others (probably larger ones that sell to lots of industries, not primarily the power industry) might assign someone to start studying this issue and figuring out what it would cost to register and become CIP compliant. If they decide the costs are less than the benefits (retained power industry customers), they’ll probably register; if they don’t, they probably won’t. In the end, you’ll have a mix of CIP-compliant and non-CIP-compliant vendors, and a mix of NERC entities that want their vendors to be compliant and others that couldn’t care less about vendor CIP compliance. All voluntary – no coercion. Life is good.

And there’s a special benefit for NERC entities that are subject to CIP-013 (i.e. ones that own or operate High and/or Medium impact assets): I think that they will be well advised to include registration and CIP compliance in all of their RFPs, as well as in future contract language (again, subject to antitrust vetting). With one fell swoop, they will be gaining a level of assurance about their vendors’ security, and – most importantly – their CIP-013 auditors are very likely to share that level of assurance (after all, CIP-013 auditors are likely to already audit the other CIP standards). If a vendor has passed a CIP audit, the entity will be assured that they have pretty good security.

Of course, this can’t be codified in the CIP-013 requirements themselves – i.e. the next version of CIP-013 can’t include a requirement that the entity’s vendors be CIP compliant; this would again be a pretty clear antitrust violation. But it won’t have to be codified: Since there will likely need to be “vendor” versions of the CIP standards (see below), it will be easy for auditors and NERC entities to weigh in on the standards drafting process, to make sure that “Vendor CIP” includes the types of requirements that the industry thinks are necessary for vendors. As a result, Vendor CIP will ipso facto be as close as possible to an “industry standard” for vendors, but the fact that compliance will be audited makes this a standard that will definitely be respected.

A couple problems, and solutions

However, all of this isn’t a straightforward proposition, for two reasons. The bigger reason is that, even though vendors typically won’t fulfill the tasks of any of the NERC Functional Model designations, they still need to register as something. My client chose GOP as the closest designation to what they actually do (and that’s not very close). But a GOP doesn’t just have to comply with the CIP standards – they have to comply with a number of Operations and Planning standards, very few (if any) of which have anything to do with what my client actually does.

Rather than have to pretend to comply with all of these other standards, my client will most likely try to get their compliance scope reduced by their Regional Entity for most of the Operations and Planning (i.e. “693”) standards, although there’s no assurance they’ll be allowed to do this. So it’s possible they will have a big compliance documentation burden for the 693 standards, even before CIP comes into the picture.

What would be much better would be if there were a NERC Functional Model designation for vendors – let’s call it VEN. The definition of VEN would be something like “an entity that has the capability to monitor or control BES Cyber Systems in real time, through remote electronic access or on-site physical access, but does not otherwise meet any Functional Model designation”. A VEN would probably only have to comply with CIP, although I’m not ruling out some other NERC standard that might apply to them. But a VEN certainly wouldn’t have to comply with the full range of O&P standards.[ii]

Of course, there’s another vendor infrastructure that needs to be secured besides the infrastructure (and people) used for remote access – and this is far more challenging. I’m speaking of the systems and processes used in designing, manufacturing and integrating the products the vendor sells, and even more importantly those used by the vendors’ suppliers (and those suppliers’ suppliers). Were a line of motherboards to be compromised in China so that they phoned home information about power flows, etc. (as was alleged to have happened by a long article in Bloomberg Businessweek. The credibility of this article has been called into question, but in principle this sounds like something that could happen. Of course, if true this would pose a very serious threat, and one that is very hard to mitigate).

So maybe there should be two designations. VE1 is the one I described above. VE2 could be something like “an entity that designs, manufactures or integrates hardware or software products used in BES Cyber Systems”. Vendors might sign up for one or both of these designations.

However, as just discussed above, I think there should really be a vendor version of CIP – although it’s certainly not out of the question that a vendor would be able to comply with the “regular” CIP standards (and my client intends to do exactly that). The problem isn’t so much that the vendors can’t comply with the current CIP standards, as that the BES Cyber System definition (including the BES Cyber Asset and Cyber Asset definitions) doesn’t apply to all of the systems that should be in scope for the vendors.

While it would be interesting to sit down and try to draw up a detailed Vendor CIP, for now I’ll just list what I think should be some of its main features (and challenges):

• There is a real applicability problem, since a vendor by definition doesn’t own BES Cyber Systems. Even if they regularly operate BCS, they’re the customer’s systems, and the customer itself needs to comply with CIP for them. There might even need to be a new type of Cyber System called a “Vendor Control System”, which could be defined as something like “A system used to monitor and/or control BES Cyber Systems, whether remotely (which would mean from the vendor’s own remote control facility) or onsite (which would usually mean a vendor laptop used by a technician to control the vendor’s systems while onsite at one of the customer’s BES assets)”.
• Since there are two types of vendors, VE1 and VE2, there could be two types of Vendor Control Systems: "Vendor Remote Access Systems" are in scope at VE1 vendors, and “Vendor Development Systems” and “Vendor Manufacturing Systems” are in scope for VE2 vendors. The requirements for these systems would obviously be very different (for one thing, Secure Development Lifecycle would be a big component of the VE2 CIP standard(s), while the VE1 standard(s) would be very focused on securing any system that would ever be used for interactive or machine-to-machine access to customer BCS).
• Now that I think of it, developing requirements for the VE2 vendors will be much more involved than for the VE1 vendors. Since the Russian attacks show that the latter are at the moment a much bigger threat to the BES, it might be best to first concentrate on developing CIP standards just for them, unless both can be done at once.
• Along with controls for Interactive Remote Access (i.e. remote access to the vendor's own critical systems, from their vendors or employees), VE1 vendors would also need controls for “Interactive Remote Monitoring or Control” (IRMC) – i.e. what they do for their customers’ systems. There would also need to be personnel controls for these vendors (especially for employees that either come onsite to customers or access customer systems remotely) and of course regular computer controls like patch management, ports and services control, etc. (since both the systems used for IRMC and those used for machine-to-machine access to BCS clearly pose a high risk to the BES).

As I said, I could go on and on about this subject, but this post is already long enough. Please let me know what you think of this idea.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013; we also work with security product or service vendors that need help articulating their message to the power industry. To discuss this, you can email me at the same address.

---

[i] Of course, there are a small number of vendors who are power market participants. These include vendors that provide outsourced services to NERC entities and other power market participants – services that those entities would otherwise have to provide themselves. For example, an organization that runs power plants owned by NERC entities – probably from the vendor’s own Control Center – is a true GOP, and I’m sure they will be compelled to register as such, if they don’t do so voluntarily. Of course, these aren’t the vendors this post is about. It’s about vendors of hardware and software products, as well as vendors who provide services on a contractual basis and don’t actually take on responsibility for the services of one of the NERC Functional Model designations.

[ii] I wish to point out here that great minds think alike, and it seems that Tobias Whitney of EPRI (and previously of NERC, as I’m sure many of you know) also came up with the idea of having vendors register with NERC and comply with CIP – he brought it up in a discussion of cloud vendors at the NERC CIPC meeting in Atlanta a few weeks ago. I discussed this with him after the meeting, and he hadn’t thought about the idea that there would need to be a new Functional Model registration for vendors. However, I don’t think many vendors will register if they may be on the hook for at least some of the 693 standards as well, even though they really don’t perform functions that need to be regulated by NERC, as cyber security does.