How does SEL address supply chain vulnerabilities?

As I’ve already mentioned, I was quite pleased with how the panel discussion that I moderated at GridSecCon last week turned out. I received lots of good feedback and compliments. The topic of the panel was – of course – supply chain security threats, and we had quite a distinguished group of presenters. One of them was Dave Whitehead, Chief Operating Officer of Schweitzer Engineering Laboratories.

The biggest problem I had with the panel was that, except for whatever notes people took while we were talking and answering questions, there was no record of what was said. I’ve appeared on a panel – the same three people – at RSA Security Conference this year and last, and both times the session was recorded and posted on the RSAC website (although both recordings seem to have been taken down now, as part of some sort of reorganization of the web site. Maybe they ought to just outsource the whole site to YouTube).

I would like to see the E-ISAC post recordings for the panels at GridSecCon next year; it’s a little discouraging to know that whatever you say is floating out into the ether, never to be heard again. Slides from individual presentations are posted on the website, but the problem is that the panels are only allowed one slide per person, which doesn’t tell you much.

To preserve at least some of what the panelists said – both in their initial four-minute presentations and in their answers to questions from the audience – I asked them all afterwards to provide me with some written record of both their presentation and their answers. But I also suggested they could go beyond what they actually said, if they’d like. They addressed a lot of interesting topics, and they might want to expand on what they actually said. I said I’d post all of these in my blog.

So far, three of the five panelists have taken me up on this (and if the other two don’t do it, that’s fine. I know that people are busy, and they didn’t sign up for doing this when they accepted the invitation to speak). I’ll publish two of them next week, but now I’m posting what Dave Whitehead sent me (this is verbatim what he sent, although I’ve made a few formatting edits). 

I found Dave’s summary to be a great guide to how a quality company secures their own supply chain – and some of these could be considered best practices for electric utilities as well. And by the way, we were extremely pleased that Dave decided to make the long flight from Pullman, WA to Atlanta to speak at GridSecCon. It’s not like he doesn’t have anything else to do with his time…

How does SEL address supply chain vulnerabilities?

Introductory Remarks by Dave Whitehead

At SEL, our supply chain is global and complex. We take a comprehensive approach to securing every facet of our supply chain – from the moment we bring on a new supplier through the lifecycle of the products we deliver to our customers.

There are eight (8) pillars that we constantly assess to ensure a dependable supply chain:

1: Build Trusted Supply Networks

• We hold an annual supplier conference at our HQ in Pullman, WA, bringing in more than 200 companies that we source components and products from to share our technical needs and strategic objectives for the coming year.
• Conduct regular ON-SITE audits of our suppliers.
• And since we invent, design, and manufacture all our products here in the U.S. (Pullman, Lewiston, and Lake Zurich), we NEVER share our bills of materials or design schematics in order to avoid disclosing vendor product and part information. 

2: Rate Suppliers Risks

• We have a custom supplier rating system that we call PQFIDS – price, quality, features, innovation, delivery, and service.
• We also evaluate our suppliers’ suppliers, and to the greatest extent possible, we source within the U.S. – even if it costs more. 

3: Ensure Component Integrity

• SEL verifies the performance of ALL purchased components against our supplier product specs.
• We continuously test our products throughout the manufacturing process.
• We take additional steps to ensure the integrity of the components in our products. For example, we use x-rays, inspect packaging, and consult the manufacturer’s design drawings. (I invite anyone who wants to come to our HQ in Pullman to visit and check out our x-ray machines, manufacturing operations, and R&D facilities). 

4: Keep Track of Components and Products

• This is simple enough. We keep a detailed record of every product we manufacture so that we know exactly where our products are installed, making it easy to notify our customers about potential quality or security issues.
• We have an outstanding warranty that is essentially good for the lifetime of the product at no cost. Outstanding warranties encourage customers to return products if there is an issue, which in turn allows our engineers to learn how to make our products even better. 

5: Ensure Component Availability

• With our detailed record keeping of products we manufacture, we are able to quickly identify which parts are at risk of becoming depleted – for example, in the wake of the 2011 Japanese earthquake, we identified the products that were at risk, purchased additional inventory, and continued to provide products to our customers around the world. 

6: Collaborate with Customers and Industry

• We encourage customer inspection and feedback. And we invite our customers to tour and observe our manufacturing facilities and processes.
• We participate in various government-led initiatives and standards development activities so we can be aware of the current best practices of others and contribute our best practices.

7: Build Security into Company Practices

• We own every line of source code, and do not share outside of SEL.
• We have robust testing practices onsite at SEL, by SEL employees.
• Our projects are compartmentalized, on a need to know basis.
• We embrace simplicity of design to create resilient control systems and product architectures.
• Cybersecurity is embedded from the earliest stages of product development, not on the back end. We START with cybersecurity. We enforce strict security practices for our employees and visitors.
• We get to the root cause on every failure, because every single one is significant. 

8: Ongoing Risk Management

• At SEL, our executives are constantly monitoring for emerging risks and threats to our supply chain. It is part of our daily lives.
• We develop the majority of the software our products use, and if we use third-party software, we always acquire the source code.
• SEL makes many of our product components in-house. This type of vertical integration allows for high quality products and sustained growth in product expertise.

Questions from GridSecCon Supply Chain Panel

1.       Q: I was told that SEL is concerned about counterfeit equipment. Is this true? How would a customer avoid purchasing counterfeit equipment?

A: We have very little concern that our customers will receive counterfeit equipment, because all our products are invented, designed, and manufactured at SEL, and shipped directly from SEL. So this is very little concern for us.

If by chance you receive a counterfeit SEL product, it is probably because you ordered it off of eBay or some other internet website and did not purchase directly from SEL. And if that is the case, I have to wonder what on earth you are doing ordering critical infrastructure equipment off of eBay?!

2.       Q: How can we assure that equipment isn’t tampered with while being delivered?

SEL provides serial numbers for products and firmware version numbers for code that goes into our electronic devices. Along with the firmware version we provide a cryptographic hash, e.g. a unique digital fingerprint, of the firmware image. The hash allows customers who receive firmware from SEL to independently verify the firmware sent by SEL hasn’t been modified.

  

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. My offer of a free webinar on CIP-013, specifically for your organization, remains open to NERC entities and vendors of hardware or software components for BES Cyber Systems. To discuss this, you can email me at the same address.