OSIsoft’s comments at GridSecCon

My last post replayed the comments that Dave Whitehead, COO of Schweitzer Engineering Laboratories, made at NERC GridSecCon two weeks ago, during the panel on supply chain threats that I moderated. Now I’m posting the comments that Bryan Owen, Cyber Security Manager of OSIsoft, made (and when I say “comments”, these aren’t some sort of transcript. These are the two people’s recollections of what they said, although I made it clear they didn’t have to limit themselves to just what they said. I’m interested in the subject matter they discussed, not some sort of historical record). I have a shorter set of comments from Francois Lemay of Hydro Quebec, which I will post later this week or weekend.

As you’ll see, Bryan’s comments took a very different approach from Dave’s. Dave described what SEL is doing to mitigate supply chain risks. Bryan – who has been doing cyber security since before joining OSIsoft in the 90’s as employee number 42, and is very involved with industry security initiatives – discussed important new initiatives for supply chain security, including Software Bill of Materials (SBoM) and Cyber Security Data Sheets (CSDS). In the comments below, I’ve inserted my own comments (identified, of course). All of the links are ones I inserted.

How does OSIsoft address software supply chain threats?

Introductory Remarks by Bryan Owen

Thank you Tom for moderating this panel discussion on supply chain and E-ISAC for another great GridSecCon! Tom asked for a brief highlight on unique aspects of supply chain security.  We’d like to pique your interest for further discussion during question and answer.

Let’s discuss software supply chain in the perspective of the 3Cs: Content, Communication, and Coordination.

Content

Software supply chain assurance is considered by many to be an endless problem.  Unlike the physical world, software tends to be opaque – it’s hard to tell what’s really inside of a digital system.

Heartbleed in 2012 was the eye opener. Everyone scrambled to find where OpenSSL was used in their code. We conducted an exhaustive search of code repositories. Hidden dependencies were found and diverse versions were in play.

In addition to the immediate remedial action, it became obvious new tooling would be needed to manage software supply chain issues in a reliable and proactive manner.  Yesterday’s workshop on software and firmware supply chain had exercises using state-of-the-art tools to decompose software packages and identify problem components.

Software Bills of Materials (SBoM)s can provide a step forward to address transparency in software components.  Today SBoMs are quite limited (Tom’s link) such as enumerating first-party dependency. Nonetheless, simply being able to provide an SBOM is a signal of good development practices even if not as actionable as ultimately needed.  Longer term, the Department of Commerce NTIA initiative is expected to harmonize SBoM with use cases in mind.  NTIA is seeking more stakeholders – it would be great to see participation from the electric sector.

Note from Tom> I have liked the idea of SBoM since I first heard about it a couple of years ago, in reference to the medical device industry (which has mandatory cyber regulations from the FDA – although they don’t cover SBoMs). However, the article I linked in the paragraph above shows how hard it will be to get these into something that is really trustworthy and understandable. But that shouldn’t stop NERC entities from asking (or requiring) their software suppliers to provide them.

As Bryan points out, the real benefit of SBoMs currently is that just asking your software supplier to produce at least a partial SBoM (i.e. one that doesn’t go down more than one level – to your supplier’s suppliers, in this case) will at least tell you something about the supplier. If they blow you off entirely, or insist that they can’t provide that information (they’ll say it’s for competitive reasons, but you could point out that you’re in competition with the Russians every day for control of the grid, so you think that should override their concern that another supplier might learn from an SBoM about a code source they hadn’t considered), you need to consider them high risk for this particular threat. Of course, they may say they don’t have any third-party code in their product, and in that case you could reply “Great. So I guess you won’t mind if we insert a clause prohibiting third party components the next time we renew your contract?”

Communication

Communication about software hazards and mitigations in a product is important.  The chemical industry standardized on Material Safety Data Sheets decades ago. We believe a similar approach can be implemented as a Cyber Security Data Sheet (CSDS).  EPRI, SANDIA and major Nuclear Power Generators are moving just such an initiative forward with the Cyber Security Technical Assessment Methodology.  Get EPRI 3002012752 and start asking suppliers for a reference CSDS.  Create and share CSDS as a community of interest.

Note from Tom> Having learned about Material Safety Data Sheets as part of my safety training when I joined Honeywell in 2010, I like the idea of CSDS. The EPRI document referenced discusses that, but if you’re not part of an EPRI member organization, you’ll have to pay $300 to get it. I’m told EPRI will have some free information on it in the near future. In the meantime, Bryan gave me a link to an OSISoft presentation that provides some information on CSDS (which is part of a larger EPRI project called TAM). Slide 26 shows an MSDS, and slides 27-44 provide information on TAM and CSDS. Slides 40 and 44 have especially useful information). See below for an email Q&A I had with Bryan this morning on CSDS and TAM.

Coordination

The need for coordination of vulnerability and incident handling is amplified by software supply chain issues.  Such incidents tend to involve one-to-many or many-to-many organizations.  Formal and professional coordination becomes a necessity.  Speculative execution and Urgent 11 are examples of the far reaching impact of supply chain issues. 

With today’s hype cycle, it’s especially important for trusted authorities to vet fast breaking news such as a breach announcement. CISA ICS-CERT and E-ISAC have proven most effective to review and disseminate facts without the hype in a timely and efficient manner. 

Questions from GridSecCon Supply Chain Panel

1.       Question> How often do you review the open source software you depend on? Open SSL was mentioned but the whole of Linux (heavily used in embedded systems) is open source. Do you take steps to financially support open source software to ensure longevity?

Bryan> Continuous integration methods and tooling are used to assess risk in open source software dependencies, typically as part of a daily build.  The assessment includes legal, operational and security risk – for instance, whether an open source project appears to be abandoned or unsupported.  Our approach to longevity is to never leave a customer behind; we ensure that support or replacement is available. 

Note from Tom> One of the five supply chain security guideline papers currently on the NERC CIPC website deals with open source software risks. You can find it by going here and dropping down the CIPC Security Guidelines tab.

2.       Question> IoT devices are plagued with security vulnerabilities due to severely outdated operating systems and other supporting software embedded into the components. Will testing verify current and valid software is used on ICS hardware?

Bryan> Findings from CyberITL.org and others confirm IoT device manufacturers lack accepted security development lifecycle (SDL) practices.  Since SDL involves cultural change as much as technical, we should expect this to continue for several more years including devices in the pipeline for DER and the greater vision for smart grid. Simple tools like Attack-surface Host Analyzer from Washington State University can help rapidly identify devices; in most cases formal testing is overkill for IoT and IIoT.    

Q&A between Tom and Bryan on Cyber Security Data Sheets (CSDS)

1. Is EPRI's program now the standard - or is it trying to be - for developing CSDS's in the power industry? 

Bryan> There are a number of threat modeling methodologies for software development, but EPRI TAM appears to be unique in addressing the chasm between suppliers and system integrators/operators.

2. Is EPRI going to endorse CSDS's that suppliers prepare on their products, if the supplier follows EPRI's methodology (takes the course, etc.)?

Bryan> The methodology expects entities and third parties to be creating most of the CSDSs – at least initially.  Suppliers that provide a reference CSDS will significantly reduce effort to fully execute the methodology.  CSDS created by suppliers and others can be distributed without permission from EPRI, but must acknowledge EPRI.

Note from Tom> Bryan let me see a CSDS for one of OSISoft’s main software products. I had two reactions: a) Calling this a “sheet” is a little bit of a misnomer. There were two Word documents of about 53 pages total, plus a voluminous Excel workbook, as well as about 15 or 20 other files; and b) There’s no way this should be in any way be made public, except to customers, and even then there will need to be strong security requirements to keep it from being released inadvertently. For example, the CSDS I saw lists all of the attack vectors for all of the components of the product. What a great present for the Russians!

3. Are there other ways to do CSDS's? I'm thinking that if a utility requires that suppliers prepare EPRI-standard CSDS's for their products, and a smaller supplier looks at what's going to be involved with the EPRI method, they might just back off from selling their product to the utility in the first place. 

Bryan> Any suppler large or small that sanctions use of their product for a high impact BES Cyber System should be developing threat models and providing security documentation to operating entities. OSIsoft got into it because of nuclear generation and found the level of rigor was appropriate. For example about a person-month of engineering for a modestly complex product.  While potentially prohibitive for custom applications, the CSDS makes sense where the same technology is being used by many operating companies.  Most importantly, without good security documentation, the operating company still has to protect the asset and would be advised to produce their own CSDS for anything being used in High or Medium impact BES Cyber Systems.

Note from Tom> I think the complexity of the process of preparing these might call for someone to develop a “lite” version of the CSDS (it would probably have to be called something else, which is fine). The problem is that, if you’re the first utility on the block to require your supplier to develop one of these and you require the whole TAM process, they’re likely to wonder “Given what it will take to do this, do we really need their business?” Of course, if all NERC entities required this of all suppliers, that would be a different story – and a sure-fire antitrust violation.

In any case, this will have to be a case where utilities start asking their suppliers about this, but not absolutely requiring it; maybe on the twentieth question a supplier will wave the red flag and develop one (Bryan suggests that you may just want to ask this of your vendors of High impact BCS components, especially your EMS vendor). Then their competitors will realize they need to do the same thing…and after perhaps years this will be standard practice for the industry. But I do think there needs to be a lite alternative – less filling, but still tastes great – to speed up the adoption rate. EPRI might want to consider that.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that if you’re a NERC entity, Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post – especially on compliance with CIP-013. My offer of a free webinar on CIP-013, specifically for your organization, remains open to NERC entities and vendors of hardware or software components for BES Cyber Systems. To discuss this, you can email me at the same address.