Wednesday, August 19, 2020

Is software bill of materials the answer to all of our problems? Part II



If you're looking for my pandemic posts, go here.

The first part of this post described a pretty scary problem: the fact that just about any software you purchase nowadays includes lots of third-party or open source components; those components contain components; those components of components contain components, etc. etc. A 2017 study by CA Veracode found that “83 percent of developers use code components to build web applications, with the average number of components per application (found to be) 73.” And a vulnerability in any one of these components might prove serious.

I used the recent example of the Treck TCP/IP stack, which was developed in the late 1990s (back in the days when internet security was hardly even an afterthought. I remember visiting one of the national labs at that time and learning that they didn’t even have a firewall. Moreover, they had internet-addressable IP addresses on all their computers, since – of course – the whole idea of the internet at the time was that it was to allow free collaboration; I don’t think it even registered with me that this might be a problem. How quaint that time seems today!).

Recently, researchers in Israel identified 19 vulnerabilities in the Treck stack, and they think there are at least twice that many still to be identified. Four of the 19 vulnerabilities were deemed to pose the highest level of risk per CISA. Fortunately, Treck is still in business and they were able to quickly develop patches for all 19 vulnerabilities.

But what’s the problem? The Treck stack is probably embedded in hundreds of millions of devices. Many – if not most - of the software suppliers whose products include the Treck stack probably have no idea it’s there. And since it’s been on the market for more than twenty years, it might very well be embedded in components that are more than ten levels down in a piece of software installed today. Let’s be honest: there’s no way more than a small fraction of the total instances of the Treck stack in use today will ever be patched. So patching really isn’t a good first-line mitigation for this risk.

What is the best mitigation? It’s pretty simple to describe: Your software supplier should have a complete software bill of materials (SBoM) that lists each third-party and open source component of their software, as well as the version number and supplier name (probably other information as well). Each of those components should have its own SBoM; each of those components should have an SBoM, etc.

If you get the full SBoM from the supplier, when a new vulnerability is announced in a particular software component, you’ll be able to do a simple search to find all instances of that component – if any – in each piece of software that your organization runs. If you have any hits, you’ll then compare the version numbers listed in the vulnerability notice to determine whether you’re running any instances of that version. If so, you’ll know exactly what machines to apply the patch to, and all will be good.

But to tell the truth (which I like to do every now and then. Breaks the monotony), even patching a few components will be a painstaking job, and in some cases literally impossible if the component’s code is compiled into your supplier’s product. But not to worry; this is really your supplier’s job, not yours. All you need to do is only to engage with a supplier that:

1.      Has a complete multilevel SBoM for their products, as described above;
2.      Will provide you the complete SBoM, so that you can contact some of the third party suppliers (or open source communities) and be on the list to be notified of new vulnerabilities and patches;
3.      Has a policy of always buying source code for any third-party component, meaning that if the third party goes out of business or for some other reason can’t patch a newly identified vulnerability in their code, your supplier will be able to develop their own patch for their customers (of course, they need a high level of expertise to do this. A 3-month course in Visual Basic™ – if they even have those anymore – doesn’t count);
4.      Will commit to patching any vulnerability discovered in any of their third party or open source components within say five days;
5.      Will commit to keeping track of the suppliers of all third-party components, to make sure they’re still in business and still able to provide patches when needed – and if your supplier determines one of their third parties isn’t long for this world, they commit to replacing that code in their software within say one month;
6.      Will commit to keeping track of the online community supporting each open source component of their software – and if the supplier determines that the community has dwindled to the point that patches aren’t being developed anymore, they will commit to replacing the code within say three months;
7.      Before releasing a new product or a new version of a product, will commit to scanning each component for vulnerabilities;
8.      When a third party supplier or open source community announces a new version of a component and end of support of the version incorporated in your supplier’s software, your supplier will either upgrade the component to the new version or find and install another component of equal functionality; and finally can
9.      Bend 6-inch wide steel bars with their bare hands, penetrate walls with their X-ray vision, and leap tall buildings with a single bound.

In other words, current reality is vastly different from the above. Here’s the true story:

·        Few suppliers could provide you with a complete SBoM of even the first level of components in their software, and that won’t often have even the name of each supplier or open source community behind a component, even if it identifies each particular component.

·       If they do have a complete SBoM (or say they do), they may refuse to give it to you for “competitive reasons” – meaning they don’t want to take the chance that you’ll give the list of ingredients in their cake to a rival cake-maker.

·       Very few suppliers would buy source code from every third-party supplier, even if it were available to them. The only reason I even know about this concept is it’s described in a NIST best practices paper about Schweitzer Engineering Laboratories (which has that policy). Plus I doubt a lot of suppliers would even have the level of expertise that SEL has, to develop their own patches for third party software.

·       I don’t have any data on this, but I’m willing to guess that few suppliers will commit to patching component vulnerabilities within any period of time, let alone five days. And even if they did, it would be a meaningless commitment if they didn’t at the same time commit to telling you about new vulnerabilities, even if they’re unpatched so far (which of course is what CIP-013 R1.2.4 requires); but it will be quite hard to get suppliers to commit to doing this. Yet if they only tell you about vulnerabilities they’ve already patched, they’re not committing to anything new when they “commit” to patching all vulnerabilities in five days.

·       The 2017 Veracode study quoted above also states “A mere 52 percent of companies reported they provide security fixes to components when new security vulnerabilities are discovered. Despite an average of 71 vulnerabilities per application introduced through the use of third-party components, only 23 percent reported testing for vulnerabilities in components at every release.”

·        You get the idea: Except for a few exceptions, your supplier is unlikely to commit to a single one of the above items. Probably the most likely one they’ll commit to is leaping tall buildings in a single bound.

So what’s a poor NERC entity (or any organization, for that matter) to do? Is there no shining knight that will ride in and save you from this menace of vulnerable third-party components? Believe it or not, there is a government-led effort underway now to develop a standard for developing and integrating SBoMs. The effort is led by Dr. Allan Friedman of the National Telecommunications and Information Administration of the US Department of Commerce. He is leading a large collaboration with industry and universities to develop a standard for generating and accessing software bills of material. This document from NTIA provides a good introduction to the SBoM concept, as well as links to other information about the NTIA initiative.

However, let’s be honest: It will be years before this initiative will lead to widely-available SBoMs. Meanwhile, the threats described above are here now. What can you do in the meantime? I asked two software experts I know for their thoughts on this question. In this post, I’ll describe what George Masters of SEL told me. I’ll discuss what the other expert said in part III of this post.

I know George primarily from working with him last year on the CIPC Supply Chain Working Group. We both led the development of guideline papers. His was on “Risk Considerations for Open Source Software”. The paper is excellent and it (as well as the webinar George presented on it) can be accessed here. I learned a lot just from the meetings he led while developing the paper.

I asked George whether it would be realistic to ask software suppliers to provide at least a first-level SBoM. Here is what he replied:

My advice to people who mention this in conversation is “be careful what you ask for”, because using SBOM information to form a competent risk assessment requires information that a customer just won’t have.  He or she would not, for instance, be able to recognize an older component that had been selectively patched to address particular vulnerabilities, nor would they be able to know about other controls that mitigate an issue, or that vulnerable code may not be reachable at all.  

Given the reliability expectations for power system equipment, “churn” in the code is something to be avoided.  Every change should be carefully reviewed and thoroughly tested.  Entirely updating a component that may contain hundreds of changes involves some degree of risk – and cost - and that has to be balanced against security gains.

Yes, the supplier should be able to demonstrate knowledge of the product composition, and that they are tracking those components’ version releases, but one shouldn’t expect to be able to second-guess them with an SBOM.  If the supplier is competent, this is wasted effort.   Suppliers may also want to keep composition data proprietary, disclosing details only under NDA.  

Reading the foregoing, as you might expect, I’d say that getting beyond a ‘top level’ SBOM would make things worse.

I continued by asking George “Do you think there's value in even asking for an SBoM? If for no other reason than it shows whether or not the supplier even knows what's in their software.” I said this because I’d read another article that said the only real value to be obtained from asking for an SBoM at the current time is to see if the supplier can respond to you at all – that is, whether they even have any idea what the h___ is in their software. If they clearly don’t know that, then you should question why you’re even doing business with this supplier in the first place.

George’s reply was succinct:

I think that’s the *only* value, for the reasons detailed before, though there may be other ways to accomplish its purpose.   In particular, opening up a tracking effort for 2nd level configurations (now tracking versions of device and software components in addition to tracking versions of equipment and software in their inventory) is complexity that utilities don’t need to add to their burdens.

George concluded with this statement of his credo: “I try to be the kind of person my dog thinks I am.” A high standard indeed!

If I can sum up what George said, it’s:

1.      There’s no point in even trying to ask for more than a first-level SBoM.
2.      It’s especially pointless for you to try to track versions of the components in that SBoM (although the supplier should know them!). Because your knowledge of the components and how they’re configured in the software is quite limited, you’re likely to waste a lot of time tracking down new patches and bugging your supplier about applying them, when for example the functionality that the patch addresses isn’t even used by your supplier.
3.      The primary value of asking for the SBoM is to see how much knowledge the supplier has about what’s in their software.

The second expert takes a different take on this. He has a very simple (if you have a basic knowledge of software development, which I don’t) method for, in some cases, being able to generate SBoMs, and he’s also involved with the NTIA SBoM program. Stay tuned.


Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Are you hot at work – or should be – on getting ready for CIP-013-1 compliance on October 1? Here is my summary of what you need to do between now and then.


No comments:

Post a Comment