If you're looking for my
pandemic posts, go here.
The first
part of this post described a pretty scary problem: the fact that just
about any software you purchase nowadays includes lots of third-party or open
source components; those components contain components; those components of
components contain components, etc. etc. A 2017 study by CA Veracode found that
“83 percent of developers use code components to build web applications, with
the average number of components per application (found to be) 73.” And a vulnerability
in any one of these components might prove serious.
I used the recent example of the Treck
TCP/IP stack, which was developed in the late 1990s (back in the days when internet
security was hardly even an afterthought. I remember visiting one of the
national labs at that time and learning that they didn’t even have a firewall.
Moreover, they had internet-addressable IP addresses on all their computers,
since – of course – the whole idea of the internet at the time was that it was
to allow free collaboration; I don’t think it even registered with me that this
might be a problem. How quaint that time seems today!).
Recently, researchers in Israel
identified 19 vulnerabilities in the Treck stack, and they think there are at
least twice that many still to be identified. Four of the 19 vulnerabilities
were deemed to pose the highest level of risk per CISA. Fortunately, Treck is
still in business and they were able to quickly develop patches for all 19
vulnerabilities.
But what’s the problem? The Treck
stack is probably embedded in hundreds of millions of devices. Many – if not
most - of the software suppliers whose products include the Treck stack
probably have no idea it’s there. And since it’s been on the market for more
than twenty years, it might very well be embedded in components that are more
than ten levels down in a piece of software installed today. Let’s be honest: there’s
no way more than a small fraction of the total instances of the Treck stack in
use today will ever be patched. So patching really isn’t a good first-line
mitigation for this risk.
What is the best mitigation? It’s
pretty simple to describe: Your software supplier should have a complete
software bill of materials (SBoM) that lists each third-party and open source component
of their software, as well as the version number and supplier name (probably
other information as well). Each of those components should have its own SBoM;
each of those components should have an SBoM, etc.
If you get the full SBoM from the supplier,
when a new vulnerability is announced in a particular software component, you’ll
be able to do a simple search to find all instances of that component – if any –
in each piece of software that your organization runs. If you have any hits,
you’ll then compare the version numbers listed in the vulnerability notice to
determine whether you’re running any instances of that version. If so, you’ll
know exactly what machines to apply the patch to, and all will be good.
But to tell the truth (which I
like to do every now and then. Breaks the monotony), even patching a few components
will be a painstaking job, and in some cases literally impossible if the component’s
code is compiled into your supplier’s product. But not to worry; this is really
your supplier’s job, not yours. All you need to do is only to engage with a
supplier that:
1.
Has a complete multilevel SBoM for their products,
as described above;
2.
Will provide you the complete SBoM, so that you
can contact some of the third party suppliers (or open source communities) and
be on the list to be notified of new vulnerabilities and patches;
3.
Has a policy of always buying source code for
any third-party component, meaning that if the third party goes out of business
or for some other reason can’t patch a newly identified vulnerability in their
code, your supplier will be able to develop their own patch for their customers
(of course, they need a high level of expertise to do this. A 3-month course in
Visual Basic™ – if they even have those anymore – doesn’t count);
4.
Will commit to patching any vulnerability
discovered in any of their third party or open source components within say
five days;
5.
Will commit to keeping track of the suppliers of
all third-party components, to make sure they’re still in business and still
able to provide patches when needed – and if your supplier determines one of
their third parties isn’t long for this world, they commit to replacing that
code in their software within say one month;
6.
Will commit to keeping track of the online
community supporting each open source component of their software – and if the
supplier determines that the community has dwindled to the point that patches
aren’t being developed anymore, they will commit to replacing the code within say
three months;
7.
Before releasing a new product or a new version
of a product, will commit to scanning each component for vulnerabilities;
8.
When a third party supplier or open source
community announces a new version of a component and end of support of the
version incorporated in your supplier’s software, your supplier will either
upgrade the component to the new version or find and install another component
of equal functionality; and finally can
9.
Bend 6-inch wide steel bars with their bare
hands, penetrate walls with their X-ray vision, and leap tall buildings with a
single bound.
In other words, current reality is
vastly different from the above. Here’s the true story:
· Few suppliers could provide you with a complete
SBoM of even the first level of components in their software, and that won’t often
have even the name of each supplier or open source community behind a component,
even if it identifies each particular component.
· If they do have a complete SBoM (or say they
do), they may refuse to give it to you for “competitive reasons” – meaning they
don’t want to take the chance that you’ll give the list of ingredients in their
cake to a rival cake-maker.
· Very few suppliers would buy source code from
every third-party supplier, even if it were available to them. The only reason
I even know about this concept is it’s described in a NIST best practices paper
about Schweitzer Engineering Laboratories (which has that policy). Plus I doubt
a lot of suppliers would even have the level of expertise that SEL has, to
develop their own patches for third party software.
· I don’t have any data on this, but I’m willing
to guess that few suppliers will commit to patching component vulnerabilities
within any period of time, let alone five days. And even if they did, it would be
a meaningless commitment if they didn’t at the same time commit to telling you
about new vulnerabilities, even if they’re unpatched so far (which of course is
what CIP-013 R1.2.4 requires); but it will be quite hard to get suppliers to
commit to doing this. Yet if they only tell you about vulnerabilities they’ve already
patched, they’re not committing to anything new when they “commit” to patching
all vulnerabilities in five days.
· The 2017 Veracode study quoted above also states
“A mere 52 percent of companies reported they provide security fixes to
components when new security vulnerabilities are discovered. Despite an average
of 71 vulnerabilities per application introduced through the use of third-party
components, only 23 percent reported testing for vulnerabilities in components
at every release.”
·
You get the idea: Except for a few exceptions,
your supplier is unlikely to commit to a single one of the above items. Probably
the most likely one they’ll commit to is leaping tall buildings in a single
bound.
So what’s a poor NERC entity (or
any organization, for that matter) to do? Is there no shining knight that will
ride in and save you from this menace of vulnerable third-party components? Believe
it or not, there is a government-led effort underway now to develop a standard
for developing and integrating SBoMs. The effort is led by Dr. Allan Friedman
of the National Telecommunications and Information Administration of the US
Department of Commerce. He is leading a large collaboration with industry and
universities to develop a standard for generating and accessing software bills
of material. This
document from NTIA provides a good introduction to the SBoM concept, as
well as links to other information about the NTIA initiative.
However, let’s be honest: It will
be years before this initiative will lead to widely-available SBoMs. Meanwhile,
the threats described above are here now. What can you do in the meantime? I
asked two software experts I know for their thoughts on this question. In this
post, I’ll describe what George Masters of SEL told me. I’ll discuss what the other
expert said in part III of this post.
I know George primarily from
working with him last year on the CIPC Supply Chain Working Group. We both led
the development of guideline papers. His was on “Risk Considerations for Open
Source Software”. The paper is excellent and it (as well as the webinar George
presented on it) can be accessed here. I learned a lot
just from the meetings he led while developing the paper.
I asked George whether it would be
realistic to ask software suppliers to provide at least a first-level SBoM. Here
is what he replied:
My advice to people who mention this in conversation is “be
careful what you ask for”, because using SBOM information to form a competent
risk assessment requires information that a customer just won’t have. He or
she would not, for instance, be able to recognize an older component that had
been selectively patched to address particular vulnerabilities, nor would they
be able to know about other controls that mitigate an issue, or that vulnerable
code may not be reachable at all.
Given the reliability expectations for power system
equipment, “churn” in the code is something to be avoided. Every change
should be carefully reviewed and thoroughly tested. Entirely updating a
component that may contain hundreds of changes involves some degree of risk –
and cost - and that has to be balanced against security gains.
Yes, the supplier should be able to demonstrate knowledge of
the product composition, and that they are tracking those components’ version
releases, but one shouldn’t expect to be able to second-guess them with an
SBOM. If the supplier is competent, this is wasted effort.
Suppliers may also want to keep composition data proprietary,
disclosing details only under NDA.
Reading the foregoing, as you might expect, I’d say that
getting beyond a ‘top level’ SBOM would make things worse.
I continued by asking George “Do
you think there's value in even asking for an SBoM? If for no other reason than
it shows whether or not the supplier even knows what's in their software.” I
said this because I’d read another article that said the only real value to be
obtained from asking for an SBoM at the current time is to see if the supplier can
respond to you at all – that is, whether they even have any idea what the h___
is in their software. If they clearly don’t know that, then you should question
why you’re even doing business with this supplier in the first place.
George’s reply was succinct:
I think that’s the *only* value, for the reasons
detailed before, though there may be other ways to accomplish its purpose.
In particular, opening up a tracking effort for 2nd level
configurations (now tracking versions of device and software components in
addition to tracking versions of equipment and software in their inventory) is
complexity that utilities don’t need to add to their burdens.
George concluded with this statement
of his credo: “I try to be the kind of person my dog thinks I am.” A high
standard indeed!
If I can sum up what George said, it’s:
1.
There’s no point in even trying to ask for more
than a first-level SBoM.
2.
It’s especially pointless for you to try to
track versions of the components in that SBoM (although the supplier should
know them!). Because your knowledge of the components and how they’re configured
in the software is quite limited, you’re likely to waste a lot of time tracking
down new patches and bugging your supplier about applying them, when for
example the functionality that the patch addresses isn’t even used by your
supplier.
3.
The primary value of asking for the SBoM is to
see how much knowledge the supplier has about what’s in their software.
The second expert takes a
different take on this. He has a very simple (if you have a basic knowledge of
software development, which I don’t) method for, in some cases, being able to
generate SBoMs, and he’s also involved with the
NTIA SBoM program. Stay tuned.
Any
opinions expressed in this blog post are strictly mine and are not necessarily
shared by any of the clients of Tom Alrich LLC. If you would like to comment
on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.
Are
you hot at work – or should be – on getting ready for CIP-013-1 compliance on
October 1? Here is my summary of what you need to do between now and then.
No comments:
Post a Comment