I want to thank Kevin Perry for
forwarding a post
by Patrick Miller from last week on the Ampere Security blog. You should read
it, but I’ll summarize the post by saying it describes the dangers posed by two
“zero-day” vulnerabilities that two researchers found in “…a
Moxa Ethernet-to-serial converter with the newest firmware.” Of course, the
problem with zero-days is they won’t be picked up by vulnerability scanners, so
– absent intrepid researchers who are willing to reach out to the supplier
before they exploit the zero-days they found (and aren’t planning on exploiting
the vulnerabilities themselves, of course) – neither the supplier nor their
customers will have any way of learning about those vulnerabilities.
I don’t want to downplay the
importance of this development, since zero-days pose a big risk to any organization,
and especially critical infrastructure organizations like electric utilities. An
Ethernet-to-serial converter is very unlikely to be found in any but a critical
infrastructure organization, since only these organizations are likely to be “blessed”
(or cursed, one of the two) with serially-connected devices. If Ethernet-to-serial
converters installed in multiple electric power transmission substations were
to be compromised, this could potentially cause a serious outage, and
potentially the worst kind of outage: a cascading one.
However, while this is a real
problem, in May I learned, from discussions sparked by Tom Pace of NetRise in an
informal group of SBOM practitioners that I’m part of, that there are at least
two even worse problems than individual zero-day vulnerabilities. One is
currently only theoretical, but could in the longer term (a few years – that’s
not that long) turn into perhaps a huge issue worldwide, were some creative
person or nation-state decide to devote a little time to exploit it. It’s hard
to imagine any quick practical solution to this problem (perhaps banning production
of open source software? That shows how serious the problem could turn out to
be). That problem is described in this
post
The other problem is an immediate
one and is currently huge, although it will be hard even to learn how big it is
until the evidence becomes all too clear through successful attacks. But this
problem can be solved with relatively easy measures - easy technically,
although I’m not sure about politically. This is one of those problems where the
cause can be easily traced to the wetware between the seat and the keyboard. I’ll
focus on this problem in this post.
In the first
post of the series, I described Tom’s initial presentation to our group, in
which he described his experience with one in a family of infrastructure devices
from a certain manufacturer. These devices are used in critical infrastructure
and in – shudder! – the military. Were you to do your due diligence on this
device before procuring it, you might well think it had perfect security. After
all, when you search on the name of either the device or its manufacturer in
the National Vulnerability Database (NVD), you will receive this response: “There
are 0 matching records.”
Sounds good, right? Before you purchased
the product, you wanted to see if the NVD listed any serious vulnerabilities for
it. Not only did no serious vulnerabilities appear, but no vulnerabilities at
all appeared. However, in his initial presentation to our group, Tom said he knew
that, in a single piece of firmware in the device in question, there are 1,237
unpatched vulnerabilities due to open source firmware components. In fact, Tom
later calculated,
this time through examining all of the firmware and software installed in the
device, that there were at least 40,000 unpatched vulnerabilities[i] in the one device – which of
course appears to have no vulnerabilities at all, if you just go by the NVD search
results.
These 40,000 are all known
vulnerabilities, not zero-days. But the problem is that end users will never
learn about these vulnerabilities if they’re not listed in the NVD - and this manufacturer
has never done so. As far as the users are concerned, these are zero-day
vulnerabilities. In fact, this manufacturer appears to make around 50 devices
and not a single one of them has any vulnerabilities listed in the NVD. Moreover,
this supplier doesn’t even mention security or vulnerabilities on their web
site. They obviously think they have the security problem licked in their
devices, since they’ve never identified any vulnerabilities. Perhaps they
missed the memo that said if you don’t look for vulnerabilities, you won’t find
them…or perhaps their lawyers sent that memo.
What’s even worse is that the
supplier itself isn’t hiding anything. They haven’t lied to the NVD or anyone
else, since they have probably never identified any vulnerabilities in any of
their products. Unfortunately, a search in the NVD that doesn’t find the
product at all (as in this case) returns the same error message as if the
product was found, but there were no vulnerabilities listed for it. Someone who
hasn’t been schooled in the pitfalls of relying too heavily on what the NVD
says will likely come away believing the product has zero unpatched vulnerabilities,
and they should go ahead and buy it. However, their number is just a little off
the true number of unpatched vulnerabilities – by about 40,000.
Here’s the reason why I say this
is a bigger problem than any discovery of zero-day vulnerabilities. Even though
these 40,000 vulnerabilities are known (meaning they have been assigned a CVE
number or another recognized vulnerability identifier), the fact that they’re
in this device (and lots of others, to be sure) will never be known to any end
user, unless the supplier reports them to the NVD.
However, the supplier won’t report
them until they’ve found them; but they’re not looking for them, so it seems
they won’t find them. Darn the luck! To the users of this device, these might
as well be 40,000 zero-days. But don’t worry; they’ll learn about them when
they get hacked.[ii]
What can be done about this
problem? I’m glad you asked. The group to which Tom Pace made this presentation
(the group is completely informal, but we call ourselves the “SBOM Forum”) is
now – largely because of that presentation – hot in pursuit of a solution to
the problem behind this and other aspects of the “naming
problem”. The naming problem is perhaps the biggest problem plaguing
software supply chain security, and SBOMs in particular. It’s definitely a huge
part of the reason why SBOMs aren’t distributed to customers in any more than a
few trickles today (the exception to this is the European auto industry, which uses
SBOMs for open source license management, not software vulnerability
management. They are also exchanging their SBOMs through a unique arrangement
that would probably – in my non-lawyerly opinion - not even be legal under US
antitrust laws).
When we started to pursue a solution
(although we didn’t dare use that word at the time) to the naming problem in
May, I thought we should be realistic and break the problem into two parts: the
part that can be addressed in the “near term” of 2-3 years and the part that will
take 5-10 years to address. I advocated that we put off the latter for the
moment.
But guess what: It turns out that
the near-term solution is the same as the long-term one. And neither is
technically hard at all: If everyone were agreed on what needs to be
done and were on board with how to implement their part of the solution, it
would be up and running in about a week – and I’m only saying that partially
tongue-in-cheek.[iii]
Of course, “everyone” will never be in agreement on everything, so I’m sure that
there will never be a 100% solution to the naming problem. But, as we see it
now, there are no big technical obstacles in the way of what we’re proposing,
and the funding required (all of which will need to come from the government,
of course) is much less than I imagined originally – although I’ll admit we’re
not considering cost in our solution. The problem is already exacting a huge
cost on the software industry, as well as the general public, due to not being
able to easily (or even at all, in some cases) learn about most software supply
chain vulnerabilities.
I’m hoping that in a couple weeks,
I can share some details on what we’re proposing, although the formal push won’t
come until September. We’re already starting to socialize it with people in the
government and elsewhere who can help with the implementation (and we’re getting
a good initial response). At the moment, we need to finalize the 12-page document
we’re working on, before we can seriously discuss this with anybody. I hope we’ll
have that done in a week or two.
Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.
[i] Of course, these are 40,000 vulnerabilities that are identified in software and firmware components found in the product. Since at least 95% of these vulnerabilities are probably not exploitable in the product itself, this means there are “only” 2,000 exploitable vulnerabilities in the device. And since maybe 2/3 (to be generous) of those vulnerabilities are perhaps not serious enough to patch, that leaves over 600 exploitable vulnerabilities that are serious enough to patch, and of course haven’t been patched yet – since this manufacturer evidently never even looks for vulnerabilities, let alone patches them.
[ii] I know that Tom has recently been in contact with the manufacturer about this problem, although there hadn’t been any resolution when I heard from him more than a month ago. But the problem is that this manufacturer is far from alone. There are tons of products that have zero vulnerabilities listed in the NVD, mainly because the supplier has never reported a single one. This is why I said, in one of the posts linked above, that it’s actually safer only to purchase software products or intelligent devices that have at least some vulnerabilities listed in the NVD, since their suppliers obviously take vulnerability management seriously. This isn’t a joke.
[iii] I’ll
admit I’m glossing over the fact that we’re solving two naming problems:
software and hardware. What I say above is related to software. The hardware side
of the problem isn’t technically hard, but it will require a lot of grunt work
merging databases. Nothing like that is required on the software side. However,
since the NVD includes both hardware and software products and a lot of the
components of the naming problem apply to both hardware and software, they both
need to be solved, although not necessarily at the same time.
I agree with your thoughts;)
ReplyDelete