The joy of VEX, part I

I’ve said it before, and I’ll repeat it now: Even though the VEX document was developed to be a companion document to software bills of materials (SBOMs), I think it’s very possible it will end up being even more useful than SBOMs – which is saying a lot, since SBOMs themselves are quite useful. This is because having VEX available allows a software supplier (or a government agency) to answer, in a machine readable format, a lot of vulnerability management questions that are probably not even being asked now – and if they are, they can only be answered through a long phone call with someone in tech support. Here’s what I mean.

The VEX format was developed in order to answer one question: If CVE 123 is found in a component of Product ABC, does that mean that XYZ is actually exploitable in ABC? There are many reasons why a vulnerability in a component of a software product won’t be exploitable in the product itself – meaning an attack that utilizes that vulnerability won’t succeed. These include a) the component is a library, and the library module that’s vulnerable wasn’t included in the product; b) even though the vulnerable code is present, there’s no code path to it; c) the report that the product is vulnerable is mistaken (e.g. I’ve heard that Juniper gets calls every time Cisco announces a vulnerability in one of their products); etc.

To answer this question (often before it’s asked) in the negative, the supplier of ABC will put out a VEX that essentially says “Even though one of our components is susceptible to CVE 123, our product ABC isn’t susceptible to it, because (insert reason).” Why would a supplier put this out? For both a selfish and an unselfish reason.

The selfish reason is suppliers fear that, when a serious component vulnerability is announced, their support lines will be overwhelmed with calls from customers demanding to know when they’ll patch this vulnerability, even though in most cases the vulnerability isn’t exploitable (it seems to be agreed in software circles that more than half of component vulnerabilities aren’t exploitable in the product itself). The unselfish reason is that the supplier doesn’t want their customers to waste a lot of time scanning their product for a vulnerability that isn’t there.

Of course, can’t a supplier do this now, using non-machine readable means? Since most now notify users of exploitable vulnerabilities in their products using PDFs, web site postings, emails, etc., why can’t they use those same means to notify users of non-exploitable vulnerabilities?

They certainly can do that. For example, when a widely-publicized set of vulnerabilities comes out, like Ripple 20 or Amnesia:33, some suppliers put out notices saying in effect “You don’t need to worry. We have examined our products carefully, and to the best of our knowledge, these vulnerabilities aren’t present in our products.”

However, it’s safe to say that suppliers aren’t currently even bothering to issue a statement saying that a vulnerability isn’t exploitable in a particular product, despite the fact that it’s found in one of the components of the product. Why aren’t they doing this? Their customers aren’t likely to be concerned about component vulnerabilities, since few suppliers are even providing SBOMs to their customers. So most software customers don’t even know about the components in their software, let alone vulnerabilities in those components.

But this situation is changing rapidly. It’s safe to say that, by this time next year, software users will be much more likely to know about components of the software they use, and also about vulnerabilities that apply to them. This is due to recent events, including the May 12 Executive Order 14028 and Microsoft’s recent blog post, which said they will start providing SBOMs for all of their products. And since a) the average software product has 135 components and each of those components can have vulnerabilities, while at the same time b) perhaps more than 90% of component vulnerabilities aren’t exploitable in the product itself (meaning 90% of component vulnerabilities might require a VEX), it’s clear that software suppliers have good reasons to be worried about their support resources being overwhelmed with calls about non-exploitable vulnerabilities.

But I’ve come to realize lately that there are a lot more questions about software vulnerabilities that aren’t being asked or answered now by any means – machine readable or not. Yet these questions aren’t being overlooked because they’re not important ones. They’re probably being deliberately put aside, because the resources that are required to answer them, including a customer spending a lot of time on the phone with one or more support people, are substantial. So in most cases, asking the question doesn’t seem to be worth the trouble for users.

Having a machine readable means of proactively answering the question for all customers of a product – as soon as the need to answer it becomes apparent – will make it much more likely that these questions will be asked and answered in the first place. What are some of these questions? And what would be the best way for the supplier to proactively answer them, using their powerful new VEX tool?

Note that I’m assuming below that there’s either a vulnerability management tool or a third-party service on the customer’s end, which is capable of reading VEXes and incorporating information from them into the list of open vulnerabilities in each software product in use in the organization. There isn’t such a tool or service now, but I know there’s more than one in the works.

Question 1: I’ve heard a lot about the new serious vulnerability CVE 123. Have you (i.e. the supplier) released a patch for CVE 123 yet?

Let’s say the supplier has already released a patch for CVE 123 in the current product, which is on version 1.30. Normally, they’ll release a patch notification in a PDF or some other non-machine readable format; this might or might not reach the right people in the organization. On the other hand, the VEX will go directly to the organization’s vulnerability management tool or service, so there’s no doubt it will be “heeded”. Here are the steps the supplier should take:

1.     When they patch the current version of the product, increment the version number. If the current version is 1.30, the patched version might be 1.31. Note that incrementing the version number should be a normal procedure. If the version number doesn’t change yet the code does change, the product becomes unsupportable.

2.      Send out VEX 1, which applies to v1.31. It indicates that CVE 123 has a status of “not affected” in the product (for a description of the VEX status designations, see this one-page VEX overview. Note that the “fixed” status could also be used in this case).

3.      Every “not affected” status in a VEX must be accompanied by a textual “impact statement” that describes why the product isn’t affected by the vulnerability in question (the impact statement isn’t mentioned in the VEX overview just referenced, but it’s mentioned in this CSAF Profile of VEX. I’ll freely admit that the Profile isn’t easy reading. I hope there will be more explanatory material out on VEX in the not-too-distant future). In this case, the supplier might write “CVE 123 was patched by Patch XYZ” for the impact statement.

4.      Assuming application of the patch automatically incremented the version number of the installed instance of the product from v1.30 to v1.31, the customer who asked that question will be able to ascertain whether or not their version has been patched, by simply checking the version number.

Question 2: I may have missed a patch notice. Does CVE 123 affect version 2.5? If so, do you have a patch for it?

1.      The supplier should issue VEX 2 indicating that CVE 456 has a status of “affected” with respect to version 1.30 of the product (i.e. the version that was patched, making it v1.31).

2.      Because a status of “affected” requires that the supplier provide a textual “action statement” that describes mitigation steps for the vulnerability, the supplier should enter something like “Apply patch XYZ (which of course is the patch for CVE 123).” If any other mitigation steps are required – and whether or not a patch is required – they should be included in the action statement as well.

Note that, with the actions taken to address questions 1 and 2, the supplier has effectively provided two different patch notifications. VEX 1 is like “traditional” patch notifications: it says “Here is the patch for CVE 123.” VEX 2 is unlike traditional notifications, since it says “Version 1.30 is vulnerable to CVE 123.”

Here’s an extra credit question: Why is it that you don’t currently see software suppliers issuing notifications like VEX 2, that state “Yup, we’re vulnerable to CVE 123, which has a CVSS score of 9.8. Have a nice day”? You’re right! It’s because they’d probably be out of business in a week, since every hacker in the world would now know exactly how to compromise their product.

In other words, when the supplier issues a “traditional” notification that they’ve patched CVE 123 in their product, they’re implicitly also stating that the product was vulnerable in the first place. They would never issue a notification that stated explicitly that their product was vulnerable, unless the patch was already available.

However, by using VEX1 and VEX 2 in place of their traditional (non-machine readable) notification, the supplier is able to accomplish more than they could without VEX. Specifically:

1.      By saying that v1.31 includes a patch for CVE 123, VEX 1 implicitly states that v1.30 is vulnerable. The message is clear for all who have ears to hear: If you still have v1.30 (meaning you haven’t yet applied the patch), you need to get off your a__ and apply Patch XYZ right away. A traditional patch notification works in a similar fashion.

2.      But what if someone didn’t get the traditional patch notification, so they don’t know about the new patch, and they also don’t know that v1.30 is vulnerable? Traditionally, their status would become SOL. However, VEX 2, which states that v1.30 is vulnerable to CVE 123 and a patch is available, will automatically notify the customer’s vulnerability management system that a patch needs to be applied. It will be much harder for that to slip through the cracks.

The point of this post is that use of VEXes provides the supplier (and government cybersecurity agencies, if they want to take advantage of VEXes) with capabilities they didn’t have before: in this case, the capability to provide a “failure-proof” notification to their customers that they need to apply a particular patch ASAP.

I’ve thought about other such cases as well, and it seems there are a number of them. I’ll hope to discuss one or two of these in the near future. In the meantime, I wish to call your attention to an old Irish limerick (the original Gaelic is listed in a comment below):

There once was a format named VEX,

Which we thought too opaque and complex.

When we learned it was free,

We shouted with glee,

"This is certainly better than sex!"

While I can’t vouch for the accuracy of the last assertion, this strikes me as a remarkably prescient piece of poetry.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the CISA’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

The Russians are at it again

 

It seems that Nobelium, the wonderful Russian government-sponsored attacker group that brought you SolarWinds, hasn’t been deterred by President Biden’s threats – or mollified by his entreaties - to Vladimir Putin. Microsoft reports they’re now conducting a large-scale campaign to infiltrate cloud service providers and IT services organizations, and they’ve already had some success at it.

Of course, they’re not primarily interested in those organizations themselves, but their customers. As we all have known for a while, Russia long ago gave up on the idea of attacking targets like critical infrastructure and government agencies using full-on frontal attacks. They know that the organizations they’re most interested in attacking have all figured out what they need to do to keep themselves safe from those attacks.

But what we haven’t figured out yet – and I’m not sure we’re learning too quickly, either – is how to protect ourselves against supply chain attacks. There are so many possible vectors for those attacks, and the average organization has so many suppliers that they trust in lots of ways, that it will be a long time, if ever, before we’re as protected against supply chain attacks as we now are from frontal attacks (the only other cyberattack category that’s increasing recently is phishing, which is the main vector for ransomware attacks. But Kaseya showed that it’s possible to realize huge efficiency gains in ransomware attacks by running them through the supply chain as well. The single penetration of Kaseya led to at least 1500 organizations being compromised. Talk about ROI!).

But I’m not writing this post just to marvel at how clever the Russians are. I’m really writing it to wonder how long we’re going to wait before we hit the Russians with some really strong sanctions for their various acts of piracy. And here, I’m not just talking about recent cyberattacks. I’m talking about shooting down a civilian airliner in 2014 and causing $10 billion worth of damage with the not Petya cyberattack in 2017 (which was a supply chain attack on the Ukraine that ended up getting a little out of hand. You know how those things go…).

The Russians haven’t paid a dime of restitution for either of those attacks. Let’s start with them. And if that doesn’t deter the Russians, we can look at retaliation for SolarWinds and many more recent attacks (including the fact that, according to the FBI and CIA’s Worldwide Threat Assessment in 2019, the Russians have penetrated the US grid – and this obviously means Control Centers – and could cause outages at any time. This charge has never even been investigated, by the way. It seems that investigating Russia wasn’t a good career move in Washington in 2019 and 2020. The question is whether that’s still the case in 202).

Let’s start with the airliner. There’s no question the Russians were behind that, even though it was one of their proxies that pulled the trigger. Let’s do what we should have done then: ban Russian planes from all international airspace until they pay say $5 million to the family of all 293 passengers and 15 crew members who were killed. And until they’ve paid the full costs of the Malaysian and Dutch governments for property damage, and especially the costs of investigating the crash.

Then we’ll start with the other attacks. By the time they’ve paid for those as well (and suffered other sanctions like ending Russian bond sales in the US), maybe Uncle Vlad will be a little more cautious about his attacks in the future.

As they say, tragedy repeated is farce. 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the CISA’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

Data centers are critical infrastructure, and need to be regulated as such

Nextgov ran an excellent article last week, pointing out that data centers are very dependent on OT (especially power and cooling systems), even though the systems running in those data centers are all IT systems. The lesson I drew from the article (although it wasn’t stated like this) was that, not only are data centers critical infrastructure, but their OT side should be regulated as critical infrastructure. In other words, while I don’t think critical infrastructure regulations should be applied to the IT systems in the data centers, they should be applied to the OT systems that keep the IT systems running.

There’s another relevant lesson that the US learned the hard way this summer: OT systems aren’t limited to a bunch of strange-looking devices that don’t run an OS you’ve ever heard of (or in some cases, don’t run any OS at all), and that don’t normally run a networking protocol you’ve ever heard of. As the Colonial Pipeline attack showed, and as a 2018 ransomware attack on a large US electric utility also showed, a serious attack on IT can shut down systems that are critical to operations, forcing the OT network to be shut down as well.

So any systems that support the operations of a critical infrastructure provider – whether it be a pipeline, a utility, a key software supplier like SolarWinds, a cloud services provider, a data center provider…and other industries, I’m sure – should be in scope for critical infrastructure regulation. And this applies to systems that aren’t actually OT systems, like the Intel-type servers running Windows and Linux in the electric utility’s control centers discussed in the link above. These all had to shut down in 2018 due to a devastating ransomware attack on the utility’s IT network, even though they weren’t infected by the ransomware (of course, control centers in electric utilities are already subject to NERC CIP compliance, and it could certainly be argued that the CIP requirements are aimed much more directly at IT-type systems in control centers, than they are at OT-type systems in substations and generating stations).

I define critical infrastructure systems as those that are necessary for the smooth and continuous operation of critical processes – large scale data processing, production and delivery of electric power, etc. Both IT and OT systems can be critical infrastructure systems. In my opinion, all critical infrastructure systems should be regulated.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the CISA’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

How can SBOMs help with compliance?

Yesterday, Tobias Whitney of Fortress Information Security introduced to the NERC Supply Chain Working Group a white paper entitled “Enhancing Cybersecurity Best Practices with Software Bill of Materials (SBOM)”. This is a well-written paper that discusses how utilizing supplier-provided SBOMs for cyber risk management purposes can help an electric utility (or other power market participant, such as an IPP) in its many cybersecurity compliance obligations, both those imposed by NERC/FERC (i.e. the NERC CIP standards) and those that may be “imposed” by the utility itself, such as NIST 800-53, NIST 800-171 (CMMC), and the National Defense Authorization Act (NDAA).

Fortress doesn’t pretend that SBOMs are a requirement for compliance with any of these standards; in fact, none of these standards even mentions SBOMs. However, the paper identifies particular requirements in each standard whose purpose could be better fulfilled with SBOMs than otherwise. In other words, Fortress asks, for each of these requirements, “Can SBOMs help utilities address the best practice that is the goal of this requirement? If so, how?”

For example, for NERC CIP-007-6 R2 (which had never occurred to me to be one in which SBOMs might help), Fortress points out that having a recent SBOM from the supplier would help the utility “evaluate security patches for applicability”, as required by R2.2, as well as evaluate the security impact of applying the patch.

I recommend that you download and read this excellent paper.

Also, I recommend that you attend the bi-weekly SBOM Energy PoC meeting tomorrow (Wednesday, October 20) at noon ET. No sign-up is required, although if you’re not already on the PoC mailing list, I recommend you join it by sending a request to sbomenergyPOC@inl.gov. The URL for the meeting is here.

The topic of tomorrow’s meeting will be VEX, the “companion” document to SBOMs which now seems as important as SBOMs themselves. Dr. Allan Friedman of CISA, leader of the Software Component Transparency Initiative, will describe why VEX was developed in the first place, as well as how it works. There should be ample time for Q&A.

See you tomorrow!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared CISA’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

This software supply chain flaw killed 346 people. That's a lot.

I have been following the saga of the 737 Max with fear and loathing since it started. As you probably remember, the story became public with a crash in Indonesia in October 2018, that killed 189 people. As this NY Times article from last November says, Boeing “quickly diagnosed the problem: faulty software. The company immediately promised to fix the code, reminded pilots how to handle a potential malfunction and insisted the Max was safe.” But they didn’t ground the plane worldwide, and the FAA didn’t force them to.

When a second 737 Max crashed five months later in Ethiopia – costing 157 more lives – the FAA finally ordered the planes to be grounded. Since both crashes happened shortly after takeoff and were preceded by a stall caused by the nose of the plane being pushed down against the pilots’ will, attention had immediately focused on the software that caused this to happen, called MCAS. In both crashes, the pilots fought mightily with the software, repeatedly pulling the nose up after it had pointed down. But MCAS won the battle in both cases.

Clearly, nobody at Boeing designed MCAS so that it would cause crashes. Does this mean a nefarious actor penetrated the development process and implanted the code that caused MCAS to behave as it did? There was never any question of that, either. It seemed that – for some reason – the designers of the software didn’t properly consider all the cases in which it might be activated. Even more importantly, they didn’t notify Boeing’s customers that there was a simple way to fix this problem if it ever occurred: just turn MCAS off.   

The story came back into the headlines this week when Mark Forkner, who was chief technical pilot of the MAX, was indicted for misleading the FAA about MCAS. One of the more appalling stories that came out of this disaster last year was that he had bragged in emails that he’d “inadvertently” misled the FAA. If he’d misled the FAA inadvertently, why was he indicted? And how did it happen that such a terribly dangerous piece of software was developed and deployed in the first place? This clearly wasn’t just a case of a semicolon being inserted at the wrong place in the code.

Today, Holman Jenkins, Jr. of the Wall Street Journal editorial page, published a good – to a point – analysis that asks both of these questions. Spoiler alert: Mr. Jenkins answers the first question, but considers the second to be a mystery that will probably never be solved – when it was already answered in the Times article linked above. But of course, people who work for the WSJ editorial page would no more read a Times article for its content than they would have read Pravda (the official government mouthpiece of the Soviet Union) during the bad old days of the Cold War (note my lack of regard for the WSJ editorial page writers is the opposite of the high regard in which I hold the WSJ news reporters, especially those who report on cybersecurity and electric power issues, like Rebecca Smith. She reports on both those issues, especially when they intersect. BTW, there’s still a story to be told about the article I just linked, and the quite unfortunate chain of events it led to. If there’s interest, I’ll tell it sometime).

Jenkins explains that it was a change in the software that caused MCAS to become so deadly, as well as why Forkner didn’t know about this when he talked to the FAA:

MCAS had been designed to counter the tendency of the plane’s nose to rise in a maneuver that would never be experienced in normal operations, a high-speed, ever-tightening turn that would cause pandemonium in the passenger cabin if it were tried during a commercial flight. Mr. Forkner discovered only in the simulator, and then by contacting a Boeing engineer, that the system had been belatedly altered to intervene even during low-speed maneuvers.

So why was Forkner indicted? Because he subsequently did learn about the change, as shown by the famous email I mentioned above. But did he call up the FAA and tell them he’d been wrong? This might well have led to an investigation of MCAS, and the discovery that Boeing had made a major software change without doing much if any analysis to figure out what might happen if a pilot hadn’t been alerted to the possibility of MCAS pushing the nose down soon after takeoff.

Now we get to the second question, which is such a huge mystery to Mr. Jenkins: Why was this change made? Here’s what the Times article says:

In 2011, Boeing learned that American Airlines, one of its most important customers, was poised to place a major order for new jets with Airbus. Boeing had been considering designing a new midsize passenger jet, but the threat of losing out on the American deal forced the company’s hand.

Boeing decided to redesign the 737 — a plane that was introduced in 1967 — once more. To make the new Max more fuel efficient, Boeing needed bigger engines. But because the 737 was an old plane that sat low to the ground, those engines needed to be mounted farther forward on the wings, changing the aerodynamics of the plane. To compensate, Boeing introduced the ill-fated MCAS.

And even when the company knew that the software had caused the first crash, Boeing kept the Max flying until another plane fell from the sky.

But this change in itself wouldn’t have caused the accidents, if the pilots had been trained in simulators to recognize this problem and turn MCAS off. But it turns out that the FAA didn’t require any pilots to be trained in simulators for the 737 MAX, as long as they’d already been trained for “regular” 737s. But of course, the MCAS problem could only occur in the MAX. Of course, this doesn’t mean the airlines couldn’t have required this training on their own, but few (if any?) did – it would have cost a lot of money, and delayed when they would be able to start flying the MAX.

But even without simulator training, it would have been very helpful if Boeing had put out a detailed alert about this problem. This would certainly have gotten the attention of management of the airlines, who would have made sure the pilots knew what to do when MCAS kicked in at the wrong time. But Boeing didn’t do that either, perhaps because it might have made the airlines realize they did need MAX simulator training, even though it wasn’t required.

And Mr. Jenkins points out another reason why not requiring simulator training was such a serious mistake (if “mistake” is the right word. The phrase used by an Ethiopian man in the Times article is “corporate manslaughter”): In developing the training for customers (Boeing obviously had the simulator for their employees, since that’s how Forkner learned about the change in MCAS), Boeing managers might have learned of the problems that some engineers, and Forkner, hadn’t told them about, and realized that having a couple planes fall out of the sky would probably be a little more expensive for Boeing than the cost of requiring simulator training and notifying their customers of the problem.

So Boeing made the MCAS changes to compensate for the fact that their redesign of the 737 – which they undertook to prevent American Air from placing a big order with Airbus – had made the plane likely to go nose-up and stall in high-speed situations (well after takeoff). But the redesign of MCAS to activate in some low-speed situations made MCAS into a deadly weapon, aimed directly at pilots who didn’t know about these problems, and of course the unsuspecting crew members and passengers.

As I asked in this post recently (although in a very different context, I admit), “Is there any doubt that software is the biggest source of supply chain cyber risk?” True, the MAX crashes weren’t due to a supply chain cyberattack, but they could have been (perhaps due to hackers who had shorted Boeing’s stock and were looking for a huge negative event to cause it to tank). When you’re buying planes, the impact of those risks being realized can be much greater, but there’s risk in any software used for any purpose.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the National Technology and Information Administration’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

A big day for SBOMs!

A few hours ago, Dick Brooks of Reliable Energy Analytics sent around a link to an important announcement on Microsoft’s developer blog, to the main mailing list for the NTIA software component transparency initiative. Not being an avid follower of developer blogs, I must have missed this one, so I was pleased that Dick shared it.

The post is of course filled with developer-speak, and if you’re not much more familiar with that arcane dialect than I am, you won’t understand a lot of it. Here are the important points that I see:

The first paragraph says “…Microsoft has chosen to use Software Package Data Exchange (SPDX) for all SBOMs we generate, and we have embarked on the mission to do this for all software we produce.” In other words, all software that Microsoft produces will come with a software bill of materials, and it will be in the SPDX format. SPDX is a project of the Linux Foundation and is one of the two leading SBOM formats (the other is CycloneDX, an OWASP project). As the first paragraph also notes, SPDX was recently designated an ISO standard.

The author lists three reasons why SBOMs are useful:

• Software transparency: SBOMs provide a list of ingredients used in the creation of a piece of software, such as open source software, components, and potentially even build tools. This enables producers and consumers to better inventory and evaluate license and vulnerability risk.
• Software integrity: While code signing is still the industry standard for trusting software and its integrity, SBOMs contain package and file checksums to enable consumers to validate the hashes, which can be useful in scenarios when signatures aren’t present.
• Software identity: When vulnerabilities (CVEs) are created, they are assigned to a Common Platform Enumeration (CPE) identifier, which can have issues attributing a CPE to a specific piece of software. Software IDs within SBOMs provide a much more accurate way to identify software.

The first of these, software transparency, is without much doubt the reason most often cited when it comes to a discussion of whether SBOMs are needed. The other two are important, although frankly I’m not quite sure what the author has in mind in citing these two. I expect to learn more about that in the coming days.

Clearly, Executive Order 14028, which mandated that government agencies require SBOMs from their software suppliers (this provision will be in effect next August, according to OMB), was the biggest driver for Microsoft’s decision – or at least I interpret the fact that the first words of the post refer to the EO to be evidence of that motivation. Of course, now that Microsoft has broken the ice, I expect a lot more software suppliers to make similar announcements soon.

But there’s another motivation for Microsoft’s announcement. This can be seen in the last two sections of the paper, and especially in the diagram at the bottom – which is pretty much a diagram of the SolarWinds hack. I’m not referring to the Sunburst malware that was used to attack around 250 SolarWinds users (unfortunately, including some very high-profile government agencies). I’m referring to the Sunspot malware, which the Russians used to plant Sunburst in seven SolarWinds updates.

Along with Stuxnet (and perhaps ahead of it), Sunspot was one of the most amazing pieces of malware ever created (Microsoft estimated that the Russians had 1,000 people working on it). The red text in the diagram at the bottom of the post is almost exactly what Sunspot did to plant Sunburst in the SolarWinds build process (the Russians had to carry out the process completely through Sunspot, working remotely). This was so ingeniously covered up that an SBOM produced when they normally are – at the end of the build process – would never have shown anything to be amiss.

Thus, Microsoft is going far beyond simply producing their SBOMs at the end of the build process. Instead, they’re producing and validating them at various stages in the process, so that an attack like the Russians mounted on SolarWinds won’t succeed – it will be discovered, and the build process will be halted. While this isn’t to say that nobody will ever come up with a way to fool Microsoft’s SBOM process, it is to say that Microsoft is setting a great example for the rest of the software industry.

At the moment, I don’t expect many, if any, software developers to duplicate Microsoft’s SBOM process. However, I do expect that the combination of Joe Biden and Microsoft will provide the impetus for the software industry to finally start releasing SBOMs to their users. Just having a single SBOM – produced at the end of the build process – gets the user light years ahead of where they were without any SBOM at all, in terms of being able to identify and mitigate software vulnerabilities. And having SBOMs for a good percentage of the software products that your organization is running…well, that will put you megaparsecs ahead!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the National Technology and Information Administration’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

Q: Is the TSA pipeline security directive a step forward or backwards? A: Backwards

I was recently given the opportunity to review a redacted copy of the TSA pipeline cybersecurity directive issued July 19. I was eager to do this, in order to answer two questions: A) Is this a step forward from existing OT cyber regulations – especially NERC CIP? If it is, it could provide a guide to how CIP can be remade to become much more effective and efficient than it currently is. Or B) is it a step backwards, meaning that if anything it will furnish a good object lesson in how not to regulate OT cybersecurity?

My answer….drum roll, please…is that the TSA directive is a big step backwards. As such, it will provide us with some great lessons on how not to regulate OT, and especially how not to revamp NERC CIP. I’ve listed below what I think are some of the big failings of the directive, but I’ve restated them as lessons learned for any agency or organization that might someday find themselves writing or rewriting OT cybersecurity standards.

1.      Involve the industry in writing/rewriting the standards.[i] A Washington Post article quoted an industry executive as saying the pipeline industry was only given three days to respond to a draft of the standards. Of course, giving them only three days was the equivalent of saying, “Here you go, you bastards. This is all you deserve, and this is all you’ll get!” I’m sure if the industry had had more time to respond, they would have said things like I’m saying below, and maybe the TSA directive would turn out to be more effective than it will almost certainly prove to be.

2.      Don’t make the standards secret. What purpose does that serve? And more importantly, how could the standards ever be implemented unless the whole organization knows about them, and each person understands what role they can play in making sure the implementation is successful? As it is, I believe that whoever drafted the directive (Darth Vader?) somehow got the idea that cybersecurity for a large organization can be achieved by having a tiny group implement a relatively small number of very technical – and untried at scale – measures, while keeping the rest of the organization completely in the dark about them. Here’s the real story: This_approach_could_never_work. Period. Unless you involve everybody in the organization to some degree, you’ll end up spending a lot of time and money implementing a bunch of controls that will be obsoleted within months. The only lasting change will come from the organization itself. No organizational change, no improvement.

3.      Don’t impose requirements that are literally impossible to comply with. There are requirements that use words like “ensure”, “prevent”, “prohibit”, “eliminate”, “block”, etc. As if there were any way that an organization could promise that they will ensure, prevent, prohibit, eliminate, or block anything in the constantly-changing world of cyber attacks. Requirements like these will probably end up with zero compliance as written. However, this result may be hidden because “compliance” isn’t defined at all! Which leads me to the next point.

4.      Define how compliance with the requirements will be assessed. There are a few of the requirements that state how they’ll be assessed: through self-assessment within a certain time period. But there’s no discussion about auditing, and it seems that it will be entirely up to the pipeline operators to report on their state of compliance with each of the requirements. As well as to determine how often they will report, etc. -  since I see little or nothing in the directive that requires any ongoing reporting (although it will certainly be required). There is a requirement for an annual third-party evaluation of the “Owner/Operator’s” OT “design and architecture”. As far as I can see, those are the only occasions when someone other than the pipeline operator will look at what they’ve put in place. But there’s no indication that these evaluations will in any way constitute an “audit”. I’m not a big fan of prescriptive audits, but I think there does need to be something more than simply a review of documentation.

5.      Don’t take new ideas – that haven’t been discussed a lot in public forums - and turn them into requirements. The directive includes very specific requirements based on technologies about which I’ve seen very little (if any) public discussion: “passive DNS”, “known malicious IP addresses” (is it really likely a bad guy would keep using the same IP address?), “SOAR” (I know generally what this means, but there must be some specific meaning in order for this to be the basis for a prescriptive requirement), and others.

6.      Finally, make sure you’re addressing the really important requirements. For example, the widespread outages caused by the Colonial Pipeline ransomware attack (the inspiration for this directive, in case you hadn’t guessed that) were due to the fact that Colonial felt compelled to shut down their OT network, even though the ransomware had never spread to it. Obviously, there was some link between the two networks that necessitated the OT shutdown. What was that link? The directive goes through all of the usual suspects and tries to address every one of them. There are requirements for physical and logical segmentation, a DMZ, isolating “unrelated sub-processes” (a wonderful undefined term), monitoring and filtering traffic between different trust levels, implementing zones and conduits, prohibiting OT protocols from traversing the IT network – just about every remedy you might find in an ICS security textbook. Yep, they addressed everything except for what caused Colonial’s OT network to be shut down. That one they missed, although you can read about it here. Oh well, that’s the only one they missed. Not so bad, no?

I don’t think I’ll surprise you by saying I don’t think very much of the TSA directive for the pipeline industry. What would I suggest they could have done instead? Here’s a radical idea: Why couldn’t they have ordered pipeline companies to implement a cybersecurity risk management program based on the NIST Cyber Security Framework? Admittedly, this wouldn’t have been seen as an innovation, and it wouldn’t have resulted in a Full Employment Act for cybersecurity consultants who can now put themselves out as experts familiar with the arcane niches of the TSA directive.

But it would have been a better way to secure the pipeline industry, on both IT and OT sides. And that ain’t hay.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the National Technology and Information Administration’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] Note the TSA document is a one-time directive, not a set of ongoing standards. DHS will institute a formal rulemaking process at some point, to draft the ongoing standards. If so, I hope they’ll take a look at these lessons learned.

A good upcoming webinar on SBOMs

The Canada-based cybersecurity company Cybeats has been putting on an excellent – and very well-informed – series of webinars on software bills of materials on LinkedIn. The next one (the third) looks like another winner. The webinar will be at 1PM EDT on October 19^th. The link to the webinar is here: https://lnkd.in/dyiaYECc. No signup is required, although Cybeats would like you to register here.

The webinar consists of a three person panel, including:

1.      Cassie Crossley of Schneider Electric, who has without doubt been one of the foremost advocates for SBOMs in the critical infrastructure sector. She has been a very active participant in the NTIA’s SBOM efforts, and has provided (with a couple colleagues from SE) two very informative hour-long presentations on SBOMs to the Energy SBOM Proof of Concept.

2.      Shuli Goodman of Linux Foundation Energy, who promotes open source projects aimed at the energy sector all over the world and who has been a great supporter of the DBOM (distributed bill of materials) effort. DBOM is somewhat related to SBOM, but is a really great idea in its own right. I continually compare it to the internet in the early 1990s: Everyone knew it would be big someday, but nobody was exactly sure what the spark would be (I’d say the Netscape IPO in 1995 – when a bunch of geeks, doing something that very few people understood, gathered the unheard-of sum of $1 billion – was that spark).

3.      Philip Tonkin of National Grid, who I don’t know but I’m sure will have something interesting to say, since he wouldn’t be on the panel if he didn’t!

The moderator of the panel is Chris Blask of Cybeats, the inventor of DBOM (which is an open source project) and someone who has played a leading role in cybersecurity for a long time. For more information on the webinar, go here.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the National Technology and Information Administration’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

The grid has been compromised! You need to buy my stuff!

Forbes Magazine seems to be making a specialty out of publishing stories with the following simple marketing scheme, requiring that just two boxes be checked:

1.      Box One: The grid has been compromised! No evidence is presented for this assertion, but hey – would Forbes publish somebody who lies? Answer: Yes.

2.      Box Two: The only thing that can really save the industry – and therefore the whole country - is my (insert name of product here).

There’s almost a crystalline beauty to this scheme. It seems people want to believe the power grid can be brought down with the wink of an eye by some guy in China. And, of course, everybody wants to think there’s a simple solution to their problems, especially when the problems are imaginary ones.

So Forbes, being – according to their own logo – the “capitalist tool” par excellence, was the perfect venue to publish a story (written by a woman who sums up her work as "I distill energy policy and technology movements", whatever that means) about a box that’s going to save the grid. As Ron Indeck, CEO of Indeck Security, himself said, “What’s being used now is not working. We’ve seen water supplies compromised. We’ve seen oil and gas delivery systems compromised, and we’ve seen the electrical grid compromised.”

So there you have it. Ron Indeck – you know him, don’t you? – says the grid has been compromised. That’s all you need to know. Check the first box.

What can save the industry? Would it astound you greatly if I told you that it’s a product that Ron just happens to sell? And it’s just icing on the cake that it’s literally a box – in this case a very simple box with just two ports: “Network” and “Endpoint” (and if you don’t believe it’s that simple, look at the picture in the article. After all, pictures don’t lie, even if Ron does). What could be simpler? You just plug the network into the box, and according to what Ron says – remember, Ron doesn’t lie, except about grid compromises – up to 2,000 endpoints will be protected.

Of course, Ron’s box is so simple, yet so powerful, that no software reconfiguration is required on any of the devices (and shame on you if you wonder whether this means the device doesn't do anything at all! You're obviously not enlightened enough to understand someone who "distills energy policy and technology movements"). All you have to do is plug in the network. Check the second box.

Now, if I were a skeptical person, I might ask how the box is going to handle all the really critical devices found in substations and generating plants that only connect via serial protocols, and therefore won’t be able to connect to Ron’s box at all. But fortunately, I’m not a skeptical person.

And I would also be quite skeptical about the very helpful statement from a former FERC commissioner, Branko Terzic, who says “They would be plant in service [rate base] and be depreciated over their economic lives. The cost of the Q-Box like other assets is part of the revenue requirement upon which rates are based. As rate base the Q-Box both earns a return and creates a depreciation expense.”

I’ll note that people trying to sell into the power industry often get enamored with the idea that the product they’re selling will just go right into the rate base, so it’s essentially free (it always helps your sales if your product is free. That’s why this blog is so successful). Anybody who’s actually worked for a regulated utility will tell you that, just because the utility wants to put something in the rate base, that doesn’t mean the PUC will let them, and even if they do, it will be years before the money comes back to them. Being a former utility regulator, one would expect Mr. Terzic to know this. But fortunately, I’m not skeptical.

I also might have been skeptical when the article brought up yet another member of its rogue’s gallery, this time the CEO of Sedulous Consulting Services (although I wonder if that really means “Credulous”). Under a section heading reading “Who you can trust”, he – believe it or not – states that “We hear about Colonial [pipeline hacking] but those in the news are only a quarter of what’s really going on”. Wow! He sure sounds like he knows about the grid. Does he work with power sector companies? No, he's primarily a military contractor. But he has done some work (unspecified) for DoE - that's all you need to know.

So why should you trust him? Well, it seems “By November, DOD could add Sedulous to its list of four U.S. companies certified by the U.S. government to judge the capability and integrity of other cyber software companies.”

Very impressive! This guy’s company might get added to a list of companies certified to judge other software companies! That’s great. But…what if he doesn’t make the list? And more importantly, what does it mean to judge the “capability and integrity” of a “cyber software” company, anyway? It’s a good thing I’m not skeptical.

Is he a big supporter of Ron's box? No, he never mentions it. So why is he in this story? Beats me. I guess he's there to support the idea that the grid will collapse any day now, based on the fact that, well, he uses electricity. He must know what he's talking about!

I’ll stop here. It’s late now, and I need to go to bed and think about how I can get on this gravy train that Forbes is perfecting. Let me see…I just need to write a post that says two things:

1.      Because of my special knowledge of the US power grid and the idiots who run it without the slightest thought about security, I can absolutely guarantee that – if we don’t do something differently – the grid will go to hell in a handbasket by Thursday of next week.

2.      Fortunately, I just happen to know the secret tweak that every utility can make to its network, which will make the grid absolutely immune to further attacks. If only a thousand utilities will pay me $5,000 by next week, I’ll reveal the secret and save the country (and forget what I just said about being idiots. You guys are really geniuses. That’s why you will all immediately send me $5,000. You can even use PayPal).

How could you possibly lose?

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the National Technology and Information Administration’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

How can the power industry really collaborate with vendors?

On September 17, Robert Walton of Utility Dive published a well-researched article that asked the question why vendors weren’t invited for the upcoming GridEx exercises.  There were quotations for and against the idea of vendors participating. One of the “for” votes was from Nick Cappi of Hexagon PPM, who said "If we want transformational events in improving the security of our critical infrastructure, then we need to break down the silos of information and obstacles to collaborate."

I was one of the “against” votes, and was quoted as saying

"GridEx is an operational exercise for responding to a hypothetical attack, so only the operators of the [bulk electric system] can participate. When an attack happens, it's too late [for] most vendors to be able to help — the software has already been written, the control systems have already been developed, etc."

Of course, my point was that, when it comes to a situation where the grid is being defended against a serious attack, there’s very little that vendors can do – given the very short (in some cases, literally infinitesimal) timeline for possible action.[i]  If there’s a vulnerability in some software, the software supplier can’t patch it in real time. If a router, server or any hardware device is compromised, the best thing is usually to shut it down and remove it from the network, not leave it operating and try to block or remove the avenue of compromise.

But later I found myself thinking, “You know, there really isn’t a lot of collaboration between electric utilities and industry suppliers, to solve security problems they jointly face. It’s certainly not as exciting as a 2 or 3-day exercise like GridEx, in which the entire American way of life is assumed to be on the line. But it’s still very important, and it can be quite satisfying to collaborate to find a solution to a specific cybersecurity problem. Maybe the solution won’t make everybody happy, but if it leaves everybody better off and no significant group worse off, that’s something worth doing.”

I thought about the collaboration that goes on between vendors and asset owners today. It seems the biggest collaboration nowadays revolves around CIP-013 compliance. Of course, it’s the utilities who have to comply, not the vendors. But CIP-013 won’t work if the vendors refuse to cooperate in filling out questionnaires, improving their practices when needed, etc. (fortunately, they do seem to be cooperating). This is all good, of course, but collaboration on compliance, where the ultimate goal is to help utilities pass audits, isn’t the same as collaboration on security, where the goal is to find ways to make the power grid more secure.

I wondered, “What would be an example of real security collaboration between vendors and asset owners?” Then I realized that I’m involved in one right now: the Energy SBOM Proof of Concept. Here’s why we started this initiative:

1.      There isn’t a huge amount of disagreement in the cybersecurity community that one of the biggest sources of cyber risk is software supply chain risk.

2.      There also isn’t a huge amount of disagreement – although there is some, I’ll admit – that software bills of materials (SBOMs) are a crucial tool for mitigating software supply chain risk. This is because, numerically speaking, there are many more component vulnerabilities in the average software product, than there are vulnerabilities in the relatively small amount of code that was actually written by the company whose name is on the software.

3.      While the supplier bears responsibility for preventing those vulnerabilities from appearing in the first place, and fixing them when they do appear, the customer plays a crucial role, too. They need to let the supplier know (in contract language where possible and feasible, but that’s certainly not the only way to do it) that they want component vulnerabilities fixed as fast as the supplier (usually) now fixes vulnerabilities in their own code. Only an SBOM allows the customer to do that, since that’s the only way they can track the supplier’s progress (or perhaps lack thereof) in fixing component vulnerabilities.

4.      But if SBOMs are so obviously a good thing, and if software suppliers are using them now for their own risk management purposes (which they are, in large part), why aren’t they giving them to their customers?

5.      The reason is there’s still uncertainty about:

a.      The “rules of the road” for production, distribution and use of SBOMs (as well as VEXes. Just as SBOMs are crucial for enabling software supply chain cybersecurity, VEXes are crucial for enabling SBOMs – and it turns out they’re good for a lot more than that). These rules don’t need to be etched in stone and handed down at the summit of Mt. Sinai, but they do need to be agreed upon between suppliers and their customers. And not necessarily on a country-wide or world-wide basis; on the basis of a particular industry or say one part of one industry would be fine.

b.      Tools available to “ingest” SBOMs, so that organizations can use them to learn about vulnerabilities found in the components in the software they use.

c.      Services available to help software-using organizations “consume” SBOMs and use them for vulnerability management, the number one use case for SBOMs.

d.      Support for SBOMs from the vendors of tools for vulnerability and configuration management.

As I said, large numbers of software suppliers – including probably all of the bigger ones – are producing SBOMS for their products now, and using them to manage their own product risk. For example, tools that produce SBOMs will often search the NVD for vulnerabilities that apply to components in the product, giving the supplier a chance either to patch the vulnerabilities or replace a risky component with a less-risky one – before they ship the product. On the consumption side, there’s very little in the way of tools or services now. Fortunately, they’re on the way.

There are two goals of the energy SBOM proof of concept. The first is education. While it’s certainly not necessary for everyone who uses SBOMs to become an expert in them, it does help to have a general understanding of what they are and how they’re put together. That’s why the PoC is currently running “cooking classes”, in which very knowledgeable – but accessible – SBOM practitioners demonstrate how their tools work.

On September 22, Steve Springett, co-leader of the OWASP project that developed and supports the CycloneDX format, demonstrated production of an SBOM in that format, based on an open source project. It was a great presentation, and the video should be up soon on the web site (along with the videos of most of our meetings so far). On October 6, I believe Kate Stewart of the Linux Foundation, who has been the leader of the SPDX project for a number of years (SPDX and CycloneDX are the two “full-featured” SBOM formats. SPDX was just designated an ISO standard), will provide a similar presentation using her favorite format.

We’ll have other cooking classes – including on VEXes – after that. We currently have over 200 people on our mailing list for this track. You don’t have to be in the energy industry to attend the education meetings – you just have to be a user of electricity.

The other track of our project has been slower to start up, but we expect it will start in October. In this track (the meetings for the two tracks are held every other week on alternating weeks), individual asset owners will sign NDAs with individual suppliers. We currently have about six asset owners and six suppliers interested in doing this. The suppliers are all very large ones, whose products are widely used in the electric power industry. The suppliers produce both intelligent devices and separately-installed software.

Each supplier/asset owner pair will decide on one or more products for their exchange. The supplier will provide SBOMs and VEXes for those product(s), and the asset owner will consume them. As I mentioned, the tools and services available on the consumption side are much fewer than on the production side. However, both tools and services are available, and an asset owner won’t be required to create their own software tools to ingest and parse the SBOMs and VEXes.

So here’s where the cooperation comes in: These exchanges won’t work unless the suppliers and asset owners can agree on the rules for the road on SBOMs and VEXes: the fields included in the documents, how the information will be extracted from the SBOMs and VEXes, how that information will be integrated with current vulnerability management processes, etc. As issues come up, the group will discuss them together. And as we learn what works and what doesn’t, we’ll make that information available to the NTIA initiative and to the general public (since every document the NTIA produces is public).

And to be honest, this strikes me as not a bad model for supplier/vendor cooperation in cybersecurity in general. For example, it would be great if NERC put together a team of suppliers and asset owners to work on a document on good security practices for suppliers.

However, that idea has been floated a few times in the past few years, and it always runs into the same hurdle: Everyone at NERC is sure it will never be allowed by the lawyers (although I kind of doubt that they run to the lawyers every time questions like this come up). The reason why they’re sure it won’t be allowed is that it could be seen as providing compliance guidance on CIP-013. Providing guidance is a firm no-no for employees of NERC and the Regional Entities.

Of course RE employees, including auditors, provide compliance guidance all the time. But – except for one employee of one of the RE’s, whom I have quoted many times in these posts – the guidance is never written down and always delivered with the disclaimer that “These are my personal opinions, not those of NERC or my Region.” And ironically, nobody has ever been able to point out to me where in the NERC Rules of Procedure (or even GAGAS) the ban on compliance guidance can be found. Nevertheless, it’s firmly written in the mind of every NERC or Regional employee, even if they provide guidance anyway. And in cases where NERC has provided real guidance on NERC CIP – as in some of the Lessons Learned provided during the CIP v5 implementation period – those were slapped down by the lawyers, and the documents were withdrawn. The most egregious example of this had to do with the controversy over the meaning of the word “programmable”.

So don’t look for much help from NERC on the idea of a forum on supplier best practices. Who else might do this?

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the National Technology and Information Administration’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] I was referring to hardware and software vendors, though. As Ben Miller of Dragos pointed out in the article, service vendors do participate behind the scenes sometimes, since they can easily provide consultation to their utility clients who are directly participating in GridEx.