What if SBOMs are included in the new EO?

As I mentioned in Friday’s post, Reuters carried an article on Thursday that said there would soon be a new Executive Order regarding software security, and that it would include a requirement for software bills of materials. Of course, given how much I’ve advocated for SBOMs since last summer, it would be hypocritical of me to say that I don’t welcome the idea that there might be a mandate driving their production and use. Since I’m sure SBOMs will come sooner or later – they have to – I don’t mind the government putting their weight behind them, to speed up that process.

But here’s the problem: It would be an unmitigated disaster, were the feds (and I’m not sure who would enforce this order. DHS, and CISA specifically?) to simply mandate that certain software suppliers to the federal government start delivering SBOMs for all of their software products within say the next six months to one year. This is because there are still far too many unanswered questions about SBOMs, for this to be even remotely possible.

Here are two questions that have been answered (there are many others that have been answered as well):

Q: SBOMs need to be machine-readable, given the huge volume of components that software end users might want to track for vulnerabilities. For example, the average software product contains over 100 components (and some contain thousands). If an organization uses 200 different products and they get SBOMs for all of them, that means they should track 20,000 components. Can SBOMs be made machine readable?

A: This problem is solved. There are currently three generally accepted formats, all of which are machine-readable: SPDX, CycloneDX and SWID; it is likely that more will appear in the future.

Q: Speaking of SBOM formats, which is the best one?

A: Each of the three formats provides what the National Technology and Information Administration (NTIA) has defined as the minimal fields required in an SBOM. Beyond that, each format provides capabilities that are important to some suppliers and users but not others. The NTIA Software Transparency Initiative members have decided that there is no need to choose one format over the others – given that tools are already available to translate among formats.

But here are some questions that haven’t been answered yet and are unlikely to be definitively resolved in the next year. I wrote about two of the biggest last fall: the “naming problem” and the “problem” caused by the fact that the majority of vulnerabilities in components aren’t in fact exploitable in the end products that include those components, which I discussed in this post.

There are also some questions that can only be answered through experience – that is, through actual production and use of SBOMs. This is what the industry Proofs of Concept are doing. The healthcare industry has been conducting PoCs for SBOMs since 2018, and PoCs for electric power and autos will start in the near future. There will be other PoCs as well, as momentum picks up.

Answers to all of these questions are emerging from the PoCs, but trying to rush answers now would inevitably require finger-in-the-air-type guesses, rather than answers based on experience, which is what the PoCs are providing. These questions include:

1.      How often should SBOMs be re-issued?

2.      How can SBOMs be provided to end users in a way that doesn’t overwhelm them with seemingly random documents?

3.      What kinds of “intermediary” services would be required, to make SBOMs issued in one of the three current formats usable for vulnerability management purposes in the near future? This may be one way of “jump starting” successful use of SBOMs, vs. having to wait for the current vendors of vulnerability management products to modify their products to ingest SBOMs directly.

4.      What about other potential uses of SBOMs, like for license management? The fields for this are already in the SBOM formats, but in most cases they’re not being used now. There are certainly other uses for SBOMs as well. Facilitating these use cases doesn’t require technical changes, but it does require agreement on procedures.

5.      How can access to the data in an SBOM be protected, so that only the organizations (usually customers) that the supplier wants to have access in fact do so?

There are many more of these questions, of course. My point is that it would be a big mistake for the feds to mandate that SBOMs be produced and used until these questions are much closer to being answered than they are now.

Fortunately, one federal agency has already faced these issues and pointed out the best way to address them, while at the same time making a firm statement that SBOMs will be mandatory. The FDA in 2018 was concerned about securing the software in medical devices used in hospitals (e.g. infusion pumps) and stated that they would require SBOMs for those devices in the future. However, they didn’t set a date for this, or provide any specifics about what they would mandate.

The FDA did this because they knew full well that trying to issue specific regulations for SBOMs at that time would have led to nothing but confusion and bad feelings. Instead, they supported the NTIA Software Transparency Initiative as the best road to resolving all of these questions. They have still not set a date for regulation and they continue to support the initiative, including the healthcare Proof of Concept.

This approach turned out to be very successful. The healthcare industry – both the hospitals and the medical device makers – realized they had to cooperate to make SBOMs a reality, rather than just a vague aspiration. That is what the NTIA Software Transparency Initiative is doing, and especially the healthcare PoC. Other industries, including energy, can help advance this effort by starting their own PoCs.

If SBOMs are included in the EO, I think the best approach is to follow the FDA model: State that SBOMs will be required in the future (or perhaps even set a date to begin rulemaking, say 2-3 years from now).  This puts all the pressure needed on industry to resolve these and other questions, so that in a few years it will make a lot of sense to talk about some regulation of SBOMs (if it’s still needed. Perhaps SBOMs will be in such wide use already, that there will be no need to mandate anything more than that).

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

Next (and final) introductory Energy webinar on SBoMs scheduled!

Dr. Allan Friedman of the NTIA announced today that the final in the series of four webinars to introduce the energy industry to the concept and uses of software bills of materials will be held at the following virtual location and time:

SBOM Info Session – Energy Perspectives

Monday, April 12, 12-1pm ET

Teams Link

Dial-in: +1 202-886-0111,,617460066#     United States, Washington DC

Phone Conference ID: 617 460 066#

Find a local number

This webinar will consist of discussion by people involved with (or at least planning on) producing SBOMs (software and intelligent device suppliers) and consuming them (electric utilities and industry organizations). The topic will be why they think SBOMs are important, both for software suppliers and software users. If you would like to receive a calendar invite to this meeting, or if you would like to be placed on Allan’s energy mailing list, email him at afriedman@ntia.gov.

Allan also announced that the recording of this Wednesday’s webinar is available here. You can find the recording of the first webinar in the series here and the second here.

One more thing. This article appeared in Reuters yesterday. To cut to the chase, it says the new Executive Order regarding software supply chain security will include a mandate for SBOMs. I know nothing more about that and Allan can’t talk about it, whatever he might know. But we all might know about it next week, or if not soon thereafter. If nothing else, this shows SBOMs are in the wind. You shouldn’t wait for a regulation to at least start learning about them.

 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Reminder: SBOM webinar coming up on Wednesday

Just a reminder: The third in the series of informational webinars on use of software bills of materials in the energy industry will be presented this Wednesday (March 24) from 12-1 Eastern Time. As with the others, it’s sponsored by the National Technology and Information Administration of the US Department of Commerce. This session will focus on lessons learned from software and security experts who have been working to build and utilize SBOMs.

The webinar will include:

·        Bruce Lowenthal, Director of Product Security for Oracle, discussing why SBOMs are important for Oracle (Bruce is an active participant in the NTIA Software Transparency Initiative).

·        Dr. Jess Smith, Senior Cyber Security Research Scientist at Pacific Northwest National Laboratories, discussing use of SBOMs and HBOMs (hardware bills of materials) in the DoE CyTRICS project. PNNL and Idaho National Labs will participate in the upcoming Energy Proof of Concept.

·        Representatives of a few of the medical device makers and large hospitals that have been participating in the ongoing Healthcare PoC describing their experiences and why the PoC has been valuable to them.

I hope you’ll be able to join us! Here’s the contact information:

SBOM in Energy Info Session – Lessons Learned

Wednesday, March 24, 12-1pm ET

Teams Link

Dial-in:+1 202-886-0111,,239096326#   United States, Washington DC

Phone Conference ID: 239 096 326#

Find a local number

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. 

 

The wrong way to promote SBOMs

On one of the mailing lists of the NTIA Software Transparency Initiative this week, this article was passed around and got a lot of attention. On the one hand, it’s the best summary of the (long-term) benefits of SBOMs that I’ve seen in the press so far (it was written by two people from MITRE Corporation, one of whom is a former Deputy Director of the NSA). The authors both firmly believe that having SBOMs widely available is important for national security. No argument from me on that.

On the other hand, there are right ways and wrong ways to promote something. The wrong way is to a) not do much research on the subject yet assume that the little you’ve read about it makes you an expert, and b) set huge expectations that SBOMs won’t possibly be able to deliver. Unfortunately, this is a textbook example of the wrong way to promote SBOMs.

Ironically, it doesn’t seem that the authors of the article even talked with a very respected member of MITRE’s own staff, who regularly participates in some NTIA meetings and the email discussions. Nor, for that matter, do they seem to have read any of the NTIA documents on SBOMs, or participated in any of the meetings. If they had, they wouldn’t have made the following mistakes:

First, they imply that SBOMs might have in some way alerted users to the SUNBURST malware in the SolarWinds Orion updates they were installing (or even better, alerted SolarWinds to the fact that the malware was present in all the updates they sent over a 2-3 month period). Actually, while the SolarWinds attack (along with others) shows that software supply chain security is without doubt the most important supply chain cybersecurity consideration now, the fact is that the super-sophisticated way in which the Russians conducted the attack (Microsoft thinks that up to 1,000 people were involved) couldn’t have been detected by SolarWinds (although the fact that the Russians were operating at will inside their network for 15 months is inexcusable). Even though the Russians in fact implanted a component (SUNBURST) in about seven builds of Orion, they did it in such a way that it would have never been included in an SBOM that was generated at the end of the build (which is when they’re usually generated nowadays).

Second, they speak of a “standard” for SBOMs coming this year. This is wrong in many ways:

1.      The NTIA doesn’t develop standards, mandatory or otherwise. They also don’t develop guidelines, best practices, fatwas, Papal Encyclicals, Executive Orders, or anything else. The purpose of the Software Transparency Initiative is to get software suppliers and users together to determine what would be the best way to remove barriers that are currently preventing implementation of SBOMs. The Initiative has decided that the best mechanism for figuring these things out is to conduct Proofs of Concept, in which software and suppliers and users in a particular industry collaborate to work out the different issues of production, distribution and use (and if you want to learn more about the upcoming Energy Proof of Concept for SBOMs, attend the webinar next week, and also email afriedman@ntia.doc.gov to be put on the mailing list).

2.      There is already a widely accepted format for SBOMs; in fact, there are three widely accepted formats: SPDX, CycloneDX and SWID. Moreover, MITRE (funny thing about that!) is rumored to be working on a fourth. There’s nothing wrong with having multiple accepted formats, since there is already a tool that does a good job of translating between the formats. In fact, I’ve already observed the SPDX format adopting innovations in CycloneDX and vice versa (both are open source projects, one conducted by the Linux Foundation and one under OWASP. The leaders of both projects are active participants in the NTIA Initiative. They certainly don’t view each other as competitors).

3.      The issues with SBOMs are really around questions like: How often should an SBOM be generated? How should components be named in the SBOM, so that users can use the names to identify vulnerabilities (this is the Naming Problem)? How can suppliers avoid being overwhelmed with support calls for component vulnerabilities that aren’t in fact exploitable in their product (the issue I discussed in this post)?

4.      Allan Friedman, who leads the Software Transparency Initiative for NTIA, has said several times that he’s had to talk a government agency (or two or three) out of the idea of making SBOMs mandatory in some way. The fact is that, with all these important questions still to be answered (and perhaps addressed in slightly different ways by different industries), there simply is no way there can be a standard for SBOMs in the near future. Perhaps in five years there will be enough agreement on the answers to these questions that a standard would be possible. But I think it will be closer to ten years before use of SBOMs is so widespread that there needs to be regulation of them.

Third, the authors overpromise what SBOMs will do, or at least when they will be able to do it. For example, they say “When a new hack is uncovered, cyber defenders can use these software manifests to identify precisely which systems in the U.S. government are vulnerable.” It will be at least ten years before SBOMs are widely accepted and used, to the point that it might be possible to talk about identifying systems that are affected by a new vulnerability “precisely”. Until then, when serious vulnerabilities in components are identified (such as the Ripple 20 vulnerabilities), we should be happy if we can identify some fraction of the systems affected by a particular component vulnerability. That fraction will undoubtedly grow over time, but it will never be 100%. But having maybe ten out of 100 systems protected is better than having 0 out of 100.

Here are the three major changes I would have suggested in the article:

1.      Get rid of the talk about a coming standard for SBOMs. That’s simply not true, and more importantly it will make some people get into a compliance mindset regarding SBOMs. They will conclude “If there’s a standard coming, I’ll certainly be given plenty of time to come into compliance with it. I’ll focus for the time being on the 20 other standards that I have to comply with now.” This mindset can literally hinder adoption, not promote it. The fact is that the case for SBOMs is very compelling, with or without regulation: The average software product has 135 components, although some products contain thousands of components. Just like the product itself (i.e. the code written by the company whose name is on the software) can develop vulnerabilities, each of those components can do the same. Without an SBOM, you’ll never know about those vulnerabilities, unless your supplier is willing to tell you (fortunately, a large portion of them are willing, but it’s nowhere near 100%. A 2017 study by Veracode said that only 52% of software suppliers even bother to patch component vulnerabilities, let alone tell their customers about them).

2.      Don’t talk about solving our problem in any absolute sense. And especially don’t say (as unfortunately the authors did) that “Software vulnerabilities have been with us for more than three decades, since the Morris Worm crippled internet servers in 1988. We cannot tolerate it (sic) any longer.” Guess what? SBOMs will never solve the problem of software vulnerabilities, just like cold medicine will never solve the problem of common colds. What they will do is provide visibility into what is without doubt the largest source of vulnerabilities in any modern software product: the third-party and open source components it’s made of.

3.      Start to learn about SBOMs (especially if you plan to write articles promoting them). Above, I provided links to the document page of the NTIA Software Transparency Initiative site, as well as information on the upcoming third in NTIA’s series of four webinars on SBOMs for the energy industry. And if you’d like to review videos of the first two webinars in that series, go here and here.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

Finally, a little light at the end of the tunnel in Texas

 

Yesterday, I wrote this post about the continuing financial mess in Texas; it was based on an article yesterday in UtilityDive. The post described how the Lieutenant Governor has done a 180-degree shift and now is trying to get refunds for everyone who was overcharged due to the high prices for power during the week of Valentine’s Day. That’s a great idea – but the only problem is it’s illegal, period. My post ended with the implication that there will be no end to the uncertainty in the near future, and perhaps for years.

However, I realized early this morning (when most of my posts get written, at least mentally) there was another article yesterday that pointed to what isn’t a solution to the whole problem, but may be a solution to part of it (at least $3 billion, maybe more, of the entire $29 billion potentially overcharged. Hey, it’s a start, anyway). Of course, it’s not a solution that the people of Texas will like, since it will require them to pay most or all of it – either as taxpayers or ratepayers. But guess what, folks? You’re going to pay most or all of the entire amount of overcharges anyway. This way, at least you’ll know how much you may end up paying for the $3 billion (and possibly higher) reduction in the magnitude of the problem.

That article was in the Wall Street Journal, and discussed two news events:

·        ERCOT is in talks with Goldman Sachs to tide them over a $3 billion shortfall in power payments. They need this because some distributors of power aren’t paying ERCOT for it (ERCOT, besides managing the grid in Texas, serves as a clearing house for wholesale power payments. The power distributors pay ERCOT for the power they buy from generators, and ERCOT pays that to the generators).

·        Normally, unpaid bills are paid by all of the other participants on the grid, as dictated by ERCOT’s charter. However, a number of power buyers are disputing the charges (or have declared bankruptcy, as in a couple cases, with more certain to come), and it’s unlikely ERCOT will be paid in full soon, if ever. This leads to the second event:

·        CPS Energy, the huge municipal utility that serves San Antonio and surrounding areas, sued ERCOT last week, saying they shouldn’t have to pay for other participants’ shortfalls. They said the “extreme confiscatory prices have caused many providers within the Ercot system to become insolvent.” The municipal utility for the city of Denton has filed a similar suit.

CPS is both a power generator and a power distributor (probably most large municipal and cooperative utilities in Texas, as well as in many other states, are similar). In fact, CPS is a very large generator. Yet CPS “ran up $1.1 billion in costs buying electricity and natural gas last month”, according to the lawsuit.

Why did CPS have to spend so much money buying power, when they generate so much of it themselves? That’s simple: because they don’t generate enough to serve their whole service area, and because the fact that most Texans use electricity to heat their homes meant that demand skyrocketed during the extremely cold (for Texas) weather during Valentine’s Day week.

But here’s a more interesting question: Didn’t CPS make a lot of money on the power they generated that week? If so, the fact that their costs went up so much would almost certainly have been offset by the amount they received for selling their power – in fact, their receipts from selling power should have been some huge multiple of the costs they incurred buying it. Why are they trying to kill the goose that laid the golden egg that’s enriching them?

Here’s where municipal and cooperative utilities differ from the investor-owned power generators in Texas: They’re owned by their customers. A cooperative is literally owned by the consumers that it serves. A municipal utility is technically owned by the municipality it serves, but since the mayor and other elected officials of the municipality owe their positions to their voters, municipal utilities are effectively “owned” by their consumers as well, since they can all vote if they want to.

Given this, it’s just about certain that municipal and cooperative utilities didn’t pass on to their consumers anywhere near the full cost of the power they had to buy during the crisis – if they passed on any of the costs at all. This is certainly why Brazos Electric Cooperative – the largest coop utility in Texas – filed for bankruptcy two weeks ago. And the article points to a couple other munis or big coops that will likely have to file, if there’s no relief from the bills they owe.

Note that CPS doesn’t seem to be asking for relief from costs they incurred for power they purchased. Rather, they’re asking for relief from ERCOT’s practice of spreading costs of payment defaults across all ERCOT grid participants (note that this isn’t synonymous with “Texas power market participants”, since the far eastern and western parts of Texas are on different grids, and not part of ERCOT’s grid. Those areas aren’t part of this mess, but that doesn’t mean they won’t end up paying part of the costs in the end, since they do – as far as I know – pay taxes to Texas).

Here’s why I believe the CPS lawsuit points to at least a partial solution to the $29 billion payments problem in Texas:

1.      If CPS is in fact willing to absorb the $1.1 billion in costs they directly incurred during the crisis, this means those costs will be paid by their “owners” – the ratepayers of CPS’ service area (although the costs won’t be billed in one bill, as is the case with customers of investor-owned power providers. If they spread this over say ten years and get financing during that period, it may well be fairly manageable).

2.      Note that CPS’s take on this is different from Brazos’ take, since the latter has filed for bankruptcy – meaning they see no easy way to pass all of their costs on to their consumers over a reasonable period of time. But it seems that at least some of the municipals and coops will be able to weather this storm, although their ratepayers/owners/voters aren’t going to jump for joy at the prospect of sharply increased bills over a long period. This reduces the amount of the ultimate shortfall in payments, although how much is uncertain at best.

However, it’s clear to me that ERCOT might as well give up trying to collect the $3.1 billion payments shortfall from the grid participants. They’ll presumably get the loan from Goldman Sachs, and they’ll have to pay it back either by surcharging power sales for say the next ten years, or by asking the state to sell bonds to make them whole. And guess who will ultimately have to pay off those bonds? Why, the good citizens of Texas will. Who else did you think will?

In fact, my guess is the citizens of Texas will ultimately pay a good deal more than $3.1 billion for the very expensive week in February, and the poorly designed market system that led to it.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

Things are playing out in Texas (not improving, mind you, just playing out)

For those of you who haven’t been able to devote your life to following what’s happening in Texas in the wake of the Valentine’s Week Storm (my name for it, although there hasn’t been a lot of love lost so far), this article from UtilityDive helps you update your scorecard. To summarize for you:

·        The star of the article is Lt. Gov. Dan Patrick, who made a remarkable statement last April in regards to Covid: “there are more important things than living and that’s saving this country.” My guess is the 45,000 Texans who have died so far in the pandemic might disagree with him, but they’re currently unable to comment.

·        Not content to make light of the lives of his fellow Texans, he followed up that statement with one on Feb. 24 (on Fox News), making light of their loss of life savings: “We have in Texas, you can choose your energy plan and most people have a fixed rate. If they had a fixed rate per kilowatt-hour, their rates aren’t going up…. But the people who are getting those big bills are people who gambled on a very, very low rate…going forward, people need to read the fine print in those kinds of bills.”

·        My guess is that, unlike with the first statement, Patrick got a lot more pushback from the second one, because he seems to have experienced a remarkable (political deathbed?) conversion. As the UtilityDive article recounts, in a couple of weeks he’s changed from a cold-blooded defender of free markets into a wild-eyed consumer advocate, who’s pressing the lone remaining member of the Texas PUC to retroactively change the price of power during the week in question.

·        As the PUC member points out (actually, I should call him “the PUC”, since he’s all that’s left of it now), there’s only one problem with this request: there’s no legal basis for it. 

So that’s the current state of this crisis: Texas is caught between an immovable object and an irresistible force. The only outcome I can see is that ultimately the Texas citizens and ratepayers will bear some of the (perfectly legal) costs, and those that have paid the bills already will be only partially reimbursed, so they’ll share the costs as well. Nobody will be happy, but hopefully that will provide some momentum to fix the problems that led to this human and economic disaster.

Speaking of the economic problems, if you found this post interesting (or if you hated it and want to see me receive my comeuppance), you might enjoy the Energy Central conversation I’m having with Robert Borlick. He’s someone who obviously knows much more about the Texas power market than I do, and he wrote in to correct me on a few important points. I replied to him, and I’m hoping he’ll continue the conversation (I’ve been posting almost all of my blog posts on Energy Central since last August, and they’ve proven popular there, too. One difference with EC is I get a lot more comments – including long ones like Robert’s – there than I ever have on my blog. You can read me on either medium – I earn the same princely sum either way. Anyone can read any EC article and its comments, but if you want to comment on your own you have to be a member).

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

See you at RSA 2021 – without the hotel bill!

 (or in my case, without the AirBnB invoice)

This year, RSA will be all virtual. It will run from May 17-20. And I’ll again be speaking on a panel led by Mark Weatherford, former NERC CISO and now CSO and Board Member of the National Cybersecurity Center. My fellow panelists will be Jennifer Bisceglie, Founder of Interos Inc, and Chris Blask, Global Director of Industrial Security of UNISYS.

Our panel will run from 2:40 to 3:20 PM Pacific Time on May 20 (and don’t give me an excuse that you have to miss our session to catch a flight home from SFO! Normally, the last afternoon of the conference is pretty quiet because so many people have left). However, please stick around afterwards, because there will be an “Additional Audience Engagement” session, starting at 3:20. I have never heard of one of these before at the conference (if they’ve even had them before), but RSA says it’s a “40-minute interactive QA discussion” on the topic of our panel. This is great, because I’ve never been on a panel where it was possible to get into the topic in much depth, or to take many questions. This will be an exception.

Well, I guess that’s about it…What, you want to know what the topic is? Sure, it’s “DBOM and SBOM: New Options for Better Supply Chain Cybersecurity”. The description is “The global supply chain includes a mystifying accumulation of digital and software components that generate perplexing cybersecurity risk management challenges. These supply chain risk management challenges can be addressed through the focused application of both Digital Bill of Materials (DBoM) and Software Bill of Materials (SBoM) to document component provenance to consuming organizations.”

If you’ve been reading this blog at all lately, you’ve certainly heard of SBoM. But what’s DBoM? DBoM is…well, here’s a very succinct summary of it by Chris; you can also find a few podcasts on DBoM by searching on his name. But I’ll admit: I’ve been going to (at least) weekly meetings discussing DBoM since last August, and I’m just now beginning to realize how revolutionary it is – and how it can lead to great improvements in security and efficiency of supply chains, which I (and the others in the group, frankly, including Chris himself) are just beginning to understand. It's definitely worth a few minutes of your time to learn about this.

Oh, and in case you think this is a (poorly) disguised product sales pitch, I’ll point out that DBoM is an open source product.

I hope to see you there on May 20! And be sure to leave enough time to find the room where we’re speaking (for some reason, I don’t see the room number on the email from RSA). The Moscone Center is a big place.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

A great article on SBOMs and the Energy Proof of Concept

Today, Utility Dive published a great article discussing SBOMs, and especially the upcoming Energy Proof of Concept exercise. To be honest, it’s the best article I’ve read yet about SBOMs, and the only one that intelligently describes why the NTIA Software Component Transparency Initiative has latched onto proofs of concept as the primary tool for driving acceptance of SBOMs in the US, and frankly worldwide.

I’ve been quite impressed with how Robert has taken the time to really understand what SBOMs are about, as well as what the Initiative is about. Some reporters take the “Get a quote and end the call” approach to interviews, but that’s not what Robert does.

I’ll let you read the article, but I want to repeat a post that Virginia Wright of INL put on LinkedIn today, as a comment on Allan Friedman’s post calling attention to the UtilityDive article (I do this with her permission):

The Idaho National Laboratory (INL) is delighted to join with Allan and NTIA, DOE, and multiple energy sector vendors and asset owners to explore the application of SBOM to the enumeration of the energy sector digital supply chain. Your ideas have been an inspiration for the CyTRICS program (http://www.inl.gov/CyTRICS). Thanks, Allan for your tireless advocacy of SBOM and supply chain transparency!

Of course, it’s quite exciting to have INL committing to join the energy Proof of Concept. If you know anything about CyTRICS, you’ll know it can never succeed without SBOMs.

And here's another comment to Allan's post, added by Cheri Caddy, Senior Advisor - Cybersecurity Strategy and Policy at DoE (CESER):  "The Department of Energy is pleased to be a partner in this effort. BOMs are a great way to get to digital subcomponent illumination. Thank you for your leadership, Allan!"

Good stuff happening!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

The new administration gets it: Software is where the risks are!

Christian Vasquez of E&E News published on Monday a very good article that reviewed Anne Neuberger’s keynote address for the SANS (virtual) ICS Summit last Friday. In it, she said, according to the article, that “Biden is planning to unveil an executive action that aims at ‘building standards for software, particularly software that's used in critical areas.’”

I was quite happy when I read that (I hadn’t attended her keynote for the Summit on Friday). It’s been clear to me for quite a while that the real supply chain security risks – for any industry – are in software (in fact, Robert M. Lee said exactly this in his keynote to the Summit on Thursday). In fact, I know of no true hardware supply chain attack ever (and I’m not talking about an attack on firmware, as in the Mirai botnet. Firmware is just software installed on a different medium, and mitigating firmware risk is a lot like mitigating “regular” software risk) – that is, one in which the attackers altered the microcode in a processor.

I’m certainly not saying that there could never be a hardware supply chain attack, but we need to look at the record. On the software side, you have SolarWinds, Equifax, Delta Airlines and other very successful software supply chain attacks. And on the hardware side you have…rumors that the NSA might be altering microcode in network devices being sent to some foreign countries. On which side should you allocate your scarce time and money resources?

Ms. Neuberger mentioned two different concerns that the new order will address. The first is “in response to the massive Russian-linked espionage campaign that has affected nine agencies and around 100 organizations by exploiting a commonly used software product from Austin-based SolarWinds.” Unfortunately, that’s all the information she gave on this topic.

Here's what I’m hoping the EO will do regarding software supply chain security: Scrutinize the controls the supplier has on their software development environment. The appalling thing about SolarWinds is that the Russians penetrated the SolarWinds development network (they were presumably already in the IT network) in September 2019. As documented in a great article by Crowdstrike, they had free rein in that network, and were able to first plant “proof of concept” code (i.e. a non-malicious version of Sunburst) in two or three updates for the flagship Orion product.

Having demonstrated that they could do that undetected, they went on to build Sunspot, perhaps the most sophisticated malware since Stuxnet - or even more sophisticated that the latter - and installed it in the build network to guide the rest of the effort. In March 2020, Sunspot started planting Sunburst in Orion builds. This continued until June, by which time Sunburst was in about seven Orion updates that had been provided to customers. In June, the Russians decided the fun was over, covered their traces and pulled out.

But they didn’t do this because they’d been detected or feared being detected – they did it because they already had been so successful that they knew they weren’t ever going to be able to exploit more than a fraction of the 18,000 organizations that had downloaded a version of Orion containing Sunburst (it seems they actually used Sunburst to attack about 100 organizations). And indeed, SolarWinds never detected them. The world might never have known about Sunburst if some observant person at FireEye hadn’t noticed that an unauthorized device had been added to an account.

Clearly, SolarWinds dropped the ball on this, and there’s no reason to believe they won’t continue to drop the ball from now on. They – and a number of their peers, including the cloud providers – need to be regulated just like any other critical infrastructure. Because that’s what they are.

The other concern Ms. Neuberger mentioned was the need for monitoring of OT networks. Here, the example was the Florida water system that was compromised. The compromise wasn’t discovered by any sort of monitoring system, but only because an operator noticed his cursor was moving on the screen, even though his mouse wasn’t. This is another problem, but it’s probably much easier to solve than the first one.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

 

After all, what’s $29 billion between friends?

It now looks like the financial storm that resulted from the meteorological storm in Texas three weeks ago has turned into an absolutely huge cluster you-know-what. For those of you who aren’t keeping score at home, here’s my take on the situation. You might want to read my previous post on this subject, since I’m building on what I said there.

1.      The bitter cold that came to Texas over the Valentine’s Day weekend caused a perfect financial storm: Demand for electric power soared because most people in Texas heat their homes with electricity, while at the same time supply was severely constrained because so much generation was put out of service.

2.      Monday, Feb. 15 was the Day from Hell in which ERCOT, the Texas grid operator, had to order large-scale outages in order to keep the system from being completely overwhelmed (which would have resulted in long-term damage to a lot of big generation – meaning probably many months of outages after that).

3.      At the same time, the wholesale market price for power jumped from its normal level of around $20 per megawatt-hour to $1,200 per MWH. That alone was pretty bad. Unless you compare it to what happened next.

4.      The Texas Public Utility Commission was meeting at the same time. Being all very knowledgeable about the basic principles of microeconomics (although not much more than that, it seems), they concluded that the fact that the market price was “only” $1,200 – while at the same time a lot of people were sitting in cold homes and couldn’t get power no matter how much they would have been willing to pay – meant only one thing: The price was being artificially constrained by the fact that so much demand was being curtailed (load was being shed, in power industry terms). They decided the best thing they could do would be to raise the maximum price to $9,000 (the highest price allowable) – a more than seven-fold increase.

5.      And they were absolutely right – at least as far as their Microeconomics 101 course went. If not all demand is being satisfied (i.e. some people can’t buy the commodity at any price), this normally means that the true market price is higher than the apparent market price (namely, the $1200 price at the time). The way you clear the market in such a situation, according to a highly simplified view of microeconomics, is you allow the price to rise to the level at which quantity demanded will fall, because people decide they don’t need the commodity badly enough to pay the higher price. At the same time, quantity supplied will increase, because suppliers who didn’t think they could make money at the old price realize that they can at the new price. Quantity supplied will equal quantity demanded, markets will clear, and we’ll all live happily ever after.

6.      Unfortunately, those PUC members never took Micro 102. If they’d done that, they might have learned:

a.      When supply is constrained by non-economic factors (such as the effect of cold weather on generation facilities which aren’t prepared for it), it doesn’t matter how high you make the price – you’re not going to get any more supply. That was certainly the case in Texas on Feb. 15.

b.      The end consumers of electric power (i.e. Joe and Jane Consumer) would never be willing to pay the $1,000+/day prices that would have been required to clear the market anyway. My guess is even at $300/day, most consumers would prefer to endure some cold, rather than start to cut into their life savings. In other words, the $1200 price was probably pretty close to the true market price. It was certainly a lot closer than $9,000 was.

c.    More importantly, consumers almost never know the real-time price of the electric power they're using; they learn what it was when they get their monthly bill. So there was no way they could make an informed decision whether or not to turn their thermostats down to freezing (or even below, if that's possible), when it was always possible that the cost of leaving the thermostat where it was would later prove to be negligible.

7.      So the PUC was clearly wrong in basing their decision to raise the price on the fact that ERCOT had ordered so much demand to be curtailed. But anyone with half a brain (a commodity evidently in short supply at that time and place) should have at least realized that the rationale for that high price would go away once the curtailments ended. On the 18^th (Thursday), ERCOT stopped ordering curtailments (although because a number of utilities were still struggling to meet demand, there were still lots of people without power). Yet they didn’t lower the price by a nickel until Friday, even though the market price returned to normal by Thursday. To quote a Wall Street Journal article from March 4, “On Feb. 18 at 5:30 p.m., for instance, the market price of electricity was $22 per megawatt hour—and the utility commission’s additional price on top of the real-time market price, as imposed by Ercot, was $8,979 per hour.”

8.      So the good news is that by Thursday, ERCOT was no longer imposing a price that was seven times the market price. The bad news is that they were imposing a price that was 409 times the market price. In other words, they made the PUC’s original decision much worse than it already was.

9.      Let’s fast forward to last Thursday morning. The WSJ, in the article I just linked, pointed out that a market monitor who had been hired by the PUC said that the market price should have been restored – according to the PUC’s rules - on Thursday, when ERCOT stopped ordering curtailments. The Financial Times also had a good article that touched on other aspects of this story (thanks to my college roommate Dave Reed for forwarding me that link). That was 33 hours before the market price was actually restored. They said that the $16 billion surplus in receipts that was generated (pun intended) during those 33 hours should be returned to the entities that paid it (and ultimately, hopefully, to the end consumers whose huge bills funded it).

10.   The tone of both the WSJ and FT articles was something like “Here’s a sensible proposal that won’t make everything right, but it will at least help some of the victims of this mess.” Neither article quoted anybody who didn’t believe that the PUC would take their advice and order the $16 billion to be refunded.

11.   So I was very surprised to read in the WSJ on Saturday morning that, while the PUC (now down to just two members, since the chairwoman resigned two weeks ago, and deservedly so. The ERCOT CEO was fired last week, although he’s still on the job for 60 days) hadn’t made a final decision on this idea, they’re currently very much leaning against it. The new chairman worded their reasoning very succinctly: “It is impossible to unscramble this sort of egg”.

12.   And indeed he’s right. This isn’t a case of having a bunch of suppliers on one side, who are sitting on this huge hoard of accounts receivable, and a vast, unwashed (since they haven’t been able to take showers because their pipes broke) mass of consumers on the other side, who are facing huge bills. Instead, there are all sorts of nuances to the story. Here are a few:

a.      A lot of power suppliers are themselves the biggest victims. That’s because they were under contract to deliver power during the crisis, but since their plants (or wind turbines) weren’t operational, they had to go to the spot market for their obligations – meaning they were paying $9,000/MWH for the power they were delivering, vs. earning maybe $25/MWH in normal times. The big power producer Vistra said they lost over $1 billion in the crisis. Exelon’s generation arm lost more than $500 million; Exelon then decided to pull out of the Texas market altogether. And Brazos electric cooperative, the largest cooperative in Texas, declared bankruptcy two weeks ago, because they had to pay such high prices to deliver on their obligations to their customers (who are also their members).

b.      Prices in the natural gas market also spiked by huge amounts during the crisis. As in the power market itself, this was due both to supply constraints (most gas wellheads were no better protected against the cold weather than the power plants were. And because it’s usually not necessary, gas pipeline companies in Texas don’t normally take all of the moisture out of the gas before putting it in the pipe, as happens in colder climes. Thus, many of the gas pipelines froze), and to demand spiking (because the gas generation plants that were still working were demanding as much gas as possible, as their owners tried to make all the hay they could while the sun shone). But this meant that a gas plant owner who was being paid an astronomical price for the power they were producing was also paying an astronomical price for the gas they used to produce that power. So if just the power price was rolled back but not the gas price, they would be in a deep financial hole.

c.      Large purchasers of power (manufacturing plants, utilities that don’t generate the power they distribute, municipal utilities that don’t generate much if any power themselves, etc) will often (or even usually) hedge their purchases on the futures market – meaning they essentially buy the power they will need in advance at an agreed price (and it’s certain that the futures price, since it was probably set months ago, was much closer to $20/MWH than $9,000). So the big bills those purchasers received a couple weeks ago would be paid by the speculators on the other side of the contract; those purchasers just had to pay the much lower cost of the contract.[i]

d.      On the other hand, the speculators would have lost a ton of money. Normally, they would just blame themselves for the losses, since obviously when you play that game you have to be prepared to lose as well as win. However, if some of the charges get reversed as suggested, it’s just about certain that the speculators are going to want some of that for themselves. And it’s likely that the power futures exchanges (the largest of which by far is the Intercontinental Exchange in Chicago) have rules allowing those funds to be recouped from the buyers of the contracts.

e.      But the problem is that the large purchasers who bought futures contracts wouldn’t receive any funds if ERCOT ordered the power producers to cancel their big bills to consumers; they just wouldn’t have to pay more than the price of the futures contracts they bought. Yet now they would be hit with an order from the futures exchange to reimburse the speculators. They might in turn go to the power producers for reimbursement, but the producers will likely argue that, since they’ve been ordered to return whatever payments they received at the high rate, they certainly shouldn’t be ordered to reimburse their customers for the fact that they hedged their purchases.

f.       In other words, it’s likely the large purchasers will be caught in a really tough legal fight with the futures exchanges, of which the outcome will be far from certain.

And that last statement pretty well sums up the electric power forecast for Texas in the coming years: Massive lawsuits and political fights. Ultimately, I’m guessing there will be some sort of huge settlement that won’t make anybody whole, but will put most of the cost on – you guessed it – Texas citizens, in the form of higher taxes and high power bills. I certainly hope they’ll decide that changes need to be made to keep this from happening again.

One other thing: You may have noticed the title of this post refers to $29 billion, but the amount that is being talked about for a refund is $16 billion. The market monitor decided that the original PUC decision to raise the maximum rate to $9,000/MWH was made according to the PUC’s rules, and continued to be so until Thursday, when the fact that ERCOT ended demand curtailments meant the price should have reverted to the market price of about $22/MWH (Note: The fact that the PUC had to pay somebody to point this out to them is incredible in itself). The $16 billion is just the extra cost for 33 hours from Thursday into Friday.

But as I hope I’ve shown, if the PUC had just applied some common sense on Monday the 15^th, they would have realized there was no case for trying to make the rate go higher than the market rate, which at $1,200 was certainly high enough to induce whatever remaining supply might have been sitting on the sidelines. I read somewhere that, over the entire Monday-Friday period, the difference between the actual price ($9,000) and the market price led to $29 billion in unnecessary charges (i.e. if the PUC had just left the actual price at the market price, power purchasers would have saved $29 billion compared to the bills they actually received).

Given the recklessness and incompetence that the PUC exhibited in boosting the price far higher than what the market said it should be on Monday, and the terminal cluelessness exhibited by ERCOT in keeping the actual price at that level through Friday (remember, the $9,000 was the maximum allowed. The PUC didn’t order ERCOT to set the price at that level. I doubt they had that power, anyway), I think the entire $29 billion should be reversed. Of course, that makes a huge legal mess even worse. But expecting some power buyers to simply eat the $13 billion unjustified extra cost for Monday-Thursday isn’t exactly a solution, either. Get everything on the table and treat everyone equally poorly. And remember the people who got you into this mess the next time you go to the polls.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

---

[i] This assumes the purchasers were 100% hedged, meaning they’d bought futures contracts covering their entire needs. Since that’s not the usual practice (after all, there’s certainly a cost associated with buying futures contracts), these purchasers almost certainly faced large power tabs for the week of Feb. 14; but those were much lower than they would have been if they hadn’t hedged at all.

The third SBOM webinar is scheduled!

Allan Friedman of the NTIA has scheduled the third in a series of informational webinars on use of software bills of materials (SBOMs) in the electric power community. It will be presented on March 24 at noon Eastern time. Connection details are pending. If you would like to be on Allan’s mailing list for this and subsequent energy events, send him an email at afriedman@ntia.doc.gov. Here are the connection details:

SBOM in Energy Info Session – Lessons Learned

Wednesday, March 24, 12-1pm ET

Teams Link

Dial-in:+1 202-886-0111,,239096326#   United States, Washington DC

Phone Conference ID: 239 096 326#

Find a local number

The third webinar will include discussions by (I believe) two large suppliers (one hardware, one software) to the power industry, on why it is important to them to produce SBOMs and distribute them to their customers. It will also feature members of the healthcare software community – both medical device makers (MDMs) and hospitals, who call themselves healthcare delivery organizations or HDOs – discussing their experience with proofs of concept for industry use of SBOMs. The community started their first PoC in 2018 and it continued into 2020. They then started their second PoC, which continues today, although it’s now in its third “iteration”.

In each iteration, the participants in the PoC – there were about five HDOs and five MDMs in the first PoC, although the numbers have grown since then and are continuing to grow – decide at the beginning what questions they want to answer in that iteration. In the first PoC, the questions were very simple: Can SBOMs be successfully produced in a standard format by suppliers, and can the hospitals use them successfully as part of their software supply chain risk mitigation efforts? The answers to both of those questions were yes.

In each subsequent iteration, the participants have become more ambitious, especially in setting up procedures and validating machine-readable formats for automatic production of SBOMs by the MDMs, as well as automatic “ingestion” of them by the HDOs. They are moving closer and closer toward the Holy Grail they’re aiming at: demonstrating an (almost) fully automated program for producing and utilizing SBOMs.

But there have been a number of surprises along the way; it’s impossible to state at the outset all of the different obstacles you’ll run into in any new technical endeavor, but it’s inevitable that you will. I wrote about two of these obstacles last year: the “naming problem” and the problem of “vulnerability exploitability”, or VEX for short. Both of these are hard problems, but neither of them is insurmountable. In both cases, working groups within the NTIA’s Software Component Transparency Initiative have worked on solutions, and the healthcare PoC has put those solutions to the test.

A PoC for the autos sector (where the “consumers” of SBOMs are the automobile manufacturers, and the producers of SBOMs are the suppliers of the electronic components that make the modern car more like a computer on wheels every day) will start soon. Of course, we hope to start the energy PoC soon as well. The webinar will feature MDMs and HDOs sharing their stories about why their organizations felt it was important to participate in the PoC, and what they’re getting out of it. 

If you missed the first two webinars, you can watch the first one (an introductory session) here. And today Allan posted the recording of the second webinar (essentially SBOM 101, although there was more than a little 201 and even some 301 in it) here. Even though I attended the second webinar (and participated in the first), I’m going to review it this weekend, since there was a lot of material to absorb.

I hope you’ll join the third webinar, and consider participating in the PoC itself. I will point out that, if you and your organization just don’t have the bandwidth to be direct participants, there will still be regular meetings (perhaps weekly) that are open to anybody. In those, the participants will discuss lessons they’ve learned and problems they’ve encountered, as well as collaborate to create a document describing what they’ve learned, like the document produced by the first healthcare PoC. Since I’ve been participating in the public meetings for the healthcare PoC, I can assure you they’re quite interesting, and get into some really interesting questions.

And speaking of interesting questions, in case you’re wondering why people are going to all this trouble over something called SBOMs, when you don’t exactly wake up in the middle of the night wondering how you can survive another day waiting for SBOMs to arrive on the scene, I recommend you view the recording of Robert M. Lee’s keynote address to the SANS virtual ICS Summit 2021, which he delivered today.

The address was wonderful in all sorts of ways (and I will write a post on it soon), but I want to point out that Robert wasn’t ten minutes into his presentation before he pointed to Ripple 20 as a stark reminder that most organizations have no idea what’s inside the software products that their organizations depend on. And he said the way to fix this problem is for SBOMs to be widely available and widely used. That’s where you come in.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.