Regular readers of this blog (both of you!) know that for more than a year I have been concerned about the “showstopper” problems that are inhibiting wide (or even narrow) distribution and utilization of SBOMs by organizations whose primary function is not software development (you know, banks, restaurant chains, baseball teams, police departments, architecture firms…the rest of us).

I say “non-developers”, because there are some developers who are using SBOMs heavily now to manage risks in the products they’re developing. However, what is strange is that I honestly can’t identify a single developer that is regularly providing SBOMs to their customers (since an SBOM can no longer be trusted once the software has been upgraded or even patched, the developer needs to provide a new SBOM with at least every major and minor update of their product) – although I’ll admit I have no visibility into what military and intelligence agency suppliers are doing.

What’s holding SBOMs back? One is the naming problem. An informal group I lead, the SBOM Forum – which is, I’m happy to say, now the OWASP SBOM Forum[i]! – developed a paper almost exactly a year ago that describes how to significantly mitigate this problem in the NVD. The NIST team that runs the NVD has told us they would like to implement what we describe in the paper, but they obviously have a lot on their plate nowadays, such as wondering whether they’ll have a paycheck next week (for the second time this year, I might add).

Thus, while we will certainly engage with the NVD whenever they’re ready and able to do that, the OWASP SBOM Forum isn’t losing sight of the real goal: The US and the rest of the world need a truly global vulnerability database, which is maintained and funded internationally.

We’re now talking with people at ENISA (the EU cyber agency) about their potentially using our recommendations for NIST as the basic design for the vulnerability database they’re developing from scratch – and which they hope to have up and running in two years (this is eminently doable, in my opinion). The project is already funded (although probably not on the scale required for a global database) by Article 12 of the EU NIS 2 legislation, which came into effect last year.

Thus, we’re making progress on the naming problem. But, until this week, we hadn’t even started to address the second serious showstopper problem for SBOMs: the fact that, three years after the VEX idea was first discussed in the NTIA SBOM initiative, there is still no clarity on what VEX is and how tools for VEX production and consumption should work. Therefore, VEX is literally going nowhere at the moment.

However, there is clarity that SBOMs will never go very far until VEX is made to work. This is because suppliers have seen the statistics stating that over 90% of component vulnerabilities in a software product aren’t exploitable in the product itself. The suppliers are worried that, the day after they release their first SBOM, their help desk will be overwhelmed with calls and emails from angry customers demanding to know when they will patch CVE-2023-00666, which has a CVSS score of 10.0 and will probably get them fired if they don’t get it fixed by…how about 2PM this afternoon?

Of course, the help desk people will patiently explain to caller after caller that they don’t need to worry about this vulnerability because the vulnerable module in the library that contains the vulnerability was never used in their product. But maybe, after they reach the 70^th such caller in one morning, they will begin to lose their patience…and by the end of the day they’ll submit their resignations.

Yet, I don’t know of a single supplier that has started to produce VEXes on anything but an experimental basis. Cisco recently announced they will start producing VEXes for customers (not for public viewing), but didn’t say how often they will be updated.

While it’s nice to have one-off experiments, VEXes will be needed even more frequently than SBOMs, since new vulnerabilities appear all the time and customers need to know about them in as close to real time as possible (which is why I’ve proposed a VEX server and API. These will eliminate, or at least greatly reduce, the need for VEX documents, as well as take a huge burden off the suppliers). It’s not at all unreasonable to expect VEXes to be updated or issued multiple times a day.

But even among the very small number of experimental VEX documents that have been produced and published, there is no consistency regarding what is covered in a VEX, or even the format for it. This isn’t surprising, given that there are two “platforms” on which VEX can be produced now - CycloneDX (CDX) and CSAF – and in neither one of them has a format for VEX been specified. This has led to wildly differing documents, all bearing the name “VEX”. Given that there is no consistency in the VEX documents being produced, is it any surprise that there are literally no tools available to ingest and utilize VEXes? Yes, there are plenty of tools that can read the JSON of a VEX and make it look a little prettier for the reader, but why should a supplier produce a machine-readable VEX document if it’s just going to be read by humans? They could save a lot of time (and some money) by putting what they have in a PDF and emailing that to their customers.

Last week, the OWASP SBOM Forum decided it was time for us to lead development of VEX “playbooks” – one for CycloneDX VEX and the other for CSAF VEX (the two platforms are so different that there could never be a single VEX playbook that will encompass both platforms). Each platform will have a Producer’s Playbook and a Consumer’s Playbook.

These playbooks will be intended mainly to describe to the toolmaker how VEX documents will be produced and utilized. They will be constructed with enough rigor and detail that there should be no question about what needs to be in the VEX and how it should be presented. Since we will be creating the producer and consumer playbooks at the same time, we will hopefully avoid the common pitfall of not having the production and consumption tool specs exactly in sync.

The two platforms will require very different levels of effort. CycloneDX VEX is much easier to understand than CSAF VEX, since it is built on the same platform that CDX SBOMs utilize (as well as CDX HBOMs, SaaSBOMs, MLBOMs, OBOMs, etc. See the CDX site for the full list of document types that are built on the same framework). There are already plenty of tools that can both create and read CDX documents; the problem is those tools need to be constrained so they only produce and consume VEX documents, rather than making the supplier figure out for itself how to create a VEX in CDX format.[ii]

However, understanding CSAF is much harder, since the spec is about 80-100 pages long and published in small type. If you look at the VEX profile in CSAF, it seems simple, since it just lists a small number of fields. However, it leaves out two mandatory fields (required in all CSAF documents) that are by far the hardest to understand in CSAF: “product tree” and “branches”. The description of these goes on for about 10-15 pages in the spec, and even then, it’s not likely that organizations with no experience in CSAF will be able to understand them quickly.

Fortunately, the group working on the playbooks will include staff members with long CSAF experience from Red Hat, Oracle and Schneider Electric, as well as perhaps staff members from Microsoft, Google and Cisco. Our challenge won’t be to understand CSAF, but to determine the optimal configuration of the “product tree” and “branches” fields (these seem to be related to other fields as well, although my understanding of CSAF is very limited). We may decide that we need to constrain those fields very tightly, perhaps by constraining the VEX document to only addressing one product (although it will have to address multiple versions of the product – and being able to represent and interpret version ranges automatically would be very desirable). Again, our goal is to have a VEX spec that won’t need much if any interpretation on the consumption side.

Why am I so worried about constraining the spec? It’s simple math: a consumption tool will need to be able to interpret every combination of options that might be thrown at it – and the number of combinations is the factorial of the number of independent options. If there are three options, there are 3X2X1=6 possible combinations. If there are 5 options, there are 5X4X3X2X1 options = 120 combinations.

How about if there are 10 options? There it gets trickier: there are 3.6 million possible combinations. And how about 20 options (which may well be the case if you consider the entire CSAF 2 spec)? Not a big deal…then there are only 2.4 quadrillion possible options! My guess is most developers aren’t going to feel like devoting the next couple of millennia to writing a CSAF consumption tool unless the options are severely constrained.

I’m telling you this because this will be a very interesting exercise as well as a very important one, since as I’ve said many times, “No VEX, no SBOMs.” VEX has to be fixed if we will ever have widespread distribution and use of SBOMs. Next week our group will organize itself, and we’ll aim for two weeks from now for our first meeting (I’ll send out a Doodle poll next week to judge a proper time). I think biweekly meetings will be fine. In fact, the documents we create – starting with a document on VEX use cases – will be available on Google Docs for comments and suggested edits at any time. If you want to join us or at least be on the mailing list, send me an email.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Fortress Information Security, an organization I have worked for as a consultant for over three years, has recently developed two great resources for anyone interested in or involved with software bills of materials: a webinar and an e-book. I recommend both of these.

WEBINAR: Software Risks: Understanding your Software Supply Chain Security

The webinar takes place tomorrow at noon ET. The link for both registration and attendance is here. If you can’t attend tomorrow, you’ll be able to view the recording later.

The speakers are both very knowledgeable about SBOMs: One is Tom Pace, the Co-founder and CEO of NetRise (and a familiar name to readers of this blog. My most recent post featuring him is here) and Bryan Cowan of Fortress, who is Product Owner for SBOMs at Fortress.

The topics to be covered in the webinar “include SBOM adoption drivers, SBOM risk insights, example use cases, and a business case for managing risk with SBOMs.” One interesting aspect of this webinar is that it only lasts half an hour. I used to believe that would be too short for a webinar, but now I think – after seeing a few recent hourlong webinars that lost my interest after about 20 minutes – this might be the right idea.

One reason that this may be a better idea is some people will rule out spending an hour viewing a webinar but will be more inclined to listen to a half-hour one – since they know that the message will be delivered in that time. But even more importantly, there’s the “I didn’t have time to write you a short letter, so I wrote you a long one” truism. In other words, it takes much more effort to concentrate what you have to say into a short period of time than to write something that takes longer to read but can make its arguments less succinctly (I would use the word “sloppily”, but that hits too close to home for this blog!). In the unfortunately few times I have been forced to concentrate what I say into a small number of words or period of time, I’ve almost always been pleased with the result.

I know that both Tom and Bryan can say a lot about SBOMs and supply chain security, but since they’re doing it in half the usual webinar time, my guess is what they say will be more memorable than anything they (or I) could say in an hour. And best of all, you get a whole half an hour back. I hope this is a trend!

Whitepaper: SBOM Use Cases for Asset Owners

Bryan has been busy recently, since he co-authored (with Ty Short) the above white paper, which is available here. This takes a very different approach to SBOMs than just about anything I’ve seen in writing (including my posts) so far: Instead of focusing on the use cases of licensing (the original SBOM use case) and software vulnerability management (the use case behind Executive Order 14028 and most articles on SBOMs, including my posts), the paper is clearly based on real-world research into the possible uses of SBOMs by public and private organizations. The result is quite good and very readable.

I recommend you look at both of these! And I promise that neither one will require an excessive amount of your time.  

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Warning: The post below starts on an optimistic note but ends on a pessimistic one. Parental discretion is advised.

Yesterday, Cisco made an important announcement, although people who aren’t VEX addicts probably didn’t see it and if they did, it didn’t register as important. I had been looking for the short announcement about VEX that Cisco made a few weeks ago and found it odd that it didn’t appear in searches on Google or the Cisco web site. However, I see now that yesterday’s announcement filled in the details of the previous announcement, which was just a placeholder for the real thing.

The announcement was written by Omar Santos of Cisco, who is also the leader of the OASIS CSAF project. CSAF is the leading general vulnerability reporting format, and the VEX “profile” in CSAF adapts that format to the more limited needs of VEX (mainly, it limits the large number of status designations in the full CSAF format to four: “affected” (i.e., exploitable), “not affected”, “fixed” and “under investigation”).

The announcement is of the “Cisco Vulnerability Repository” (CVR), although I want to point out that name is a mistake. The software world has enough vulnerabilities that we don’t need to have Cisco create a repository to protect them 😊! The best name would be “Cisco VEX Repository” (which doesn’t require a change of acronym, BTW), since that’s what it is. A second-best name would be “Cisco Vulnerability Advisory Repository” or just “Vulnerability Advisory Repository”.

In any case, why do I think this is a significant announcement? For one thing, this is just the second announcement by a major software or device supplier (Cisco is both, of course) that they will be producing VEX documents for customers regularly. The first announcement was by Red Hat in February; however, within the last two months Red Hat changed their mind about VEX (nothing wrong with that, of course!) and renamed the documents they were calling VEX back to their previous name: security advisories.

However, I believe that RH will soon announce VEX documents that are a lot like the ones Cisco just announced – since they are big supporters of CSAF themselves and they have a seat on the CSAF board.

The announcement lists three types of vulnerabilities that will trigger VEX notifications:

• Vulnerabilities found in the Cybersecurity and Infrastructure Security Agency (CISA)’s Known Exploited Vulnerabilities (KEV) Catalog
• Vulnerabilities that Cisco has determined to be high-risk
• Vulnerabilities whose status has been requested using CVR 

This clarifies what Cisco's VEX strategy is (and I think it's a good one): Instead of putting out VEX documents that discuss the status of multiple vulnerabilities in a particular product (which is the strategy that a lot of people have been thinking of), this lists the status (exploitable or not) of one significant vulnerability in one or more products. The example in the announcement utilizes the "product family" capability of CSAF, since it applies to the entire Catalyst 9800 Series family of wireless controllers.

Perhaps the biggest problem with VEX is that user organizations already face a huge backlog of unpatched software vulnerabilities. If they are to learn about new ones, they need to know just the ones that pose a significant risk to them. Meanwhile, software suppliers who have looked at creating VEXes (and very few have even looked at it, I’m sure) have realized that determining whether a particular component vulnerability is exploitable in a particular product/version is hard and time-intensive (unless they want to adapt the “shortcut” I described in my most recent post, which will require them to patch a lot more component vulnerabilities than otherwise. I don’t know whether that’s a good trade-off or not, of course; that will depend on lots of different factors).

Cisco is clearly quite aware of this problem, which is why they have stated up front that they will only consider certain component vulnerabilities for remediation, meaning they won’t even bother to determine whether any others are exploitable or not. I think other suppliers should emulate what Cisco has done here: state very clearly which vulnerabilities they will consider for remediation and which they won’t. Their customers should appreciate this, since they won’t be burdened with a lot of patches that mitigate little risk (of course, the risk to Cisco in this strategy is that a high-risk vulnerability like log4shell will slip through their net and not be patched. But given the level of awareness of vulnerabilities in general nowadays, it’s likely that any such vulnerability will quickly be pointed to by their customers).

Regarding Cisco’s three criteria for considering a vulnerability:

1.      Almost by definition, the KEV catalog is a list of the most risky vulnerabilities

2.      Cisco’s own list of high-risk vulnerabilities is important, since presumably Cisco knows best which risks to its products are high ones; but,

3.      I especially like the third category: "vulnerabilities whose status has been requested using CVR". This gives Cisco’s customers a voice in deciding which vulnerabilities to consider for remediation. Moreover, notice the announcement doesn’t say that customers will only be able to access VEX information for products they own (in fact, it doesn’t sound like Cisco would even have any way to implement that policy, since they’re reporting VEX information by vulnerability, not by product). And because a large percentage of all organizations in the world are customers of at least one Cisco product or service, in theory they will all be able both to make suggestions for vulnerabilities to address and to see all VEX notifications (although I admit I’m guessing this is the case).

So I think this announcement might set a pattern for a lot of other organizations; at least I hope it does.

But there’s a downside to this announcement as well. It can be summarized in three questions:

1. Omar Santos is chairman of the CSAF committee. Pete Allor of Red Hat is one of the most active participants on that committee. Oracle has also started putting out CSAF VEX documents, and they also have a long history with CSAF and its predecessor format CVRF. That’s great.

However, CSAF is an extremely complex format, as a quick skim through the 100-page-plus[i] draft specification will quickly reveal. Even more importantly, figuring out how to fill out the mandatory “product tree” and “branches” fields requires close study of about 15 pages of very dense text within the CSAF format, and even then there are so many options available that a supplier that doesn’t benefit from long experience with CVRF/CSAF is unlikely to devote the time to do that. After all, nobody has a gun to their head, requiring them to produce VEX documents at all, let alone CSAF VEXes. Why go through the aggravation?

Aha, but you don’t need to know all of CSAF to create a VEX in that format. What about the VEX profile? That limits the fields you need to put in the document (which is good), but it omits mention of the need for the “baseline” elements required in every CSAF document, including product tree and branches.

2. Then, what about a tool to create a VEX document in CSAF? There are no VEX-specific tools; in fact, I know of no tools that create any type of CSAF document – meaning they ask the supplier questions about what they want to include (products, vulnerabilities, etc.) and then put them into a CSAF document, requiring no direct CSAF knowledge on the VEX author’s side. The only CSAF tool I know of is Secvisogram. This is a good editor, but it requires knowledge of the CSAF format. Again, it will be quite a hard row for an organization to come up to speed on creating CSAF documents of any type, unless they have somebody that can devote a substantial block of their time to learning the format and creating documents with it – using Secvisogram, faute de mieux.

3. But let’s say a supplier has created a CSAF VEX document? What are their customers going to do with it? Are there low-cost, commercially-supported tools available that will ingest SBOMs and CSAF VEX documents, then provide the user with a continually updated list of exploitable component vulnerabilities in a product/version they use, which has a recent SBOM? No, there aren’t. How about open source tools that do that? Again, no.

So, how about any tools that utilize CSAF documents for security analysis purposes? Here is the official list of CSAF tools. Do you see anything that meets that criterion?[ii] I don’t.

In fact, I’ve asked several large suppliers that produce CSAF documents (whether VEX or just vulnerability advisories – which is the main use case for CSAF, of course) what their users do with them. They honestly don’t know – except that some customers may have developed their own tooling for that purpose.

So, here are the two big problems with CSAF VEX (these each have several parts, and a small number of those parts also apply to the CycloneDX VEX format. However, I know it is many times easier both to create and to utilize CDX VEX documents than CSAF VEX documents):

1.      Any software or device supplier that wants to learn how to create VEX documents in CSAF needs to be prepared to spend a lot of time doing so. I doubt many suppliers will be interested in doing that.

2.      Those suppliers that distribute VEX documents would be better off developing a PDF format and sending PDF VEXes out by email (or just putting them on a portal).  I know one very large device maker that is doing that with SBOMs: They produce both JSON-based and PDF-based SPDX SBOMs and make them both available on their portal. They report much better uptake for the PDF files.

So – as often happens in this blog – we have run up into one of the “showstopper” problems for SBOMs and VEX: the lack of tooling, especially the lack of low cost, commercially supported tools that are essential for widespread use of both SBOMs and VEX documents by organizations whose primary business isn’t software development (i.e., at least 98% of organizations, public and private, worldwide).

Can this problem be solved? It can be partially mitigated, but substantial mitigation (I won’t even use the word “full”) also requires addressing the other showstopper problems, especially the naming problem. One partial mitigation would be a “playbook” for CSAF VEX, like the “playbooks” that were developed during the NTIA SBOM initiative (here are the two most important of these: for producers and for consumers).

I know that a draft playbook for Cyclone DX VEX has already been developed. However, I don’t know of any current attempt to develop a playbook for CSAF VEX. One is sorely needed.

I told you this post would be depressing, but you didn’t believe me!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

This is the first in a two-part series about VEX. Currently, VEX is going nowhere fast. I now know of just one organization – Cisco - that has even announced they’re producing VEXes for their products, although the only examples they linked to in their announcement were the same CSAF examples that appeared in CISA’s VEX Use Cases document, published in April 2022 (which also had CycloneDX VEX examples).

Another organization that was producing documents that they called “VEX” – Red Hat – has changed their mind about whether they are really VEXes, and has changed their name back to their original name: security advisories. They explained this to the SBOM Forum in a presentation more than a month ago, but they also stated they will be putting out a new type of document soon that they are calling a VEX (and since the only published VEX definition – near the beginning of the same Use Cases document – essentially just says a VEX makes statements about vulnerabilities, it doesn’t make sense now to even debate what is or isn’t a VEX. If the author calls it a VEX, it’s a VEX. End of story).

I want to point out that I don’t blame Red Hat for any of this, since they have always been a leader in producing machine readable vulnerability notifications (you’ll notice that the notifications on the page I referenced above go back to 2001), as well as doing lots of work of benefit to the software vulnerability management community. For instance, they sit on the boards of both the OASIS CSAF group and CVE.org.

Moreover, they are one of a handful of Root CVE Numbering Authorities (CNA) and are available to help open source software projects report their own vulnerabilities, as well as proactively reporting vulnerabilities in their own products. Pete Allor, Director of Product Security for Red Hat (and an active member of the SBOM Forum), told me once that Red Hat almost considers this their duty, because of their unique relationship with the open source community.

But other than Red Hat and Cisco, I know of no organization that has made any public announcement about producing VEX documents, and I have seen no other VEX documents from any source, except for a few demonstration documents.[i]

One of the biggest problems with VEX is that proving that a vulnerability has “not affected” status (which is, of course, the main reason that VEX was developed) often requires proving a negative. For example, if the supplier believes there is no “code path” that would allow an attacker to reach the vulnerable code (in order to exploit it), they would literally need to be able to show that none of the perhaps thousands of paths an attacker might utilize to reach that code will allow them to reach it, in order to “prove” the vulnerability isn’t exploitable in the product.

However, the five “status justifications” offer a way out of this problem. Whenever the status of a vulnerability in VEX is “not affected”, a status justification needs to be provided to justify the “not affected” status. Note that these are called “justifications” in CycloneDX VEX, and there are nine of them, vs. five in CSAF. The CDX justifications are a superset of the CSAF status justifications and were developed almost two years before the CSAF ones.

While three of the status justifications suffer from the need to prove a negative (they are “Vulnerable_code_cannot_be_controlled_by_adversary”, “Vulnerable_code_not_in_execute_path”, and “Inline_mitigations_already_exist”), two of them do not. They are “Component_not_present” (meaning that the vulnerable component that appeared in the SBOM isn’t actually in the product) and “Vulnerable_code_not_present” (meaning that, even though the component is present in the product, in this case the vulnerable code has been removed from the component).

Neither of the latter two status justifications requires proving a negative. If someone is skeptical about either justification, it won’t be hard for the supplier to prove it is valid. If a supplier asserts that either one of these is the reason why they used “not affected” status, they aren’t likely to be questioned.

Therefore, a way to “tighten up” the use of “not affected” in VEX would be to require that it only be used in a case where the reason for the “not affected” designation is either “Component_not_present” or “Vulnerable_code_not_present”. In all other cases, every vulnerability should be given the status of “affected”.

Of course, there may be some suppliers who aren’t bothered by the fact that three of the status justifications require proving a negative – plus some of their customers may be quite happy to accept all five status justifications as valid.

In this case, there wouldn’t need to be any change from what is now assumed in VEX: that all five of the status justifications are valid. In fact, when there is tooling available to utilize VEX information, there could be a simple software option: Those software users (e.g. hospitals, electric utilities, and military contractors) that believe they require a higher assurance level can choose the first option described, in which there are only two valid status justifications. Those users that are comfortable with the “looser” standard can accept all five status justifications, which will probably be the default option when and if VEX “consumption” tooling appears.

How about the nine “justifications” in CycloneDX? They are:

• code_not_present = the code has been removed or tree-shaked.
• code_not_reachable = the vulnerable code is not invoked at runtime.
• requires_configuration = exploitability requires a configurable option to be set/unset.
• requires_dependency = exploitability requires a dependency that is not present.
• requires_environment = exploitability requires a certain environment which is not present.
• protected_by_compiler = exploitability requires a compiler flag to be set/unset.
• protected_at_runtime = exploits are prevented at runtime.
• protected_at_perimeter = attacks are blocked at physical, logical, or network perimeter.
• protected_by_mitigating_control = preventative measures have been implemented that reduce the likelihood and/or impact of the vulnerability

The CDX equivalent of “component not present” is “requires dependency”. The CDX equivalent of “vulnerable code not present” is “code not present”. High-assurance users might want to require that only these two justifications be allowed to accompany a “not affected” status, while the other seven justifications would require “affected” status.

How should these two options be positioned – i.e., the high assurance and “normal assurance” VEX options? When and if end user VEX “consumption” tooling appears, it should offer both options, although in general it would be better to make the “normal” configuration (i.e., all five status justifications are acceptable for “not affected” status) the default, with the “high assurance” option (i.e., only the “component not present” and “vulnerable code not present” justifications are acceptable for “not affected” status) available for organizations that believe they need that.

Perhaps the best aspect of this tooling flexibility is that it places no burden on the supplier. As long as they provide one of the standard five CSAF status justifications (or one of the nine CDX justifications) whenever the status of a vulnerability is listed as “not affected”, the end user will make the decision, through their tooling choice, of how it will be interpreted.

I feel that uncertainty regarding questions like the above is inhibiting production of VEX documents by suppliers; what I’m suggesting here may somewhat alleviate that uncertainty, although it is hardly the only problem with VEX. In the exciting Part II of this series, I’ll tell you about another important development affecting VEX, which may also help to make VEX more palatable to suppliers.

An article in today’s Washington Post led off with this sentence: “An elderly North Carolina woman’s death when her oxygen machine failed during a power outage has been ruled a homicide by the state medical examiner and blamed on what authorities said was intentional gunfire that hit power substations in her area.”

The “intentional gunfire” in question occurred during shooting attacks on two power distribution substations in Moore County, NC on December 3, 2022. They left over 40,000 people without power for multiple days (which itself was a problem, since there’s so much redundancy built into the power grid that any outage should be remediated within hours, not days. At least, that’s the idea…). I wrote about these attacks in this post on Dec. 5 and this one on Dec. 7 (a date that is remembered for the far more serious attack on Pearl Harbor). These attacks were imitated soon afterwards in Oregon and Washington state, although no outage was caused there.

In the second post, I included this sentence, “(The outage) certainly caused damage, and people were injured in car crashes…. We may hear later about people on oxygen at home, etc. that were victims as well.” I don’t claim prescience, but that seems to be exactly what happened in this case. An elderly woman’s oxygen compression machine, which helped her breathe at night, didn’t work because of the outage; she died as a result. I agree that’s murder, given that the outage was human-caused.

Of course, nobody has been charged with murder because the police have so far not found whoever shot up the substations – and of course, they may never find them. But at least this serves as a warning to whoever thinks that causing a power outage is a way to make a political point. This was the reigning theory in December for the attacks in all three states.

These attacks were coordinated. People get drunk and shoot up substations all the time, but they don’t make a point of shooting up multiple substations at once – and they don’t usually know exactly where to shoot in order to cause an outage. These people seemed to know that.

I think that this type of attack – a physical attack on distribution substations – is the most likely to cause an outage compared to any cyberattack, or even compared to a physical attack on generation or transmission substations (which are much better protected now, thanks to the Metcalf attack and the subsequent development and implementation of NERC CIP-014). Of course, an attack on distribution would almost by definition never cause a cascading outage like the August 2013 event (which wasn’t caused by a deliberate attack. Why bother to attack if carelessness and lax procedures can produce the same effect?).

In any case, I’m glad the medical examiner ruled the woman’s death a homicide. That alone may prevent some future attacks.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Recently, Dale Peterson interviewed Steve Springett, leader of the OWASP Dependency Track and CycloneDX projects, for his podcast. I have watched a number of podcasts with Steve, and I never come away without a lot of notes; for this podcast, I took six pages’ worth. This was in part because Dale asked some really excellent questions, which showed he had done a lot of research beforehand. Between the two of them, it was quite a show.

I recommend you listen to the whole podcast, but I’d like to address three topics that came up – among many.

Contractual challenges to sharing SBOMs

Steve and Dale both agreed early in the podcast that, while SBOMs are being heavily used by software developers to learn about and manage vulnerabilities in products they’re developing, they’re hardly being distributed to or used by end user organizations at all (i.e., organizations whose primary business isn’t developing software, which is of course well over 99% of the organizations on the planet).

One of the reasons Steve pointed to for this problem (and it is a problem!) was contractual. He didn’t elaborate on that, but I can guess that, since there are no standards or official guidelines for producing or distributing SBOMs, any organization that tries to negotiate with suppliers on the terms under which they’ll provide their SBOMs won’t find this to be easy at all.

However, I’ve already discovered the solution to this problem. It’s a lot like the solution the doctor proposed to his patient who complained that if he performed an unusual arm motion, it hurt. The doctor’s solution? “Don’t do that.”

And that’s my solution to the problem of contracts for SBOMs being hard to negotiate: Don’t bother with them. Given the complete lack of standards or official guidelines for producing or distributing SBOMs, it’s simply way too early to even consider using contract language to “force” the supplier to give you exactly the type of SBOM you want, in exactly the way you want it.

However, let’s say the supplier gives you the perfect SBOM that you want. What are you going to do with it on your own? There are currently no low-cost, commercially supported tools that ingest SBOMs (and VEX documents, although they’re hardly being distributed at all today) and output a list of component risks for the user, including a list of exploitable component vulnerabilities. Yet, this is the use case that most end users have in mind for SBOMs.

This is why I recently suggested that we forget about using contract language for SBOMs for the time being (except for perhaps an NDA) and simply focus on mutual learning: the suppliers will produce the SBOMs they think their users want and their users will tell them whether and how they’re able to use them. This will be effectively a large-scale proof of concept and, as with any proof of concept exercise, there will need to be mechanisms for gathering and aggregating the lessons learned.

I know of only one active SBOM proof of concept that is exchanging SBOMs today - the healthcare PoC being sponsored by the Healthcare ISAC. Moreover, that PoC has just around ten participants (medical device makers and large hospital organizations). Wouldn’t it be great if we could get a lot more suppliers and users, in many industries, exchanging SBOMs and learning from them – and then sharing what they’ve learned from the experience? We can do that if we forget about contract language for now, and simply focus on what we can learn.

The only way SBOMs are going to provide value to end users in the near term

After the above discussion, Dale pointed to the one solution that can lead to SBOMs (or at least the information to be derived from them) being widely used by non-developers in the next few years. The solution won’t be to have low-cost, commercially supported tools that every organization can use. I used to think that those tools would someday magically appear, but they haven’t appeared yet and I know of none that are even on the horizon.

More importantly, there are serious issues like the “naming problem” (which Dale brought up later in the podcast) and how to get useful VEX information out (I plan a post on this problem next week), which are currently standing in the way of easy-to-use consumer tools. These problems won’t ever be completely solved, but they can be addressed to a degree that makes usable consumer tools possible. We’re simply not there yet, and won’t be for years.

However, Dale pointed out that he sees a more realistic path to SBOMs being usable to consumers in the near term, and that’s third party service providers; I agree with him 100% on this. The fact is that the tools required to analyze SBOMs and VEX information, and then report component risks to end users, are available now. However, they’re almost all not low cost or commercially supported or both – and that’s what will be needed for true consumer tools. Moreover, an end user today would need to string together several open source tools (and address data formatting issues, etc.), to get the required functionality. Few businesses or government agencies have the technical chops and available time to do this.

However, third party service providers are a different story. They know that whatever tooling they put in place will allow them to provide this service to many end user organizations. The economic picture will be very different for them.

Exactly who will these service providers be? I honestly don’t know now, but I do know that, given the high and growing interest in learning about software component vulnerabilities, they will appear. The important technical problems have all been solved in principle. Even the naming problem has been “solved” by hundreds or even thousands of software suppliers and their consultants, since there are lots of ad hoc workarounds available, including AI/ML routines, fuzzy logic, etc. No non-software suppliers will want to invest in developing these routines for their own use, but again, the economic picture changes drastically for service providers who can amortize the cost of doing so over hundreds or thousands of customers and products.

There’s one important aspect of these service providers that didn’t come up in the podcast, but is one that I’ve written about once or twice: It’s clear to me (at least) that end users shouldn’t be responsible for paying the service providers for their analysis. After all, the supplier is the one that chose the component and included it in their product. If they chose a component that poses big risks for end users, why should the end users have to find this out for themselves? More importantly, if there are 10,000 customers of a product, why should they each have to pay a service provider to tell them that CVE-2023-12345 is exploitable in the product they use, and they should immediately contact the supplier to find out when they’ll patch it?

Instead, why doesn’t the supplier pay the service provider to do this and distribute the results to their users? I remember when it first became clear in the 1990s that software vulnerabilities weren’t rare events (as had been the general opinion), but could be counted on to appear constantly in just about any product. At first, software suppliers tried to charge their users for the privilege of receiving patches for the vulnerabilities in their product – often through saying that only users who pay for maintenance would receive patches.

Of course, this idea quickly faded, and now I don’t know a single supplier that doesn’t develop and distribute security patches to their customers for free. I expect the same thing to happen with SBOM analysis. While I don’t think most suppliers will want to make the investment in providing the service I described above for their customers, I think they will gladly pay a third party to provide that service on their behalf. Especially when all their competitors start doing it and they realize they won’t be in business much longer unless they do as well. Just like what happened with security patches.

Side-channel attacks

Toward the end of the podcast, Dale brought up the topic of VEX. He had heard Steve say on a different podcast that “VEX is a missed opportunity because it doesn’t represent risk.” He asked what Steve meant by that.

Steve’s first response was fairly complicated (and led to a more complicated discussion than I want to address at the moment), but his second response was very straightforward: VEX considers a single vulnerability in isolation and provides a binary answer to the question of whether that vulnerability is exploitable in a particular product (and not just a particular product, but a particular version of that product).

However, Steve pointed out that hackers often don’t just try to exploit one vulnerability and then give up if they don’t succeed in compromising the product using that vulnerability. Instead, they often try to exploit a different vulnerability. Then, if they’re successful with that attack, they “pivot” to exploiting the vulnerability they had originally aimed for. This is known as a “side channel” or “chained” attack. When you consider these attacks, then VEX’s answer to whether a vulnerability is exploitable in the product or not becomes more complicated – since you need to consider which side channel attacks might be used to exploit that vulnerability, not just direct attacks.

This was a question that was discussed in the VEX working group when it was under NTIA (I don’t think we have discussed it much under CISA). Our feeling at the time was that, if we started to consider side channel attacks, then essentially every vulnerability would be exploitable. Why even have a VEX in that case? Instead of spending their time putting out VEX information, a supplier would just need to admit they need to patch every component vulnerability, even though they know that over 90% of them probably aren’t exploitable.

Moreover, end users will also need to admit that, even though they on average are currently only applying about 15% of the patches they receive for products installed on their network, they’ll now face a huge influx of additional patches, which they will have to add to their already voluminous patching queue. Since we (the VEX working group) didn’t want to provide those answers either to suppliers or end users, we decided to proceed on the assumption that only directly exploitable vulnerabilities need to be designated as exploitable in VEX.

However, enough people (besides Steve) have raised this issue that I now think we need to admit that there are more sophisticated hackers out there who know how to use side channel attacks. So, for some users (especially those in high assurance use cases, like military contractors, hospitals, and OT in general), it does make sense to patch any component vulnerability that is known to be present in the product, and for users to apply those patches.

Fortunately, there are some cases in which even a side channel attack won’t be able to exploit a particular vulnerability in a product, meaning that VEX can provide an acceptable solution, even for high assurance users. I’ll discuss that soon, hopefully next week.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

In the past couple of months, it seems events have been conspiring to tell me two things:

1.      Security in almost all types of intelligent devices is in terrible shape and borders on being nonexistent. However,

2.      Even if you want to take security matters for a device into your own hands (e.g., do your own vulnerability management), you’re going to have a hard time doing that.

The first event was when I wrote this post, which reported that a person involved in product security for a major medical device maker told me he had never reported a vulnerability for one of their devices, and in fact he wouldn’t know how to report one. Since most CVE reports (which form the basis for all listings in the NVD and the great majority of them in other vulnerability databases) are created by the developer of the product reporting the vulnerability (either a software product or an intelligent device), this probably means that none of that company’s products lists any vulnerabilities in the NVD.

In fact, since a CPE name is only created when a vulnerability is reported for a product (more correctly, a particular version of a product), this probably means nothing will be found when someone searches for that product in the NVD. This isn’t good, but it’s made worse by the fact that the message that displays when that happens is “There are 0 matching records.” This is the same message that displays when there is a CPE name for the product, but there aren’t any vulnerabilities reported against the product. In other words, a user that searches on a product and receives that message should either be elated that it’s a “vulnerability-free” product or worried that it might be loaded with vulnerabilities, but the supplier of the product doesn’t report them. How do you tell which is the correct conclusion to draw? Perhaps a coin flip?

My friend defended the fact that his company doesn’t report product vulnerabilities by saying they would only report a vulnerability if they hadn’t patched it. However, this gets the situation completely backwards: A responsible supplier (either of software or devices) usually only reports vulnerabilities they have patched, since to report an unpatched vulnerability might invite hackers to try to exploit that vulnerability whenever they encounter that product. But the responsible supplier also patches vulnerabilities in their products as quickly as possible.

The above scenario - in which a customer is told about a vulnerability in a product they use and therefore applies the patch that has been developed to mitigate the vulnerability (usually the patch is linked in the same report that announces the vulnerability) – falls down in the case of intelligent devices, since the customer almost never has the option of applying a patch for a particular vulnerability. Instead, the customer has to wait for the next time the supplier updates all the software in the device (or at least all that’s changed since the last device update).

I used to think that it wasn’t such a terrible thing that device users would have to wait for the next device update to learn that a vulnerability had been patched. I thought this, because I assumed that device manufacturers update the software in their device about every month. If they did this, it might even justify not revealing the vulnerability until after it’s patched on the next update, since waiting a couple of weeks for a patch to be applied is probably close to normal in the software world.

However, when I asked my friend how often his company updates the software in their devices (which are medical devices, remember. They aren’t baby monitors or recipe servers), he said they do it annually. In other words, let’s say a device manufacturer discovers a serious vulnerability in one of their devices the day after they released their annual update. Assuming they’re on an annual update schedule, this means the vulnerability won’t be patched in the field for 364 days.

Will the manufacturer let their customers know about this serious vulnerability and the fact that it won’t be fixed in the device they use for another 364 days – so they can at least apply some alternative mitigation to the device, even perhaps pulling it from their network altogether? If they’re like my friend’s company, the answer seems to be no. My friend said they won’t issue a notification for the vulnerability until it’s patched in the field. Where does that leave their customers if they’re attacked using that vulnerability during those 364 days? How about SOL?

So, the manufacturer of a device needs to report vulnerabilities for the third-party software and firmware products included in their device, as well as for the code in the device that they wrote themselves (the first-party software). Moreover, they need to report all these vulnerabilities to a CPE that identifies the device, not to one of the myriad software or firmware products contained in the device.

The reason for doing this is obvious (to everyone except the device manufacturers, it seems): Just like for a “standalone” software product, the customer of an intelligent device needs to be able to learn about all the vulnerabilities in their device in a single search. Even if they have a current SBOM for the device (a dubious assumption today, to be sure), they shouldn’t have to invest all the time and effort required to look up each of those products in a vulnerability database (assuming they even have a searchable identifier, which means a purl or CPE). The supplier should be tracking all vulnerabilities for third- and first-party software and firmware in their device, and posting them all to the single CPE for the version of the device where those vulnerabilities are found.

What about the fact that a lot of the vulnerabilities that a manufacturer learns about (in a vulnerability database like the NVD) for the individual software and firmware products installed in their device won’t in fact be exploitable in the device itself – so they don’t actually pose a risk to their customers? Should the supplier still report every vulnerability regardless of its exploitability status, and then issue VEX information to let their customer know which of those vulnerabilities aren’t exploitable?

Were there any low cost, easy-to-use and commercially supported tools available for software users, which would use VEX information to narrow down a list of vulnerabilities in a particular product/version to just the exploitable ones, I would say by all means, that’s the way to do it. However, given that this is far from being the case, I’m willing to concede that the manufacturer can apply the exploitability information on their own and just report the small percentage of component vulnerabilities that are exploitable in their device, not the huge percentage (usually around twenty times the percentage of exploitable vulnerabilities) that aren’t exploitable. That will at least make the job much more manageable.

However, another thing that device suppliers need to do is provide much more regular updates to their devices. Quarterly updates would be a big improvement over annual ones, so maybe that’s what they should aim for today. But the manufacturers also need to let their customers know about important new vulnerabilities in their devices and not wait until the next device update to do this. That way, if a customer wants to implement an alternative mitigation during the time remaining before the next update, they’ll be able to do this (the manufacturer should suggest alternative mitigations in their vulnerability report).

At the end of his presentation to the SBOM Forum, Tom admitted that, given the large number of vulnerabilities that are likely to be found in most intelligent devices (as he pointed out to our group last year, he examined one device – that had no vulnerabilities reported for it in the NVD – that very likely had at least 40,000 unpatched vulnerabilities), it was unrealistic to expect the manufacturer to be able to quickly patch all the vulnerabilities in the device; they will need to figure out a way to triage the vulnerabilities (using perhaps EPSS score or KEV ranking) and patch the most important ones first.

But he did point out that some device manufacturers have an alternative strategy for reducing the number of vulnerabilities they’ve found in their devices, without having to patch them at all. He showed us a message his company had received, when they tried to follow up on a previous notification to a manufacturer about a serious vulnerability they’d identified in one of their devices. It read, “Your message to security@(manufacturername).com has been blocked.”

Yep, that’ll do the job, too. And it’s a lot cheaper than patching. 😊

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

…consistency is the hobgoblin of little minds.

- Emerson

Whenever you identify problems in an existing organization, program or device, you always face the question: Will it be easier to fix what is currently in place, rather than have to start over from scratch? Or will it be easier – or at least better – to start from scratch and build something of your own design, which hopefully avoids all the problems with the existing system?

Early in 2022, I got together a group of people who were committed to making SBOMs successful, but were concerned that this will never happen unless certain obstacles are removed from their path; we call ourselves the SBOM Forum. I knew there were a number of problems to address (including maybe 4 or 5 “showstoppers”), but I wasn’t terribly concerned about which one we started with – as long as we started somewhere.

However, there was one thing I was sure of: I didn’t want us to start with the “naming problem” – that is, the problem that a single software product can have many different names in different contexts, and these names can correspond to subtle (or not-so-subtle) differences in the product (small code variations, source code vs. compiled binaries, etc.). It seemed to me that, if we started with the naming problem, we’d work on it for five years and still be no closer to solving it than we were when we started.

But I didn’t want to constrain the conversations we had in our weekly meetings, so I let them go in whatever direction people wanted to take them. Guess what topic we ended up focusing on within a couple of weeks? You’re right…the naming problem. So much for planning, eh?

However, I was surprised by how much progress we made when we started discussing the naming problem. There are many aspects to the problem, but the one that’s probably on the minds of most people in the software security community is the National Vulnerability Database (NVD), which is by far the most heavily used vulnerability database worldwide. The NVD’s biggest problem is the software identifier it uses: CPE.

As we started discussing these problems, I was amazed that a) the problems became easy to identify and describe (I had thought it would take a year just to describe the problem we were addressing), and b) the solutions became apparent very quickly, at least at a high level (I’ll admit we had deployed a secret weapon that greatly speeded up identification of the solutions: Steve Springett, leader of the CycloneDX and Dependency Track projects and someone who – from what his family tells me – never goes to bed unless he’s solved at least one software security problem that day).

The result was that, five months after we started discussing the subject, we published a paper that has received a good amount of attention. It described what we saw as the major problems with CPE, as well as the broad outlines of a solution to the problem. Early this year, we started talking with Tanya Brewer, leader of NIST’s NVD team, about implementing our solution. She said they had read our paper with interest and would like to collaborate with us (as well as a few other organizations that have reached out and offered to help them).

These three posts - first, second, and third – described our conversations with Tanya. In the third post, I relayed the news that NIST would soon come out with a video asking for suggestions on a public-private partnership to fix the problems (as well as accomplish other goals, to be sure), and they would probably post a Federal Register notice about their plans in late August. Tanya thought the PPP could be up and running by early 2024.

How is that coming? Well, it’s now about six weeks after the date Tanya said she’d have the video out and…there’s been no word at all. This means the whole schedule is probably pushed back two months already…and counting.

What happened? I know they were having a serious problem that might have required “all hands on deck” (the last time we talked to her, her head count was down about 20%, since she hadn’t even begun to receive her 2023 funds – this was in late June. Of course, given that the US almost defaulted on its debt a few weeks earlier, I guess this isn’t a huge problem in the scheme of things), although the notification about that serious problem has been removed from their website, as well as the admonition not to ask them when it would be fixed. I also know that the video was first delayed by a couple of weeks because somebody who had to sign off on it was on vacation (does NASA work the same way? “Sorry, we’ll have to postpone the moon launch for ten months since Mr. Smith has to have surgery, and nobody else has authority to sign for him”).

Of course, delays like this are par for the course in the government game, and we haven’t given up on the NVD; we’ll pick up the phone whenever they want to talk. However, we’ve now decided that it’s time to work on Plan B, which is a Global Vulnerability Database.

Why have we moved from Plan A to Plan B? Because B has always been our goal. The whole world is becoming more and more concerned about software vulnerabilities.  We’re sure there will be a lot of interest in Europe, the UK and Japan (and perhaps other countries like Singapore), both in the public and private sectors, in having a single, robust global vulnerability database. In fact, we’ll have a first meeting with ENISA next week.

Given the upcoming (although still not finally approved) EU Cyber Resilience Act, as well as the rapid growth of SBOM use among software developers (plus the fact that SBOM use by organizations whose primary business isn’t software development has – to put it gently – nowhere to go but up. But it will go up, mark my words), it frankly makes no sense not to start mapping out a global vulnerability database.

The paper we put out last fall is no longer our wish list for the NVD (although we’d be glad to see it implemented at the NVD, bien sur), but rather the start of a blueprint for the GVD. Of course, we’re not going to walk away from the wealth of vulnerability information already in the NVD, namely the existing vulnerability listings. It’s easy to obtain; some organizations download the entire NVD multiple times a day, with time off for a leisurely lunch. And we’re not going to walk away from the CVE.org database, which is where the NVD’s vulnerability information ultimately comes from (in fact, one of the board members of CVE.org is a very active member of the SBOM Forum).

Fortunately, putting together this database doesn’t require solving any problem that hasn’t been solved hundreds or thousands of times already. And frankly, I think a lot of organizations worldwide feel a little guilty about never contributing to the NVD – which has never even asked anyone for money, as far as I know. I think the money to get the GVD up and running will be fairly easy to obtain.

Who will run the GVD long term? It could be a government body (although I’m not interested in trying to shake Washington for money, especially since there’s a good chance the government will be shut down through at least some of the fall. After all, shutting the US government down is becoming almost a pre-holiday tradition, like getting a Christmas tree) or even a private industry group. However, maybe the best long-term solution is IANA, which already assigns IP addresses and runs DNS. There are lots of options.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

August 14 is the 20-year anniversary of the 2003 Northeast blackout. This event had a profound impact on the world we live in today in many different ways; one of them was mandatory regulation of the power industry. Here is my summary of what happened:

·        The Wikipedia article states, “The blackout's proximate cause was a software bug in the alarm system at the control room of FirstEnergy.” Frankly, this is a stupid statement. It’s kind of like saying that the cause of World War II was the German invasion of Poland. There’s a big difference between a triggering event and a cause. This bug was a triggering event.

·        The causes were multiple: An important one was the fact that compliance with the NERC reliability standards at the time was voluntary. First Energy, as well as probably other utilities, had violated the NERC requirements for trimming trees under high voltage transmission lines. Since high levels of load cause lines to sag and the load levels were high on that hot day, the lines sagged into treetops, which caused them to be tripped by their protective relays. When one line shorted out, its load was automatically distributed to other lines, which themselves began to sag, encountered trees, and tripped. And so on.

·        Finally, when the last major line into the area tripped, northern Ohio became a virtual black hole for electric power, trying to suck in as much power as possible from all its neighbors. In turn, they tried to get all the power they could from their neighbors – and voila!, the cascading outage began. Within six minutes after the last line failed, the event was over. Much of the Northeastern US (excluding New England), Detroit and eastern Michigan and most of Ontario up to Hudson Bay had blacked out. 508 generating units at 265 power plants shut down during those six minutes.

·        But lack of tree trimming wasn’t the only cause. What made the blackout much worse than it had to be was that the protective relays protecting many generating units from grid instability were (in retrospect) set to trigger at a much lower level of instability than necessary. NERC’s monumental six-volume study of the blackout (which I skimmed through once, not that I could have understood most of it anyway) pointed out that (again in retrospect) many of those units would not have had to shut down if the settings had been more forgiving.

·        The resulting total blackout required activation of “blackstart plans” in many areas. Since most generating units require some amount of external power to start up, the blackstart plans lay out a complicated path for utilizing the small amount of power still available (e.g. power from hydroelectric plants or power from generating units that have diesel-powered emergency generators available) to one-by-one energize lines, then more generating units, then more lines, etc. – until the generation in the area covered by the blackstart plan is fully operational.

·        However, all of this could have been avoided if there had been better visibility into what was going on in Ohio. The software bug in the First Energy control center caused the alarms – which would otherwise have been flashing deep red – to be suppressed. The operators in the control center saw nothing but green screens and were happy as clams…until they weren’t. But even that might not have mattered, had a technician at what was called at the time the Midwest Independent Transmission System Operator (now the Midcontinent ISO) not left for lunch after fixing a problem with the “state estimator” system.

·        That system would have detected the growing divergence between electricity load and supply in northern Ohio (exacerbated by a couple of major plant outages in Cleveland) had it been active. However, the technician forgot to turn the system back on when he left. When it finally was turned on, it immediately became clear that northern Ohio was going to hell in a handbasket very quickly.

·        On the other hand, it’s not clear that the human beings monitoring these systems would have reacted correctly if they knew how bad things were. To end the imbalance between load and supply, they probably would have had to black out the entire city of Cleveland and the surrounding area. Would they have been authorized to do this and if not, would they have been able to get that authorization quickly enough to do any good, given how rapidly the imbalance between load and supply was increasing? Of course, Cleveland and many other cities ended up being blacked out, so it would clearly have been better if they system operators had taken this step.

Of course, all these causes (and others besides them) were correctable and since have been addressed with policies, standards, etc. But that required something else: mandatory standards for utilities to follow. That’s what Section 215 of the Electric Power Act (EPAct) of 2005 provided. It required the Federal Energy Regulatory Commission (FERC) to enforce mandatory reliability standards for the electric power industry.

To facilitate this, Section 215 ordered FERC to engage an “electric reliability organization” (ERO) to draft mandatory reliability standards as well as audit compliance with them, all under FERC’s oversight. Of course, there wasn’t much question that NERC was the only organization that could possibly be the ERO, and FERC chose them.

Section 215 also ordered that mandatory cybersecurity standards be developed for the power sector. At that time, the main NERC cybersecurity effort was a set of voluntary requirements called Urgent Action 1200, which applied just to large control centers. NERC was working on an update called Urgent Action 1300 but in 2016, NERC rolled that effort into development of version 1 of the CIP standards.

So the CIP standards, which were (I believe) the first cybersecurity standards that applied to Operational Technology (OT), outside of perhaps military standards, were very much a consequence of the Northeast blackout. While I have certainly expressed my disagreement with many aspects of CIP over the years (you might go back and read the at least 400 posts I’ve written on CIP, starting with my first post on this blog in early 2013), and while I continue to believe that compliance with the CIP standards is much more expensive than it should be, there’s no question in my mind that the standards have done a great job of securing the North American power grid against cyberattacks.

Moreover, I think all the NERC reliability standards (which cover all aspects of operating the power grid, including – thanks to the blackout – relay settings) have proven very important, not just the CIP standards. While local outages happen all the time (with squirrels being one of the main causes), there has not been a cascading outage since 2003, or even just a large outage that wasn’t caused by a natural disaster like a hurricane. The dedicated people of NERC (and the six NERC Regional Entities that explain and audit the standards) deserve all our thanks.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

One of my biggest eye-opening experiences of 2022 was when Tom Pace of NetRise described to the SBOM Forum (a group I lead) a maker of networking devices used by the military and critical infrastructure, which has never reported a single vulnerability for any of their devices – so naturally, a search on one of their devices in the NVD will leave many users with the impression that their devices are all vulnerability-free!

Unfortunately, the reality is different: Just a single device that Tom’s company examined likely had at least 40,000 unpatched vulnerabilities in components included in the device’s firmware. Of course, I was full of righteous indignation at this supplier (whose website doesn’t even mention “security” or “vulnerability”). How dare they shirk their obligation to report vulnerabilities to CVE.org (which is where the NVD, as well as any other database that uses CVEs, learns about its vulnerabilities)? After all, I’m sure (or was sure at the time) that all major intelligent device manufacturers (and software developers) diligently report all their vulnerabilities to CVE.org.  

Since a lot of people who work with vulnerabilities all the time don’t understand how they get into a vulnerability database (and I didn’t, until about a year ago), I’ll give a brief summary:

1.      CVE.org is part of DHS and is funded by CISA. It is staffed by the MITRE Corporation and used to be known just as “MITRE”. However, it is now run by an independent board which includes representatives of private organizations, CISA, NIST (which runs the NVD), and MITRE.

2.      New vulnerabilities are usually identified by the developer of a software product in their own software. If the developer is what’s known as a CNA (see the CVE.org website for a discussion of CNAs), they will be able to write up the CVE report and submit it to CVE.org. If they aren’t a CNA – and only about 200 organizations are CNAs; becoming a CNA requires training – they can go to a CNA for help if the CNA’s “scope” includes them. For example, Red Hat (a longstanding CNA) will help any open source project (I believe); GitHub will help any GitHub project; there are country- and industry-specific CNAs; etc.

3.      If a developer that wants to report a new vulnerability doesn’t think that their organization can be served by any of the CNAs, they can always approach MITRE directly (through CVE.org, of course), since they are the “CNA of last resort”. The CNA will help the developer (or device manufacturer) create the CVE report. The report will include information on the affected product, but not a CPE name; that name will be assigned by the NIST people later.

4.      The report is submitted to CVE.org and if approved, it will be passed to the NVD, where it automatically becomes part of the database. Around 20,000 new CVEs were identified in 2022.

5.      When the CVE report reaches the NVD, someone on the NIST team that runs the NVD will assign a CPE name to the product. Unfortunately, they don’t always do it correctly and the product becomes hard (if not impossible) to find in the NVD; the developer is left to apologize to their customers for NIST’s mistake. One of the changes the SBOM Forum has discussed with NIST is allowing the developer to assign the CPE name themselves.

Currently, there won’t be a CPE name for any product for which a CVE report hasn’t been submitted; that is, NIST currently only creates a CPE name when a vulnerability is reported for the product. This in itself wouldn’t be terrible were it not for the fact that, when a user creates a CPE name (following the specification) for a product they want to learn about and enters it on the search bar, they will receive the same message - “There are 0 matching records” - if no CPE exists with that name (which will be the case if the NIST person made a mistake when they created it) as they will if the name is valid, but there are no vulnerabilities reported for it.

In other words, the user will have no way of knowing whether the product really has no vulnerabilities (and has a correct CPE name) or whether it has 40,000 vulnerabilities, but was entered under an incorrect CPE name.

(now, back to our regular programming) When I heard Tom Pace’s presentation last year, I initially thought that intelligent device manufacturers – at least the larger ones (including manufactures of IoT, IIoT, medical devices, networking devices, etc.) - generally do a decent job reporting vulnerabilities in their devices, since I believe that’s the case for software developers. The main evidence that software developers are doing a good job is the number of software vulnerabilities that can be found in the NVD. Aren’t device manufacturers doing the same thing?

Some are. For example, when I just searched the NVD for “Cisco ASA” (Cisco’s firewall appliance), I received 581 CPE names for different models, different versions, etc. This means that Cisco™ has reported at least one vulnerability against each of these CPE names. Of course, the CPE name refers to “adaptive security appliance software”, not the device itself.

And this is how I think vulnerabilities should be reported for devices. They should be reported for the device’s software (and firmware) as a whole, not according to the individual software or firmware products installed in the device (since most devices include at least some third-party software, not just software developed by the manufacturer itself). If instead, vulnerabilities are only reported for the individual third-party products, then the only way someone will be able to learn about vulnerabilities in a particular device is to a) be a customer, b) receive a current SBOM that lists all the software in the device (and I know of no device manufacturer that is regularly releasing SBOMs to its customers. BTW, the new FDA requirement is unlikely to change that), and c) go through the work of looking up each of the third party products in the NVD – based on the infinitesimal-to-faint hope that the supplier of each of those third party products is diligently reporting its vulnerabilities.

This week, I was quite happy when I got a chance to have a one-on-one conversation with someone I’ve known for three years, who is involved with product security for medical devices made by one of the largest medical device makers (known as “MDMs” in the industry). One of the many questions I asked him was how often they report a vulnerability for one of their devices.

I wasn’t expecting him to say they report vulnerabilities all the time, but I didn’t expect the answer I received: that they have never reported a vulnerability for any of their devices, and he isn’t even sure how to report one.

He justified that answer by saying – and I believe him – that they have never found a vulnerability that they didn’t patch, so their products haven’t had vulnerabilities. But here he was simply wrong. Few if any software developers or device manufacturers would ever report a vulnerability unless they had a patch for it. The normal procedure is that the developer discovers the vulnerability, develops the patch, then submits a CVE report (and usually a report to their customers) that lists the vulnerability but also provides information that allows the user to find the patch and apply it. If the MDM doesn’t submit anything to CVE.org, nobody will be able to learn of the vulnerability.

If this were a standalone software product, not a device, there would be no question that the developer needs to report the vulnerability (as well as the patch), both to CVE.org and to their customers, since the customers need to apply the patch ASAP. However, usually there is no way for the customer to patch a device they utilize; instead, the supplier needs to do that.

Of course, if suppliers patched vulnerabilities as soon as they appeared in the device (or at least the serious vulnerabilities), there wouldn’t be any problem. I realized that this idea is unrealistic, since device manufacturers can’t be expected to push out each patch the day they create it. I assumed they push out monthly updates – which is probably frequent enough to catch all but the most severely exploited vulnerabilities before the bad guys can get into the device.

However, I asked my friend how often they update their devices, and was amazed to hear it was usually once a year. In other words, even if the manufacturer develops a patch for a vulnerability the minute after they learn of it, their users may never benefit from that patch for up to six months. Even worse, since the manufacturer doesn’t immediately report the vulnerability to the NVD (and presumably not to their customers either. I’m betting that the customers wouldn’t be pleased as punch to learn there was an important vulnerability in the device they rely on, but it won’t be fixed for six months. And remember, the devices my friend secures are used for sometimes life-sustaining patient care. They’re not baby monitors).

I’ll stop here and let you contemplate this situation. By the way, do you or a loved one have any surgery scheduled? Just wondering.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

As I made clear in this recent post, I no longer think it makes any sense for the end user of a software product or intelligent device to bear the responsibility for analyzing SBOM and VEX information to identify exploitable component vulnerabilities in a product they use. Instead, the supplier should bear this responsibility for five main reasons:

1.      The supplier chose to include the components in the product. They did this because utilizing pre-built components is now the only realistic way to create software, due to the huge savings of money and developer time that result from this practice. While some of the economic benefits of using components accrue to the end users of software because of the overall lower level of software prices that results from this widespread practice, it is hard to deny (at least for me) that the greater part of the benefits of using components accrues to the supplier. This is due to the savings that developers realize through accelerated development time and lower overall development costs.

2.      Currently, there are no tools available to software end users (which I define as consumers of software that are not utilizing the software they acquire to create downstream products, but instead utilize it to fulfill the mission of their organization) that are a) free or low-cost, b) commercially supported, c) easy to use, and d) complete[i]. By a “complete” tool, I mean one that:

                           i.          Ingests an SBOM for a particular software product and version, for example version 1.5 of product ABC;

                          ii.          Resolves any component naming issues, meaning it identifies a valid CPE or purl identifier for every component listed in the SBOM. Learning just the common name for a component is close to useless for a user concerned about component vulnerabilities in the software they use. Identifying a valid CPE for a component requires manual searching, fuzzy logic, dumb luck, etc. Identifying a purl for an open source component should be straightforward if the repository or package manager is known, along with the product name and version string registered in that repository. However, these should always be verified by the product supplier before including them in an SBOM);

                         iii.          Looks up each component CPE or purl in a vulnerability database like the NVD or OSS Index and includes every identified component vulnerability in a list of vulnerabilities found in ABC v1.5. Each vulnerability on the list is initially assigned an exploitability status of “under investigation”;

                         iv.          Repeats these lookups at least daily, adding any newly identified component vulnerabilities to the list with the “under investigation” status;  

                          v.          Receives VEX information from a supplier, either in the form of machine-readable documents or using an API that retrieves the information from a VEX server. VEX information consists of a) exploitability status for every vulnerability in the list for ABC v1.5 (“affected”, “not affected”, “fixed” or “under investigation”), b) a status justification for every “not affected” status designation, and c) mitigation information (e.g. patch or upgrade information) for every “affected” status designation;

                         vi.          Continually uses new VEX information to update the vulnerability list for ABC v1.5;

                       vii.          On demand from the user of the tool, provides a list of exploitable component vulnerabilities (i.e., those with “affected” status) in ABC v1.5, as well as mitigation information for each one (which is also retrieved from the VEX server)[ii]. The list can be provided in either a generic format or in a format required by a particular vulnerability or configuration management tool; and

                      viii.          Continues this process for ABC v1.5 until the user upgrades or removes the product from their network. If the user upgrades to ABC v1.6, the process will start over with step 1 above. If the user still has any instances of ABC v1.5 running, the tool will maintain the component vulnerability lists for both v1.5 and v1.6. Of course, the tool will be able to maintain vulnerability lists for many different products/versions from many suppliers.

3.      Despite the SBOM Forum’s recent incremental progress in getting NIST (which runs the NVD) to recognize the importance of addressing the NVD’s problems with CPEs, the naming problem is hardly solved. This means that, unless suppliers take it upon themselves to do the work required to identify a CPE or purl for as many components as possible, SBOMs (especially those created through an automated process) will continue to be mostly useless for the purpose of vulnerability management.

4.     VEX information is essential for the component vulnerability management use case, yet suppliers aren’t producing VEX documents for their customers. I know of only one supplier, Red Hat™, that makes documents called VEX available to customers now. However, much though I admire the great work Red Hat does in the area of vulnerability management and reporting (and the work that Pete Allor, Red Hat’s Director of Product Security, does as a member of the CVE.org board), because Red Hat has such a unique business model, the type of VEX documents they now produce would be of limited usefulness to the customers of any other supplier.[iii] Note that, if the supplier were doing the SBOM analysis themselves, they wouldn’t need to produce VEXes, since the exploitability information could just be passed – in any desired format, or even just as a list in an email - from the department that produces it to the department that uses it.

5.     Most importantly, it makes no sense to require that every customer, in order to learn about exploitable component vulnerabilities, perform exactly the same analysis on their own that the supplier should already be performing for themselves. Isn’t the supplier the one that needs to know first about exploitable component vulnerabilities – so they can patch them as soon as possible? Given that they have this information (or should have it), why can’t they just share it with their customers, rather than requiring each one of them to derive exactly the same information on their own?

Fortunately, I think that software users will start requiring that their suppliers perform this analysis and make the results available to them for free – just like they do for patches (suppliers initially resisted the idea of distributing security patches for free as well. That resistance quickly faded when they realized they would end up with exactly zero customers if they didn’t do it).

The suppliers will provide the analysis, although, unlike with patching, this work can be accomplished much more efficiently by third party service providers. Those providers will utilize tooling they have developed for their own use (and not for end users, which means it doesn’t have to be made foolproof or user-friendly. Given some big issues that need to be addressed, such as the naming problem, this is currently the only way to develop such tooling. My guess is it will be years before there will be any true consumer tools available for tracking software component vulnerability status). They will make the results (lists of exploitable vulnerabilities) available to the supplier’s customers for free, while charging the suppliers for their services.

Will the end users need to perform any component vulnerability analysis on their own? They will need to in cases where:

a.      The supplier simply refuses to pay a third party to perform the analysis, but cannot or will not perform it themselves; or

b.      The organization doesn’t trust the third party service provider to perform the analysis properly and wishes to do it themselves; or

c.      Someone at the user organization simply enjoys performing the analysis on their own (maybe utilizing open source tools they’ve chained together).

Thus, the “professionals” in the software component vulnerability analysis world will be the third parties that specialize in performing these services for suppliers. The organizations that perform the services on their own will be “keeping score at home” (not that there’s anything wrong with that, of course).

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Monday’s meeting of the CISA VEX working group led off with a rather disturbing exchange: the two leaders of the group – in obviously coordinated comments - both wondered whether VEX even belongs in the CISA SBOM effort. They asked whether it might be “better” to host it somewhere else. Where? They didn’t suggest any alternative venue. Fortunately, one of the most active participants in the group strongly objected to this idea and the leaders said they would be “happy” to continue the meetings (as in “I’ll be happy to take this medicine that I hate taking”).

But they also made it clear that CISA management is now asking questions about what they’re getting from their 19-month (so far) commitment to the VEX working group (the group started in 2020 under the NTIA and was the only NTIA workgroup that continued under CISA, although its exact status under CISA has never been clear). To be honest, if I were CISA, I would ask the same questions. Here’s a little history to explain what I mean:

1.      When the NTIA Software Component Transparency Initiative ended at the end of 2021, the VEX meetings continued under CISA without any interruption in January 2022.

2.      Under the NTIA, the group only produced a single one-page Introduction to VEX (which I drafted). It was a decent effort, but it was out of date by 2022 (e.g., it assumed there was just one VEX format: CSAF); plus, there’s only so much one can say in one page.

3.      Under CISA, the group immediately produced two well-written and important documents on VEX in the spring of 2022: VEX Use Cases and VEX Status Justifications.

4.      Last summer, the group entered a period of wandering in the wilderness, which this year produced two documents that add just about nothing to our understanding of VEX: VEX Minimum Requirements and the close-to-being-finalized When to issue VEX information. Both of these are classic examples of a “document for the sake of having a document”. While neither one contains untruths or misrepresentations (except for the title “Minimum Requirements”, since nothing in that document is “required” by anything other than the document itself), there is nothing I can point to in either document that wasn’t already obvious or wasn’t stated better in one of the 2022 documents.

Given the above, I can certainly understand why CISA is concerned that the investment they’ve made in the VEX working group (primarily in the form of employee time) during at least the past 12 months has been unproductive. So, why do I think this group shouldn’t be discontinued?

That’s simple: The VEX effort started because several huge software and intelligent device suppliers had realized, early in the NTIA SBOM effort, that distributing SBOMs to their customers regularly would inevitably lead to their help desks being overwhelmed with calls from customers about “false positive” component vulnerabilities. This is because more than 90% (some say more than 95%) of component vulnerabilities – which a customer can identify by running an SBOM through a tool like Dependency-Track – are not exploitable in the product, even though they are exploitable in the component when it is considered to be a standalone product. Not only would these calls overwhelm their help desks, but they would be a source of endless (and unnecessary) frustration, both to customers and to help desk personnel.

It gets down to this: Software users won’t be interested in receiving SBOMs if they know that a) over 90% of the component vulnerabilities they learn about will be false positives, yet b) they won’t know how to distinguish these from the ten percent that are real; they will just want to learn about the real ones. And suppliers won’t be interested in providing SBOMs to their customers unless there’s some way they can let them know, in an automation-friendly fashion, which component vulnerabilities are exploitable (along with information on downloading a patch for each, or an upgrade path to a patched version) and which aren’t exploitable.

A recent meeting of the Healthcare SBOM Proof of Concept (which started under the NTIA but is now sponsored by the Healthcare ISAC and the Healthcare Sector Coordinating Council) produced these two relevant statements, one from a large healthcare delivery organization (HDO) – aka hospital chain - and one from a very large medical device maker (MDM):

1.      The HDO said they had recently looked up vulnerabilities for the software components identified in the SBOM for a medical device they use and found 2,000 vulnerabilities in that one device! Of course, at most 200 of these are exploitable, and the actual number of exploitable vulnerabilities is probably less than 100. But knowing this is cold comfort if the user organization has no way of knowing which of the 2,000 vulnerabilities are exploitable and which aren’t. They made it clear: “We need VEX!”

2.      The MDM said they won’t issue SBOMs until they can also issue VEXes – due no doubt to the fear of being overwhelmed with false positive help desk calls. More importantly, they need to issue VEXes with the knowledge that they will be used by the HDO in an automated fashion to “winnow down” the list of component vulnerabilities to only those that are exploitable. Obviously, if a user organization identifies 2,000 component vulnerabilities in a single device, they will need a tool that will ingest VEXes (as well as SBOMs) and do the winnowing for them. Currently, no such tools exist.

There you have it: Suppliers won’t be interested in distributing SBOMs to their customers, and software users (meaning organizations whose primary function is not developing software) won’t be interested in using SBOMs to learn about software component vulnerabilities, unless VEXes are made available and can be used effectively, in an automated fashion.

Absent VEX, SBOMs will go nowhere. This is why we need VEX.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Yesterday, the White House announced the long awaited cybersecurity labeling program for IoT devices, called Cyber Trust Mark. I had written about this in my post in May, as well as in previous posts. This program was mandated by Executive Order 14028, issued in May 2021. It is a voluntary program, which is expected to be “up and running” in 2024.

There were several surprises in the announcement, that went beyond what had been made public before:

1.      The agency that will implement the program will be the Federal Communications Commission (FCC), not the Federal Trade Commission (FTC), as I had expected (as well as some others). I hope this has been thought through, since the FCC’s experience with consumer products has mostly to do with technical communications standards and not cybersecurity. The FTC, since one of its roles is enforcing commitments consumer products companies make regarding privacy of personal information they hold, has already done a lot of work in the cybersecurity area.

2.      “NIST will also immediately undertake an effort to define cybersecurity requirements for consumer-grade routers…” Of course, routers are an IoT product, and there has been a lot of concern (mostly justified) about their security in the last couple of years. It’s certainly a good idea to require a higher standard of cybersecurity for routers. NIST is obligated to draft requirements for routers by the end of 2023.

3.      The US Dept. of Energy announced a project to work with the National Labs to develop cyber labeling requirements for smart meters and power inverters. Both of these are devices that are installed in (or on) homes, although they have lots of applications in industrial and commercial facilities (in more industrial-strength incarnations, to be sure).

However, there were some negative surprises as well. The biggest was that the labeling program, after being initially discussed as a way to communicate to consumers the degree to which an IoT product met certain criteria for cybersecurity has now become a way for “..Americans to confidently identify which internet and Bluetooth-connected devices are cybersecure”, according to Anne Neuberger, the deputy national security adviser for cyber and emerging technology at the National Security Council.

Note that the label was originally not intended to provide an up-or-down judgment on a product, but instead just point out areas of strength and weakness, allowing the consumer to determine for themselves whether it’s safe to buy. However, now it seems the White House has come up with a Roman emperor-style thumbs up-thumbs down label, which is based on some extraordinary insights they have into what constitutes a cybersecure product; I certainly hope they’ll share those insights with the rest of us, especially the many manufacturers who think they’re doing a good job on security, only to get a thumbs down when the labels are awarded.

More importantly, the document supposedly defers to NIST’s judgment on what should be in a cybersecurity framework for IoT. Even though it’s not mentioned in the announcement, NIST came out with what seems to me to be an excellent IoT cyber framework, NIST.IR.8425, last year. I wrote about it in a blog post for my French client Red Alert Labs in November; RAL works with manufacturers to secure and certify IoT devices.

However, like all NIST frameworks, exactly which provisions an organization complies with and how they comply with them are up to the organization, based on their assessment of their own risk environment. This is not compatible with an airy statement that a product is “cybersecure” or not.

I’m going to choose to believe that Ms. Neuberger’s statement was a bit of hyperbole inserted in her address by an over-zealous intern, who seems to think that the device labeling program is a game-changing innovation, when in fact it’s an initial step on a long journey towards an unattainable goal that might be called “complete cybersecurity of IoT devices”. But if it’s not, and the FCC decides to go through with the binary label idea, I’m sure that will fail. What manufacturer is going to meekly tuck their tail between their legs and walk away after being told they’re not being awarded the label because they didn’t meet some undefined criteria, while their competitor down the street received the label? They’re going to raise h___ and rightly so, in my opinion.

However, if I found Ms. Neuberger’s statement amusing, another statement (which appeared in the NextGov/FCW article on the announcement without attribution, although I heard it said by another source as well) chilled my blood: “The administration is also working with the Department of Justice to develop liability protocols for manufacturers working with the Cyber Trust labeling program.”

Yes, boys and girls, this is our good friend the Liability Monster, once more rearing its head after making an initial appearance in the National Cybersecurity Strategy last March. I thought I might have succeeded in driving a wooden stake through its heart in my 4 or 5 posts on the subject, but I must have grabbed a plastic stake instead.

It seems that some people in the White House have come to believe that our nation’s cybersecurity problems are far too pressing for us to deal with them through regulation (as everybody knows, regulations take a lonnng time to develop and implement) or through the court system (after all, you never can be sure how a judge or jury will rule on liability for a breach. It’s much better to ensure the outcome before the trial starts).

Instead, we need to place our finger on the scales of justice from the beginning and ensure that the parties we all know are responsible for cyber breaches – the developers of the software or manufacturers of the devices – are assumed to be liable right from the start. This will substantially reduce the workload of some very busy people in the White House and assure that they might even be home for dinner every now and then. What higher goal can there be than that?

Just like the thumbs up-thumbs down idea, the idea that the Department of Justice is even going to consider amending the centuries-old principle that liability is determined by a judge and/or jury in a court of law, instead of being determined by someone in the White House, will fail. I just wish people who should know better didn’t waste so much time on these pursuits. The IoT device labeling program needs to be treated seriously, since it has an important role to play in national security.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Almost on the last day of 2021, I wrote a post in which I stated my newfound belief (after a “road to Damascus” experience, although unlike St. Paul I didn’t hear the voice of God speaking to me) that suppliers should be responsible for performing the analysis of SBOMs and VEX documents, in order to produce a continually-updated (at least daily) list of exploitable component vulnerabilities in a particular product/version. Even though suppliers may choose to work with third-party service providers to perform this service (since the service providers can amortize the cost of their services across a large user base), it should be their responsibility, and they should pay for the service provider.

I’ve repeated that belief at various times since then, but I’ll admit that I’ve often forgotten about it, and spoken as if this analysis is really the responsibility of software end users. While that may be true in the short run, I anticipate that in maybe 5-10 years, the party universally believed to be responsible for analysis of SBOMs will be the supplier.

While there are several compelling reasons why suppliers should bear this responsibility, here’s the most compelling:

In order for end users to be able to make use of SBOM and VEX data to manage component vulnerabilities in the software products they utilize, somebody needs to ingest an SBOM, look up component vulnerabilities in the NVD or another vulnerability database, and ingest VEX information to learn how the supplier views the status of each of those component vulnerabilities. It makes no sense to force thousands, tens of thousands or even millions of customers to perform exactly the same set of steps that the supplier could perform on their own, and just distribute the results to their customers.

For example, suppose a software product has 10,000 users, all of whom are concerned about managing vulnerabilities due to components in the product. Let’s say there are low cost, easy-to-use, commercially supported tools available that will perform the required analysis, so the cost of tooling is not an important factor here (hey, a guy can fantasize, can’t he?). And assume the users have all been utilizing these tools for a long time, so they don’t need to “learn on the job” while performing this analysis.

Now, let’s suppose that performing the required analysis across the useful life of the software requires five hours of time for a single version of the product. If all 10,000 users do this, the total cost to them will be 50,000 hours. Ideally, if they all work with the same information from the supplier (i.e. both the SBOM and the VEX documents), they will all end up with exactly the same results for this product: a list of exploitable component vulnerabilities in the product and version, which is updated daily to reflect new vulnerabilities found in a major vulnerability database and new VEX documents received from the supplier.

Now, let’s suppose the supplier performs this same analysis themselves, using the same tool as their customers do (in fact, in the author’s opinion, the supplier would be negligent if they weren’t performing this analysis themselves, at least daily and perhaps more often). They will also spend five hours on this and achieve the same results as each of their 10,000 customers.

Here’s the hard question: Which is more, 50,000 hours or five hours? You’re correct! 50,000 hours is more than five hours.

Now, here is the even harder question: How could it ever make sense for a supplier to require each of its customers to perform an analysis that the supplier could perform just as cheaply on its own, while simply distributing the results to their customers (probably in a customer portal, so they don’t have to push any documents out)?

And here’s the answer to that even harder question: It probably never makes sense for a supplier not to do this analysis themselves and make the results available to their customers. So why is everybody in the (expired) NTIA and CISA SBOM initiatives talking as if the supplier’s only responsibility is to toss the SBOM over the wall to their customer and let the customer figure out what to do from there?

Beats me.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

The phrase that seemed to be on everybody’s lips when the FDA finally received authority to regulate cybersecurity in medical devices, which was granted in the Omnibus spending bill at the end of 2022 but had been specified in an earlier targeted bill called the PATCH Act, was “game changer”. This was because one of the likely consequences of the FDA’s new authority (it wasn’t directly mandated by the bill itself) is that they will require medical device makers (MDMs, in industry parlance) to provide a software bill of materials (SBOM) with their “pre-market submissions”.

The latter refers to a package of documentation that – if the MDM has done their homework correctly – will assure the FDA that the device the MDM seeks permission to market to hospitals or other end users is both safe (which has always been a criterion for approval) and cybersecure (which is a new criterion, due to passage of the Omnibus Bill). This provision (for a cybersecurity review, not just an SBOM) came into effect at the end of March (I believe), but the FDA said they won’t enforce it until October 1 – although they’re requiring submission of an SBOM now, and they’ll have a discussion with the MDM about any shortcomings they find.

What will actually be required come October 1? The MDM will be required to submit a single SBOM for their device. It will be scrutinized as part of the review of the entire submission, although no criteria have yet been stated for what will be considered an acceptable SBOM. Most importantly, the SBOM will never be shown to any person or organization outside of the FDA, including any customers or potential customers of the device.

Folks, this is the big “first SBOM regulation” that everybody is so excited about! Of course, it’s hard to see how anybody would get excited about just that. The reason people even use the phrase “game changer” is because the FDA hasn’t released any guidelines for what should be in an SBOM, how often it should be released, who should receive it, etc. A lot of people, most of whom have an economic stake in MDMs being forced to utilize (insert name of startup services vendor or startup cybersecurity tool vendor here) to help them produce and distribute SBOMs, have worked mightily at convincing themselves that this one small step will inevitably lead within months (or at least before their seed funding runs out) to industries of all stripes facing onerous regulations that will cause them to start banging on their door, open checkbooks in hand, begging to be allowed to buy their product or services. A true “game changer”.

I don’t deny that these people have achieved a lot of success – in convincing themselves of this quite dubious proposition. However, I have no idea what game they’re talking about changing, unless it’s TiddlyWinks. I’ve been working in the cyber regulation field – specifically in NERC CIP, which is no game at all – for a long time, and I’ve noticed one funny thing about regulation: The organizations that are being regulated and face onerous fines for violations don’t take kindly to being told to comply with a list of “requirements” that are poorly worded, based on ambiguous terms, make assumptions that appear to be taken from The Chronicles of Narnia, etc.[i] They tend to push back and demand clarification or wholesale rewriting of any requirement that’s ambiguous or misconceived. And if the agency that imposed the requirements pushes forward and implements the objectionable regulations, any penalties they levy are likely to be immediately reversed by highly skeptical judges, who will issue strongly worded opinions suggesting that perhaps whoever drafted those regulations should consider a career change to shoe sales.

In other words, if any game at all is going to be changed on October 1, whatever additional “requirements” are imposed by the FDA’s guidelines to be issued in September will need to be clear, practical and based on an understanding of what is in fact possible as of October 1, 2023. And they cannot be based on what some person wishes were in place, without being very concerned with whether that’s actually the case.

Moreover, the fact that the September guidelines won’t be requirements raises the question whether they will have any impact at all. However, I don’t deny that an agency with as much power over MDMs as the FDA has will probably meet with a remarkable “compliance” rate with these “guidelines”. After all, the executives of MDMs are (presumably) well paid to develop a keen understanding regarding on which side their bread is buttered.

What about other industries? Will they immediately start to require SBOMs from their suppliers? Perhaps. As long as there’s a federal agency that has the authority today to impose mandatory cybersecurity (not just safety) requirements on vendors to the industry. And how many industries are blessed (?) with such an agency today? As I discussed in this blog post, the only industries that I know of in which a federal agency very likely has that power now are nuclear power and the military. For any other industry, “changing the game” will first require putting in place an agency that will be staffed by experts (both in cybersecurity in general and in the specific circumstances of the industry in question), that will be granted all the authority required to enforce whatever regulations they determine to be necessary, and will act without any hint of partisan game-playing, as well as…hello, did somebody just cut the connection?

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

To my many readers who don’t know much if anything about NERC CIP, I’ll explain the title of this post: A large number of important systems that operate the North American power grid are indirectly forbidden from being located in the cloud, because of the nature of the current NERC CIP requirements – which no cloud provider could ever comply with. 

It seems that every few years, there’s a lot of talk about moving Medium and High impact BES Cyber Systems (BCS) into the cloud. The reason that topic keeps coming up is it would undoubtedly be much more efficient and cost-effective to do that, and probably more secure, not less. However, the problem is also that any utility that did that wouldn’t be able to prove CIP compliance with a huge percentage of the CIP requirements.

I have believed for a long time that medium and high BCS will never be allowed in the cloud until the whole CIP compliance regime, which is now based on compliance for individual cyber assets (both physical and virtual) changes to being based on compliance for systems (i.e., BCS needs to be the foundation of CIP compliance, not BCA).

Ironically, the CIP Modifications drafting team outlined almost exactly that approach in 2018 in one or two webinars and started to work on redrafting the CIP requirements as needed to implement that approach. However, it seems that effort got set aside, perhaps because a lot of NERC entities have a substantial investment in compliance with the CIP standards as they are – which they’re reluctant to throw away (I might well agree with them, were I in their shoes). The team then turned to the more conventional approach to virtualization (i.e., basing it on cyber assets, but virtual ones as well as physical), which it continues to pursue today. See this post for a description of the unhappy 2018 experience.

There is a proposal being circulated now that would essentially create a “parallel CIP” for medium and high BCS that are in the cloud (lows can be there now without a problem, as far as I know). Of course, compliance will be much easier for the NERC entities that pursue that approach, since their evidence in all cases will probably be the CSP’s FedRAMP certification. The inevitable result of implementing this will be that all medium and high impact Control Centers (and perhaps even parts of substations) will be moved to the cloud, if it’s at all possible for the utilities to do that. Obviously, that might make the grid more vulnerable to cyberattack, not less.

Two of my posts from 2021 flesh this out: from August and November.

I don’t see any way to change this situation, other than starting to contemplate wholesale changes to NERC CIP. But I currently doubt there’s the will to do that.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

CycloneDX 1.5 arrives!

June 26 was a watershed day for the “SBOM industry” (if I may be so bold as to declare this an industry). On that day, two important things happened, both of which I discuss in this new post on FOSSA’s blog:

1.      CycloneDX 1.5 was released.

2.      The CycloneDX Authoritative Guide to SBOM was published.

Which of these do I think is more significant? While CDX 1.5 represents a solid advance for what was already an excellent SBOM standard, the Guide (which I never even heard was in development, since – sniff! – Steve Springett never mentioned it to me) is simply the single best document on SBOMs that I’ve ever read.

I’ll point out that this isn’t a technical guide to CDX 1.5 (here’s the technical guide) and doesn’t even mention anything about v1.5, v1.4, etc. Instead, it introduces SBOMs, their use cases, and important features. Of course, all the examples in the Guide are from CycloneDX and the topics were clearly developed with CDX in mind; this just shows that the people that wrote the Guide, the CDX development team, ain’t no fools. But SPDX users will find the Guide very useful as well, so I recommend it to everyone.

What I especially liked in the Guide were the three items I discussed at the end of the post, under the heading “Three Important SBOM Problems and Their Solutions”. These are three of the hardest questions regarding SBOMs (the SBOM Forum just spent two meetings discussing only the first of these, and we never reached a conclusion). I’ve been wondering about all of them since the NTIA days, and none of the NTIA or CISA workgroups has ever seriously discussed them, let alone looked for a solution.

The Guide shows how each of these problems can be solved – yes, I’m saying “solved”, not “mitigated” - in CycloneDX (I assume they can be solved in SPDX as well, since the problems are certainly not specific to CDX). I planned to start discussing all three of them in blog posts, although I wasn’t sure I’d find an answer. Now the Guide’s published the answers. It seems they’re trying to put me out of business…

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Just a few hours ago, I got an email letting me know that Dale Peterson had mentioned me in a comment on LinkedIn. Since I always find Dale’s comments on my posts – both positive and negative – to be a great learning experience, I went to see what he had said. And when I saw he was commenting on a topic that I was planning to write about today anyway (but was thinking of putting off ‘til tomorrow), that was it – I had to write it now. Once I have this post up, I’ll find something to eat for dinner.

Dale had posted a video of a presentation by Lindsey Cerkovnik of CISA that mentioned the naming problem – something I’ve written about a few times and will write about much more in the near future. In his post, Dale said that one of the biggest challenges to SBOMs is “naming (well, actual identity) of software and software components. Then Lindsey talks about three possible solutions, CPE, SWID Tags, and Purls, and the challenges using each of these. Where is the global identifier solution?” I couldn’t agree more with Dale’s statement about naming being one of the biggest challenges to SBOMs. In fact, I’d rank it the second most serious of the three showstopper problems. Of course, there are other problems that aren’t showstoppers, but are still serious).

As always happens with Dale’s LinkedIn posts, there were a number of comments. He added one himself, which read:

I believe Tom Alrich wrote recently that NIST claimed to not have the funding or project for this mission….I keep hearing that identity is the big technical issue that isn't solved. Was hoping to get a solution in Lindsey's session. Maybe I will get some great ideas in and on stage at S4x24.

I’ll say it right off: the global identifier solution is purl. This is evidenced by the fact that, as of today, it’s hard to find a single vulnerability database (or other type of software “database” like Google GUAC) that is not based on purl. The big exception to this? The NVD, which is based on CPE, an identifier that was developed by NIST sometime in the 00’s (long before purl appeared, of course). The big difference between CPE and purl is that the former requires a centralized database and requires CPE names to be assigned by trained staff.

Purl, on the other hand, doesn’t require any centralized database at all and doesn’t require any trained staff. In fact, there aren’t any purl “staff” that I know of. Purl is an OWASP open source project, like CycloneDX. In fact, Steve Springett, the leader of the CDX project, is a maintainer of purl. It’s funny how Steve seems to have his hand in some of the most important developments in software supply chain security – for example, the fact that he developed Dependency-Track, to read software BOMs and look up vulnerabilities for components, a little more than ten years ago. This was at least 3-4 years before the term “SBOM” started to be used.

Of course, CDX can be used to produce many types of BOMs, the latest being MLBOM (machine learning BOM), which debuted as part of CycloneDX 1.5 today (literally!). And Dependency-Track is going stronger than ever, being used over ten million times a day to look up a component from an SBOM in a vulnerability database.

I don’t doubt that in 2011, CPE was the state of the art for software naming. And the main challenge to CPE - open source repositories that would proliferate like rabbits in the coming decade or two - was still barely on the horizon. However, CPE hasn’t worn well in recent years, given the open source challenge. So, when Philippe Ombredanne conceived of purl, its advantages quickly became obvious. Rather than try to repeat them here, you can read the post on purl I referenced just above.

You can also read the paper that the SBOM Forum wrote on the naming problem in the NVD last September (the NVD is just one source of the naming problem, but given that it’s now the most widely used vulnerability database in the world and that it uses a very problematic identifier – see pages 4-6 of our paper – makes it by far the biggest locus of the problem).

And that paper leads me to the real topic of this post (I usually get around to the topic at some point in the post): the fact that the SBOM Forum held our second meeting with the NIST NVD team last Friday. In our first meeting, the idea of a public-private partnership to enhance and update the NVD came up. This new meeting was to discuss what the NVD team has learned internally in NIST about PPPs, and their ideas of how such a partnership will be structured.

The NIST team is going to release a video about this within a couple of weeks, so I won’t try to steal their thunder. But I will say they’re very much interested in working with private industry partners from all over to fix problems in the NVD and make it a much more robust database. The video will outline their general ideas for the program and ask for any comments or questions (purely informal). They will then mull over the comments they get and post concrete plans in the Federal Register at the end of the summer. And knowing that things in the federal government take time (I was astounded to hear this, of course. I’ve always thought of the federal government as a nimble, agile dancer, able to pivot on a dime and go in a completely different direction 😊), they say they’ll be happy if the program is running by the end of the year.

So, look for the video. When it comes out, I’ll of course post a link and write about what they say in this blog. I will point out, though, that they have been reading our paper, which I referenced above. They have decided they will implement purl in the NVD, although there are several other important moving parts that have to be put in place as well. Fortunately, the biggest of those – the CVE 5.1 JSON spec – is underway now, I believe (CVE is operated by CVE.org, an independent board funded by CISA, which is of course part of DHS. The NVD is part of NIST, which is in the Dept. of Commerce. All of the NVD’s current funding comes from DoC, through NIST. Like most people, I always thought CVE was part of NVD, but that’s far from being the case).

So if you’re in private industry (say, a software developer or tool vendor) and you’ve been complaining about the NVD for a long time, you’ll now have a chance to contribute to the solution – or at least the mitigation – of the NVD’s problems. But you also need to keep in mind that there’s a much bigger goal that needs to be kept in mind, even though it’s not attainable at all in the near future.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

There has been confusion regarding the postponement of the date the software security attestations by suppliers are due under Executive Order 14028, as interpreted by OMB. The attestations were originally required for this month in last fall's OMB memo M-22-18. Most software developers that sell to the federal government will need to fill out these attestations. The recent OMB memo M-23-16 provides a new timeline for the attestations being due. Unfortunately, the new memo isn’t exactly a model of clarity, to say the least. The relevant paragraph is:

This memorandum modifies the deadlines by which agencies must collect attestation letters. Agencies must collect attestations for critical software subject to the requirements of M-22-18 and this memorandum no later than three months after the M-22-18 attestation common form released by the Cybersecurity and Infrastructure Security Agency (CISA) (hereinafter “common form”) is approved by OMB under the Paperwork Reduction Act (PRA). Six months after the common form’s PRA approval by OMB, agencies must collect attestations for all software subject to the requirements delineated in M-22-18, as amended by this memorandum. 

From this, I’ve derived the following rough timeline: 

1. CISA releases their approved version of the attestation form. The comment period for the original form won’t close for a week or two. Then it will probably  take at least 3 months before the CISA technical staff approves a revised version. Given that this form is likely to be very controversial, with a lot of pressure put on CISA from software suppliers and device manufacturers, this might well be an underestimate.
2. CISA lawyers approve the form. I strongly doubt CISA will be able to release the form without the lawyers’ approval (heck, I wouldn’t be surprised if the CISA lawyers have to approve every change to the lunchroom menu). From what I’ve seen so far about getting those lawyers’ approval for SBOM and VEX documents, it will take them about 2 months to approve the form. Thus, I'm guessing CISA will take 5 months at a minimum to develop and approve the new attestation form.
3. OMB reviews the form and approves it under the Paperwork Reduction Act. I allocate at least 2 months for that. We’re now at 7 months minimum.
4. Agencies collect attestations from suppliers of “critical software”. This occurs 3 months[i] after OMB approval, so now we're at 10 months minimum.
5. Agencies collect attestations for all software which is “subject to the requirements delineated in M-22-18…”  This is 6 months after OMB’s PRA approval date (and 3 months after the deadline for critical software), meaning that the deadline for attestations for other software will be 13 months from now, at a minimum.

Is it reasonable – other than in the bureaucratic sense of “reasonable” – that this should take so long? Actually, I think this is very reasonable. Unfortunately for CISA, being told to develop an attestation form for the NIST Secure Software Development Framework (SSDF) is something like being told to develop a perpetual motion machine, and a procedure for squaring the circle for good measure. The SSDF – like everything else that NIST puts out - was developed as a risk management framework and certainly not a compliance framework.

Thus, the SSDF includes no information on what would constitute "compliance" with any of its provisions, or what criteria might determine whether a provision applies to a supplier at all (since, when a risk management framework like SSDF is developed, the developers of the framework assume the organizations that follow it won’t be compelled to address provisions they believe don’t apply to them. That’s how risk management works). This and other questions will all need to be answered by CISA before the form is ready.

I'm sure CISA will be pressured to put in measurable compliance parameters, since only by doing so will federal agencies be enabled to determine whether or not a supplier has provided a valid attestation. However, each of these parameters is certain to be quite controversial. For example, consider the provision “Separating and protecting each environment involved in developing and building software” in the current form. If that were to survive in the new form, CISA would need to:

1.      Define “environment”. Presumably, it doesn’t refer to whether the devices that build the software are located in the desert vs. a big city (although that could certainly be part of the calculation). More importantly, what constitutes the border of this environment? Unless that’s clearly defined, the software developer might have to provide these protections to every device that’s on any of its networks, even if they’re properly segmented from each other.

2.      Define what measures constitute “Separating…each environment”. Does that mean just separating it from other networks? Does it mean the development network needs to be air-gapped from the rest of the world? And if that’s too drastic, what is at least an “adequate” level of separation?

3.      Define what measures constitute “protecting each environment”. Obviously, a single network firewall provides a good deal of protection, which is why just about every network on the planet has one today. Is that adequate protection? And if it is adequate, how does it have to be configured? If every rule reads “any/any”, is that enough? Certainly not! And if the single firewall isn’t enough protection, what is?

Those are just the first three questions that come to mind. I’m sure that, as they get answered, that will raise other questions, e.g. whether the internet routers are protected against DNS attacks. Some of these might be irrelevant, but the CISA staff won’t want to take any chances with this form, since if they leave out an important risk, they’ll be criticized for not being "tough enough" on suppliers (of course, some suppliers will always say they’re being too tough, no matter what CISA proposes. There simply isn't any happy medium).

Of course, I’m not recommending that any supplier wait a year before they create their attestation(s). They should all start on it as soon as the form is released by CISA (i.e. in five months, according to my estimate above). OMB isn’t going to make any substantive changes to the form, even if they have objections – they’ll send it back to CISA for remediation.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

The CISA SBOM-a-rama last week was quite good; I attended it virtually. The meeting started with presentations by three people leading SBOM programs in their industries: Jon Meadows of Citi (financial), Jennings Aske of NY Presbyterian Hospital (healthcare), and Charlie Hart of Hitachi Automotive (autos). They all discussed their industry’s experience with SBOMs and expressed a lot of confidence that SBOMs will help make their software more secure, given some time (speaking of time, there was general agreement that all three of them should have been given more of it for their presentations. I’m sure they – and maybe others – will have more time in the next SBOM-a-rama, which might be this fall).

However, there was something else that all three of them had in common: none of them pointed to any actual use of SBOMs in their industry. And even though all three of them are in leadership positions for proofs of concept in their industry, none of the proofs of concept is currently actively exchanging SBOMs (the healthcare PoC started exchanging SBOMs in 2018. There was a second healthcare PoC in 2020 and 2021. The third PoC started this year and is on a temporary hiatus but promises to return).

So, what was different about the three presenters? Both Jon and Jennings had a positive attitude, yet were clearly disappointed that they are encountering serious problems in their PoCs. In Jon’s case, the problems have to do with the “quality” of the SBOMs they’ve received as well as the naming problem, which is serious but may be on the road to at least a partial solution. In Jennings’ case, the problems are uncertainty about how to approach VEX, and issues regarding how to integrate SBOM and VEX data with existing asset and vulnerability management systems.

But Charlie was very different. He was quite upbeat, and he was – dare I say it? – clearly having fun. The autos PoC, which he runs, isn’t exchanging SBOMs between suppliers and customers any more than the other two are. However, the industry as a whole (or at least a significant percentage of the companies) is conducting tabletop exercises, where representatives from automotive suppliers and OEMs get together in one place, throw a bunch of SBOMs on the table (which don’t relate directly to products made by any of the suppliers, thus avoiding any legal issues) and discuss them – why they’re good, why they’re bad, etc. If a supplier doesn’t get something right – and it’s almost guaranteed that something won’t be right – everyone can learn from their mistake, and nobody has to feel bad about it.

What allows this to happen? I haven’t talked with Charlie about this specifically, but my guess is the Autos PoC had both suppliers and end users sign an NDA that says something like (in legal language, of course), “We’re entering into this exercise with the clear understanding that we’re all learning about something new. There are still many unknowns about SBOMs and VEX information, as well as the policies and procedures that need to surround them. If we pretend any of this is well-understood practice, we’re simply fooling ourselves. More importantly, we’ll spend all our time trying not to make a mistake and we won’t learn anything. Therefore, we’re going to jointly examine publicly available SBOMs for a variety of products and see what we can learn about creating and utilizing them”.

In listening to Charlie, it occurred to me (although not for the first time) that we ought to apply the approach of the Autos PoC[i] to the whole software community (at least in the US. I’m not competent to decide what other countries should do). We ought to agree (in NDAs between suppliers and their customers. This might be a standard form or a customized one) that we’ll treat SBOMs as a big tabletop exercise, where suppliers promise customers they’ll do their best to produce usable SBOMs (perhaps for legacy products, in order not to reveal information about current products that they may not want revealed, for competitive or security reasons) and the customers promise not only not to sue the suppliers for mistakes, but to provide feedback to them that will help them improve. In a year, suppliers and their customers will decide together how this is going, and whether they want to continue the “exercise” or start moving toward an approach based more on legal guarantees.

What’s the alternative to taking this course? It’s the state we’re in now, where I don’t know of a single software or intelligent device supplier in the US (other than in the military realm, into which I have no visibility) that is providing SBOMs to its customers (who aren’t themselves developers) on a regular basis (which I define as with at least every major new version). At the moment, we’re learning almost nothing about SBOMs, because only experience can settle most of the big questions. Could learning at least something, and having fun while doing so, be all that bad? It would certainly be a change of pace…

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.