On November 10, CISA issued a blog post called “TRANSFORMING THE VULNERABILITY MANAGEMENT LANDSCAPE”. It got a lot of attention and widespread approval. It describes three techniques whose implementation will, according to the post, lead to “more efficient, automated, prioritized vulnerability management.” While these should be of interest to both software (and intelligent device) suppliers and end users, the recommendations are aimed primarily at the suppliers – since they need to take the initiative in all three of these areas.

Since the third recommendation, use of the new SSVC vulnerability categorization framework (which, not at all coincidentally, is CVSS spelled backwards) isn’t closely related to the first two, I’ll just focus on those two, although SSVC strikes me as a very good idea.

CISA’s first recommendation is that software suppliers should “Publish machine-readable security advisories based on the Common Security Advisory Framework (CSAF).” CSAF is currently on v2.0, which is also its first versoin. CSAF is the replacement for the Common Vulnerability Reporting Format (CVRF) version 1.2, which has been available since 2017. CVRF was developed and maintained by the CSAF technical committee of OASIS. I don’t know the full story, but at some point the committee presumably decided that the new version they were contemplating was going to be so different from CVRF that they should just name it after the committee. Sounds like a smart move to me. After all, it’s all about branding.

Unfortunately, branding alone isn’t enough when it comes to a machine-readable advisory format. You need two more things. First, you need at least one software tool to create the advisories, for use by suppliers who aren’t intimately familiar with the format. If you spend five minutes looking through the CSAF format, I think you’ll agree that it would take a lot of study for someone with no prior knowledge of, or experience with, CSAF to create an advisory without the help of a tool. Currently, the only available tool is Secvisogram, a CSAF editor (this was also the only available tool when the VEX working group approved CSAF as the VEX format in the spring of 2021). It will count brackets for you and perform similar tasks, but you need a full understanding of the CSAF format in order to use the tool to create a CSAF document.

There are a small number of mostly very large organizations – including Oracle, Cisco and Red Hat – that have announced support for CSAF 2.0. However, I could only find actual CSAF files published by Red Hat (and Red Hat labels the CSAF 2.0 format as “beta”, even though it was finalized and approved more than a year ago). Presumably, those organizations have been able to make the substantial investment of time required to create CSAF advisories (I know that Cisco and Red Hat have been part of the CSAF technical committee for years).

However, CISA’s blog post didn’t limit its recommendations just to large, well-resourced companies. They clearly want all software and device suppliers, large and small, to start issuing CSAF vulnerability advisories. And this is where I see a problem: Expecting every supplier of any size, large or small, to start creating CSAF advisories without having to invest a huge amount of time learning the CSAF format requires a “CSAF for Dummies” tool. I define this as a tool that prompts the user for the information they want to represent in the advisory – the CVE or CVEs in question, the affected products and versions, remediation advice for each affected version (which might include a patch or upgrade, but might also include something else), etc. To use the tool, the user shouldn’t need to understand the details of the format; they should just be required to answer questions about what they want in the advisory.

Currently, no such tool is available for CSAF (one is under development, and evidently has been for a long time); if a supplier wants to create CSAF advisories now, they have to go Full Monty and learn CSAF well enough to be able to use Secvisogram intelligently. And there are clearly a number of required fields – like “product name” (and the related concept of “product tree”) and “branches” – whose meaning is anything but self-evident. Any supplier wishing to create CSAF documents will need to have a good understanding of the huge number of options available for creating these and other fields, as well as understand how versionsare represented, which is anything but straightforward.

Second, a machine-readable advisory format, in order to be usable, requires tools that read the format. And here the story is the same: There are no tools that read CSAF documents, even a simple parser tool. There is a parser tool under development, but that’s all. And frankly, a parser tool isn’t going to do end users a lot of good. Since most medium-to-large organizations utilize some tool or tools to manage vulnerabilities on their network (the tool may go under the name “scanner”, “vulnerability management”, “configuration management”, “asset management”, and probably other names as well), just parsing a CSAF file so that it’s readable by ordinary humans doesn’t get the information into their vulnerability management tool, where it’s needed. At the moment, the user is going to have to key the information in by hand, unless they’ve created their own tool to do this.

You probably get the idea: CSAF is a machine-readable vulnerability reporting format, but currently no machine can create or read CSAF documents – which raises the question what “machine readable” means in practice. If CSAF 2.0 had just been recently released, I wouldn’t be too bothered by this fact. Indeed, I wasn’t bothered a year ago that there weren’t any tools for it, since it was then only three months since CSAF 2.0 had been approved (although it had been under development for at least a couple of years).

But to be honest, the fact that it’s a year later and there still aren’t tools available for producing or utilizing CSAF documents makes me wonder why that’s the case and when this situation will change. The VEX working group has been repeatedly assured that such tools are under development (as we were 18 months ago, when we decided to base the VEX format on CSAF); and it’s certainly true that they’re under development. But I won’t recommend that any software supplier even start learning to create CSAF documents until someone can provide me a firm date by which a CSAF creation tool will be ready (and the date had better be before say July 1, 2023. Anything beyond that isn’t a firm date; it’s a wish and a prayer).

Of course, it still won’t help when there’s a CSAF creation tool, if there aren’t also CSAF consumption tools. And those need to be something more than a parser, as described above. But I’ll cut CSAF some slack: If there’s a parser ready by next July 1 along with a CSAF production tool, I’m willing to stipulate that it might not take a huge additional effort to put together the required interfaces to vulnerability management tools (in fact, the tool vendors themselves might create them). So, by the end of 2023, end user organizations might be able to actually utilize CSAF advisories, if the production and parser tools are ready by July 1.

Getting back to the CISA blog post, it seems clear that CISA should never have advised that suppliers start putting out CSAF vulnerability advisories without also warning them that a) they need to be prepared to invest a lot of time learning the CSAF format, and b) they have to understand that currently no end user organization will be able to do anything with the advisories when they get them.

CISA’s second recommendation was “Use Vulnerability Exploitability eXchange (VEX) to communicate whether a product is affected by a vulnerability and enable prioritized vulnerability response”. So how about this? Are there tools to produce and utilize VEX documents? Fortunately, the picture is much brighter there. There is at least one open source tool now, Dependency-Track, that both creates and reads VEX documents. Plus, I know of at least one other tool today, Stack Aware, that reads VEX documents. Moreover, I’ve been told there are a number of other tools, for both creating and ingesting VEX documents, that will be available soon.

So, it seems that CISA’s second recommendation was a good one. However, there’s one small problem in what they recommended: They only mentioned the CSAF based VEX format. Since that format is just a subset of the full CSAF format (and that subset still hasn’t been spelled out, beyond a short NTIA document that I prepared the first draft of in 2021, but which I now realize isn’t complete), any tools to create or read CSAF VEX documents will need to wait on tools that do the same for all CSAF documents  – thus, what I just said about CSAF documents in general applies directly to CSAF VEX documents as well.

However, there is another VEX format. In January 2022, Steve Springett and Patrick Dwyer, co-leaders of the OWASP CycloneDX (CDX) project, surprised everyone (including me, who had by then become friends with Steve) by announcingthat CycloneDX v1.4 includes VEX capability. I can confirm that the CDX VEX implementation is quite straightforward, so it doesn’t surprise me that there are already tools both to read and write CycloneDX VEX documents.

So, CISA provided great advice in their second recommendation: Software suppliers should create VEX documents in CDX format, since there are currently tools both to create and read them. In fact, the same applies to the first recommendation: The CycloneDX format can be used to create Vulnerability Disclosure Reports (VDR) just as easily as it can be used to create VEX documents. As long as they’re not committed to CSAF and nothing else, a supplier will be able to “Publish machine-readable security advisories” per CISA’s recommendation.

Funny how these things work out… 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

In September, I announcedthat the SBOM Forum, the informal discussion group I started last March, had published a “proposal” to address the naming problem. This problem is one of the most important obstacles currently limiting use of software bills of materials (SBOMs) by end user organizations (although I pointed out, as I do frequently, that software developers are already using SBOMs heavily to learn about vulnerabilities in the software they’re building. This shows the benefits are real).

In thisrecent post, I described the problem, which centers on the CPE identifier that’s the basis for the National Vulnerability Database (NVD), using excerpts from our proposal. While there are six particular causes of the problem, their result is the same: Only a small percentage - probably under five percent - of component names found in an SBOM can be located through a search of the NVD, even using a properly-constructed CPE name. If a developer wishes to find more than this small percentage of names, they need to utilize various heuristic schemes like AI and fuzzy logic to increase the percentage.

The good news is that, by all reports I’ve heard, it’s possible to find a respectable percentage of component names in the NVD using these schemes. But the bad news is that having to do this means that, until at least a partial solution is implemented (and ours is the only proposal I’ve heard about, although I know CISA will be soliciting other proposals as well - something a government agency has to do), producing SBOMs is always going to require a substantial “manual” effort; this process can’t be fully automated until the component names produced by the automated process can in principle always be found in a vulnerability database like the NVD or OSS Index.

How does our proposal address this problem? It proposes that the NVD start accepting other identifiers for software and hardware besides CPEs – although there is no need to replace CPEs. Existing CPEs can be retained and new CPEs can be created. We are proposing that the NVD accept new identifiers for software and hardware. I’ll discuss our software naming proposal in this post, and our hardware naming proposal in a future post.

For software, we’re proposing that the NVD accept an additional identifier called purl, which stands for “package url”. While there is no way to know for sure, we believe that accommodating purl will address 70-80 percent of the software naming problem in the NVD. Purl is already in widespread use in the open source community; it is the identifier used in Sonatype’s OSS Index and most other open source databases.

Purl was developed primarily to address the problem that the “same” open source component can often be found in multiple repositories (package managers), and the code may be somewhat different in each one. Additionally, the same component might be found with different names in different package managers. Purl provides a name that is tied to the download location; if the user knows where they downloaded the component from and the name in that repository, they should always be able to create a purl that matches the one used by the component supplier (which is often an open source community, of course) when they reported a vulnerability for the component; thus, they should always be able to find the vulnerabilities reported for that component. That certainly can’t be said for CPE, or any other identifier that has to rely on a centrally maintained database.

Since 90% of software components are open source, purl was the natural choice to be the basis for the SBOM Forum’s proposal. However, as you’ll see if you read our whole proposal, we have utilized the unique properties of purl to incorporate names for proprietary (or “closed source") software into the proposal as well, utilizing SWID tags (I plan to discuss this in a post in the near future).

Purl is unique among identifiers in that it doesn’t require a centralized database. How is that possible? Here’s how we answer that question in our proposal:

Our solution is based on…the distinction between intrinsic and extrinsic identifiers. As described in an excellent article^[1], extrinsic identifiers “use a register to keep the correspondence between the identifier and the object”, meaning that what binds the identifier to what it identifies is an entry in a central register. The only way to learn what an extrinsic identifier refers to is to make a query to the register; the identifier itself carries no information about its object.

A paradigmatic example of an extrinsic identifier is Social Security numbers. These are maintained in a single registry (presumably duplicated for resiliency purposes) by the Social Security Administration. When a baby is born or an immigrant is given permission to work  in the US, their number is assigned to them. The only way the person “behind” a Social Security number can be identified is by making a query to the central registry (which of course is not normally permitted) or by hacking into the registry.

By contrast, intrinsic identifiers “are intimately bound to the designated object”. They don’t need a register; the object itself provides all the information needed to create a unique identifier. What  intrinsic identifiers need is an agreed-on standard for how that information will be represented. Once a standard is agreed on, anyone who has knowledge of the object can create an identifier that will be recognizable to anyone in the world, as long as both creator and user of the identifier follow the same standard.

An example of an intrinsic identifier is the name of a simple chemical compound. As the article states, “We learned in high school that we do not need a register that attributes different identifiers to all possible chemical compounds. It’s enough to learn once and for all the standard nomenclature^[2], which ensures that a spoken or written chemical name leaves no ambiguity concerning which chemical compound the name refers to. For example, the formula for table salt is written NaCl, and read sodium chloride.”

In other words, simply knowing information about the makeup of a simple chemical compound is sufficient to create a name for the compound, which will be understandable by chemists that speak any language - as long as you follow the standard when you create the name. NaCl refers to table salt, no matter where you are or what language you speak^[3].

An example of an extrinsic identifier that is very pertinent to our proposal is a CPE name. CPE names are assigned by a central authority (NIST) and stored in a register. Whenever a user searches for a CPE name in the National Vulnerability Database, the register is searched to determine which entry  the name refers to.

Only CVE Numbering Authorities (CNA) authorized by NIST may create CPE names; currently, there are around 200 CNAs, mostly software suppliers or cybersecurity service providers. Organizations that wish to report a CVE (vulnerability), but do not themselves have a CNA on staff and are unable to identify an appropriate CNA to do this, may submit their requests to the MITRE Corporation “CNA of Last Resort”, which will identify an appropriate CNA to process the request.

In most cases, a software supplier applies to NIST for a CPE name for a product. When the name is assigned, the supplier can report vulnerabilities (each of which has a CVE name, assigned by CVE.org^[4]) that apply to the product. Even though software products are named in many different ways as they are being developed and distributed, the product name used in the CPE is based on a specification^[5] that has no inherent connection to the product itself.

The solution we are proposing for the software naming problem is based on an intrinsic identifier called purl^[6], a contraction of “package URL”. As in the case of a simple chemical compound, knowing certain publicly available attributes of a software product enables anyone to construct the correct purl for the product. Moreover, anyone else who knows the same attributes will be able to construct exactly the same purl, and therefore find the product in a vulnerability database without having to query any central name registry (which of course does not exist for purl, since it is not needed).

Purl was originally developed to solve a specific problem: A software product will have different names, depending on the programming language, package manager, packaging convention, tool, API or database in which it is found.^[7]Before purl was developed, if someone familiar with, for example, a specific package manager (essentially, a distribution point for software) wanted to talk about a specific open source product with someone familiar with a different package manager, the first person would need to learn the name of that product in the second package manager, assuming it could be found there. This would always be hard, because – of course – there is no common name to refer to.

This situation is analogous to the case in which an English speaker wishes to discuss avocados with someone who speaks both Swahili and English. The English speaker doesn’t know the Swahili word for “avocado”. To find that word, the English speaker uses an English/Swahili dictionary to learn that the Swahili word for avocado is parachichi. Neither avocado nor parachichi has any connection to an actual avocado – they are simply names. They are extrinsic identifiers, and there is no way to know that they refer to the same thing without a central register, which in this case is the dictionary.

However, what if the two speakers didn’t have ready access to an English/Swahili dictionary, either in hard copy or online? Most likely, the English speaker would find a picture of an avocado and show it to the Swahili speaker. The latter would smile broadly and say they now understand exactly what an avocado is. In fact, even if a dictionary were available, it would probably be easier for the English speaker to show the picture to the Swahili speaker. The picture is an intrinsic identifier; it is based on an attribute of the thing identified – in this case, what the thing looks like. Even if there were no English/Swahili dictionaries in existence, the picture would be a perfectly acceptable (in fact, preferable) identifier.

If you want to know how we envision purl working in the NVD, go to the discussion starting on page 10 of our proposal. In future posts, I plan to discuss how our proposal addresses proprietary software and intelligent hardware devices.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

To say there is confusion about what constitutes VEXis a big understatement. Conservatively, I’d say upwards of 95% of what’s been written about VEX or discussed in webinars or live presentations is wrong or misleading; this includes a lot of what I’ve written.

However, I’m not alleging this is due to some vast conspiracy to bamboozle the cybersecurity community. Nobody making these wrong or misleading statements is acting in anything other than good faith. But good faith doesn’t make those statements correct.

There are two primary reasons for this situation:

1.      The VEX working group, which started under the NTIAin 2020 and continued under CISA this year, has only published two documents on VEX. These can both be found at cisa.gov/sbom[i]. While these are both good, neither one attempts to provide a definitive definition or description of VEX.

2.      More importantly, almost all of the false or misleading statements were based on expectations – really, hopes – for events we all thought would occur, but in fact didn’t. But as often happens, the words didn’t change nearly as quickly as events on the ground. This is why statements that were neither wrong nor right when they were made are now wrong. It is also why people whose knowledge of VEX is based on those erroneous statements make more erroneous statements, in classic whisper-down-the-lane fashion.

Because of this situation, VEX documents – which many people believe to be in wide production and use at this time – are neither being produced nor used, to the best of my knowledge. How could this be otherwise, given that no parties interested in producing or using VEXes could possibly understand how to do either?

Since the human race has existed for approximately 200,000 years without VEX documents, the fact that this situation isn’t changing currently might not be considered high tragedy. However, the VEX idea came about because of an important realization: Software bills of materials (SBOMs) will never be widely produced or used until VEXes are also produced and used. No VEXes means (almost) no SBOMs. It’s literally that simple. Thus, if we want to have SBOMs, we have to figure out a way to have VEXes. Or at least we have to figure out how to have the informationthat we’ve been thinking (erroneously, I now believe) needs to be delivered in VEX documents.

How this situation came about and how it might be remediated are the subjects of a web event I’ll be participating in on November 30 at 10AM Eastern Time. It’s sponsored by Scribe Security. I hope you can join in!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

I confess that I and almost every other person who writes or speaks about software bills of materials has been misleading you for years. We have done this when we’ve spoken about SBOM as something that is well defined. The two main formats are well defined, but almost all the best practices regarding production, distribution and use of SBOMs are still far from being defined, or at least agreed upon.

We have also been misleading you when we’ve implied or stated that SBOMs were being widely (or even narrowly) used for the purpose that most people talk about (including in Executive Order 14028): to enable software end users to learn about most important exploitable component vulnerabilities in the software they utilize, so they can coordinate with the suppliers to patch or otherwise mitigate them.

There are many issues standing in the way of widespread SBOM use, but there are a few that are show-stoppers, meaning I don’t see any way SBOMs will be used widely for this purpose until those problems are addressed. One of those is the naming problem, which can be summarized as, “When an SBOM is generated by an automatic process, only a small percentage of component names will be found through a search of the National Vulnerability Database (NVD).” In other words, in very few cases will a user who looks for a component name in the NVD find the component – meaning the user (or even a software tool acting on their behalf) will only be able to learn about a small percentage of component vulnerabilities through an NVD search, despite having an SBOM listing all the components.

How small a percentage is this? The Director of Product Security for a very large software supplier, who has participated for years in the SBOM and VEX discussions under the NTIA and now CISA, as well as the informal group I started called the SBOM Forum, had previously used the figure of 20%. Of course, that’s bad, since it means the user won’t be able to learn (through searching the NVD) about vulnerabilities applicable to 80% of the components in an SBOM they receive.

However, that same person was challenged regarding this figure by another very large supplier (of both software and intelligent devices) at one of our SBOM Forum meetings. Did the other supplier question how the 20% figure could be so low? No, they asked why it was so high, because their experience was that it was below 5%. The original supplier admitted that 20% was a very conservative number and agreed that 5% is closer to their actual experience.

So, the good news is that it isn’t true that 80% of component names can’t be found in the NVD; the bad news is the figure is really more like 95%. This means you’ll only be able to find about 5% of component names from an SBOM in the NVD.

Needless to say, SBOMs wouldn’t be used at all if users could only find 5% of component vulnerabilities. Yet, even though end users are barely using SBOMs at all today, suppliers are using them very heavily. How can they do that? It’s because every supplier who needs to use SBOMs – or in many cases, the consultants who help suppliers use SBOMs – has some method, based on AI, fuzzy logic, throwing bodies at the problem, prayer, etc., to get around this problem. The good news is that every supplier or consultant that I’ve talked to about this problem says they’ve been able to get the matches up to an acceptable level, although it’s nowhere near 100%.

So while this is an acceptable workaround for those suppliers willing to invest the time, money or both that is required, the fact is that it makes it impossible to “operationalize” SBOM production; every SBOM produced will need its own care and feeding, rather than being a completely automated process. SBOMs will never be produced in the volume required, if every one of them needs to be massaged in this way.

This is why the SBOM Forum developed a workable “solution” for maybe 70-80% of the naming problem in the NVD, which we described in this paperthat we released in September. It’s now being evaluated by CISA and MITRE, and I’m reasonably optimistic it will be implemented at least in part. In this post and (probably) my next one, I’ll lay out the basics of the argument that we made in the paper, since I’ll admit that it’s densely written and a lot of people may have given up on reading it.

The NVD’s problems lie mostly with CPE names, the only identifiers supported by the NVD. Here are the six main problems:

1. Vulnerabilities are identified in the NVD with a CVE number, e.g. “CVE-2022-12345”. A CPE is typically not created for a software product until a CVE is determined to be applicable to the product. However, many software suppliers have never identified a CVE that applies to their products, so they have never created a CPE for them. This is almost certainly not because the products have never had vulnerabilities, but because the suppliers, for whatever reason, have not submitted any vulnerability reports for those products for inclusion in the National Vulnerability Database.

The worst part of this problem is that the result of an NVD search will be the same in both cases - the case where a vulnerability has never been identified in a product and the case where the supplier has never felt inclined to report a vulnerability, even if their product is loadedwith them. The search will yield “There are 0 matching records” in both cases. Someone conducting a search won’t know which case applies, so they may believe the product has no vulnerabilities, when the truth is very different.

2. There is no error checking when a new CPE name is entered in the NVD. Therefore, if the CPE name that was originally created for the product does not properly follow the specification, a user who later searches for the same product and enters a properly-specified CPE will receive an error message. Unfortunately, it is, once again, the same error message that they would receive if the original CPE name were properly specified but there are no CVEs reported against it: “There are 0 matching records”.

In other words, when a user receives this message, they might interpret this to mean that there isa valid CPE for the product they’re seeking, but a vulnerability (CVE) has never been identified for that product - i.e. it has a clean bill of health. However, in reality it would mean the CPE name was created improperly. In fact, there might be a large number of CVEs attached to the off-spec CPE, but without knowing that name, the user will not be able to learn about those CVEs.

Another explanation for the “There are 0 matching records” error message is that the user had misspelled the CPE name in the search bar. Again, the user would have no way of knowing whether this was the reason for the message, or whether the message means the product has no reported vulnerabilities.

It is to avoid problems like this that most organizations that use the NVD employ advanced search techniques based on AI or fuzzy logic^[1]. While that can greatly reduce the number of unsuccessful searches, having to resort to this makes it impossible to conduct truly automated searches. Considering that an average-sized organization might easily need to conduct tens of thousands of NVD searches per day and a service provider doing this on behalf of hundreds of customers would need to conduct some large multiple of that number, the magnitude of this problem should be apparent.

3. When a product or supplier name has changed since a proprietary product was originally developed (usually because of a merger or acquisition), the CPE name for the product may change as well. Thus, a user of the original product may not be able to learn about new vulnerabilities identified in it, unless they know the name of the current supplier as well as the current name for the product. Instead, this user will also receive the “There are 0 matching records” message.
4. A similar consideration holds true for supplier or product names that can be written in different ways, such as “Microsoft(™)” and “Microsoft(™) Inc.”, or “Microsoft(™) Word” and “Microsoft Office(™) Word”, etc. A user searching on one of the variants of a supplier or product name may learn about just the CVEs that are applicable to the variant they entered, rather than all of them.
5. Sometimes, a single product will have many CPE names in the NVD because they have been entered by different people, each making a different mistake. For this reason, it will be hard to decide which name is correct. Even worse, there may be no “correct” name, since each of the names may have CVEs entered for it. This is the case with OpenSSL (e.g. “OpenSSL” vs “OpenSSL_Framework”) in the NVD now. Because there is no CPE name that contains all of the OpenSSL vulnerabilities, the user needs to find vulnerabilities associated with each variation of the product's name. But how could they ever be sure they had identified all the CPEs that have ever been entered for OpenSSL?
6. Often, a vulnerability will appear in one module of a library. However, because CPE names are not assigned on the basis of an individual module, the user may not know which module is vulnerable, unless they read the full CVE report. Thus, if the vulnerable module is not installed in a software product they use but other modules of the library are installed (meaning the library itself is listed as a component in an SBOM), the user may unnecessarily patch the vulnerability or perform other unnecessary mitigations. In fact, it’s likely that at least some of the patching performed for the log4j vulnerabilities was unnecessary, for precisely this reason.

What is needed is to be able to name software and hardware components in a BOM with an identifier that, when entered in the NVD, will

1. Almost always match to the correct product, if the product is listed.
2. Almost never match to an incorrect product.
3. Not require that the identifier already exist in the NVD. This is almost always required today, in order for the user to get a correct response. If the user searches on a CPE name that doesn’t exist in the NVD, the error message they receive, “There are 0 matching records”, is the same one they would receive if the CPE does exist, yet it has no reported vulnerabilities.
4. Never yield a result that might be interpreted to mean the product was found but there are no applicable vulnerabilities, when in fact one of the following is the case:
1. The wrong identifier was entered in the search bar; or
2. An off-spec CPE was initially created for the product, so the product cannot be found by searching on a CPE that was created according to the spec; or
3. The name and or/supplier of the product has changed due to a merger or acquisition. Thus, the CPE entered by a user of the original product won’t match the current CPE name.
5. Identify the vulnerable module in a library rather than just the entire library, so that, if that module isn’t installed in a product but other modules are installed (meaning the product will appear to be vulnerable when in fact it isn’t[i]), users will not patch or perform other mitigations that are not necessary.
6. When a supplier and/or product name has changed for a product, allow there to be separate identifiers - and thus separate locations to report CVEs - for the different supplier or product names; thus the different supplier/product names will be treated as separate products.

Who or what is the hero that will tame the CPE beast! Keep tuned to this blog for the exciting conclusion!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

For the last six months or so (maybe more), Brandon Lum of Google has been sometimes participating in two or three of the CISA SBOM workgroups, especially the VEX workgroup. His title has something to do with open source software, and since a lot of people in the workgroups are involved with OSS, I wasn’t surprised that he would be attending these meetings.

A little more than a month ago, he started making announcements in some of the meetings about a new Google project called GUAC and asking for people to participate in it. I didn’t pay a lot of attention to the details, but I knew it had something to do with software supply chain security, and with SBOMs in particular. Since open source software depends on volunteers to develop and maintain the projects, this wasn’t unusual, either.

Moreover, there was one reason why I deliberately didn’t pay a lot of attention to what Brandon was saying about GUAC: Last year, Google announced another project aimed at software supply chain security called SLSA, which has been very well received by the developer community. It’s essentially a framework that will allow developers to identify and take steps to prevent attacks on the software build process, which nobody (that I know of, anyway) even thought of before SolarWinds.

(SolarWinds fell victim to an extremely sophisticated 15-month attack conducted by – according to Microsoft’s estimate – about 1,000 people working out of Russia. There’s a really fascinating article on CrowdStrike’s website about SUNSPOT, the malware that the Russians purpose-built for this attack. In fact, they tested it during a three-month proof of concept conducted inside the SolarWinds software build environment, then deployed the malware for 5 or 6 months, without ever being detected. This was easily, along with Stuxnet, the most sophisticated malware ever developed. If only the Russians would start putting all that great expertise toward a good use, for a change! BTW, don’t try to understand everything in the article. It’s just amazing to see what Sunspot was able to do, all without any direct Russian control).

But I digress. When I heard Google was following a project called SLSA with one called GUAC, I found this a little too cute for my taste (can Google CHIPS be far behind?). So, frankly, I tuned Brandon out when he brought this up.

However, two weeks ago I saw a good article in Dark Reading – which linked to a great Google blog post - about GUAC. I also found out that my good friend Jonathan Meadows of Citi in London – a real software supply chain guru, although very focused on how ordinary schlumps like you and me (OK, maybe not you) can secure our software supply chains without having all of his knowledge and experience – was involved with GUAC from the get-go. These two datapoints convinced me that I should be paying a lot more attention to GUAC.

So I did. And this is what I found:

1. The project intends to present to users a “graph database”, which in principle links every software product or intelligent device with all of its components, both hardware and software components, at all “levels”. You might think of the database as being based on a gigantic SBOM dependency tree that goes in all directions – i.e., each product is linked with all its upstream dependencies, as well as the downstream products in which it is a component (or a component of a component).
2. One of the important functions of this database is to provide a fixed location in “GUAC space” (my term) for software products and their components. Artifacts necessary for supply chain analysis, like SBOMs and VEX documents, can be attached to these locations, making it easy for the user of a software product to learn what new artifacts are available for the product (actually, the nodes of the database are versions of products, not the products themselves).
3. While the database will incorporate any artifacts created in the software supply chain, the three types of artifacts incorporated initially will be SBOMs, Google SLSA attestations, and OSSF Scorecards. The idea is that, ultimately, all the documents required for an organization (either a developer or an end user organization) to assess their software supply chain security will be available at a single internet location (and I don’t think the location would change – just its attributes).
4. The artifacts can be retrieved by GUAC itself – the supplier of the artifact won’t have to put it in place “manually”. Google says, “From its upstream data sources, GUAC imports data on artifacts, projects, resources, vulnerabilities, repositories, and even developers.”
5. Artifacts like SBOMs can be contributed and made available for free, but they don’t have to be. The Google blog post says, “Some sources may be open and public (e.g., OSV); some may be first-party (e.g., an organization’s internal repositories); some may be proprietary third-party (e.g., from data vendors).” In other words, a vendor that has prepared documents or artifacts related to security of a product can attach a link to the product in GUAC space. Someone interested in one of those artifacts can follow the link and, if they agree to the price, purchase it from the vendor.
6. Thus, one function of GUAC can be enabling a huge online marketplace. However, unlike most markets related to software, the user won’t have to search on the product name to find what’s available for it. Instead, the user will just “visit” the fixed location for the product and look through what’s available there.

I can imagine the inspiration for GUAC might have occurred when some Google employee involved in software supply chain security grew frustrated at the number of different internet locations they had to visit to get the artifacts they needed to analyze just a single product. Instead of a person running ragged while searching for the most up-to-date artifacts, how about having the computer do the legwork in advance? The user would just have to go to the right location, to find everything they need in one place and with one search.

In fact, Google has done this before. Those of you who were involved with computers in the later 1990s (once the internet had supposedly arrived, but was often proving to be slower than just doing some things by hand) may remember that the search engines – Yahoo, DEC’s Alta Vista, etc. – just searched for character strings. If you were good (plus lucky) and entered a string that would return you just the items you were interested in but no others, searching was a pleasant experience.

However, if you weren’t good and lucky, and just searched on a topic like “mountain vacation”, you would get hundreds of pages of results, with no assurance that what you were really looking for wasn’t on the very last of those pages. Google came out with a very intelligent search engine that used all sorts of tricks – like ranking results by their popularity with others – to make it more likely that what you were looking for would be on the first page or two. The rest, as they say, is history.

I certainly don’t think GUAC will have anywhere near the success that the search engine had. After all, just about everyone in the world can use a general search engine, but only a small fraction of the world’s inhabitants are involved with software supply chain security – although given how I spend my time nowadays and the people I meet online, I’m sometimes tempted to think it’s actually a very large percentage.

On the other hand, a look at the GUAC project’s statistics on GitHub shows that GUAC has already become at least a minor phenomenon in the few short months of is existence (if even that): over 10,000 code repositories, close to 700 users, 2,000 issues under discussion, 31,000 commits (i.e. code insertions or revisions). These indicate a huge level of early interest.

My guess is that in five years, anyone involved in software supply chain security will be spending a lot of their time navigating the highways and byways of the GUAC world. For some, the need to do this will become apparent sooner rather than later. For example, if you’re involved with one of the companies for which distribution of SBOMs is an important part of the business model, you should be figuring out how you can incorporate GUAC into that model – although I’m certainly not saying you should abandon whatever you’re doing, or planning to do, now.

Another idea: While I haven’t tried to look at tech specs on the project yet, I’m sure there must be some sort of fixed address for a software product (or a component, which of course is just a product that’s been incorporated into another product) within GUAC world. I can see that address becoming a kind of universal name repository for a software product, which of course can have multiple names over its lifecycle. Currently, if you have an old version of a product whose name has subsequently changed, and you want to learn about vulnerabilities that have recently been reported for the product, you’re out of luck, unless you happen to know the current name of the product. That (and a lot of other things) may change with GUAC.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

In the past 3-4 years, NIST has produced a dizzying array of guidance documents for IoT security. This isn’t because they can’t make up their minds. It’s because a) the IoT marketplace has been going through rapid change, and b) NIST has been ordered to undertake at least two “regulatory” mandates for IoT (in the IoT Act of 2020 and the IoT device labeling program mandated by Executive Order 14028), even though they’ve never been a regulator. The mandates changed multiple times after they were given to them; in both cases, they’ve finally been withdrawn from their purview.

Thankfully, now NIST’s duties regarding IoT seem to have returned to their traditional role, which they are quite good at: developing risk-based cybersecurity frameworks that are mandatory for the federal government and merely advisory for the private sector. The end product of the various NIST documents and initiatives regarding IoT security in the past few years is a new Interagency Report: NIST.IR 8425.

This is a very good document, and it was worth the many interim documents, workshops, etc. that led to it. Don’t get thrown off by the fact that it’s titled “Profile of the IoT Core Baseline for Consumer IoT Products”. “Consumer” is in the title because these were originally intended to be the guidelines behind the device labeling program in EO 14028, not because the NISTIR only addresses security of consumer (i.e. household) IoT devices.

It seems the folks in the White House decided a few months ago that they now have a better idea for the device labeling program, than what they included in the EO. They would like to run it themselves and they’re going to gather ideas for the program from a group of industry folks, rather than leave NIST the task of designing and running it – which, truth be told, didn’t go very well.

This move by the White House (which of course wasn’t publicized, but seems to be the only explanation for their recent actions) freed NIST to take the guidelines they’d published in February for the device labeling program and turn them into their final set of guidelines for both business and consumer (i.e. household) IoT products. NIST did this with only minor changes to the document, based in part on comments they’ve received since February

The fact that NISTIR 8425 is intended to address both consumer and business (i.e. commercial and industrial) products is made clear at the beginning of the Abstract on the first page. The first sentence states that the publication “…identifies cybersecurity capabilities commonly needed for the consumer IoT sector (i.e., IoT products for home or personal use)”. However, the second sentence reads, “It can also be a starting point for businesses to consider in the purchase of IoT products.”

With the words “starting point”, NIST is clearly leaving the door open for more extensive business-oriented guidelines in the future. Any businessperson who yearns for more extensive guidelines today can always go back to NISTIRs 8259A and 8259B, both of which are good documents and go beyond what’s included in NISTIR 8425 (in fact, most of the guidelines in 8425 were selected from 8259A and B).

But NIST’s choice of words means that businesses wanting to base their IoT program on a recognized framework (at least, recognized in the US) would do well to follow NISTIR 8425 now, since any further NIST guidelines for business IoT are likely to build on that, not replace it. This applies both to manufacturers of IoT and IIoT devices that are designing new products or beefing up security on existing products, and to commercial and industrial firms that are looking for guidelines they can use to assess the security of the devices they buy.

There’s another use case for NISTIR 8425 in those commercial and industrial firms: as the basis for contract language. But please do me a favor: Before you write a contract term that reads something like, “Vendor must comply with NISTIR 8425”, keep in mind that “comply” is a meaningless term when it comes to this document (and to most other NIST documents).

For example, look at the first requirement in the “Data Protection” section on page 7: “Each IoT product component protects data it stores via secure means.” Of course, there are lots of ways to “protect data” by secure means. Which one should the manufacturer implement in the device?

That will very much depend on the destination of the device. If the device is for dry cleaning establishments, its security requirements can be much less strict than if it were going into, for example, a nuclear power plant. A device intended for a nuclear plant would probably need to employ the latest encryption technology to secure the data stored by the device, while a device intended for a dry cleaner might not need encryption at all. For them, just requiring the user to enter a password when they first log into the device might be all the protection required.

This is why simply asking a device manufacturer for “compliance” with 8425 will never work. Instead, the term might read, “Vendor shall be prepared to provide evidence that they have implemented a product risk management plan based at minimum on the guidelines provided in NIST.IR 8425” – or something like that. That way, the manufacturer will be assessed on whether it is taking steps in each area of capability listed in the NISTIR (there are ten “capabilities”, all but one of which have between one and three subordinate capabilities. The “Documentation” capability lists a large number of subordinate capabilities, which isn’t surprising).

However, there is one requirement that I think should be addressed by all device manufacturers, regardless of who their users are: They should register their device with the National Vulnerability Database (NVD) and report any vulnerability they learn of, in either the software or firmware installed in the device, as being found in the device (even though the software vendor may have already reported the vulnerability for their software). It seems very few IoT manufacturers are doing this now; yet it is the only realistic way for users of a device to learn about the vulnerabilities found in it.

My reason for requesting this is described in the last section of thisarticle, which I wrote with Isaac Dangana of Red Alert Labs. RAL is a client of mine that works with IoT manufacturers to secure their devices, and with IoT consumers to verify that the devices they buy are and remain secure. While the article officially discusses SBOMs for IoT and IIoT devices, the need to report vulnerabilities for the device itself has nothing to do with SBOMs. Rather, it’s needed so that IoT and IIoT users can properly secure their devices.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Practitioners of the black arts of software security are collectively sighing and saying, “Here we go again”, since the announcement[i] late last week by the OpenSSL Project that they’re going to release a new version of the almost ubiquitous OpenSSL cryptographic library on Tuesday. It patches a “critical” new vulnerability in that library, about which they won’t release any information.

Since there’s no patch available and the Project didn’t describe any temporary mitigation (presumably because even that might have provided enough information to the bad guys that they could have figured out how to exploit the vulnerability – although this is just my speculation), the question naturally arises, “What was the point of announcing it now? Was it so security professionals could quickly submit their resignations and move on to their new career in fast food delivery?”

No, the reason was that the OpenSSL Project knew that organizations that use OpenSSL will experience the same problem they had with earlier vulnerabilities like Heartbleed (the 2014 OpenSSL vulnerability that led to a massive effort to find and patch it – although there are probably still a lot of unpatched instances of the vulnerable versions out there), Apache Struts (which unfortunately Equifax didn’t get the memo on, resulting in the compromise of private information of more than ¾ of adults in the US. My information was also compromised), Ripple20, and of course the beloved log4j.

That problem is that OpenSSL is a component used in other software products, or in components of products, components of components of products, etc. – yea, unto the 20^thgeneration. No organization has an invoice from a software supplier that identifies OpenSSL as something the organization bought and installed. How does the organization figure out how many instances of OpenSSL they’re running and where they’re running?

The answer is simple: They look at the most recent (hopefully quite recent) SBOM, which stands for “software bill of materials”, which they received from each software or intelligent device supplier whose product(s) they use. These will tell them what are the components (but probably only the first-level ones) in each product. So, while you think of it, go find the SBOM for every software product or device that your organization operates. This should only take you five minutes, right?

Of course, this was a joke. I doubt there are many organizations outside of the military and intelligence agencies, which have virtually unlimited budgets, who have an SBOM even for the majority of software products they operate – and if they do, they probably created the SBOMs themselves (or a consulting organization they hired created them). In fact, I’m sure there are many large organizations that don’t have an SBOM for anyof the software they operate.

If you were around for the log4j effort at the end of last year, you’ll know that almost nobody had SBOMs then. But your probably heard assurances from a number of people (which could have included me, although I don’t recall giving such assurances) that by now SBOMs would be much more widely available than they were then. There was a specific reason why we said this: We knew that the SBOM provision in Executive Order 14028 was going to come into effect in August. It required federal agencies to start asking for an SBOM from each supplier of “critical software” by this past August 10.

Of course, the EO only applied to federal agencies. But we all supposed (or I did, anyway) that software suppliers weren’t going to start providing SBOMs to one large group of their customers (the agencies), but not to an even larger group (private sector organizations). Surely, we reasoned, this would be the beginning of a “beautiful friendship” (to quote Humphrey Bogart at the end of Casablanca) between software suppliers and end users – that is, an area of close cooperation between them, to address the common problem of software component security.

It turns out that we reasoned wrong. While I’m sure that at least some suppliers did provide an SBOM on request, I know of no supplier of software or intelligent devices anywhere on this planet (I’ll admit I haven’t checked with other planets) that’s regularly providing SBOMs to their users. It’s great that some of them provided one SBOM when asked.  However, SBOMs age like milk, not wine.

The official guidance is that an SBOM should be re-issued whenever the software changes, including a new patch, new build, etc. In practice, a software user should consider themselves quite lucky if one of their suppliers provides a new SBOM with every new major version release. Of course, if the only SBOM you have for a product is an old one, don’t throw it away; it’s still worth looking at. But before you take action (or don’t take action) based on something you find in an old SBOM (like OpenSSL), you’ll need to check with each of your software suppliers to find out whether OpenSSL 3.0 is present in their product and whether the vulnerability (to be announced on Tuesday) is exploitable in their product – since in over 90% of cases a vulnerability in a component isn’t exploitable in the product itself.

Why aren’t SBOMs being released? It’s not because suppliers don’t have them; they’re using them very heavily now to manage component vulnerabilities in the products they develop. See this post, which describes how just one open source tool, Dependency-Track, is being used 270 million times a month to look up around 27 billion open source software components in the OSS Index vulnerability database; almost all of that use is by software developers.

Rather, the main reason SBOMs aren’t being released is that end users aren’t asking for them. Why aren’t they doing that? There are a number of reasons, which I divide between “show stoppers” (problems that need to be addressed before SBOMs are widely distributed and used by non-developers), and others. Here are some of the show stoppers (I think there are around 6 or 7 in all, but I might be neglecting one or two. Note that all of these problems don’t need to be solved before SBOMs can start being used with regularity. However, users and suppliers will need to be confident that a solution is at least in the works):

1.      There are no tools or subscription-based third-party services that import SBOMs and VEX documents and output a list of exploitable component vulnerabilities, which is continually updated. If this is available, it means that, at any time, a user can obtain an up-to-date list of all known exploitable component vulnerabilities in a particular version of a product. Dependency-Trackcomes the closest to doing that, but even that tool doesn’t provide a continually-updated list.

2.      Even if the user has the above tool or service, they can never be sure that the list of exploitable vulnerabilities for a product/version they use doesn’t contain a number of CVEs that aren’t in fact exploitable in the product, which the supplier has already identified but which they haven’t yet communicated to customers. I have proposed “real-time VEX” as a solution to this problem. I’m happy to say that the idea will most likely be submitted to the IETF as a proposed standard – part of a larger submission – this year.

3.      When an SBOM is created, fewer than 10% (and maybe fewer than 5%) of components listed won’t be found – without a lot of manual effort – in the National Vulnerability Database (NVD); this is according to a few major software/device suppliers, but it is widely echoed throughout the software security industry. This is the problem that an informal group that I started, the SBOM Forum, addressed in thispaper. That document is being considered by appropriate authorities now; we are quite optimistic that at least the main part of our proposal will be implemented in a year or a little more than that.

4.      There are many software products that have had multiple names over their lifetime, due to mergers, acquisitions, rebranding, etc. – or simply due to an open source product being made available in multiple package managers, with a different name in each. A search on one of those names won’t find CVEs that have been reported for any of the other names. The SBOM Forum has just decided to take this problem up next.

5.      There aren’t coherent “playbooks”, describing how a software user can utilize SBOMs and VEX notifications to manage risks due to software component vulnerabilities in the products they utilize. This makes users reluctant to forge ahead with SBOMs and it makes tool providers reluctant to develop SBOM/VEX consumption tools. This is because they can’t currently be sure that what a tool is required to do won’t change (this is more of a problem for VEXes than it is for SBOMs).

6.      None of the major (or minor, as far as I know) vulnerability, asset or configuration management tools can ingest SBOMs or VEX documents, in order to learn about exploitable component vulnerabilities in products installed on an organization’s networks. Once the first problem on this list is solved, it will be easy to solve this problem, since all of these tools presumably can ingest a list of CVEs linked to products.

As I stated above, these are far from the only problems holding up widespread distribution and use of SBOMs (I have a spreadsheet with about 40 problems including the above, but I won’t say that’s complete, either). However, these are the most serious problems, and I believe they will all have to be on their way to solution, before we can say we’re making real progress on the road to ubiquitous SBOMs. As you can see, 2-3 of them are already on the way to being solved (at least for the most part), and none of the others looks insoluble.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Last week, the White House held a workshop to discuss developing a program for securing IoT devices, scheduled to start in 2023 and to apply at first to two “particularly vulnerable” types of consumer devices: WiFi routers and home security cameras. What’s most important is how the device manufacturers will be motivated to comply with the program. Instead of threatening them with terrible consequences unless they meet certain cybersecurity requirements, suppliers will be “persuaded” to meet the requirements by the fact that, if they don’t do that, they won’t receive the device label that consumers will be trained to look for when they’re buying an IoT device.

In other words, the government will in effect warn the manufacturers, “If you don’t want to make any changes at all to your current security measures (or lack thereof), you’re free to follow that course. However, assuming we’re successful in making the public aware of the importance of looking for the cybersecurity label on any device they buy, you may find you don’t have as many customers as you might have been expecting.”

The news articles on the workshop made clear that the explicit model for this program is the Energy Starprogram, which uses a label to let consumers know which appliances meet certain energy efficiency standards. That program has been very successful.

Of course, a cybersecurity device labeling program is a fairly new idea. The most successful implementations of that idea so far have been in Singapore and Finland. Both of these are very small markets, so it’s not possible to draw real conclusions about what the program’s success means for the US. That being said, both programs have been successful, and Singapore’s has recently been extended to medical devices. Both programs require third-party testing in order to obtain the label.

Another country that has implemented a device labeling program is Germany. However, that label is an informational one. It indicates that the manufacturer attests they meet about five security requirements. If the manufacturer is willing to make these attestations, they will receive a label (the program is voluntary, so no manufacturer has to participate at all).

Admittedly, relying solely on attestations isn’t ideal, since the manufacturer could always lie. However, if the manufacturer suffers a breach and it becomes apparent that they had lied in one or more of the attestations, their label can be revoked. Since having attested falsely about their cybersecurity would undoubtedly reflect very negatively on the manufacturer, it’s reasonable to assume that most attestations will be truthful. If a manufacturer has terrible security, they won’t bother to apply for the label at all, but I strongly doubt they’ll lie to get it.

Note that this the meeting last week wasn’t the first time the White House has talked about an IoT device security labeling program. In Executive Order 14028 of May 2021, the WH ordered NIST to “…identify IoT cybersecurity criteria for a consumer labeling program…” within 270 days. Almost exactly on the 270^th day, NIST published thisdocument. It had two parts.

The first part was a set of “criteria” (i.e. requirements) for the device labeling program. In my reviewof a predecessor to that document in December (which remained mostly unchanged in the final version), I said I thought they were exactly what was required: NIST called them “outcomes-based” criteria and I would call them “risk-based”.

But the two terms are synonymous: The manufacturer is required to achieve a general outcome, but the exact steps by which the manufacturer does that are up to the manufacturer, and need to consider the level of risk posed by the device. That is, the steps a manufacturer needs to take for a security camera at a bank are more rigorous than what is required for a baby monitor, although the outcome might be considered the same. The post I just referenced discusses this idea in more depth.

The manufacturer also needs to consider the environment in which the device will be used. A nuclear power plant is inherently much riskier than somebody’s back porch, even though the same security camera might be used in both locations. Obviously, the security measures taken will be much more severe at the nuclear plant, even though the device being protected is exactly the same as the one on the back porch.

The second part of NIST’s February document discussed how the device labeling program would work. It listed three possible types of labels:

1.      Informational, which isn’t based on an assessment, but simply provides information on security measures taken for the device (e.g., the German label referenced above)

2.      Tiered, in which there are multiple levels at which the product can be evaluated. The level attained by the product is shown on the label.

3.      Binary, which is essentially a “pass/fail” designation. NIST indicated in the document that they preferred this label type. Energy Star provides a binary label.

In my December post I noted that, while I don’t have a problem with a binary label per se, I do have a problem with trying to combine a binary label with outcomes-based criteria. The reason is simple: Outcomes-based criteria require the organization to tailor how they comply with the criteria according to the degree of risk posed by the device (also, by the environment). It will be up to the assessor to determine whether the manufacturer’s compliance actions were appropriate for the risk posed by the device and its location.

On the other hand, a binary label doesn’t allow for any considerations of risk or anything else, in determining whether the device deserves the label or not. The assessor needs to be able to make an up-or-down decision, period. That’s only possible with prescriptive requirements, not outcomes-based ones (the December post provides examples of both types of requirements, to illustrate this point).

How did I recommend that NIST resolve this contradiction? I didn’t. I said NIST had to choose outcomes-based criteria or a binary label, but they couldn’t have both. Since I strongly favor outcomes-based (risk-based) requirements in general (and I’ve probably written 50 posts about this idea, with reference to different aspects of the NERC CIP standards), I didn’t want NIST to sacrifice those. So I suggested that NIST use an informational label, not a binary one.

And what did NIST do (drumroll, please)?...Last month, they published an IoT security framework called NIST.IR 8425, which is very close to the set of criteria in the February document (the categories of criteria are exactly the same, while the criteria in each category differ slightly). It’s safe to say that NIST decided to stick with outcomes-based criteria, which is good. But what happened to the labeling program in the February document? Did NIST stick with the binary label?

When I published my most recent post(on IoT security certification) in LinkedIn, Dale Peterson asked in a comment why I hadn’t mentioned “the very recent USG announcement that they will define IoT security labeling.” I hadn’t seen the story on the White House conference yet, so I thought he was referring to the EO. In my reply, I pointed to the February document and the fact that NIST seemed to have been taken out of the device labeling business, since the criteria from the February document had been made into their own NISTIR, with no mention of labeling.

But I now realize Dale was referring to the meeting last week. My response should have been:

What was announced by the White House last week seems to be the end of the idea that NIST can run a certification program (my last post was about the ioXt device certification program, which is of course not a government effort). NIST is quite good at writing nonprescriptive security frameworks, but they showed in December and February that they’re not good at all in developing up-or-down certification programs.

However, that doesn’t mean the White House will be good at developing a device certification program, either. They may very well make the same mistakes that NIST did – although they may boldly break new ground and make completely new mistakes!

My advice to the White House (not that I’ve been asked, of course) is the same as what I gave to NIST (and that was in response to a request for comments in December, although it wasn’t an official comment period): If you try to marry a label that’s really a certification (as NIST wanted to do with their “binary” label) with a set of non-prescriptive guidelines (like NIST.IR 8425), you’re trying to do the impossible. You might as well try to square the circle or invent a perpetual motion machine.  

Make the label an informational one, and make sure the label provides real information (perhaps attestations, like the German label). Then let the consumers make up their own minds about whether the product is secure, based on what they read on the label.

Some consumers won’t look at the label at all, of course. That’s too bad, but there’s no point in even pretending that cybersecurity is anything but a risk management exercise (it’s definitely not a matter of scientific calculations, like Energy Star, although it’s amazing the number of people who think there’s some sort of formula that will make you secure). A risk-based decision has to be made by the individual, period.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

One of my recent posts pointed out that many (and perhaps most) connected devices (aka IoT devices) are far from secure. Furthermore, it pointed out that because these devices are inherently “black boxes”, meant to be managed by the manufacturer and the manufacturer alone, it’s difficult for a user organization to learn what controls might or might not be in place in the device. While a user can easily scan “standalone” software (i.e. software the user installs on standard hardware and O/S platforms) and learn a lot about vulnerabilities in it, it’s very hard to do that for a sealed device, especially if the user doesn’t even know what software is installed inside it.

This is why, even more so than with standalone software, it’s important for the user organization to learn what’s inside the box they’re buying – both the good stuff (the capabilities included in the device, including what is shown in a software bill of materials, or SBOM) and the bad stuff (the software vulnerabilities that were also “included” in the device); it’s also important for the user organization to learn about the manufacturer’s security practices.

The more the user can learn about the technical security controls and manufacturer practices before they purchase the device, the easier it will be to decide a) whether or not to purchase a particular device in the first place, and b) after they’ve decided they will purchase it, what controls – both technical and procedural – they will need to put in place, to compensate for security weaknesses that may be found in the software or firmware installed in the product (or in the manufacturer’s practices).

Questions about technical security controls include:

1.      Does the device allow a universal password? Of course, if it does, that’s not a good thing.

2.      Are security updates applied automatically when possible?

3.      Is standard cryptography utilized?

Questions about manufacturer security practices include:

1.      Does the manufacturer disable unused services?

2.      Has the manufacturer published an end-of-life notification policy?

3.      Is there a “bug bounty” program for security researchers who find vulnerabilities in the product?

The best way for a user organization to learn what technical controls and manufacturer practices are in place for a connected device is if the device has received a previous certification from an independent third party certifier. Because this requires testing the device, not just asking the manufacturer about their practices, it should be performed by a testing laboratory. Americans aren’t too familiar with cybersecurity testing laboratories, but the labs are more common in Europe, where they’ve been testing connected devices for years.

For more than one year, I’ve had the pleasure of working with Red Alert Labs (RAL), a leading European cyber testing lab based in Paris. They provide assessments and certifications of connected devices based on multiple standards, including IEC 62443, Common Criteria and ETSI 303 645. RAL has also worked with the European Union and the European Network and Information Security Agency (ENISA) to develop assessment criteria for cloud security.

Recently, RAL became one of only 8 organizations worldwide that have been selected to provide certifications based on the standards developed by the ioXT Alliance, the global standard for IoT security. Backed by the biggest names in technology and device manufacturing, including Google, Amazon, T-Mobile, Comcast and more, the ioXt Alliance is the only industry-led, global IoT device security and certification program in the world. Devices with the ioXt SmartCert give consumers and retailers greater confidence in a highly connected world.

Besides assessing and certifying connected devices and their manufacturers, Red Alert Labs helps organizations that usethese devices to assess the cybersecurity risks they face from devices they are considering for procurement. After procurement, RAL helps those organizations assess and mitigate vulnerabilities identified in software and firmware installed in the devices they use.

RAL will soon be helping user organizations assess the IoT devices they use, based on the recently released NIST.IR 8425, “Profile of the IoT Core Baseline”. This document was explicitly developed in response to Executive Order (EO) 14028 of May 2021, and draws on a number of earlier IoT security documents – including the NIST.IR 8259 series. It also draws on the many comments received by NIST last year in response to their request to the public after the EO charged NIST with putting in place a program for IoT device cybersecurity.

NIST.IR 8425 is clearly meant to be NIST’s main IoT cybersecurity guidance document for the foreseeable future, as well as perhaps their answer to the IoT Cybersecurity Improvement Act of 2020. I believe that US-based public and private sector organizations will standardize on this document as their primary means of assessing IoT device risk for the foreseeable future.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Since the beginning of the NTIA Software Transparency Initiative, there was always an understanding (sometimes explicitly articulated in meetings) that, even though everyone knows that computing is moving more and more to the cloud, it was too early to start discussing what an SBOM for a cloud service should contain; first we needed to figure out what an SBOM for on premises (“on-prem”) software should contain.

When the NTIA Initiative ended last year and “moved” to CISA, there was general agreement that we needed to start thinking about SBOMs for cloud services. Thus, one of the five current CISA SBOM/VEX working groups is dedicated to that topic. I’ll admit that I wasn’t too excited about this topic, mainly because I thought we might end up having long discussions which in the end wouldn’t lead anywhere.

However, I’ve been surprised by how productive the group has been and how quickly we’ve come to a rough consensus (although it’s subject to change at any meeting) on what should be in an SBOM for cloud services (aka SaaSBOM, where SaaS means “software as a service”), even though we’re very far from writing a spec at this time. Here are what I believe to be the general points of our rough consensus (most of these were originally brought up by Steve Springett, the co-leader of the CycloneDX BOM format – which has supported SaaSBOMs for at least a couple of years):

First, a BOM – whatever flavor it is – needs to address the risks that are inherent to the environment in which it’s used, and specifically risks due to third parties. A software BOM is designed to address the risk that a vulnerability in a third party component of a software product running on an organization’s premises will be exploited by an attacker who will use this “foothold” to attack software and/or data found on the network, causing harm to the organization.

However, software running in the cloud doesn’t normally have a direct impact on the organization’s network (the exception brought up by one of the members of the SBOM Cloud working group is the case where an app might drop tainted JavaScript code on a user’s desktop). Therefore, it’s not very important that the user organization receive an SBOM listing the components of a cloud app that they use. This is for the better, since cloud apps are constantly changing. Because an SBOM needs to be re-issued whenever there’s been any change in the code of an app, the user would need to receive and analyze many SBOMs every day for the same cloud app; obviously, they couldn’t do their day job if they had to do this.

But it is important that the provider of the cloud service itself examines up-to-date SBOMs for whatever apps they offer in the cloud. The cloud service provider (CSP) should provide their customers an attestation describing various cybersecurity measures they’re taking, including generating and analyzing SBOMs for their apps.

Thus, a BOM distributed to users of a SaaS product doesn’t need to list actual software components, because they pose minimal risk to the user organization. What does pose a risk to the organization? Steve Springett pointed out early on that the main risk to the user organization from cloud software that they utilize runs through the third party services called out by the SaaS product.

What is the nature of this risk? Steve additionally pointed out that the risk to the organization, due to use of a SaaS product in the cloud, relates to data: both a) the risk that the organization’s data will be misused by a third party service called by the SaaS product, and b) the risk that false or misleading data will be provided by a third party service to the SaaS product, with some sort of deleterious consequences for the organization.

An example of a) is if a hospital would provide personal health information (PHI) to a third party app, but the third party would make it available to criminals who would sell the information. An example of b) is if a SaaS SCADA system for a natural gas pipeline system would be “informed” by a third party app that one of their lines was seriously underpressured, meaning they need to greatly increase pressure in the line; this might cause an explosion (this is somewhat like what happened in the San Bruno explosion in 2010, which caused eight deaths – although that wasn’t due to an online SCADA system, of course).

Given this, what should the SaaSBOM look like? It will obviously need to list every service that’s called out by the app itself (Steve Springett provided the workgroup this “services BOM” for Dependency-Track, the open source product he developed in 2010, which now is used over 250 million times a month to identify vulnerabilities for components listed in an SBOM. Dependency-Track isn’t SaaS, but this does illustrate what a BOM of services looks like. Of course, I’m sure Steve regularly makes SBOMs for D-T. In fact, both the software BOM and the services BOM are needed for any on-prem product that makes calls to external services).

However, just having a list of the services called out by a SaaS app doesn’t by itself help identify risks, any more than having an SBOM listing components in a software product by itself identifies risks. Just as vulnerabilities applicable to software components listed in an SBOM need to be identified in the NVD or another vulnerability database like OSS Index, so risks applicable to services need to be identified.

But how does a user identify services risks? Are there “service vulnerability databases”? I had never heard of one, but in yesterday’s workgroup meeting, someone posted exactly that in the chat. I’m sure there’s a lot that could be put into this database, especially regarding risks that aren’t exactly vulnerabilities. For example, I would argue that, if a service is operated out of North Korea, that in itself poses a risk, which could be included in this database as well.

My guess is that whatever the workgroup finally decides should be the format for a SaaSBOM (and that may not be the name for it, either. I’d be happy with “Cloud SBOM”) will include more than just a list of services. For example, Steve suggested that there should be a data flow diagram, listing what types of data (HTTP, PHI, etc.) flow into and out of the SaaS app. Obviously, that would also be important for an end user that wants to know what risks they face.

The bottom line is I’m now optimistic that there might be a SaaSBOM format available next year. What’s important now is to populate the Cloud Vulnerability Database (and similar databases, if any appear), so that, as SaaSBOMs appear, cloud users will be able to learn about the risks inherent in the cloud services that they use.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

In case you haven’t noticed, I’ve written a lot about software supply chain security in the past couple of years. I’ve focused on this topic because that’s where some of the biggest cybersecurity attacks have appeared. SolarWinds, Kaseya, Log4j, ransomware in general, etc. were (and are, since none of them have ended yet) all software supply chain attacks, or at least incidents.

However, even though software currently poses the biggest supply chain security threat (and certainly one of the two or three biggest cybersecurity threats in general), I think there’s ultimately a much bigger threat which we’re just beginning to deal with now: the supply chain of intelligent devices. This includes devices that have been sold for specialized purposes – electronic relays in electric power substations, infusion pumps and heart monitors in hospitals, controllers of all sorts in factories, networking equipment like firewalls and switches, etc. But it also includes a lot of devices used for much more pedestrian purposes, including security cameras and even baby monitors.

I first realized how much greater the risks are with devices through a presentation (and follow up discussions) by Tom Pace of NetRise to the SBOM Forum this spring. I wrote about the presentation itself in this post; I followed it up with 3-4 further posts. The most relevant of those further posts to our current discussion is this one.

NetRise specializes in firmware security, so obviously devices are their main concern. Tom’s initial presentation discussed a network device that, believe it or not, is used in critical infrastructure environments, including the US military. In his original presentation, he noted that he had identified, through analyzing the components of firmware in the device, at least 1,237 unpatched vulnerabilities; however, he was sure there were more. In the second post linked above, I noted he had later told me he was now sure there were at least 40,000 unpatched software and firmware vulnerabilities in that one device, which (using a general guesstimate of 5% of component vulnerabilities being exploitable) probably includes 2,000 exploitable vulnerabilities.

Of course, I doubt you’ve ever heard of a software product with 2,000 unpatched exploitable vulnerabilities; I certainly haven’t. How could there be so many in one device? For one thing, the device contains lots of software and firmware products. But I’m sure we’re talking about 50-100 products, not 1,000. This would still mean there are at least 20 exploitable vulnerabilities per product. A software product with 20 unpatched vulnerabilities would meet a lot of resistance. How is it that a device can have 2,000 vulnerabilities and still be on US military purchase lists?

The clue to this miracle is revealed in both of the posts: The manufacturer of this device has never reported a single vulnerability for it. In fact, the manufacturer has never reported a vulnerability for any of the 50-odd devices it sells. If this were a software product, anyone could scan it and discover at least some of these vulnerabilities. The software supplier would never get away with not reporting any vulnerabilities at all.

But the situation is quite different for a device. It’s a sealed box. The end user can’t even find out what software is in the box, let alone scan it for vulnerabilities. It would certainly help to have a software bill of materials (SBOM) for the device, but – other than a small number of medical device makers, who are facing a mandatory requirement to provide SBOMs to their customers later this year – I have never heard of a device manufacturer that regularly distributes SBOMs to users.

In fact, I know of one key device manufacturer, that sells into an important critical infrastructure industry and has a huge market share, that seems to be actively campaigning against SBOMs. They’ve pulled out the old chestnuts about “roadmap for the attacker”, etc. I always used to have a high opinion of this company for their cybersecurity posture (which is otherwise exemplary), but having seen – and experienced in online and phone conversations – their opposition to SBOMs, I honestly have to wonder: What are they hiding? I sure hope it’s not 40,000 unpatched vulnerabilities in one product.

A few months ago, I and Isaac Dangana of Red Alert Labs published an article in the Journal of Innovation. It noted that, when it comes to vulnerability management for intelligent devices, the device manufacturers hold all the cards. The user can’t usually even learn what software is installed in the device. The manufacturer normally distributes all updates, for all software and firmware in the device, in a single “lump”. When the update gets applied (often “over the air”, without the user being involved at all), the user has no control over what products are patched or not patched; whatever is in the lump gets applied, or else nothing gets applied.

In our recommendations at the end of the article, Isaac and I said that device manufacturers should distribute an SBOM whenever they update the software and firmware in their device. But that wasn’t the most important recommendation, which was that the manufacturer needs to register the device as a single product and report any vulnerabilities in any of the software in the device to that single product.

The way that would work today is:

1.      The manufacturer obtains a CPE name for the device from CVE.org (aka MITRE Corp.). This would normally be done when reporting a CVE (vulnerability) as applicable to the device, but I believe it can be done even before a CVE is reported.

2.      Whenever the manufacturer learns that a CVE is applicable to (i.e., exploitable in) any of the software or firmware products included in the device, they will report this to CVE.org. If the supplier of the individual software product hasn’t reported themselves that their software product has this vulnerability (meaning that an NVD search on the product name and version would uncover the CVE), then the device manufacturer might do that themselves.

3.      However, it’s not too important whether or not the manufacturer reports that the individual software product is vulnerable. What is important is that they report the vulnerability as applicable to the device – i.e. the device’s CPE name. That way, a user of the device will simply have to enter the CPE name in the NVD, in order to see all reported CVEs for any of the software or firmware in the device.

4.      As an aside, I’d like to note that, when the proposal regarding product naming in the NVD is implemented - which the SBOM Forum released recently – there will be no need to use CPE names with the NVD (although that will still be an option). Instead, hardware devices will be identifiable using their GTIN number, which is a set of international standards for naming intelligent devices. One of the GTIN standards is UPC, meaning that, in order to search for a device in the NVD, the user will just need to scan the UPC barcode on their device. Any applicable CVEs should appear immediately (or the user could enter the UPC code manually, which wouldn’t be too hard. The point is that there should be no need to search for the product in the hit-or-miss fashion that NVD users know and love today[i]).

In other words, in order for intelligent device users to realistically be able to manage vulnerabilities in devices, the device needs to be listed in the NVD and any CVEs that are applicable to any software or firmware product included in the device need to be listed as applicable to the device itself. I’ll admit that this won’t be an easy policy to sell to the manufacturers. And while, as a former student of Milton Friedman, I don’t like to see government regulation used in cases where it isn’t absolutely necessary, in this case it may be. Which do you want with your IoT device: 40,000 unpatched vulnerabilities or government regulations?

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

My most recent postquoted Tony Turner saying (in response to a previous postof mine) that SBOMs aren’t being used much by non-developers (that is, organizations whose primary business isn’t developing software), because non-developers are already overwhelmed with vulnerabilities that they’ve learned about without having SBOMs. If they can’t deal with those vulnerabilities, what good does it do them to learn about a potentially much larger set of vulnerabilities that are due to components of the products they use – which they’ll only learn about if they have SBOMs?

I concluded that post by writing:

What’s the solution to this problem? Obviously, it’s certainly not a great situation if software users ignore an entire class of vulnerabilities – those that are due to third-party components included in software products they use – simply because they’re too busy handling another (perhaps smaller) class of vulnerabilities, namely those that are due to the “first-party” code written by the suppliers of the software.

The best situation would be if users could take a holistic view of all vulnerabilities, including both those due to first-party code (which they learn about through vulnerability notifications from the supplier or from looking up the product in the NVD) and those due to third-party components (which they learn about from receiving SBOMs and finding component vulnerabilities in the NVD). They would then allocate their limited time and resources to identifying the most important vulnerabilities of either type and mitigating those. They wouldn’t feel they have to ignore half of the vulnerabilities in their software, because they don’t even have time to learn about them.

So, the question becomes how software users can prioritize their time addressing vulnerabilities in order that they mitigate the maximum amount of software risk possible. The answer needs to take into account the fact that they don’t have unlimited time or unlimited funds available for that effort.

I know some people will answer this by saying, “That’s simple. The user should just find the CVSS score for each exploitable vulnerability and rank the vulnerabilities based on their scores. Then they should start by mitigating the vulnerabilities with the highest scores. When they have exhausted their vulnerability management budget for the year, they should stop and do something else.”

But I also know other people who will say that CVSS score is an almost meaningless number, so it should never be used to prioritize vulnerabilities to mitigate. If so, what’s the solution? Is it another score like EPSS? Is it no score at all, but a different way of ranking software vulnerability risk?

I honestly don’t know. I’d love to hear your ideas.

To summarize what I said, it’s a shame if an organization decides to shut out a potential source of vulnerability information (software bills of materials, or SBOMs) simply because they already know about too many vulnerabilities. This is something like an army deciding they don’t need to conduct intelligence activities anymore, since they already face too many threats for them to deal with easily.

What both the army and the organization need to do is learn about all the vulnerabilities and threats that they face, then figure out how to prioritize their response to them. In responding, they need to use their limited resources in a way that will result in the maximum possible amount of risk being mitigated, given their available resources for responding to the vulnerabilities and threats. It’s likely that the majority of organizations, or at least the majority of organizations who try to prioritize their responses to vulnerabilities in a rational way, will base that response in whole or in part on CVSS score.

Partly in response to my request for ideas, Walter Haydock, who has written a lot about how to prioritize vulnerabilities for patching in his excellent blog, put up this poston LinkedIn. The post began with an assertion frequently made by Walter: “CVSS is dead”. I agree with him that CVSS isn’t the be-all-and-end-all vulnerability measure it was originally purported to be.

Why do I agree with him about CVSS? Let’s think about what we’re trying to measure here: It’s the degree of risk posed to an organization by a particular software vulnerability, usually designated with a CVE number. Risk is composed of likelihood and impact. CVSS is calculatedusing four “exploitability metrics”: Attack vector, Attack complexity, Privileges required, and User interaction required, along with three “impact metrics”: Confidentiality impact, Integrity impact and Availability impact.

However, all of these seven metrics, to varying degrees, will differ depending on the cyber asset that is attacked. For example, if the asset is an essential part of the energy management system (EMS) in the control center of an electric utility that runs the power grid for a major city, any of the three impacts of its being compromised will be huge. On the other hand, if the cyber asset is used to store recipes for use in the organization’s kitchen, the impact of being compromised will be much less.

There are similar considerations for the exploitability metrics: The EMS asset will be protected with a formidable array of measures (many due to the NERC CIP requirements for High impact systems that run the power grid) that are unlikely to be in place for the recipe server. So, even though in organizations in general, one vulnerability might be more exploitable than another and thus would have a higher exploitability metric, in most organizations, the fact that assets of different importance will have different levels of protection means that these differences will probably swamp the general differences in exploitability that form the basis for the metrics.

In other words, I agree that the CVSS score isn’t very helpful when it comes to prioritizing vulnerability remediation work in an individual organization; and since the EPSS score, which focuses solely on exploitability, is unlikely to be any more helpful (for reasons already provided above), this means I now doubt there is any single metric – or combination of metrics – that can provide a single organization with definitive guidance on which vulnerabilities to prioritize in their remediation efforts.

At this point, Dale Peterson, founder of the S4 conference and without doubt the dean of industrial control system security, jumped in to point out that vulnerability management efforts should be prioritized based on “Impact to the cyber asset...” That is, there’s no good way to prioritize vulnerability remediation efforts based on any consideration other than the impact on each cyber asset if the vulnerability were successfully exploited. I asked,

Dale, do you have an idea of how an organization could assess vulnerabilities across the whole organization (or maybe separately in IT and OT, since they have different purposes) and prioritize what they need to address on an organization-wide basis?...

It has to be a risk-based approach, but CVSS and EPSS obviously aren't adequate in themselves. Like a lot of people, I've always assumed this is a solvable problem. However, I'm no longer so sure of that. This would change how I've been thinking about software risk management.

I was asking Dale how a vulnerability management approach based on assets would work. He replied:

Yes for the ICS world. ICS-Patch decision tree approach, see https://dale-peterson.com/wp-content/uploads/2020/10/ICS-Patch-0_1.pdf

I highly recommend you read the post Dale linked. It provides a detailed methodology for deciding when a patch should be applied to a cyber asset in an OT environment. Dale also points out that his methodology derives inspiration from the Software Engineering Institute/CERT papercalled “Prioritizing Vulnerability Response: A Stakeholder-Specific Vulnerability Categorization”, which deals with IT assets (he points out how differences between OT and IT account for the differences between the two methodologies).

Of course, neither Dale’s methodology for OT nor CERT’s methodology for IT is easy to implement; both cry out for automation in any but the simplest environments. But both can clearly be automated, since they’re based on definite, measurable criteria.

Of course, it would be nice if there were a single measure that would solve everybody’s vulnerability prioritization problems. It would also be nice if every child in the world had their own pony. However, neither is a likely outcome in the world we live in. It’s time to focus on a real-world solution to this problem. 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

My last postdiscussed the fact that software users clearly aren’t demanding SBOMs from their suppliers now; the post described my idea for why that’s the case. While I still agree with that post, I have to admit that Tony Turner stated in four sentences (which Tony placed as a comment on my post in LinkedIn) a much more compelling reason than mine – and he did it using about 100 fewer sentences than I used!

I had stated that the reason users aren’t requesting SBOMs is that they don’t see much chance of success when they get one, because of a combination of three low probabilities:

1.      The probability that they’ll be able to find any particular component in the NVD when they look for vulnerabilities. This is currently very low, although many software suppliers and service providers have developed various ad hoc tools to improve their odds in this regard;

2.      The probability that any given component vulnerability is exploitable in the product itself (probably around 5%); and

3.      The probability that there will be VEX documents quickly available from the supplier of the software described in an SBOM, to let the user know of all unexploitable component vulnerabilities. This is currently 0%, since no suppliers are producing VEXes now. It presumably won’t always be zero, but the VEX model depends on suppliers regularly producing lots of VEXes, perhaps hundreds for every SBOM released; I now have strong doubts that will ever happen. My proposal for real-time VEX should substantially increase this probability by making VEX notifications almost trivially easy for suppliers, and not requiring blizzards of VEX documents to be distributed to hundreds of thousands of software-using organizations every day.

Of course, probabilities need to be combined by multiplying them, and multiplying three low percentages will result in an even lower total percentage; the combination of these percentages will be not far from zero. In other words, I’m saying that software users aren’t demanding SBOMs because they think that having them won’t improve their security posture enough to justify the time and effort that will be required for them to analyze them.

While Tony didn’t disagree with what I said, he doesn’t think that’s the main reason. And to be honest, I think he’s right. Tony said:

The biggest issue is they are still struggling to handle all the scanner vulnerabilities and SBOMs just pour salt into the wound. Just operationalizing all of this is a huge undertaking. So yes, it’s lack of SBOM tools, but it’s not just that. It’s a core vulnerability management pain point.

What’s the solution to this problem? Obviously, it’s certainly not a great situation if software users ignore an entire class of vulnerabilities – those that are due to third-party components included in software products they use – simply because they’re too busy handling another (perhaps smaller) class of vulnerabilities, namely those that are due to the “first-party” code written by the suppliers of the software.

The best situation would be if users could take a holistic view of all vulnerabilities, including both those due to first-party code (which they learn about through vulnerability notifications from the supplier or from looking up the product in the NVD) and those due to third-party components (which they learn about from receiving SBOMs and finding component vulnerabilities in the NVD). They would then allocate their limited time and resources to identifying the most important vulnerabilities of either type and mitigating those. They wouldn’t feel they have to ignore half of the vulnerabilities in their software, because they don’t even have time to learn about them.

So, the question becomes how software users can prioritize their time addressing vulnerabilities in order that they mitigate the maximum amount of software risk possible. The answer needs to take into account the fact that they don’t have unlimited time or unlimited funds available for that effort.

I know some people will answer this by saying, “That’s simple. The user should just find the CVSS score for each exploitable vulnerability and rank the vulnerabilities based on their scores. Then they should start by mitigating the vulnerabilities with the highest scores. When they have exhausted their vulnerability management budget for the year, they should stop and do something else.”

But I also know other people who will say that CVSS score is an almost meaningless number, so it should never be used to prioritize vulnerabilities to mitigate. If so, what’s the solution? Is it another score like EPSS? Is it no score at all, but a different way of ranking software vulnerability risk?

I honestly don’t know. I’d love to hear your ideas.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

I realized about a year ago that the reason why SBOMs aren’t being used by non-developers for vulnerability management purposes – even though the developers themselves are using them very heavily for those purposes, as shown in this post – isn’t that most developers don’t want to supply them. Rather, it’s because the users aren’t asking for them.

I have until recently blamed that reticence on the fact that there are few low-cost or open source tools to “process” SBOMS (i.e. ingest them and look up component vulnerabilities in the NVD or another vulnerability database) available for end users; moreover, there’s only one tool (at any price point), Dependency-Track, that ingests SBOMs and VEX documents, and these tools have to do both. Of course,  the whole point of creating a machine-readable SBOM (as opposed to just a listing of components in a PDF) is that…strange but true!...it needs to be read by a machine. I can attest that SBOMs in JSON or XML can be read easily by someone with a little knowledge of English, but there’s still no reason to create such an SBOM if a machine isn’t going to read it.

And when you get to SBOMs with hundreds or thousands of components (the average software product now has about 150 third-party components, but some have thousands or even tens of thousands), there’s no question that you need to process it using a tool. That is, you probably have better things to do with your time than print out an SBOM and look up 150 components in the National Vulnerability Database (NVD) using “manual” searches.

But just having a tool that can “process” SBOMs isn’t enough. The end user should look not just for component vulnerabilities, but for the small subset of those that are exploitable in the product itself; those are the ones that pose a problem. VEX documents are required to separate the exploitable wheat from the unexploitable chaff. Since at least 90% of component vulnerabilities aren’t exploitable, an SBOM consumption tool that doesn’t also read VEXes is probably less than useful.

When Dependency-Track recently started supporting VEXes as well as SBOMs (it’s supported SBOMs since 2012, long before almost anybody else got interested in them), I happily announced that fact. And since D-T is open source and therefore free, I wondered if this development might at least get a few more end users interested in getting SBOMs from their suppliers. The result was (drumroll, please)…

Thunderous silence. Of course, when Dependency-Track (which was developed in 2012 by Steve Springett, co-leader of the CycloneDX SBOM format project) is downloaded, the person who did that doesn’t fill out a questionnaire saying what they’re going to use it for. However, Steve developed D-T for use by developers, and Steve says that what he hears from users is that they’re overwhelmingly developers.

To be honest, just changing some wording and some screens would make D-T a perfectly usable consumer product, but Steve has never had requests for that. He knows of a small number of non-developers who are using D-T to track vulnerabilities in products they use, but he doesn’t believe that number is growing.

This isn’t to say that, if someone wants to make the investment to create a consumer-oriented product that “processes” both SBOMs and VEXes, and they make it available at low cost, there won’t be takers for it. Until recently, I thought that was all it would take to make people start using SBOMs.

However, just asking myself a question recently – and answering it with some simple math – made it clear to me that the lack of a low-cost, end-user-friendly tool isn’t at all what’s holding back use of SBOMs.

The (rather long) question I asked myself was, “Let’s assume there’s a low-cost, end-user-friendly SBOM/VEX processing tool, and that a user feeds an SBOM into it. The tool looks up every component listed in the SBOM in the NVD. For every successful lookup, the user receives a list of any open vulnerabilities in the component. They then use any available VEX from the supplier, for the product and version that’s the subject of the SBOM, to determine which of these vulnerabilities are not exploitable. They remove them from the list of open vulnerabilities, ultimately yielding a list of only exploitable vulnerabilities.

“Given this, what is the probability that the user will a) learn about every exploitable component vulnerability in the product, but also b) know exactly which of those vulnerabilities are exploitable or not exploitable?” We will call these the two target probabilities.

There are three probabilities that need to be considered, to answer this question:

1.      The probability that the user will be able to find each component in the NVD when they look it up, and learn of any vulnerabilities (CVEs) that apply to the component;

2.      The probability that a vulnerability identified for a component will be exploitable; and

3.      The probability that there will be VEX documents available from the supplier, which will identify all non-exploitable component vulnerabilities. This will leave just the exploitable ones.

The first probability is based on the “naming problem”. This is the problem that the informal group I lead, the SBOM Forum, addressed in a recent paper; I discussed it in this post. Briefly, the aspect of the naming problem that we addressed can be stated as, “When a user looks up the components found in a newly-created SBOM in the NVD, they will find only a small fraction of them, unless they try other manual processes.” These processes include AI, fuzzy logic, references to other databases, etc. The point is that a purely automated process will only find a small fraction of the vulnerabilities in the NVD (six reasons for this are described in the paper); the person interpreting the SBOM will need to use these other processes, and mix in a good measure of dumb luck, in order to learn about any more than a small fraction of the vulnerabilities that apply to the product.

And what is that small fraction? Our paper quotes the Director of Product Security of a very large US-based software developer, who said that they can find fewer than 20% of the components that appear in one of their SBOMs (they’re creating SBOMs regularly for all their products, but they’re not distributing them now). However, in a recent discussion in one of our meetings, this person admitted that the real number, in their experience, is less than 10%, and maybe even less than 5%. Another participant in the meeting – who works for a very large US-based producer of both software and intelligent devices – said their experience shows that the number is under 5%.

What this means is that, even if an end user has a tool that ingests SBOMs and looks up all the components in the NVD, they will probably learn about vulnerabilities in fewer than 5% of the components. Of course, the SBOM Forum’s proposal aims to raise that percentage far higher – and it will ultimately do that. But I estimate that it will be one to two years before our proposal is fully implemented (I’m optimistic that it will be implemented, based on the reception it’s received so far).

Even after it’s implemented, it will mostly affect new vulnerability reports, not existing ones. So, it could be a number of years before we’ll be able to state that the percentage of successful NVD component searches is over say 70% (it will never be 100%, of course). The bottom line is that, for at least the next couple of years, the first probability above is no more than 10% and more likely below 5%.

I’ve already discussed the second probability, the percentage of component vulnerabilities that are exploitable, in previous blog posts. I normally use the figure of 90%, but most people I talk to believe it’s over 95%. In other words, once a user has looked up whatever components they can find in the NVD and listed all the vulnerabilities identified for those components, at least nine of every ten will be non-exploitable. This might seem like good news, until you consider that you don’t know which ones aren’t exploitable – so you have to treat them all as exploitable for the time being.

The third percentage, the probability that the supplier will produce enough VEX documents to identify every non-exploitable component vulnerability, is zero currently, since no supplier is producing VEXes, period (and BTW, don’t point to a vulnerability report and tell me that’s a VEX, even if it’s in the CSAF format). Will it become non-zero later?

I have recently come to believe there’s a good likelihood that VEX documents will never be produced, for various reasons I’ve discussed already and will in the future. This is why I’m now advocating for the idea of real-time VEX, although I’m also still working with the VEX committee as it makes slow progress toward producing a VEX playbook - which is needed no matter whether the VEX document or real-time VEX emerges as the winner. VEX information is required for SBOMs to be widely used, no matter how it ultimately gets distributed. And SBOMs will be widely used.

So, what’s my estimate for the third probability? It’s zero now, but in a couple years it will be nonzero. However, I want to remind you that the only way a user will be sure that the list of component vulnerabilities they have for a product only contains exploitable vulnerabilities is if they can be sure that the supplier has notified them of every non-exploitable component vulnerability in the product and version in question – that is, they have informed them which nine of every ten component vulnerabilities aren’t exploitable.

Now that we know these three probabilities, what about the two target probabilities we want to derive from them? These are the probabilities that the user will a) learn about every component vulnerability in the product, and b) learn exactly which of those vulnerabilities are exploitable or not exploitable.

Finding a) is easy, since it’s just the first probability listed above: the probability that the user will find every component in the NVD. We’ve already said that’s (conservatively) 10%. But finding b) is harder, since this requires receiving a VEX for each of the non-exploitable vulnerabilities in the product. So, if there are a total of ten vulnerabilities but only one is exploitable, the user will need to receive a VEX for each of the other nine vulnerabilities, saying it’s not exploitable (remember, the user initially needs to assume that every component vulnerability they identify is exploitable, until they receive a VEX stating otherwise).

Let’s say the user just receives seven VEX documents, each stating that one of the ten component vulnerabilities isn’t exploitable. Does the fact that only seven were received mean the remaining three vulnerabilities are exploitable? The supplier could always later end the uncertainty by sending another VEX, which states that the three are exploitable. However, would that be a good idea, especially if the supplier doesn’t have a patch for the vulnerabilities yet? It definitely wouldn’t be a good idea. Only if the supplier sends out a VEX stating that the vulnerabilities have all been patched, which includes the URL(s) of the patch(es), can the user be sure the remaining three vulnerabilities are exploitable in that product and version.[i]

Unfortunately, there’s no assurance that the supplier will always make sure to close the loop and inform their customers of the status of every vulnerability that the NVD lists for a component of a particular version of their product. Thus, it's very possible that a customer will waste time (theirs and the supplier’s) by contacting the supplier regarding non-exploitable vulnerabilities. This is one of the reasons why I now think real-time VEX is a better option than VEX documents.

So what is the value of target probability b)? As I said earlier, currently it’s zero, since no supplier is currently producing VEXes. However, I do think that the idea of a VEX API will ultimately be adopted. This will allow the customer to be certain whether or not each component vulnerability is exploitable. But, since we’re concerned with the next year or so at the moment, I can confidently say that b) is a very low percentage, and currently it’s zero percent.

Finally, we’re able to answer the big question: Why are so few users interested in receiving SBOMs for use in software vulnerability management – even though there’s one open source tool available that could help them in this job? There are at least two reasons:

1.      They have heard that it’s difficult to look up components from an SBOM in the NVD. An estimated maximum ten percent success rate doesn’t inspire them to try doing that, either.

2.      They know that, at least in the near term, they won’t be able to learn which component vulnerabilities are exploitable in the full product, without calling up the supplier’s help desk and running down their entire list of component vulnerabilities with the unfortunate person who answers their call, forcing them to find out whether each of those vulnerabilities is exploitable or not. Of course, this is one of the main reasons why suppliers aren’t keen on distributing SBOMs to their customers now; they suspect their help desks will be overwhelmed with calls about false positives.

You may notice a seeming contradiction in this post: I first talked about the fact that SOBMs have been hugely successful with developers, but now I’m implying they’re lying dead at the starting gate, as far as end users are concerned. Clearly, the developers know something that the users don’t. Can they share this?

As far as the first reason, the naming problem, goes, the suppliers definitely have knowledge. The naming problem has been known since the beginning of the NVD, but SBOMs have turned it from an annoyance into a crisis, because of the huge number of components that now need to be looked up. Developers and consultants have developed a whole array of tricks – including AI and fuzzy logic routines, databases that shed light on different aspects of software names, etc. – to get around the problem.

Since these tricks are very ad hoc, they can’t be shared as part of a tool. However, developers could apply them on behalf of their customers, so that the SBOMs they send out have valid identifiers (“CPE names”) that allow customers to look up components in the NVD (meaning the customers would probably still end up looking up the vulnerabilities in the NVD manually). But, more likely, the suppliers will just outsource the whole “SBOM processing” activity to third party service providers

The service providers will provide an “SBOM service” for end users. This service will take in SBOMs, as well as either VEX documents or real-time VEX information (whichever is available) and produce for their customers a list of exploitable component vulnerabilities in a product/version that is used by the customer. The customers won’t need even to look at an SBOM or VEX document, nor should they need to look up anything in the NVD on their own. They’ll receive what’s really the holy grail of SBOMs: continuously-updated lists of exploitable vulnerabilities in the products/versions in use by the user organization.

This is why I now think that, at least for the next few years, almost all “SBOM processing” for end users will be performed by third parties, not by standalone tools operated by the users themselves. My guess is the users will be quite happy with that arrangement.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

If you’re not part of the electric power industry, you may not have heard of NERC, the North American Electric Reliability Corporation. Or you may have heard of it, but all you know is the acronym.

NERC is a non-profit corporation owned by electric utilities and other power market participants. It develops and audits compliance with a number of standards for reliability of the North American Bulk Electric System (BES), including the famous (infamous?) NERC CIP standards for cybersecurity of the grid. But the regulatory muscle behind all NERC standards is provided by the Federal Energy Regulatory Commission (FERC), which is part of the Department of Energy. This unusual arrangement was mandated by the Electric Power Act of 2005.

NERC also plays a number of important educational roles, one of which is to provide guidelines on grid security, including supply chain cybersecurity. The supply chain guidelines are developed by the NERC Supply Chain Working Group, which was formed in 2018, as the industry was starting to prepare for implementation of NERC CIP-013-1, the supply chain cybersecurity risk management standard. It’s important to keep in mind that the guidelines are not aimed at compliance with CIP-013, but simply at good supply chain security practices.

I’ve been a member of the SCWG since shortly after its inception. In 2019, the group was asked to develop guidelines on (I believe) five topics related to cybersecurity. I volunteered to lead the group that developed a paper on supply chain cyber risk management lifecycle. After the first couple meetings of the group, I realized there should be a separate paper on vendor risk management lifecycle, so another group was formed to lead that – and nobody else volunteered to lead it, so I led that as well. The papers were published in 2019 and 2020 (in the end, there were 8 or 9 papers).

This year, the SCWG was asked by NERC to update the guidelines papers. Since nobody else stepped forward (hard to do in a WebEx, of course) I ended up leading both of those. Just as I’d found the experience of developing the 2019 papers to be quite intellectually stimulating, I found the same thing this year.

The two papers were published for comment by NERC on Monday; they’ll be up for 45 days, and then we’ll draft the final versions. The supply chain paper is hereand the vendor paper is here. I think only NERC members can comment, but I’d love to hear any comments or questions. I’ll make sure we take them into account when we revise the documents, so your comments will be given as much attention as a NERC member’s.

I do want to point out that there is some material that was added to the papers before publication, that’s common to them as well as the other papers (which will be put up starting soon but continuing into 2023 – since a few were written in 2020, not 2019). That’s the material before the Executive Summary and after the table of participants. That material has nothing to do with the topics, but you may find it interesting anyway (I’ve always found the topic of the power grid in general to be quite interesting, and have written about it – i.e., not from a cybersecurity point of view – a number of times, like this one and this one).

And if you want to buff up on the NERC CIP standards, you’ll find a few posts of interest by searching on CIP in the search bar. There are probably about 4-500, so I wouldn’t plan on reading every one of them in one sitting.

If you’re with a NERC entity or an IT or OT supplier to the power industry, I’d love to have a discussion with you about CIP-013 and supply chain cybersecurity. Please drop me an email.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Last week, I put up an upbeat postthat pointed out that SBOMs are already a great success among software suppliers, since they are using them heavily (literally hundreds of millions of times a month, and billions of times a month if you want to count “use” in a different way) to learn about vulnerabilities in products they’re building. I noted that this was happening in the absence of any significant use by consumers, and I further noted that this wasn’t because at least some suppliers aren’t ready to supply them to their customers, but because their customers aren’t asking for them.

I didn’t go into detail on why consumers aren’t asking for SBOMs now, although I’ve dropped various hints in my posts – lack of free or low-cost tools to utilize SBOMs for vulnerability management, lack of support for SBOMs from vendors of vulnerability and configuration management software, lack of available VEX documents to notify consumers of non-exploitable component vulnerabilities, etc. But I did attribute the fact that so many software suppliers had in recent years “gotten religion” on the need to take responsibility for preventing or fixing vulnerabilities due to components in their software, to the fact that they knew SBOMs were coming. Moreover, they knew their customers were soon going to know about vulnerabilities found in components in their software and continually ask when they were going to fix them.

However, I realized today that there’s a kind of contradiction between those two ideas: That SBOMs aren’t being significantly utilized by consumers today and that suppliers are anticipating they will be heavily utilized by consumers in the near future. I resolve this by saying that suppliers are wrong if they anticipate SBOMs will be heavily used in say the next 2-3 years, but I anticipate they will be after that.

I have a list of about 8 serious impediments (and there are many less-serious ones as well) to widespread SBOM use. I’m happy to say that one of them is on the road to being resolved, and the informal group responsible for that is now deciding what’s the next impediment we’ll take on. Anyone who’s already familiar with SBOMs and would like to join that effort should drop me an email.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

When I first started attending the meetings of the NTIA Software Component Transparency Initiative (which of course was the name for the “SBOM initiative”) in the summer of 2020, I immediately started hearing about what was called the “naming problem”. I wrote a post about the problem in November 2020 and called it the “one problem that towers over the others”, as far as SBOMs are concerned.

In other words, at that time, I considered the naming problem to be the most serious roadblock in the path to widespread (or even narrowspread, to be honest) distribution and use of SBOMs by the general community. However, the whole group seemed to have decided that this was too hard a problem to address at the time, given that they were still trying to a) figure out how SBOMs were going to work in general, and b) interest the developer community in producing them. This was still the accepted view when the NTIA Initiative ended at the end of 2021, even though a lot of progress had been made in both of those areas.

I have good news and bad news. The good news is that I no longer consider the naming problem to be the biggest impediment to the spread of SBOMs. But the bad news is that I don’t say this because I think the problem has diminished. Rather, I’ve come to realize there are at least five or six just-as-serious impediments to SBOMs (and a host of not-so-serious impediments, which are impediments nevertheless); in other words, the naming problem now has company, instead of “towering” over the others. There’s progress for you!

Early this year, it seemed to me and a few others that it was time for the private sector to take the lead in addressing head-on the most serious problems for SBOMs. The idea was that we would meet weekly and discuss one problem until we at least understood how it might be solved. Then, if there was something we could do to put in motion the solution to the problem, we would do that. After that…on to the next problem.

Why were we doing this? I’ll admit that we all had selfish reasons. Our livelihoods are all tied in one way or another to the success of SBOMs. We will all benefit if SBOMs start to be widely used outside of the development community (where they’re already widely used for product risk management purposes).

The question came up, should the participants worry about helping potential competitors? After all, when one of these problems is solved, it will benefit their competitors, as well as themselves. My answer to this question (which I didn’t hear often, to be sure) is that ultimately the SBOM “market” will include every organization on the planet. It’s difficult to imagine any organization of any size in the world today that doesn’t use software (even if it’s just the software in one person’s smartphone); it will soon be literally impossible to imagine that.

Given this, it follows that today’s market for SBOM services and tools is an almost infinitesimal fraction of what it can be. Doesn’t it make sense to focus on expanding the market so it’s at least a significant fraction of what it ultimately will be, rather than worrying about how you and your competitors are going to divide up the very small market that’s there today?

Of course, it made sense to my friends to take the latter course. The only question was, which of the five or six serious problems would we start with? I wasn’t sure which one, but I wassure of one thing: it wouldn’t be the naming problem. I honestly thought that this problem would take a year to even understand, another year to develop at least a partial solution for it, and finally about eight long, grinding years – full of political battles – to see the near-solution implemented.

My friends and I created an informal group called the SBOM Forum and started meeting for an hour every Friday. In one of our earliest meetings, Tom Pace of NetRise described an eye-opening experience he had with the NVD, in which a device, that appeared in the NVD to have never had a vulnerability, in fact had at least 1,237; Tom found these with a simple scan of just two of the firmware products installed in the device. Moreover, Tom later came to realize that the same product has probably 40,000 unpatched vulnerabilities, even though an NVD search will find nary a one of them.

The problem Tom came across was just one of the many manifestations of the naming problem. The naming problem in general refers to the fact that software products have many names in the many locations in which they’re found. However, the branch of the naming problem that causes the most consternation for the software community is centered on the National Vulnerability Database (NVD), by far the most heavily used vulnerability database in the world. The NVD uses a very problematic naming system called CPE (common platform enumeration), which is the source of most aspects of the problem.

To my horror, after Tom’s discussions with our group, we stumbled into deciding to take on the naming problem first. But I was in for a big surprise: Instead of taking two years to identify the problem and document a solution to at least 70-80% of the problem, we took a little more than four months. We published the document this week.

The six most serious aspects of the problem with CPE are described on pages 5-7 of our document. I used to think that all those aspects were going to require their own specialized measures, which was why I was sure that any “solution” we proposed would be a godawful mess. However, I didn’t realize that our group was lucky enough to have the perfect Hero to slay the foul Naming Beast: Steve Springett, co-leader of the OWASP CycloneDX project and without much doubt the most creative person in the SBOM community.

As we started talking about the problem, Steve quickly realized that a big component (no pun intended) of any solution would have to be purl, a unique type of identifier that is already in wide use - although it isn’t used so far in the NVD (Steve is a maintainer of the purl project, which has helped our group a lot already). Purl has so far been used mostly with open source software (and used very extensively. Just one tool, Dependency-Track, is currently used about 27 billion times every month to look up a vulnerability for an open source product in Sonatype’s OSS Index vulnerability database, which is based on purl).

Steve realized that purl’s unique extensibility would allow us to “incorporate” two other identifiers into purl: SWID for proprietary software, and Software Heritage ID for legacy open source software (since our document explains the various naming schemes that our solution will utilize, I won’t explain them here).

The best thing about purl: for reasons described in the document (be sure to read the discussion of “intrinsic” vs. “extrinsic” identifiers), integrating purl into the NVD will not require any database linking; every open source product that’s in a package manager already has a purl. When searching for vulnerabilities in that version of the product, the user can easily create the correct purl – 100% of the time.

While purl, SWID and SWHID cover a good portion of the software universe, they don’t cover hardware. However, the NVD includes hardware, and devices also have CPE names (along with the problems belonging thereto). I would have been fine with declaring victory with software and moving on, but Steve insisted we could do hardware as well. He and Tony Turner of Fortress Information Security recognized that the two most widely used hardware identifiers, GTIN and GMN (both part of the “GS1” family of hardware standards), could be integrated with the NVD as well.

However, in order to allow those two identifiers to be used with the NVD, various databases need to be linked with the NVD. This is the only part of our proposal that requires substantial work. I think it would amount to just a few person-months, but since there will be various groups involved, it might take longer than that.

But the work will be worth it. One of the GTIN identifiers is the UPC code. Once our proposal is fully implemented, you should be able to find out about vulnerabilities in a device by entering its UPC code – either by scanning the code itself or entering the numbers by hand – in the NVD. Whoever thought using the NVD could be this much fun?

I’m hoping that, two years from today, our “proposal” will be fully implemented; Steve has already started discussing this with the group that will be most involved in the implementation: MITRE Corporation. CISA at some point (perhaps not until early next year) will hold a meeting to discuss the question of software naming in the NVD.

What can you – and your organization - do to advance this proposal? I’d say the first thing is to read the document. I’ll admit it’s dense. I’m going to put up two or three blog posts trying to shed light on some of the harder parts. And OWASP will have a webinar soon to discuss the proposal.

Second, you can read the blog post by Steve Springett. At the bottom, he has a “How to Help” section. To what he has there, I would add that we would like to put together a set of short testimonials (1-3 sentences) on why your organization supports this proposal, including how this will enable the organization to do whatever you do better and more efficiently.

For example, the document includes a statement from a very large software maker (identified in the document) to the effect that they regularly create SBMs for all of their products, but when they look at just the output of the tooling they use to create the SBOM, only one in five components can be identified in the NVD. Of course, this means they can’t “operationalize” their SBOM production process. While they can often find a lot of those components, they – like many other companies and consultants – need to employ AI, fuzzy logic, references to other databases, prayer…whatever will help. Being able to fully automate SBOM production will greatly increase the efficiency of their software development process.

But fully automating the SBOM production (or interpretation) process is out of the question for any organization, until the naming problem is solved.[i] Hopefully, that day is now closer than it’s ever been. 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

You may know that I’ve said on various occasions (although not often in this blog) that it’s disappointing that, while SBOMs are being heavily used today for vulnerability management purposes, that usage is almost entirely by software suppliers (i.e. developers), not by end users. While it’s good to see SBOMs have been so successful with suppliers, just about everything you read about them in the general press – and the whole purpose of Executive Order 14028 – is how end users need them to secure their networks. Well, it seems the end users didn’t get that memo, since – I’ll be honest – almost no non-developer organizations are regularly receiving or using SBOMs today.

So, the title of this post may make you wonder if I’ve gone crazy: Despite what I just said, I’m certain that SBOMs have already had a huge amount of success. How could that be?

To understand this, it is important to understand the fundamental purpose of SBOMs, in the supply chain cybersecurity use case. That purpose is not to empower end users to secure the software they operate by their own efforts; this is simply not possible. Truly securing software requires being able to patch vulnerabilities. Other than in exceptional cases, end users (i.e. non-developers) can’t develop their own patches for software they purchased or downloaded; having SBOMs doesn’t change this situation.

Then why do end users need SBOMs (or at least the data from them)? What good does it do to provide users with lots of information on what makes up the software they operate, if they can’t use that information to make the software more secure? And, even if the end user could secure their software if they used SBOMs, I just said they’re not using them. How can I say SBOMs are successful?

I stand by this statement. The fundamental purpose of distributing SBOMs to end users of software was never to empower them to fix software’s problems on their own. Instead, it was to empower them to put pressure on suppliers to fix the problems. Being able to identify components listed in an SBOM, and to learn about vulnerabilities applicable to those components, empowers the user not just to make general exhortations to a supplier to follow good development practices, but to ask them what they’re doing with respect to specific component vulnerabilities. That gives users a lot more power. You might call this, “security by being a pain in the ___”.

In a 2017 study, Veracode found, “A mere 52 percent of companies reported they provide security fixes to components when new security vulnerabilities are discovered. Despite an average of 71 vulnerabilities per application introduced through the use of third-party components, only 23 percent reported testing for vulnerabilities in components at every release.” These numbers are pathetic, as I’ll bet you understand.

This study alone might have helped bring about the FDA’s statement in 2017 (I believe) that they intended to start requiring SBOMs for medical devices, which itself led to the NTIA’s Software Component Transparency Initiative starting operation in 2018. That Initiative (which ended last year) was sparked by some large hospital organizations (“healthcare delivery organizations”, to be accurate) and some large medical device makers, who decided they wanted to get ahead of the curve and start producing and using SBOMs, long before the FDA’s order would come out (it will come out sometime this year).

To my knowledge, the Veracode study hasn’t been repeated, but there is quite tangible evidence that a large percentage of software suppliers have gotten religion on the matter of software security. This evidence is something I’ve already written about: the statement this spring by Steve Springett, founder of the OWASP Dependency-Track open source project (and co-leader of the CycloneDX project), that Dependency-Track was used 200 million times in a single month to query the OSS Index vulnerability database for vulnerabilities found in components in an SBOM (although that statement from April needs to be updated. Steve says the number of queries from Dependency-Track is now up 35% to 270 million per month. And that’s total queries. If you look at the number of individual components searched on, it’s at least 100 times that, since the average software product has more than 100 components. So, this represents about 27 billion individual searches every month – all based on SBOMs and all just from one product that utilizes SBOMs. There are a number of other tools – generally called software composition analysis or SCA – that do the same thing as Dependency-Track, although these usually aren’t free).

Of course, this usage is almost certainly due to suppliers, not end users. However, it makes my point very well: a large percentage of software suppliers are now actively tracking vulnerabilities in components in their products (both those under development and those already in use in the field), so they can be proactive about securing them. They’re not waiting for the customers to start banging on their doors, demanding to know when CVE-2022-12345, with a CVSS score of 10.0, will be patched in their product; they do this because they know that component X in the product is vulnerable to that CVE. Before the customers come calling, the supplier wants to know about important component vulnerabilities, and either patch them or replace the components.

However, you might wonder why I attribute this huge increase in supplier interest in vulnerability management to SBOMs, since I’ve just said they’re not being distributed to, or used by, end users to any degree (and I’m sure that’s because the end users aren’t asking for them. A lot of their suppliers already have SBOMs and would love to share them when they see demand for them). After all, there could be lots of other explanations for this. Maybe the suppliers were visited by a pillar of fire and heard a voice telling them to start managing vulnerabilities. Or maybe their spouses all started hammering them about vulnerability management. How am I sure one of these isn’t the explanation?

As with many economic effects in the real world, I think this has to do with expectations. The suppliers know that users will start requiring SBOMs in the near future (in fact, US federal agencies have been required to ask for them since August 10). Therefore, whatever may have been a supplier’s security posture previously, nottaking a proactive approach to identifying component vulnerabilities in their software and mitigating them (either by patching or by replacing the component) is no longer an option. They need to do something; hence the 27 billion Dependency-Track vulnerability searches. SBOMs are already working!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Tom’s note: Real-time VEX is an ideaI came up with in the spring, inspired by a conversation with – who else? – Steve Springett. Since then, I’ve become more and more convinced that this is something we need to do. I say this because a) the VEXidea, while sound, is taking a lot longer to implement than I’d anticipated, and b) I’m certain that SBOMs themselves will never be widely distributed or used (outside of the developer community) unless there’s some easy and quick way for software suppliers to inform their users about non-exploitable component vulnerabilities.[i]

“Real-time VEX” is an API that will allow a device on a software user’s network to query a server that resides on a software supplier’s network (or perhaps is operated by a third party on the supplier’s behalf). The API allows the user to determine the exploitability status of a particular vulnerability (CVE) in a version of the supplier’s product, which is operated by the user.

The use case for real-time VEX is simple: When a user receives an SBOM for a software product they utilize from the product’s supplier, they (or a tool or third-party service) will search the NVD (or another vulnerability database) for vulnerabilities that apply to any of the components listed in that SBOM. Whenever they discover an open vulnerability (CVE) for one of those components, the user - or more accurately a tool operated by the user, or a third-party service provider acting on the user’s behalf - will assign the default status of “exploitable” to the vulnerability for that version of the product[ii].

Shortly thereafter, the user’s system will login to the supplier’s server and provide three pieces of information:

1. CVE number[iii]
2. Product name[iv]
3. Version string

The supplier’s server will send one of three responses:

1. This CVE is not exploitable in the product and version named.
2. The CVE is exploitable in the product and version named. Please immediately apply the patch located at…(URL).[v]
3. The exploitability status of this CVE, in the product and version named, is unknown.

The user or their system (or a third party service provider) will take one of three steps:

1. If the vulnerability is not exploitable, the CVE will be marked as such in the list of component vulnerabilities applicable to the version of the product. Because it is possible the supplier might later change the status of the CVE to exploitable, the user system will regularly query the supplier’s server regarding the CVE’s status.
2. If the vulnerability is exploitable, the CVE’s status in the vulnerability list will be left unchanged, since the default should be “exploitable” anyway. The organization’s patch management team will be notified to download and apply the patch to the product. After applying the patch, the CVE’s status in the vulnerability list will be changed to not exploitable.[vi]
3. If the status of the vulnerability is unknown, the supplier’s server will be queried regularly (probably multiple times a day) to learn if the status has changed.

What is the advantage of real-time VEX vs. relying on the supplier to prepare and transmit a VEX document that indicates the exploitability status of a CVE? It is time. When a new vulnerability appears in the NVD for a component included in a supplier’s product and they subsequently determine that the vulnerability is not exploitable, the product security team will not normally be inclined to drop everything they are doing to prepare and distribute a VEX for their customers.

Even for just a few products, this might happen ten or so times in one day. It would be very disruptive to getting work done. For that reason, it is likely that most suppliers will batch their VEX notifications and release them for example every three days. However, changing the status of the CVE in the real-time VEX server will literally require almost no time at all.

Why is time so important? As previously discussed, a component vulnerability should always be considered exploitable by the user organization when it is first identified. Since a user has no way of knowing beforehand which CVEs may later be designated “not affected” in a VEX document, they will need to investigate every one of them, especially those of higher severity. If a VEX saying the CVE is unexploitable arrives for example 2-3 days later, the user may already have wasted a lot of time on this; and if this same situation happens multiple times a week, this will be a significant problem for the organization. This is why the user organization needs to learn about non-exploitable vulnerabilities as soon as possible after the supplier learns about them.

With real-time VEX, the user will be able to learn about non-exploitable component vulnerabilities almost as soon as the supplier does – and just as importantly, the supplier will need to invest almost no time at all in the notification. Upon learning that a vulnerability is not exploitable, the supplier only needs to update the status of the CVE in the real-time VEX server (and this will likely be handled by an automatic update from the system used to determine exploitability of the CVE); the user will learn about the change the next time they query the server. The supplier will not have to do any more than this.

It is important to keep in mind that the supplier itself will benefit as much from real-time VEX as the customer. Especially for more severe component vulnerabilities, it is likely that users will not be content to sit back and wait for a supplier to issue a VEX to inform them that a vulnerability is not exploitable; they are likely to call or email the help desk regularly about this. The sooner the supplier can update the status information in the real-time VEX server, the less time their help desk will waste giving the same answer to caller after caller.

The savings of time and frustration that could be provided by real-time VEX, on both the user’s and the supplier’s side, are likely to be substantial. Remember that over 90% of component vulnerabilities are not exploitable in the full product. This means it is possible that over 90% of the time and effort required by both the supplier and end user, in responding to component vulnerabilities revealed by SBOMs, will be wasted – unless the supplier quickly determines[vii] the exploitability status of a CVE and communicates this to their customers.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

I’ve written a number of posts about VEX, but there are only one or two posts that I’ve linked when I want to explain the term itself, so that someone who hasn’t heard of VEX can get a quick education. However, I recently went back and read those posts, and realized that they don’t actually give a basic explanation of VEX. Instead, they assume basic VEX knowledge to start out. Now, I’m writing the post I should have written a while ago. I’ll use this as my “VEX education” link in the future.

VEX is an acronym for “Vulnerability Exploitability eXchange”. VEX is a simple yet machine-readable pair of formats for reporting the status of vulnerabilities found in software products and intelligent devices. While it can be used for any vulnerability reporting, it was developed specifically for a use case specific to suppliers that provide SBOMs for their products.

VEX was developed to enable a supplier to notify its customers, in a machine-readable format, that a vulnerability they may believe is exploitable in their product is not in fact exploitable. The primary reason that customers would ask this question in the first place is if they have received an SBOM and looked up its components in a vulnerability database like the NVD, then learned about one or more serious vulnerabilities that apply to those components. They might waste a lot of time (and worry) trying to find and patch vulnerabilities that aren’t exploitable, and therefore pose no danger to them.

Another reason for a VEX is that a specific vulnerability or set of vulnerabilities, like the log4j or Ripple20 vulnerabilities, has been widely discussed in the news – and users want to know whether or not it applies to a software product they use.

The consensus in the software industry is that at least 90% of component vulnerabilities are not exploitable in the product itself, usually because of how the component was incorporated into the product. For example, a library may have four modules, but the supplier might have only installed the three modules that they needed. A serious vulnerability could be identified in the library, but it happens to be found in the fourth module, which the supplier didn’t install. Therefore, the vulnerability is not exploitable in the product itself. A VEX document will state that fact.

It is likely that the great majority of VEX documents will be based on the use case of identifying non-exploitable component vulnerabilities. However, the simplicity of the VEX idea enables a number of other powerful use cases for vulnerability notifications. These use cases go beyond the case for which VEX was originally developed.

The need for VEX arose from discussions in the Software Component Transparency Initiative (hereinafter, “the Initiative”). This was a “multistakeholder process” convened by the National Technology and Information Administration (NTIA) division of the US Department of Commerce, beginning in 2018 and ending in late 2021. The purpose of the Initiative was to get together private and public “stakeholders” in the software marketplace to discuss securing software and intelligent devices from risks arising from use of third-party and open source components; of course, the main tool for accomplishing this purpose is SBOMs.

Early on in those discussions, it became apparent that there was a lot of concern about the problem that was discussed above: the fact that a large percentage of vulnerabilities identified in components included in a software product are not in fact exploitable in the product itself.

At first, this might seem to be a good thing. After all, if 90% of vulnerabilities in the components of a product aren’t exploitable in the product itself, this means that the “attack surface” of the product is reduced by 90%, compared to what it would be if all component vulnerabilities were exploitable in the product itself.

This is true, but the problem is that the end user cannot normally determine, through their own actions, which component vulnerabilities are exploitable in the product, and which are not exploitable. Not having this information, the user needs to assume that every component vulnerability they have identified is exploitable. Thus, they may waste a lot of their time trying to determine whether these vulnerabilities are actually present in the product. Moreover, they will waste the supplier’s time by continually contacting their help desk to inquire when each of these vulnerabilities will be patched.

The NTIA group discussing this problem decided that the best way to prevent this from happening is for the supplier to notify the customer that a particular vulnerability is not in fact exploitable in the product, even though it is listed in the NVD (or another vulnerability database) as applicable to a component of the product.

Note that this is a type of vulnerability notification that has not been used very much, simply because it has not normally been needed. A “normal” vulnerability notification says, for example, that CVE-2022-12345 is exploitable in product ABC version X.Y.Z. We call this a “positive” vulnerability notification, since it states that a particular vulnerability is exploitable in the product.

Of course, if this example is a positive vulnerability notification, then the notification mentioned above – i.e. that a particular vulnerability found in a component of a product is not in fact exploitable in the product – should be called a negative vulnerability notification.

The supplier is normally the party that is best able to provide vulnerability exploitability information, since they have access to the source code and know in intimate detail how the product was built. Fortunately, it is also in the supplier’s interest to get this information to their customers, since this will eliminate a lot of “false positive” calls to their help desk.  However, because it is likely that many suppliers, at least initially, will not provide either SBOMs or VEX documents, third parties might need to step in to provide VEXes on their own.

There are currently two VEX formats. The first is based on the CycloneDX SBOM format (although it can refer to SPDX SBOMs as well). The second consists of a subset of the CSAFvulnerability reporting format. Unfortunately, as of now (September 2022), the CSAF VEX format isn’t completely specified (a partial specification, which will be enough for someone who already understands CSAF, is here). A “playbook” for creating and using VEX documents is currently being prepared by the CISA VEX working group.

Because of the delay in developing proper documentation for producing and using VEX documents, as well as the fact that end users will want to know about the exploitability status of component vulnerabilities as soon as they learn about them – not several days or weeks later, depending on how long it takes the supplier to produce a VEX document – I am now advocating for the idea of “real-time VEX”.

This is essentially an API where a user will be able to learn instantaneously what the supplier currently believes to be the exploitability status of a particular component vulnerability (of course, that status may be “We’re still investigating it”). My guess is that, if this is implemented, the need for a majority of VEX documents will disappear. VEXes will still be needed for more complex use cases like patching, but the urgent use cases can all be handled by the API.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.