In my most recent post, I pointed out that it’s literally counterproductive to buy only software products for which no vulnerabilities have ever been reported – especially if the product doesn’t have a CPE name, so it’s not listed in the NVD at all. This is because not reporting any vulnerabilities is almost certainly not a sign that the product is perfect, but that the supplier doesn’t think it’s worthwhile to even look for vulnerabilities. After all, if you’re sure your software is perfect, why even bother to look? I provided a vivid example of a device that’s not even listed in the NVD, but probably has about 40,000 vulnerabilities (2,000 of which are probably exploitable) in its software and firmware, according to Tom Pace of NetRise.

However, I also pointed out that Steve Springett had mentioned that, in his day job, they not only don’t get fooled by products with zero vulnerabilities, but they literally favor products for which a lot of vulnerabilities have been reported. The reasoning is simple: There are a ___-load (excuse me, “a large quantity”) of vulnerabilities in just about any software or firmware product. Since most vulnerabilities are reported by the supplier of the vulnerable product, they can exercise discretion in the ones they report and the ones they don’t report.

In the post, I did point to one type of vulnerability that probably doesn’t need to be reported: vulnerabilities that the supplier discovers and immediately fixes. Since these vulnerabilities never even have the chance to be found in a shipped product from the supplier, I don’t see why they would need to be reported. The whole reason for reporting is to inform software-using organizations of vulnerabilities in software they use, not as a form of public flagellation of the supplier for not being perfect in all circumstances, even if there’s no possibility of harm to anyone.

On the other hand, a supplier definitely should report any vulnerability that is found in a product they have already shipped. And Steve is saying it’s inevitable there will be a lot of vulnerabilities that meet this condition (of course, the biggest reason for this is that new vulnerabilities are always being discovered, so a few lines of code that seemed totally innocuous when the software was written suddenly are identified as a vulnerability). So, if a product has a CPE name, but only a few vulnerabilities have ever been reported for it, this is usually an indication that the supplier isn’t looking very hard for vulnerabilities (the fact that there’s a CPE name means they’ve at least looked once, but that’s not much better than never having looked in the first place).

This brings me to an excellent blog post I just read, by Walter Haydock. Walter is quite an interesting fellow. I connected with him on LinkedIn last year, and a couple of months ago, he mentioned me in a list of five people that his LinkedIn followers should connect to. I was of course pleased with that, but I wasn’t expecting what came next: within two days, about 300 people requested to be connected with me, all due to Walter (whereas I’d probably never had more than two or three requests in one day, and most days none at all). I found this almost scary; I asked Walter what would happen if he told all of his followers to drink poisoned Kool-Aid™. He didn’t know, but also added that he didn’t plan to try that. Good thing! With great power comes great responsibility.

You get the point: This guy’s blog is really good. I highly recommend you subscribe (although I don’t think my recommendation will bring Walter 300 new subscribers in two days!). His latest post makes a point something like my point in my previous post: that the number of vulnerabilities itself isn’t a very meaningful metric (he writes both for suppliers and end users, so you get both perspectives on this question).

What struck me most in Walter’s post was this paragraph:

Some compliance regimes, such as the federal government’s FedRAMP standard…(establish) clear perverse incentives with respect to how organizations look at the total number of findings. For example, after initial authorization, organizations that identify an increase of 20% or more from the baseline, or 10 unique vulnerabilities, whichever is greater, are subject to a “Detailed Finding Review.” By punishing companies who identify more than an arbitrary number of vulnerabilities - regardless of their severity - during a given time frame, the government disincentivizes its vendors from looking too closely for these vulnerabilities in the first place.

This was a real eye-opener to me: The fact that, not only would a supplier not get credit for reporting a healthy number of vulnerabilities, but they would get penalized for doing so. We clearly need to re-think how we treat software vulnerabilities.

Any opinions expressed in this blog post are strictly mine, and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

I’m pleased to report that Cybellum, an Israel-based company whose mission is “…to enable manufacturers and their suppliers to develop and maintain products that aren’t just safe, but are also secure”, has just posted a podcast I taped with them a few months ago. I’m quite pleased with the results, which are due as much to their good questions as to anything I said.

The podcast – which is only 26 minutes long – ended up being very focused on what I believe is the biggest issue preventing widespread adoption of SBOMs by end user organizations (i.e. organizations whose primary mission isn’t developing software for other organizations. Developers are already heavily using SBOMs for their internal purposes, thank you very much): the current lack of tools and scalable third-party services to utilize SBOMs and VEX documents for software risk management purposes, as well as the relative dearth of written guidance on how non-developers can use SBOMs.

I’ll warn you that, if you’d rather hear happy stories about how SBOMs and VEXes are already being widely used and how it will just take a little more of what we’re currently doing to reach component security nirvana, perhaps you need to look for another podcast. There is a huge amount of work to be done, and even what I know to be in the pipeline at the current moment is totally inadequate to address what’s needed.

However, I’m also quite optimistic that what’s needed will come, and in the not-distant future. I’m optimistic because – as a student of Milton Friedman during his heyday at the University of Chicago – I believe that free markets will ultimately both a) allow consumer demand for SBOMs to rapidly grow from its current close-to-nonexistent level, and b) “monetize” the so-far-nascent (at best) sub-markets for tools and services for widespread distribution and utilization of SBOMs for vulnerability risk management purposes.

I also want to point out that the previous podcast in this same series featured Steve Springett, the creator of Dependency-Track (which I mention in the podcast, and have referred to multiple timesin these posts) and leader of the OWASP CycloneDX project (Steve is also the brains behind the current effort to solve the naming problem, one of the primary inhibitors of widespread production and utilization of SBOMs).

I recommend you also listen to that podcast, since Steve provides some very good insights into the current and future state of SBOMs and VEXes. In my opinion, Steve is theintellectual leader of the SBOM and VEX communities. The rest of us are just trying to visit the places where he’s already arrived, made a big difference, then moved on to his next challenge.

Any opinions expressed in this blog post are strictly mine, and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

In this recent post (and a couple subsequent ones), I discussed an interesting presentation by Tom Pace of NetRise. In it, he described how he’d found 1,237 vulnerabilities by identifying components in the firmware in a device that’s used in many ICS environments.

Having vulnerabilities – and sometimes a lot of them – is certainly not unique to this product. Why did I write three posts about this discovery? The problem is that, were you to look for vulnerabilities for that product in the National Vulnerability Database (NVD), you would get the message “There are 0 matching records.” Sounds good, doesn’t it? No vulnerabilities at all! That conclusion is true, except it’s off by 1,237 vulnerabilities.

Now, suppose you were comparing three products in advance of a procurement decision, and you look up vulnerabilities for all three. For two of them, you find a handful of vulnerabilities, but for the third product, you get the above message. Would you tell the first two vendors, “Thanks, but no thanks”, and write up your PO for the third vendor? I’m sure many organizations would.

Of course, this would be a mistake, since Tom found 1,237 vulnerabilities listed for components in just the firmware of this device. But it turns out the true story is worse: Tom said last week that after further analysis, he identified 2,200 vulnerabilities in components included in the device’s firmware. Even worse, after analyzing all of the software installed in the device, he estimates there are around 40,000component vulnerabilities in the whole device. That’s a lot.

To be sure, these aren’t all exploitablevulnerabilities. As I’ve mentioned often, probably 90-95% of vulnerabilities identified in software and firmware components within a device aren’t exploitable in the device itself, often because of how the component was implemented. Let’s say that the percentage for this device is 95%, meaning only 5% of the identified vulnerabilities are exploitable. That’s still 2,000 exploitable vulnerabilities in a single device

This is obviously not good, but what’s even worse is the fact that not a single one of these 2,000 exploitable vulnerabilities appears in the NVD. They don’t appear because the supplier never registered the product. If the supplier (or somebody else) doesn’t register a product, it doesn’t get a CPE name. And if it doesn’t have a CPE name, nobody can report vulnerabilities for the product to the NVD. The product will appear to be perfect – no vulnerabilities, either current or reported in the past.

In fact, this supplier is so good that they’ve never registered any of their 50 or so products – meaning everything this supplier makes has a perfect record! Moreover, they don’t even mention the word “security” or “vulnerability” on their website. Why should they, given that all of their products are perfect?

Of course, that company’s products aren’t perfect – just the opposite. And the company is hardly unique. There are lots of other companies that haven’t registered some or all of their products on the NVD, meaning that anyone searching for vulnerabilities in those products will also get the message, “There are 0 matching records.”

What does all this means? I hope you’re sitting down, since I need to give you some bad news: There are no perfect products or perfect suppliers (there’s also no Santa Claus or Easter Bunny. Might as well give you all the bad news at once). You should never interpret the fact that you can’t identify vulnerabilities for a software product (or intelligent device) in the NVD (or any other vulnerability database, of course) to mean that the product doesn’t have vulnerabilities.

But there’s more to it than that. Not only should you stay away from “perfect” products, but you should also deliberately favor products that show a lot of vulnerabilities in the NVD. Why is this? Remember, vulnerabilities are almost always reported by the suppliers themselves. Would you rather buy a product from a supplier that has only reported a few vulnerabilities in the past year or two, or from one that has reported a lot of them? If a supplier has only reported a few vulnerabilities, this doesn’t mean they’re good; on the contrary, it probably means they’re clueless in cybersecurity matters. It means the supplier isn’t looking very hard – or not at all – for vulnerabilities, so they’re not finding many.

Steve Springett, who I’ve written about a number of times and who is tasked with helping 2,000 coders produce secure software in his day job, said last week that his company deliberately favors products for which there are a lot of reported vulnerabilities. They consider this a sign that the supplier is diligently seeking out vulnerabilities, not waiting for their product to be hacked.

So not only should you avoid “perfect” products, but you should actually seek out suppliers that have reported a lot of vulnerabilities. Of course, you also want to make sure that such a supplier hasn’t left serious vulnerabilities unpatched. My guess is, if a supplier has found and reported a lot of exploitable vulnerabilities, they’ve also done a good job of patching them. In fact, the supplier should report vulnerabilities, even if they’re patched.[i] That’s the only way the rest of the world will learn about the real impact of particular vulnerabilities.

Any opinions expressed in this blog post are strictly mine, and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

I’ve wanted to have this talk with you for a long time, but I’ll admit I’ve been putting it off. However, there are some important things you need to know about VEX, and I’d rather you hear them from me than from someone else (such as one of your friends):

1.      There are now two VEX formats. They’re very different, although they have the same purpose. The first is based on the new CSAF version 2 vulnerability reporting standard from OASIS (it uses a subset of the capabilities in that standard, called a profile). The other is based on the OWASP CycloneDX SBOM format, although any SBOMs referred to in the VEX can be in either CycloneDX or SPDX format.

2.      There is no direct connection between SBOMS and VEX documents. Even though the VEX format was developed to address the issue of non-exploitable vulnerabilities identified using SBOMs, the VEX itself doesn’t need to refer to an SBOM or to a component. The question of the exploitability of a vulnerability, that is found in a particular component, only applies to the product as a whole: Is the vulnerability exploitable in the product or not?

3.      There’s no way to do a direct translation between the two formats; by that I mean not only that there aren’t translation tools, but that I don’t even see a good way to do a direct translation, other than to extract the data from one format and create a new VEX in the other.

4.      Currently, there aren’t any automated tools to create VEXes[i] in either format. Thisrecently approved document from the CISA VEX committee provides links to JSON code for various use cases, in both VEX formats.

5.      However, unlike with SBOMs, where the format used should be a concern for end users, end users really shouldn’t care in what format a VEX is provided, as long as the tool or third-party service they are using to track vulnerabilities in software components can interpret it. This is because a VEX document, even though it’s comprehensible to a human who wants to invest some time learning about the format it’s written in, will always be created with machine readability in mind.

6.      Suppliers are already providing vulnerability notifications to their users in various ways now – in emailed PDFs, website postings, etc. So, VEXes don’t give them a new capability (as SBOMs do)[ii]. When suppliers start providing VEXes, they’ll be expressly intended to be read by machines; the user will seldom read a VEX themselves. If they do, it will probably be to satisfy their curiosity about what the VEX looks like, not actually to absorb information.

7.      And speaking of machine readability, what tools or third party services are currently available to utilize VEX documents? The tool will need to do more than ingest VEXes, since VEXes alone won’t usually provide any useful information. Instead, a tool will need to do all of the following:

a.      Ingest an SBOM about a particular software product (or intelligent device) – say, product A - and retrieve all the component information from it;

b.      Using the NVD or another vulnerability database like OSS Index, create a list of vulnerabilities currently applicable to components listed in the SBOM for A;

c.      Ingest a VEX document or documents that apply to A;

d.      Based on the information in the VEX documents, remove from the list of vulnerabilities applicable to A all vulnerabilities that aren’t exploitable in A; and

e.      Repeat this process daily, if not more often.

8.      As VEXes are received, the list will be whittled down so that it contains only exploitable vulnerabilities. Thus, the user is much less likely to waste their time trying to identify and patch non-exploitable vulnerabilities, than they would be if they didn’t receive VEXes at all.

9.      Currently, there are no tools or subscription-based third-party services that perform steps a. to e. above. However, I know that the Dependency-Trackopen source tool will soon be able to ingest CycloneDX VEXes (it can createVEXes now, in the CycloneDX VEX format). Dependency-Track has for at least ten years been able to read SBOMs (in the CycloneDX format) and look up vulnerabilities in the NVD or OSS Index. It is now getting a huge amount of use from developers who want to learn about vulnerabilities affecting components in a software product they’re developing. The VEX ingestion capability (which I’ve seen demonstrated) should be available soon.[iii]

10.   Even with the current lack of tools for utilizing VEXes by end-users, I’m optimistic that the tools and the VEXes will be available before long. This is because software suppliers are the main drivers behind the VEX format. Two very large suppliers were the most interested in getting work on the format started in 2020 and are still heavily involved with that work today. Both of these suppliers – as well as many more – are very concerned about the heavy load that will be placed on their help desks if customers start calling in about the large number of vulnerabilities that will be found in the NVD for components of their products. The customers will naturally think of all these vulnerabilities as exploitable – when in fact only a small percentage of them are. One of the two large suppliers said they expected that producing VEXes along with SBOMs will end up saving them thousands of false positive help desk calls every month.

11.   In thispost, I suggested that suppliers should take responsibility for a) looking up components vulnerabilities in the NVD every day, and b) “whittling down” the list of vulnerabilities so that only exploitable ones are left on it. This would confer the added advantage that VEX documents would never have to be sent outside the supplier (or outside the third party that the supplier engaged to perform this work for them), and end users wouldn’t have to look up component vulnerabilities themselves.

12.   If you think about it, it’s somewhat silly that, if a supplier has for example 10,000 customers, every one of those customers should have to look up the same components in the NVD and get the same list of vulnerabilities, vs. the supplier just doing it once for all of them. Following the five steps listed above, the supplier would provide their customers a continually updated list of exploitable vulnerabilities in particular versions of their product or device (whichever versions the customer is using). I think this idea makes a lot of sense and will probably be realized one of these days, just due to market pressures.

13.   Another thing I would like to see happen is what I call “real-time VEX”. These are simple VEX documents focused on only one product (although perhaps multiple versions of that product), that simply list non-exploitable component vulnerabilities in the product. Because they are so simple, the supplier would put one out as soon as they discover that a particular component vulnerability or vulnerabilities isn’t exploitable in the product. Receiving these right away will allow end users to avoid what could be a lot of wasted effort looking for vulnerabilities that simply aren’t there – as opposed to waiting a week or more to receive that information.

Any opinions expressed in this blog post are strictly mine, and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

In my last post, I described how a group of friends and I are meeting weekly to discuss roadblocks that are currently preventing widespread distribution and use of SBOMS (SBOMs are already produced in huge quantities by suppliers, but they’re not distributing them to customers. They’re using them to improve the security of their own products).

In that post, I pointed out that we had decided to do what we could to remove what might be the most difficult of the roadblocks, which is usually called the “naming problem” in the SBOM community. I also pointed out that we had already made progress on identifying the cause of the problem and identifying at least a partial solution.

I’m pleased to report that this week we agreed on what looks like it could be a real solution – and even better, it probably won’t be technically hard to implement. It will be politically hard (which we always believed would be the case), but we think that, given that this problem is holding up widespread rollout of SBOMS (federal agencies are required to start asking them from their software suppliers in August, because of Executive Order 14028), this may be the time when the walls finally come tumbling down.

The background of the problem is this: The primary use of SBOMs, when it comes to cybersecurity risk management, is providing a list of components in a software product (or in an intelligent device). The end user – or a third party acting on their behalf – can look up the components in the National Vulnerability Database (NVD), to find out which if any current vulnerabilities (CVEs) apply to them. They can then “coordinate with” (i.e., bug) their supplier to patch those vulnerabilities quickly.

Looking up a component in the NVD requires knowing the CPE name for the product, which is a long character string like “cpe:2.3:a:adobe:acrobat_reader:11.0.5:-:*:*:*:windows:*:*”. The problem is that, for a large percentage of components, there is no CPE name to be found – for one of a host of reasons.

Of course, the NVD has a search function that might yield results, but SBOMs need to be fully machine-readable. Given that many software products have thousands of components, having to search on even one of those will slow down processing of the SBOM. Searching for 100 CPE names would be a lot of work, and searching for say 4,000 CPEs would be a monumental task.

At the meeting this past week, I pointed out that I’d never heard an estimate of what percentage of components in an average SBOM don’t have CPE names (or at least, constructing a CPE name according to the MITRE specification and using that to search doesn’t produce a hit. There are many CPE names that aren’t constructed correctly, but to which CVEs may be attached. Unfortunately, the NVD doesn’t provide “close matches”. It must be exact). I wondered if anybody had an idea about that.

The Director of Product Security of a very large software supplier, who actively participates in this group, said that his company also wondered about this. They studied the issue, and decided they were normally only able to find CPE names for about 20 percent of the components in the average SBOM for one of their products. In other words, it’s likely that around 80% of CPE names for software components can’t be found in the NVD, and therefore require a fair amount of “manual” effort to discover. And that’s assuming they can be discovered at all, since many software products and devices don’t have a CPE name at all, even though they may be loaded with vulnerabilities, as discussed in thispost).

Of course, currently that problem just affects the supplier, not their customers - since they and almost all of their peers aren’t currently providing SBOMs to their customers. But because they do a lot of business with the federal government, in August they’ll have to start providing SBOMs to federal agencies. Given that so many of the component CPE names are unresolved, what will the agencies do? Will they have to throw the SBOMs away?

No. Remember, getting one SBOM with say 100 components, and therefore only 20 verified CPE names, is still better than getting zero SBOMs with zero components and zero verified CPE names. The agency can still find out about vulnerabilities in those 20 components and contact the supplier about any of those vulnerabilities that are shown to be serious. And if there are 5,000 components in the SBOM (and there are many products with thousands, or even tens of thousands, of components), they’ll still have verified CPE names for 1,000 of those.

But my friends and I are determined to do better. The solution isn’t to make CPE better, since the opportunities to improve CPE are quite limited. I mentioned in my last post that the solution will probably lie at least partially with the purl(package URL) identifier. While that isn’t the whole solution, it does seem to be a good part of it. I used to think this was a problem that would take ten years to solve, but now I’d say it will be substantially solved in 1-2 years, and perhaps even less than that.

Any opinions expressed in this blog post are strictly mine, and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

About three months ago, a small group of friends (old and new ones) who are involved in the “SBOM business” got together to start talking about something that is on all of our minds: the small but important set of serious problems that are currently holding back widespread use of software bills of materials, and how these issues might be either mitigated or (preferably) solved altogether.

We all agreed that the problem of SBOM production by software suppliers is much smaller than the problem of use by general organizations, since many of the suppliers are already making extensive use of SBOMs today. But the suppliers are by and large producing SBOMs for their own use, to help them manage their own supply chain cybersecurity risks (i.e. the risks posed by the many components they include in their products). They’re not distributing them to their users, mostly because the users aren’t asking for them.

Why aren’t the users asking for them? There are a number of reasons, especially the lack of low-cost or open source tools and services that will help them identify risks found in components included in the software they use.

However, there’s one problem that overshadows all of the others. It’s one that was sometimes discussed by the NTIA Software Component Transparency Initiative, but which was generally considered to be insoluble – that is, insoluble without a persistent multiyear effort that would involve a lot of…well, lobbying of various government agencies and nonprofit organizations that would need to be involved in any permanent fix. Since there were other more easily soluble problems that could be addressed without such a huge effort, and since there are partial workarounds available that can make the big problem at least tolerable, the general consensus was to let this sleeping dog lie for the moment.

The problem has a number of names because it has many facets, but most people refer to it as “the naming problem”. Briefly, the problem is that there are a lot of problems with the “CPE” (common platform enumeration) names that are required in order to look up vulnerabilities (CVEs) that apply to a product in the National Vulnerability Database (NVD). Very often it will be difficult (or impossible) to look up a product, because the user is unable to find the CPE name under which it was entered.

More generally, it turns out that simply knowing the title and supplier name of a software product that you own won’t provide you with a universal name, that will be valid in all times and places. A really striking example is a software supplier you might have heard of, named Microsoft. Well, we may think of that supplier as “Microsoft”. However, there are many different names used with different products that we would normally consider “Microsoft products”.

If you search on “Microsoft” in the NVD, you’ll miss a lot of products that are listed under a different supplier name, like Microsoft Corporation, Microsoft Europe, etc. In fact, someone who works a lot on this issue told me they had asked people at Microsoft what company they worked for, and they received something like 27 different responses. Even more interesting, there is no central location where you can go to find all products produced by the various “Microsoft” entities.  

I wrote about this problem in 2020, soon after I joined the NTIA initiative. However, at that time I agreed with the consensus that there were other fish to be fried before we turned to that one; so I didn’t write any more posts on it until today.

To be honest, when my friends and I started meeting weekly, we didn’t really intend to tackle the naming problem right away. But, in one of our first few meetings, Tom Pace of NetRise did a presentation on a very serious problem(along with two others, discussed hereand here) that poses serious risks to intelligent device users; it turns out this is just one of the many facets of the naming problem.

The week after Tom’s presentation, the group decided (although I’m not sure “decided” is the right word; “stumbled into” might be better) to start exploring the naming problem further. In the four or so weeks since that meeting, to my surprise, we have made a lot of progress. Here are some of the things we’ve decided on.

The problem can be divided into short- and long-term aspects. While we explored the long-term problem for a week or two and made some progress on at least the outlines of a possible long-term solution, we decided that there are short-term steps we can take, that could lead to improvement in perhaps six months to a year. We’re going to focus on those steps for the time being.

While it’s tempting to think the solution is to have a central registry of suppliers or products – and while this would actually be a true solution to the problem – in practice, that would require a huge amount of resources, as well as close to endless lobbying, discussion, arm-twisting, etc. to put it into place and keep it operating. This simply isn’t going to happen.

The long-term solution has to be a distributed one, in which different groups are responsible for their own "namespaces”. What are those groups? They’re the groups responsible for different types of software. Since about 90% of software components are open source, most of these groups are open source repositories and package managers, including Maven, PyPi, NPM, etc. These all have their own naming schemes.

However, it turns out that open source namespaces are the easiest to deal with. This is because each repository maintains its own namespace – i.e. a list of all the open source products stored in the repository. In theory, if you want to find the authoritative name for an open source component and you know the repository it came from (which is usually determined by the language it’s written in), you can just search there and find the name.

The hard type of software is proprietary software – i.e. software that you usually pay for, which comes from Microsoft, Oracle, etc. One would generally think that the supplier could tell you the authoritative name for one of their products, but in practice suppliers will often have multiple names for the same product. For example, the product might have been acquired by multiple suppliers over time; they each named it using their own naming scheme; the versions that were produced by Supplier A will all have completely different names than those produced by Supplier B, etc.

And, since suppliers themselves get acquired and divested, the product name (and of course the supplier name) will often have changed multiple times for that reason. The current owner of a product will likely have assigned it their own name; just knowing the name that a previous owner assigned to it won’t necessarily be any help in learning the current name – of either the product or the supplier.

If we’re not going to have a universal namespace maintained by an army of registrars, how are we going to find the name of a component we’re concerned about? While this isn’t necessarily the solution, there is a naming scheme known as purl(package URL), that in principle can cover all open source software. The core fields in the purl are:

• type: the package "type" or package "protocol" such as maven, npm, nuget, gem, pypi, etc. Required.
• namespace: some name prefix such as a Maven groupid, a Docker image owner, a GitHub user or organization. Optional and type-specific.
• name: the name of the package. Required.
• version: the version of the package. Optional.

If you’re not sure of the “real” name of an open source component, you can in theory use the information you have about it – these four items – to construct its purl. You should then be able to look it up in databases like Sonatype’s free OSS Index, the largest open source software database. And while I’m sure there are various gotchas, you should almost always be able to positively identify the software you’re looking for.

Other vulnerability databases also use purl to identify software, but guess which database doesn’t use it? You’re right, it’s the NVD! The NVD only uses CPE names, meaning there’s a lot of open source software that isn’t referenced in the NVD at all. Plus, there’s more that isn’t referenced correctly, because of various problems constructing CPE names.

So guess what the next item is on my group’s agenda? It’s getting the NVD to start using purl, along with CPEs, to identify the products in the database. Technically, this isn’t very hard to implement, but politically it’s challenging, because of the various organizations that have to be involved in different ways. We’re now putting together our roadmap for doing this, and then we’ll start. I won’t give you an ETA for purl in the NVD (I’m sure it’s more than one year, and probably more than two years), but the point is someone is working on it.

As I mentioned, 90% of components are open source, so what about the 10% that are proprietary? While it’s not impossible that purl could be extended to them in some way, there are other naming schemes like SWID tagsthat are already in wide use, that might cover proprietary software. There will need to be cooperation from the large suppliers like Microsoft (which used to include SWID tags in all of its products), and there will need to be some group that handles products from suppliers that are out of business or just don’t respond to inquiries on this. So, this won’t be a piece of cake, either.

It will definitely be many years before every software product ever produced, and every product being produced now and in the future, will in theory be capable of being identified definitively with a single name (from one of only two or three naming schemes). But solving the naming problem isn’t an all-or-nothing proposition. The various incremental steps along the way (say implementing purl in just one part of the NVD) will make life a lot easier, both for software developers and for the users who are trying to identify and patch vulnerabilities (as well as other risks) in the software they use.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Note from Tom: I apologize for sending a post with just a title on it this morning. I was testing title formatting, and I forgot to delete the test post – which was quickly picked up and sent out by the feed service, bless their hearts.

Now, as for the 12 people that wrote in to say this was one of the best posts I’ve ever written, and I should write more like it… 

Probably the most widely-read of the 20-odd documents that were produced by the NTIA on SBOMs was “The Minimum Elements for a Software Bill of Materials”. Of course, this document was mandated by Executive Order 14028, which required that federal agencies start requesting SBOMs from their suppliers of critical software, but didn’t itself provide further information about those SBOMs.

I think this is a very well-written paper, which provides a lot of good information on what should be in an SBOM, as well as how it should be produced.[i] But literally 80-90% of the minimum elements mentioned in the paper have been totally ignored by the software community. This becomes clear if you look at the table on page 3 of the document.

The table lists three areas of minimum elements that will be discussed in the document: Data Fields, Automation Support, and Practices and Processes. Yet I’m not exaggerating when I say that literally 100% of what I’ve seen written about the minimum elements of an SBOM just addresses the data fields – i.e., the seven fields that need to be in an SBOM. Everything I’ve read ignores the minimum elements found in the other two areas.

This is too bad, since the NTIA made some excellent observations in those two areas. The reason for this neglect is quite simple: On page 9 of the document, you can find a nice table of the seven minimum data fields (called “baseline data elements”). But there’s no equivalent table for Automation Support or for Practices and Processes. You have to read a bunch of – shudder! – text and make your own tables. With hindsight, it would have been good to include summary tables for the other two areas as well.

This post is an attempt to make up for that omission, although the information won’t be in tables, but in lists. However, I first want to point out that nothing in the Minimum Elements document is “required” of federal agencies this August, when compliance with Section 4(e) of the EO is due. The only actual requirement for SBOMs under the EO is that agencies start asking their suppliers for them; that doesn’t require any minimum elements – just an email saying, “Please send me an SBOM.”

On the other hand, everything in the document is worth considering as you develop your organization’s program to utilize or produce SBOMs – and that goes for all organizations that use software, not just federal agencies.

Automation Support

Automation Support is discussed on pages 10 and 11. There’s just one “element” in this section: An SBOM should be provided in one of the three machine-readable formats: SPDX, CycloneDX or SWID[ii].

I do want to point out that there are other important considerations for automation support, that were probably left out of the Minimum Elements document because that’s all they are: considerations. One is that both SPDX and CycloneDX are available in different data representation schemes, the two most important of which are JSON and XML. Both SPDX and CycloneDX also support good ol’ XLS, which is machine-readable but is mainly intended for eyeballs – and can’t easily convey a lot of the information that’s provided using the JSON and XML schemes.

Practices and Processes

Most minimum elements are found in this category. They include the following (with references to the appropriate page of the document):

Frequency (page 12): This is very simple, but also very important. An SBOM should be re-issued whenever the software has changed. Of course, this includes major and minor updates, but it also includes patches. In my opinion, the version string should also change whenever there’s been a software change, even just a patch. Having separate “patch levels” or “build levels” isn’t going to work anymore, since none of the automated SBOM tools is likely to support either of those. So, patch levels and build levels will need to be integrated into the version string. If that isn’t done, there will be multiple SBOMs with different content, that are considered by the SBOM tooling to be identical. That obviously won’t work.

Depth (page 12): I will admit that people who engage in the reprehensible practice of writing about SBOM practices (which includes me, I’m ashamed to say) often speak of an SBOM as an inherently multilayer document. That is, we often make it seem like everyone should expect every SBOM they receive to show the direct components (dependencies) of the product itself, as well as the second, third, fourth, etc. layers – yea, unto the 15^thor 20^th generation.

There are two problems with this. First, the idea of neat, nested layers of components isn’t accurate in practice. There’s nothing that prevents component B from being a dependency of component A, while at the same time a different instance of A is a dependency of B – within the same product. However, it’s still accurate to take a particular chain of dependencies and put those in hierarchical order. In practice, it’s impossible not to talk about “layers”; you just need to understand what that means.

The second problem is much more important: It’s very hard for a supplier to get information on anything more than the first “layer” of components. And here’s where NTIA’s advice kicks in: “At a minimum, all top-level dependencies must be listed with enough detail to seek out the transitive dependencies recursively.” This means that:

1.      The supplier needs to provide their users a complete list of first-level components in their product (after all, if a software supplier doesn’t even have a complete list of the components that the supplier themselves included in their product, perhaps they ought to think about finding another line of work…); and

2.      The supplier needs to name the first-level components with enough accuracy that the user will be able to seek out an SBOM for each of those components (see below). In my opinion, the supplier should also make their best effort to acquire the SBOM for each first-level component, so they can fill in at least some of the “second layer” components in their SBOM.[iii]

For the time being, it’s better for software users not to demand too much from their suppliers. Just getting a complete list of components from their supplier (with usable identifiers like CPE and PURL for each one, so they can be looked up in vulnerability databases) will be a big improvement over the SBOMs that most users have currently, which are none at all.

Distribution and delivery (pp 12 & 13): The document doesn’t prescribe any means of distribution and delivery of SBOMs, but does say there are two components to this: “..how the existence and availability of the SBOM is (sic) made known (advertisement or discovery) and how the SBOM is retrieved by, or transmitted to, those who have the appropriate permissions (access).”

And here I disagree with the document. I believe that, once SBOMs and VEX documents are being widely produced and used (which might well be 5-10 years from now, I’ll admit), any distribution and delivery mechanism that depends on the end user retrieving the SBOM themselves, or even requesting it, will ultimately fail. What’s needed long-term is literally a mechanism to “stream” SBOMs and VEXes to users (more specifically, to the tool they operate to “process” the two documents for supply chain cyber risk management purposes), so that they don’t have to do anything at all to receive the documents they’re entitled to; they just show up. I described that mechanism (in not much detail) at the end of thispost. 

I’ve gone through all the Minimum Elements listed in the NTIA document, except for the seven minimum data elements and a couple other minimum elements that I don’t think are important. However, the document also goes “Beyond the Minimum Elements” and discusses some things that are probably not needed now, but which will be important in the future – so the user needs to be thinking about them. Rather than load them into this already-full post, I’ll discuss them soon.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

On Monday May 23, I had the wonderful opportunity to participate in a webinar on SBOMs with three people who have worked with and thought about SBOMs a lot. The panel was sponsored by six IEEE chapters in the NE Virginia/DC area, and also included Dr. Allan Friedman of CISA, Dr. Jean Camp of Indiana University, and Steve Pruskowski of CISA.

The group made some excellent observations, some of which were new to me. I recently listened to the recording, and I suggest it will be worthwhile for you to do that, too. Here is the link to the recording.

One good question had to do with SBOM formats – whether it’s sustainable to have multiple formats and whether they’ll easily interoperate with each other.  One of the other panelists said there were lots of tools for translating between the two major formats (SPDX and CycloneDX), and that it shouldn’t matter which format your SBOM is in – it will be usable in all tools and for all purposes, because it can always be translated to the other format. In fact, I would say this is the consensus opinion in the software community.

When I was asked to comment, I said I agreed with that answer to a point, but I added that it’s important to remember that the two formats both go far beyond the seven minimum fields that should be included in an SBOM. They each have 50-100 fields, and a lot of those aren’t replicated in the other format; so the information in those other fields will be lost in translation. This means that anyone who wants to convert between the two formats needs to make sure that all the fields in the original SBOM are available in the target format.

But I realized later that what I said falls far short of the real problem, which is that the two formats go about what they do in very different ways – so that addressing many of the differences between them requires doing a lot more than simply adding a field on one side or the other (if that’s all that were required, I’m sure all differences between the two formats would have disappeared a while ago).

Here’s an example of what I mean: One longstanding object of discussion (which Allan Friedman mentioned in the webinar) is the fact that a software supplier will sometimes make changes to the code of an open source component before incorporating it into their product. Of course, this is completely permissible, but a problem arises if the SBOM still lists the original name and version string of the component.

Because customers will normally look up the component in a vulnerability database like the NVD in order to find the vulnerabilities that apply to it, it’s possible – if the supplier did modify the code of the component – that they will either a) learn about – and possibly act upon – vulnerabilities that are no longer found in the component implemented in the product they use (because the supplier modified it), or b) not learn about a vulnerability that the supplier themselves inadvertently introduced into the component when they modified its code. Obviously, neither of these is a good thing.

CycloneDX (CDX) can represent this situation, as well as a few similar ones, using the “pedigree” field. Included in that field are five possible relationships: ancestors, descendants, variants, commits and patches.  Can all of these be translated into SPDX? SPDX includes 40 or so “Relationships”, and most of the CDX pedigree fields are included in those relationships.

However, there is a big difference in how the relationships are implemented in the two standards: Those “Relationships” in SPDX only apply to source code files, while “pedigrees” in CycloneDX also apply to libraries and entire operating systems. This means that, if a supplier has incorporated a modified library into their software product, they can represent that fact in a CycloneDX SBOM, but not in SPDX.

On the other hand, SPDX supports snippets, which are small bits of code that can be easily incorporated into a program in an integrated development environment. They can be similar to components, in that they’re sometimes developed by third parties; including them in software can save the programmer a lot of time. However, snippets sometimes come with licensing issues, which a developer needs to be aware of. CycloneDX doesn’t currently support snippets, meaning that, if an SPDX SBOM includes snippet information and is translated from SPDX to CycloneDX, it will not retain that information.

To sum up, you should never assume that an SPDX SBOM can be faithfully translated into the CycloneDX format or vice versa.[i] This means you need to think beforehand about what you need in a format before you standardize on one format or the other.

However, this might be a moot point. If your organization is a software supplier, I recommend you consider creating SBOMs in both formats, since it’s likely that some end-user tools that utilize SBOMs will only support one or the other format.

And if you’re an end user, I would strongly recommend that you not announce that you’ll only support SBOMs in one or the other format. You may miss out on SBOMs from some suppliers, who might feel they’re too small or under-resourced to support both formats. If you find a tool that only accepts one of the two formats,[ii] you’ll initially have to require SBOMs in only that format – and if one of your suppliers says they’re only going to provide SBOMs in the other format, you should have a discussion with them about whether they’ll change their mind.

But there’s another option on the software consumer side: In probably the next 3-6 months, there will be at least one subscription-type service available that will ingest SBOMs and VEX documents and perform risk analysis based on them (as well as on other information sources).[iii] Services like that will be able to handle both SBOM formats, so the Format Wars (and the need even to receive SBOMs) will go away forever, for their customers.

Unrelated note about the webinar: When asked about how SBOMs could gain acceptance by organizations in general, Jean Camp made the interesting point (starting at about 27:50 in the recording) that SBOMs might initially be positioned similarly to how the EnergyStar label was positioned: If a supplier is willing to provide SBOMs for their products, they obviously are committed to transparency about their security posture. Even if the user never does anything with the SBOM when they receive it, the fact that the supplier is willing to provide a regularly updated SBOM means they feel they have nothing to hide from their customers. This alone will be a great driver for improving software security.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Starting in August, federal agencies will have to comply with the provision in Section 4(e) of Executive Order number 14028, that requires agencies to start asking for SBOMs from their suppliers of software and intelligent devices. In addition, many private sector organizations are starting to put serious thought into how they will use SBOMs, once they become available from their suppliers and once the appropriate tools or services are available to utilize them. In all of these discussions, the question is always asked, “What fields should be included in an SBOM?”

The easy response to this question is, “Whatever fields are needed to achieve your purpose for using SBOMs. By the way, what are you planning to use SBOMs for?” Since the most important purpose for using SBOMs currently – and the only one that’s addressed in the Executive Order – can be described as “Managing risks that arise in procurement and use of software, which are related to components included in that software”, the answer to this second question is, “Whatever is required to manage risks arising from procurement and use of software.” Of course, at this point, the questioner isn’t much closer to having an answer to their question than when they started.

We can imagine a different Q&A occurring between a federal agency and a software developer. The agency says, "We need you to start providing us with SBOMs by this August." The developer asks, “What do you need in those SBOMs?” The agency asks, “What fields can you provide?” The developer points out that the SPDX SBOM format offers about 70 fields, and CycloneDX has more than 100, but it’s certain that the agency won’t need anywhere near this many fields. So which ones do they want included in the SBOM?

At this point, the agency is likely to ask what they’re “required” to have. The answer to that is that there isn’t a mandatory requirement for what the SBOM contains. The Executive Order simply required federal agencies to ask for “SBOMs”, but it also required that the National Technology and Information Administration of the US Department of Commerce identify “minimum elements” for an SBOM. The NTIA put out an excellent report on this topic last summer.

Besides listing seven minimum fields for an SBOM, the report contains a lot of other good recommendations. However, it also goes to great pains not to make those actually look like recommendations, so there’s been almost zero (that I’ve seen) written about them; I hope to remedy that deficiency soon. Meanwhile, it seems that most people who are following developments with SBOMs have heard of the minimum fields.

I’ll agree that each of those seven fields – Supplier Name, Component Name, Version of the Component, Other Unique Identifiers, Dependency Relationship, Author of SBOM Data, and Timestamp – should be included in every SBOM. But saying that these are what “should” be included is like saying that a car should include an engine, a steering wheel, four wheels, a drive train, and a place for the driver to sit. This car (plus some other basic items, of course) would perhaps get you from point A to point B, but it doesn’t fulfill the purposes that most of us are looking for in a car. What’s needed that goes beyond the minimum?

The real answer to this question is that what’s needed beyond the minimum fields are simply what the supplier and the user agree on. After all, SBOMs are a “good” (really, more of a service) that’s exchanged in the marketplace. If the supplier and the user agree on 60 fields that the user wants and the supplier can provide these, great. By the same token, if the supplier and user agree that only the seven minimum fields are needed, that’s great, too – although I would question how much the user will be able to accomplish in the way of software component risk management, with just those seven fields.

However, it’s also very unsatisfying to say something like, “Whatever the supplier and the customer agree on is fine.” I say this for two reasons:

1.      In many cases (and just about all cases when the supplier is a huge organization, but the user isn’t), there won’t be any negotiations on this – or any other – question. The user either accepts what the supplier is offering or they find a different supplier and product.

2.      When there is real negotiation, the user will almost always have no idea of what they need, since they probably only have a vague idea of how they’ll use the SBOMs. The fact is that there isn’t much written now about how SBOMs can be used for software component risk management purposes.

However, there are at least two documents that suggest additional data fields that users and suppliers should consider:

A.     The Minimum Elements document linked above identifies, on pages 14 and 15, three additional fields regarding software components: Lifecycle Phase, Other Component Relationships, and License Information.

B.     Fortress Information Security[i] published in April an excellent white paper that provides guidelines for using SBOMs. In addition to the seven minimum data fields, the paper recommends (starting on page 12) that these fields be included: Product Supplier Name, Component Type, Component CPE Name, Package URL (a software identifier that is important now and will likely become even more important in the coming years), Component Version String, Component Download Location, Licenses, Cryptographic Hash of the Component, and Document Digital Signature/Code Signing.

Which of these should you ask for? In general, if you’re not sure which of these you need, you should ask for them all; they’re all useful to have, although a couple of them, like Lifecycle Phase and Component Hash, may be difficult for suppliers to produce, in some cases.

If the supplier pushes back on one (or more) of these and you don’t have a strong reason why you want to have it, then don’t demand it. As you get more experience using SBOMs, you’ll get a better idea of what you need and don’t need, to meet your own requirements.

I hope to write another post soon, that discusses the other items in Minimum Elements, which aren’t as obvious – but are probably more important – than the seven minimum fields.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Last week, someone brought to my attention this “position paper” on SBOMs from the “Cybersecurity Coalition” (a group I hadn’t heard of. But then, there are lots of legitimate groups I haven’t heard of). The person pointed out to me that the paper describes roadblocks preventing SBOMs from being widely used. I read it because I’m very interested in identifying these roadblocks. I’ve come to realize in the past year that widespread rollout of SBOMs will require solving some fundamental problems – and those problems need to be identified before they can be solved.

However, in reading this paper (it’s not very long), I realized that this isn’t an honest attempt to facilitate the rollout of SBOMs; it’s an attempt to impede it by asserting that this rollout can’t happen before a bunch of impossible things happen. It’s a lot like the Wizard of Oz ordering Dorothy to bring him the broomstick of the Wicked Witch of the West. The good wizard didn’t expect Dorothy to bring him the broomstick, and he certainly didn’t have any use for it when she did.

So what’s the broomstick that the Cybersecurity Coalition – which clearly consists mainly of software developers - wants “SBOM” to bring back? Oh, just these four items:

·        Establish pilot programs involving software suppliers and agencies to demonstrate the effectiveness of SBOMs in improving vulnerability management practices, based on risk metrics, more rapidly and with less effort than existing tools and processes.

·        Drive to common standards for sharing, processing, and implementation of SBOMs and infrastructure to reduce potential confusion and inconsistency in outcomes.

·        Perform additional research and develop pilot programs to better refine how SBOMs can and should be used in cloud environments.

·        Create public/private workshops to discuss both the technical and non-technical aspects of SBOM sharing, including establishment of criteria for ensuring confidentiality where desired, and avoiding liability for software suppliers.

Of course, there’s nothing inherently bad about any of these things. But the NTIA Software Component Transparency Initiative went on for almost four years (from 2018 through the end of 2021) and took at least some of these steps. Plus, if the good people who wrote this position paper had joined the NTIA effort during that time and suggested these specific steps, I’m sure the group would have considered them. Why have they just recently realized that all of these things are needed?

Of course, the NTIA effort is over (although a couple of the groups that were working under the NTIA are continuing to work under CISA). Meanwhile, Executive Order 14028 (which the Cybersecurity Coalition expresses great love for in their document) – which was published a little more than one year ago - requires federal agencies to start asking their software suppliers for SBOMs less than three months from today. Isn’t it kind of late to suddenly decide that we need pilot programs and additional research and public/private workshops…and lions and tigers and bears (oh my!)? These people could have…you know, mentioned these things when the EO was being drafted or at least after it was released. And they might have contacted OMB to give them this input, since that agency is charged with implementing the EO.

But the items listed in the four bullet points above aren’t all the deficiencies these people have observed. I was shocked SHOCKED! to learn that:

1.      “Given that SBOMs are a relatively new and often misunderstood topic, they are not being produced routinely by many software producers, and changing that will require time and effort.” That’s certainly true. But what are producers required to do? The EO simply requires that federal agencies start asking producers for SBOMs, starting this August. If a producer says it’s too soon for them to produce SBOMs, they won’t be shot at dawn (or any other time). They’ll just need to tell the agencies when they will be ready. On the other hand, given that modern development tools can automatically produce a new SBOM with each build of the software – and given that one single SBOM-based tool used by developers is generating 20 billion requests to one vulnerability database every month - it’s a little hard to understand why there are developers who still think this is something new and difficult.

2.      “… it is important to recognize that, in the near term, there will be inconsistencies related to software produced before the Cyber EO versus software produced after.” One of the main purposes of SBOMs is to help suppliers and users understand how the components in software have changed from version to version and build to build. It seems to me that finding inconsistencies – and their causes – is one of the reasons for having SBOMs in the first place.

3.       “…it remains unclear how departments and agencies will receive and use SBOMs once they are shared with them.” My goodness! The authors seem to be saying that “departments and agencies” are used to receiving perfectly clear instructions that don’t require any thought at all, so they don’t know how to handle ambiguity. My guess is the agencies, had they been asked about this, would point out that dealing with ambiguity is a prerequisite for working in the government, just as it is in private industry. They can handle ambiguity just fine, thank you.

4.      “…overly stringent requirements related to the production and dissemination of SBOMs will require that resources be diverted from other activities.” In other words, the authors are saying, “Don’t get us wrong…we love SBOMs and we love the EO. But don’t expect us to, you know…have to balance competing resource priorities. We’ve never had to do that before.” Hmm…it seems that software developers are the only businesspeople on the planet who don’t have to make choices between (almost) equally valid uses for their funds. That’s news to me (and I expect to their employees as well). In any case, welcome to the real world.

The last two sections of the paper – titled “Automation” and “Challenges in cloud environments” – raise a couple of legitimate points, namely:

First, there isn’t much automation available now for use of SBOMs to manage vulnerabilities by software customers (although there are plenty of tools available for developers, contrary to what the paper says). Nobody has pointed this out more loudly than I have. But the paper suggests that what’s needed is “a series of pilots and test cases”, as if there haven’t been any so far (again, where were all these people while the NTIA Software Component Transparency Initiative – including several proofs of concept – was going on? In fact, the oldest of the PoCs, for healthcare, is still active. Any healthcare software or device supplier that wants to join that PoC should drop me an email; I’ll put you in touch with the leaders of the effort). 

The consumer tools will come; I’m sure of that. In fact, I just received word yesterday that Dependency-Track, the open source tool that is already widely used by software suppliers to manage component risks in their own software, now can ingest VEXes as well as SBOMs (it has ingested SBOMs for ten years, before the phrase “software bill of materials” was even being used); this makes it the first complete user tool for software component vulnerability risk management (I’ll have more to say on this subject soon).

Second, the last section points out that there are a lot of open issues having to do with SBOMs for SaaS. That’s absolutely true. However, the NTIA initiative made the calculated decision to address other issues before taking on SaaS SBOMs – so it’s not surprising these issues are still open. The CISA SBOM effort has this at the top of the agenda for this year, and even if that doesn’t happen, I’m sure another group will take it up.

But it’s simply not realistic to expect all of the capabilities that you would like to see in a new technology will be ready on day one. After all, did automobile buyers in 1910 demand that cars start without hand cranking, go at least 90 miles an hour, and by the way that there be plentiful roads and highways in place to handle cars going at those speeds? My guess is, if they’d demanded those things up front, we’d still be waiting for cars to be widely used today. New technologies need to grow in conjunction with use; no use, no technology growth.

And God help us if car buyers in 1910 had demanded Bluetooth! And especially if they’d demanded cell towers to allow them to make hands-free phone calls.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Next Monday evening, I’ll participate in a virtual panel discussion on SBOMs sponsored by six IEEE chapters in the NE Virginia/DC area. My fellow panelists will include Allan Friedman of CISA and Jean Camp of Indiana University, who I’ve heard talk previously and seems to have a lot of interesting things to say. The panel discussion is open to anybody; you can register here. They’ll encourage audience questions, and if the discussion is still going on when the hour is up, they’ll let it continue. Since I hate to see discussions ended when there’s clearly interest in continuing it, I’m game for that! 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

I’ve stated multiple times that I see just about zero likelihood of a cyberattack bringing down the US power grid (for example, in this post and this one). The reasons I’ve pointed to recently have always been related to the technical infeasibility of such an attack succeeding (although I should say “impossibility”, since I think it would be literally impossible to bring down the entire US grid with a cyberattack, no matter how massive).

But I recently remembered another reason for this opinion, which I stated in a postin 2020: Any cyberattack that succeeded in bringing down (or even substantially disabling) the US grid (actually grids, since there are three of them) would have to be carried out by a nation-state, or perhaps a huge and very sophisticated terrorist organization, like al Qaeda probably was before 9/11. Since there’s currently no such terrorist organization, the perpetrator would have to be a nation-state, like China, Russia, Iran, or North Korea.

If such an attack were carried out (whether successful or not), it would be considered an act of war, almost on the scale of Pearl Harbor. And if the US declared war on the perpetrator, there’s no way to know how far that might go – including a nuclear exchange.

Why would a nation-state, no matter what degree of hatred their leaders or people harbored for the US, risk possible total destruction, simply in order to cause a lot of damage to the US economy (and perhaps kill a lot of Americans) through an unprovoked attack? And don’t give me the “madman” story. Neither Vladimir Putin nor Kim Jong-Un, or any other world leader, is that mad that they’d risk destroying their life and the lives of their family members and friends.

If you want to worry about something that could cause a grid disaster, worry about an EMP attack or a massive solar storm (one did bring down the Quebec grid in 1989, and if the Carrington Event of 1859 had occurred today, that might well have brought down the entire US grid). Or worry about a pandemic. What would happen if one of those occurred, and the US response was so badly managed that a million people died? I agree that the idea is far-fetched, but I’d still say it’s a lot more likely than a cyberattack taking down the US grid.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

I recently wrote two posts (the second one is here) about a chilling revelation that Tom Pace of NetRise made at an informal meeting I recently attended. NetRise specializes in firmware security, and Tom has  noted that a huge percentage of software and firmware products aren’t registered at all in the National Vulnerability Database (NVD), meaning there’s no CPE name registered for the product. This means there has never been a single vulnerability reported for the product.

Thus, if you’re comparing competing IoT devices and you discover in the NVD that one device has a few serious software vulnerabilities while the other doesn’t even appear in the NVD (and thus has no identified vulnerabilities), you would be making a huge mistake if you ruled the first device out of consideration for that reason.

In fact, Tom produced software bills of materials for some firmware products (and no, an SBOM for firmware isn’t an FBOM, you naughty person!) that don’t have CPE names, and identified known vulnerabilities of components in those products. In one product (a member of a family of widely-used ICS products, that doesn’t even mention anything about security or vulnerabilities on its web site. If it’s a subject you’re embarrassed to talk about, better to not mention it at all, I guess...), he found 1,237 known vulnerabilities, and he could have found more if he’d had more time.[i]

Since I addressed the implications of Tom’s statements in the two posts on that problem, I won’t repeat them now – except to say that any supplier of products that aren’t listed in the NVD should be viewed with some degree of suspicion. Frankly, they should be considered guilty of poor security practices until they prove themselves innocent.

But at the end of both posts, I pointed out that this was only the second-most-serious problem identified by Tom Pace during his presentation, and I’d discuss the most serious one soon. Well, soon is now!

The 1,237 vulnerabilities that Tom found in one ICS product – and the probably tens of millions of vulnerabilities that are likely to be found (either by the good guys or the bad guys) in all other products that aren’t registered in the NVD – are all known, meaning they have previously been identified and have CVE numbers. These should be picked up by malware scans, so if you do those regularly, you should be able to at least bug the supplier of any product that contains them, to get them patched.

However, we all know that there’s a much more dangerous class of vulnerabilities called “zero-days”, that haven’t previously been identified and don’t have CVE numbers; therefore, scans won’t pick them up at all. The holy grail for a software attacker is to be able to implant a zero-day vulnerability in software or firmware during the software development process. This means that every organization that installs the software will potentially harbor this vulnerability without knowing it, allowing the attacker to penetrate their network and do bad things. This is exactly what happened with SolarWinds, and 18,000 users downloaded – and presumably installed – the tainted files.

Of course, it’s not usually easy to plant malware – zero-day or otherwise – in software or firmware while it’s being developed. In fact, the Russians had, according to Microsoft, 1,000 people working in the team that penetrated SolarWinds’ development environment and planted the malware. They were in the environment for 15 months and – like all well-planned business projects - spent the first three of those doing a proof of concept (I imagine they’ll write a Harvard case study on their success one of these days).

Perhaps you’re comforted by the fact that such a big effort was required to plant Sunburst. If that’s what it takes to plant a zero-day vulnerability in a software product during development, that’s not very likely to happen often in the future, right? However, Tom discovered there’s a much easier way for bad guys to plant zero-days, which will be hard to prevent without some self-regulation (or actual regulation, although God help anyone who tries to impose regulations on open source! In comparison, building a house out of Jell-O™ would be easy). In the hopes that people in charge of open source projects will start to address these issues, Tom presented the following scenario:

1.      Suppose a particular open source project is regularly used as a component in other software or firmware, and that a malicious party identifies one such project.

2.      That party implants a vulnerability in the code. But not just any vulnerability: a zero-day. Perhaps they work for a nation-state that hoards zero-days for a rainy day, rather than notifying the supplier of the vulnerability (as they should do, of course). Or perhaps they’re just talented and dreamed one up on their own.

3.      The malware-laden code becomes part of the next update of the open source product, so it’s downloaded and included as a component in some end-user products. Since the vulnerability is a zero-day, the scanners don’t identify it.

4.      The party that planted the malware reaps what they sowed, and is able to penetrate end user environments that have downloaded a software product that includes the now-malicious component.

5.      The malicious party penetrates multiple organizations, using the zero-day vulnerability. Of course, these penetrations will probably only be detected after the attackers have succeeded in achieving whatever their goal was: disruption of a manufacturing plant, spreading ransomware, exfiltrating PII, etc.

6.      If we’re lucky, the vulnerability will be identified after the first few penetrations (rather than after the first 18,000, as in the case of Sunburst) and will be assigned a CVE number and reported to the NVD. After that, it will usually be picked up by scans, so proactive suppliers (i.e., the ones who actively look for vulnerabilities in their products and report these to the NVD, and who of course patch the vulnerabilities when they meet whatever severity threshold they set for patching) will protect their products. Of course, suppliers who have never reported a vulnerability in their products, and who don’t even register those products in the NVD, won’t even find this new CVE because they don’t look for vulnerabilities at all. Their mottos are “Do not seek, and ye shall not find” and “Ignorance is bliss, as well as solid legal protection.”

7.      Even after this vulnerability is identified in the NVD and the scanners start picking it up, it won’t immediately be obvious that it’s found in a component of the affected product(s), and, more importantly, in which component (remember, there are plenty of software products and intelligent devices – including ones you may have on your desktop – that have thousands of components). If you trace a vulnerability to a product, how will you trace it to a particular component? The answer: It will be very hard to do that with just a few known infections. As the number of known infections grows, it will be possible to narrow the list of suspect components down to just one. However, this is a hell of a way to identify a new vulnerability: wait until there have been a lot of infections, so that you can draw some statistically valid conclusions as to the source of the infections.

To be honest, I think Tom’s second-most-serious problem (the one I described in the two previous posts) is the more likely to cause damage, simply because of the huge numbers of software components – and software products themselves – that aren’t listed in the NVD at all, meaning that even already-known vulnerabilities won’t be identified in them. And to be honest, Tom didn’t rank these two problems himself (in fact, I don’t think he even identified them as separate problems in his presentation).

Regardless, this is a serious issue. The second-most-serious issue can be addressed “simply” by having software suppliers (especially including open source communities) register all of their products with the NVD and start reporting all vulnerabilities for them (however unlikely this is to happen on any sort of scale in real life). However, this new issue requires a lot more than that. For one thing, it might require controls on who can contribute to any open source project. And how will those ever be enforced?

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Last week, I put up a post about a webinaron SBOMs that Fortress Information Security will present next Friday, May 13, titled “Software Bill of Materials Overview”. While that title doesn’t tell you a lot in itself, the first sentence of the description of the webinar reads

Fortress VP Tony Turner will be presenting key use cases for any software consumers looking to get started using SBOMs for procurement, vulnerability management, (incident) response, and software assurance activities.

In my post, I commented that “Fortress has put together what looks like it will be the first webinar describing how people like you and I can use SBOMs to make our organizations more secure.” In other words, I was saying this will be the first webinar that discusses using SBOMs to secure your organization from vulnerabilities and risks found in the software you use.

This idea was similar to what I said in a post a month ago, after Fortress posted an important white paper on SBOMs. I said the paper, for the first time, “provides a comprehensive narrative of what an end user organization needs to do to obtain SBOMs and VEX documents and use them for the purpose of reducing the cybersecurity risks their organization faces – specifically risks due to components found in the software they utilize to achieve their mission.”

When I put up the above-linked post on LinkedIn last week, Ron Brash of aDolus (another organization very involved with SBOMs) took exception to my assertion that this would be the first webinar that describes how to use SBOMs to make your organization more secure. He pointed to two webinars that aDolus presented last year (one with Industrial Defender and one with Verve), that were both aimed at consumers. I had attended one of these (and took notes, which I just reread); I just watched the other.

Both were excellent webinars, and I recommend you watch at least one of them (they were both presented around the same time, so the subjects were similar). And I’ll agree that both of them were intended for use by software consumers, not software producers. But I stand by my original statement that neither webinar discussed how organizations can use SBOMs to make themselves more secure, although I’ll point out below that such an omission is the rule, not the exception, in SBOM messaging so far.

To understand why I say that, look at the topics discussed (and addressed very thoroughly) in the two webinars. They include:

1.      The National Vulnerability Database (NVD) doesn’t usually identify vulnerabilities found in components of a product, but just in the product itself. This is because the NVD has no way of knowing what components are included in a product. Of course, this is why you need an SBOM, since that tells you the components in the product and you can – in theory – look each component up in the NVD, in order to learn of the risks it poses.

2.      But it isn’t easy to find vulnerabilities for the products themselves in the NVD, let alone for the components. You really need to know the CPE (Common Platform Enumeration) name for the product, and there are a lot of problems with that (this is the “naming problem”, which I pointed out in 2020 is one of the two or three most serious problems preventing widespread use of SBOMs. It still is, although a group of people, which includes both me and Eric Byres of aDolus, has very recently resolved to understand the causes of the problems – which are actually not complex - and develop a suggestion for NIST to fix the problems moving forward. Of course, that in itself won’t fix the problematic data found in the NVD today, but doing that isn’t technically hard, either – once the revised structure is agreed upon).

3.      A big component (so to speak) of the naming problem is the fact that the supplier and name of a product change frequently, primarily due to mergers and acquisitions but also due to marketing changes, efforts to harmonize product lines, etc.

4.      SBOMs don’t normally provide information on open vulnerabilities in components. To learn about those, you need to look up the components in the NVD (or another vulnerability database), where – of course – you’ll once again run into the naming problem.

I agree that all of these are problems. However, these aren’t in themselves consumer issues – they’re supplier issues. In my opinion, the software supplier needs to take responsibility for addressing each of these problems as part of fulfilling their responsibility to produce SBOMs, and to alert users to vulnerabilities that are due to components in the product.

Why is this the supplier’s responsibility? Because they chose the components to include in their product. Including components (especially open source ones) in their software saves suppliers a lot of time and money, as opposed to having to develop or purchase the code themselves. But these components also bring risks to the users of the software – as we know all too well from Heartbleed, Apache Struts, Ripple20, Log4j, etc. While users have a role to play in managing those risks, the primary responsibility is the supplier’s, starting with creating and updating SBOMs for their products.

Since the four items listed above (and others) are the supplier’s responsibility, what are the users responsible for doing, to reduce the risk posed by components of the software they use? That’s exactly what – I believe – will be discussed in the webinar next Friday. However, you can get a preview of the webinar discussion in the white paper that I mentioned. Here are some of the topics addressed in that paper (I’m sure some of these will be discussed in the webinar as well):

·        SBOM USE CASES FOR VULNERABILITY MANAGEMENT  

·        UNDERSTANDING UNVERIFIED TRUST

·        PRODUCT VS. COMPONENT VULNERABILITIES

·        SBOM SOLUTION PROVIDERS AND VULNERABILITY MANAGEMENT  

·        NEW COMPONENT VULNERABILITIES  

·        NON-EXPLOITABLE COMPONENT VULNERABILITIES  

·        USING SBOMS IN PROCUREMENT  

·        CONSIDERATIONS BEFORE RECEIVING SBOMS

·        WHAT TO DO WITH SBOMS AFTER RECEIPT?  

·        WHEN SHOULD SBOMS BE UPDATED?  

·        COMPONENT MODIFICATIONS AND SOFTWARE PEDIGREE  

These are the topics that, in retrospect, should have been addressed in the two aDolus webinars more than a year ago, given that their focus was intended to be on software consumers, not suppliers. However, I certainly find it understandable that aDolus focused on problems that were really supplier issues, in a webinar that was intended to be for end users. When I say that there’s been no previous document that’s addressed how to use SBOMs, I’m including two government-published documents that were also supposed to address use of SBOMs by end users who aren’t also software developers (the developers’ use case for SBOMs is to evaluate risks posed by components they’ve included in their products. As I wrote a couple of weeks ago, there’s now ample proof that developers are already putting SBOMs to work, in a huge way, to help them reduce their own software development risks. So the developers don’t need much guidance on using SBOMs anymore – it’s the consumers who do):

1.      Before it shut down last year, the NTIA Software Component Transparency Initiative published a “Software Consumers Playbook”. It’s well-written (although the language will be hard to understand by non-developers, its intended audience), but it focuses almost entirely on how SBOMs should be produced – which, of course, has nothing to do with the question of how a consumer can use SBOMs after they’ve received them.

2.      In December, NIST put out preliminary guidelines for implementation of Section 4(e) of Executive Order 14028; this section includes the two or three provisions that mention SBOMs. In the postI wrote on the guidelines, I pointed out that, in the approximately four pages of guidance regarding SBOMs, there are only two sentences that even mention how software using organizations (i.e. the federal agencies that are subject to the EO, very few of which are in the primary business of developing software) can use SBOMs for component vulnerability risk management purposes. They are:

Develop risk monitoring and scoring components to dynamically monitor the impact of SBOMs’ vulnerability disclosures to the acquiring organization. Align with asset inventories for further risk exposure and criticality calculations.

Regarding those two sentences, I pointed out immediately below them, “…this is the only requirement in the entire SBOM section that relates to the agencies using the SBOMs. Every other requirement specifies something that the agency needs to ask their suppliers to do or provide. I hope NIST doesn’t expect the agency to simply drop an SBOM in the bit bucket as soon as they receive it – they need guidelines for how they will use it.”

Unfortunately, up until Fortress released their white paper last month, those two sentences constituted the entirety of the guidance available – from either public or private sources - on making use of SBOMs for software component vulnerability risk management purposes, at least that I know of. If anybody knows of other previous guidance, I would love to hear about it and will publish it.

Does this mean that I don’t think software users need to know anything about the various challenges that arise in developing SBOMS? No. That’s why I recommended that you watch at least one of the aDolus webinars. I also recommend that you try to at least skim through every document found here.

But as you do that, and you hear about the many problems developers face in preparing SBOMs, you need to keep in mind that these aren’t your problems. Your problem is what you do with that nice, shiny SBOM you just received. And keep in mind that there’s no good way to utilize SBOMs (or VEX documents) for risk management purposes, except through automation – either a tool that you run on-prem, or a third-party service that runs their own tools and provides you with the results; those results should include a continually-updated list of exploitable component vulnerabilities found in each of the software products used by your organization.

As of today, there’s no tool or third-party service (other than hourly consulting services, which aren’t scalable), that will ingest both SBOMs and VEXes and produce that continually-updated list for you. And don’t fool yourself into thinking that you could do that on your own, given enough time and effort and YouTube videos. There are probably products on your desktop today that have literally thousands of components (the average software product contains close to 150 components); are you really going to go through the steps shown in the video for every one of those components, especially since new SBOMs may, for some products with lots of components, arrive daily?

Sure, you can try looking up a few components from an SBOM in the NVD; you might even succeed in learning about a few vulnerabilities that apply to those components. But don’t mistake doing this for actually using SBOMs for vulnerability management purposes. If you want to learn about that, read the white paper and sign up for the Fortress webinar on Friday the 13^th.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

A recent post described a presentation I saw last Friday by Tom Pace of NetRise, describing what seems to be a huge security problem. To summarize it:

1.      Do you think products with a lot of open vulnerabilities - as indicated by CVE’s listed for the product in the National Vulnerability Database (NVD) - are dangerous and should be avoided? If so, you’re right.

2.      By the same token, do you think a product with no open vulnerabilities – and which has never even reported a vulnerability – is one you ought to buy? After all, what could be safer?

3.      It turns out that you’re probably better off with a product with lots of open vulnerabilities than with a product that’s never reported a vulnerability. Seriously.

4.      Why? As Tom pointed out, nobody can report a vulnerability (CVE) for a software product or intelligent device, unless the product has a CPE name. So, if the product doesn’t have a CPE name, no vulnerability has ever been reported for it.

5.      There are two reasons why a vulnerability might never have been reported for a product: A) The coders who created the product – as well as the third party coders who wrote each of the 135 components (on average) included in the product – never make mistakes; and B) The supplier of the product has never even bothered to look for vulnerabilities in it, perhaps because they believe A) is true. Of course, if this is their reasoning, it’s hard to argue with it: If you’re perfect, you don’t need to worry that you’ll ever make a mistake – after all, you wouldn’t be perfect if you made mistakes! Pretty compelling logic, no?  

6.      Tom came to realize that probably the majority of software products and intelligent devices sold in the US don’t have CPE names, meaning they don’t have any CVEs reported against them. So if your organization is comparing five different vendors in advance of procuring a particular type of software, and you decide to compare the vendors based in part on how many open vulnerabilities their product has, what will you do if you find that four of those vendors have some moderate number of open vulnerabilities, but the fifth doesn’t have any at all – in fact, they don’t even have a CPE number? Will you stop the evaluation and declare the fifth vendor the winner? After all, they’re not only good…They’re perfect…

7.      It turns out that Tom Pace doesn’t believe there’s such a thing as perfect software developers, so he decided to test this idea. He identified the supplier of a widely used line of devices found in many ICS environments and started looking for their devices in the NVD. He didn’t check on them all, but none of the ones he checked on had CPE names – and therefore they didn’t have any reported vulnerabilities. In fact, the supplier’s web site doesn’t mention anything about vulnerabilities or patches – or even security in general. It seems the subject never came up for them…

8.      So either this supplier is so perfect that they don’t even have to think about security, or they made the decision early on not even to look for vulnerabilities in their products, since that’s probably the best way to ensure that nobody will ever learn of vulnerabilities. They won’t have to report vulnerabilities to the NVD, post notices on their website, etc. Sounds good, right?

9.      Tom decided to test just one of this supplier’s products, by first creating an SBOM for it and then identifying all of the open vulnerabilities that apply to those components; if he found anyway, that would constitute proof that the supplier isn’t perfect.

10.   I regret to inform you that this supplier isn’t perfect, after all. How many vulnerabilities did Tom identify? 5? 10? He actually found…drumroll, please…1,237 vulnerabilities. And he could have found more if he’d had more time. Moreover, that was 1,237 vulnerabilities in just one of about 50 products sold by this supplier – and probably none of those other products have CPE names, either.

Of course, this strategy is pure genius: Since the   supplier never reports any vulnerabilities, they don’t have to worry about patching log4j, Ripple20, OpenSSL, or any other serious bugs that keep people up at night and ruin holidays. After all, this supplier is perfect.

But for those of us who live in the real world and realize there’s no perfection (especially in software), we need to take other steps to protect ourselves. Here are some suggestions:

1.      When you’re sending out an RFP for software, first check the website of each supplier you’re considering inviting to the RFP: What do they say about security and vulnerabilities? More importantly, do they say anything at all about those topics? If they don’t, drop them off your list.

2.      Ask the salesperson for each supplier you’re considering to give you the CPE name of their product. If they can’t give you anything (meaning there is none for the product), drop that supplier off your list.

3.      If the product has a CPE name, but the NVD doesn’t seem to list any vulnerabilities for any version of the product, ask the salesperson to tell you if they’ve ever reported any vulnerability for it (and if so, verify the CPE name. It’s possible they entered the CPE name wrong and reported vulnerabilities against that name. The NVD doesn’t validate CPE names, as far as I know).

4.      But, if they tell you, “We’re so dang good that we’ve never have a vulnerability!”, thank them for their time and drop them off your list. What they’re really saying is, “We’ve gone to great lengths to make sure we don’t know about any vulnerabilities in our products, so we can tell you truthfully that we know of no vulnerabilities. You can be sure that when a big vulnerability appears in one of our products, we’ll be the last to know about it! Are you ready to sign the contract?”

Again, thank them for their time and remove them from your list. By the time you’re done with this, I hope you’ll still have at least one supplier on your list. But even if you don’t, you’re much better off staying away from any supplier that’s so convinced of their perfection that they don’t want to take the chance that they’ll be proved wrong, by…you know, trying to find out if they actually might have some vulnerabilities.

At the end of my previous post, I noted that this problem was only the second most serious problem Tom has identified, having to do with CPE names. Coming soon to a blog near you: Tom Pace’s most serious problem.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Up until recently, almost the entire focus of SBOM messaging was on software developers. The reason for that – which I don’t argue with at all – was that SBOMs have to be both produced and used. Unless the software suppliers got onboard, it wouldn’t matter whether users want SBOMs or not – they won’t get them if they aren’t available in the first place. The users need to “pull” on the SBOM string, but there won’t be any string to pull on, if the suppliers aren’t “pushing” the SBOMs out on their end.

Well, that battle has been won. Software developers are using SBOMs for their own vulnerability management purposes, and heavily (as the 202 million calls from Dependency-Track to the OSS Index vulnerability database – in one month - attest). Now it’s time to get the users interested in receiving those SBOMs. That way, the string will actually move, as opposed to being pushed from behind into a tangled ball that doesn’t go anywhere, because nobody’s pulling on the other end.

But consumers aren’t interested in having SBOMs so they can a) impress their children that they know something about technology, too; b) print them out on waxy paper and use them to wrap fish; or c) print them out on contact paper and use them to paper the walls of their new den. They want to use them for vulnerability management purposes, so their organizations can be more secure, and so they don’t have to run around for months trying to find every instance of the next log4j.

In early April, Fortress Information Security[i] put out a white paper that was the first document (that I’ve seen) that discusses using SBOMs, not producing them. And in another first, it’s aimed entirely at software consumers (who can be assumed to know very little about SBOMs), rather than software developers (who can be assumed to know a lot about them).

Now, Fortress has put together what looks like it will be the first webinar describing how people like you and I can use SBOMs to make our organizations more secure, rather than to wrap fish. While I don’t know what will be in it, I’m sure it will be good. You can read about the webinar (on Thursday, May 12 at 11 AM EDT) and sign up for it here. See you there!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

I’ll be honest: It’s been quite a while since I seriously worried about anything in cybersecurity other than software vulnerabilities. Almost every serious cyberattack you can name in the last say five years, including Not Petya, SolarWinds, Kaseya, and literally every ransomware attack, was either based on or enabled by at least one software vulnerability.

Of course, when the average cybersecurity person thinks about software vulnerabilities, they probably think of badly-trained (or simply incompetent) software coders who make mistakes and leave vulnerabilities in the software they’re writing – or else malicious actors who plant backdoors (which are just deliberately-inserted vulnerabilities) in the code during development, so they can later come back when the software is deployed and exploit those backdoors. Certainly, both types of people exist.

This was certainly what I thought of when I thought of software vulnerabilities, until I fell in with a bunch of people – loosely, the “SBOM community” – who have been on the front lines of fighting software vulnerabilities for years. These people know that no amount of training, pleading, threats, firings, etc. will ever cause developers to stop creating new vulnerabilities. As the Good Book says, “Software vulnerabilities you have always with you.”

Given that there will always be new vulnerabilities, what’s most important are identifying vulnerabilities, patchingthem as quickly as possible, and making users aware of both the vulnerabilities and the patches. But there are some serious obstacles standing in the way of accomplishing these goals – and for once, the individual developers aren’t to blame for them. In fact, the most serious obstacles may be due to problems with the various institutions that we’ve created, in a well-intentioned effort to mitigate the risks posed by software vulnerabilities.

One of these institutions is the National Vulnerability Database, which was created in 2005 by NIST and continues to be run by them (MITRE Corp. created the CVE List in 1999, and still maintains it. That list is continually incorporated into the NVD, but isn’t in fact part of it). A foundation of the NVD is the CPE (Common Platform Enumeration) name, which is meant to be a unique machine-readable identifier of a particular version of a particular software product.

CPE is a crucial component (so to speak) of the software vulnerability management ecosystem. This is because the only way a software (or intelligent device) user can learn what CVEs (and by the way, CVE used to stand for something, but now it’s NBI – nothing but initials) apply to a product they use is to search the NVD, using the CPE name for the product (although they may first have to search for the CPE name itself, using a supplier or product name).

In theory, the supplier of your software product should provide, in their SBOMs, a CPE name for every component of the product. Armed with those CPE names, you (or more specifically, a tool that you operate or that a third party operates on your behalf) can search for any CVEs that apply to each component. You repeat this for each component, and you’ll end up with a list of all the CVEs that apply to at least one component in the product. What could be easier?

Well, it turns out that there are a lot of problems with this system, which prevent the user from getting accurate answers. I addressed a few of those problems in the only post I’ve ever written on this issue; these are collectively known as “the naming problem” in SBOM circles. In those circles, the naming problem is thought to be practically insoluble, although it can be managed – just like an incurable disease can often be managed, so that it won’t automatically be fatal.

Until last Friday, I agreed with the idea that the naming problem could be managed, so that the patient wouldn’t die of it. But on Friday I saw a great presentation that Tom Pace of NetRise made to an informal meeting I was part of. While there are a lot of problems with the current CPE/CVE system for identifying software vulnerabilities, Tom pointed out one that perhaps outweighs all the others – and makes me and a few others think that the naming problem needs to be solved in the near future, not just mitigated.

Before I explain the problem, you need to know that NetRise is a company dedicated to firmware security risk management. But just like software (certain theologians argue that firmware issoftware; however, I don’t get involved in theological arguments), probably the biggest source of risk in firmware is the components included in a firmware product. Also like software, in order to learn about vulnerabilities in those components, the user needs to have an SBOM listing them, with hopefully a CPE name for every component; then the user can look in the NVD to find the complete set of vulnerabilities (CVEs) found in those components.

Now, I already knew that there were problems with looking up CPEs. I mentioned a few of these in the post from 2020 that I referenced above. And since I anticipate I’ll do more posts on this problem as the effort to solve it moves forward, I expect to discuss others. However, the problem Tom discussed was one I hadn’t heard of before, although it follows so clearly from the facts (all of which I already knew), that I wonder why I didn’t think of this.

1.      The root of the problem is an unfortunate (to say the least) quirk of the NVD: If you search for CVEs by entering a CPE name, or even a product or supplier name, and there are no CVEs associated with whatever text string you entered, you will receive the message “There are 0 matching records.”

2.      Of course, this fact alone doesn’t constitute a problem. If there aren’t any CVEs, that’s what you would want to see. But what do you see if you had mis-entered just one of the characters in a CPE name (for example, you had entered one letter wrong in the CPE name “cpe:2.3:a:altiris:report_pack_for_inventory_solution_for_windows:6.2.1047:*:*:*:*:*:*:*”)? You see the same message.

3.      Does the fact that the message is the same mean there aren’t any CVEs associated with the CPE name you intended to enter? Not at all. There could be a ton of them, and if you hadn’t made the mistake, you would have seen them. But – humans being generally an optimistic lot – my guess is most people, upon seeing that message, will breathe a sigh of relief and say, “Thank goodness, my product has no vulnerabilities!” So it’s likely a lot of people will be misled into thinking the software product they’re inquiring about doesn’t have any vulnerabilities, when the problem was that they fat-fingered one of the letters.

4.      Of course, that would be bad, but what Tom pointed to was much worse. Suppose you enter the name of a product that doesn’t exist in the database at all? You’re right, you’ll get the same message. Yet there are lots of software products that don’t have a CPE name, since they were never entered in the NVD. Just as in the fat-fingering case above, a user who searches for a CPE name that was never defined will see the same message as they would see if the CPE was defined, but there were no CVEs applicable to it; again, this is likely to give them a false sense of security.

5.      Who’s responsible for entering a product in the NVD? Anyone can do it, but in general the supplier of the product should. In fact, Tom pointed out that suppliers of software products or intelligent devices should get in the habit of creating and entering a CPE name as soon as they’ve developed a new version of their product.

6.      But most software products and devices (especially ones that are mainly used as components of other products – like firmware) don’t even have a CPE name; this is especially true, when you consider there needs to be a CPE name for every version of every product. In fact, it’s likely that most CPE names are only created when someone wants to enter a vulnerability for the product, since recording that a particular product is vulnerable to a particular CVE requires referring to the CPE for the product.

7.      This means that you will never find any vulnerabilities listed for a product if the CPE for that product/version doesn’t exist. In fact, you’ll get the same message as in the cases discussed above: “There are 0 matching records.” Once again, you’ll come away with a false sense of security.

8.      But in this case, the problem is much more serious, because the reason there were no matching records was the product was never entered in the NVD. And the product was never entered in the NVD because a vulnerability has never been reported for it.

9.      Of course, the reason that a vulnerability has never been reported for a software product might be that the product has never had a single vulnerability…Yes, I think that’s hilarious, too. In fact, if a supplier has never entered a single vulnerability for their product, it’s likely because they’ve never looked for vulnerabilities in the first place, not because the product is so well coded that it’s never had one.

Let’s recap where we stand now: For those of you keeping score at home, the problem I’ve just described is that probably most software products in the US don’t appear in the NVD. This means that a user of one of those products, who wants to learn about vulnerabilities in it, will never find a single vulnerability – but they’ll get the same message that they would get if in fact the product was in the NVD (with its own CPE name), but it was truly vulnerability-free.

You may wonder, “Perhaps most of these products never had a vulnerability, or at least they’ve had such a small number that it really doesn’t matter that they’re not in the NVD. Could that be the case?"

Tom says NetRise has found a lot of products (again, mostly products that are used as components of other products) that seem to have a lot of vulnerabilities, yet there’s no CPE for any of them. He pointed (by name) to one particular CPE-less product – used in many ICS environments – whose manufacturer has never entered a single CVE for that product. Is this because they’re such careful coders that they never make mistakes? Not likely.

But it’s even better: That manufacturer is so good that there exist no CPEs for any of their products (and they have a bunch of them). Is this because they’ve never found any vulnerabilities, despite constantly searching for them in all of their products, all of the time? Again, not likely.

And I’ll add that this supplier is so good that their web site doesn’t contain a single mention about vulnerabilities or even product security in general. Of course, this means they really have their act together, right?

However, Tom, being a naturally skeptical guy, decided to do some scanning on one of that manufacturer’s products – just to see if perhaps there might have been one or two vulnerabilities that had escaped their attention. The good news? He didn’t find one or two vulnerabilities. The bad news? He found 1,237 – in that one product. And he could probably have found a lot more if he’d had the time.

Boys and girls, this is a big problem. Here we’re worrying about individual vulnerabilities - like the log4j vulnerabilities - that are found in thousands of products. Those are legitimate worries, but what about the tens or hundreds of thousands of products that have never even reported a vulnerability – since they’re not even in the NVD in the first place. In fact, there are suppliers who make lots of products, that have never reported a vulnerability on a single one of them. And it turns out that one randomly-chosen product, which has never reported a vulnerability, in fact has at least 1,237 vulnerabilities now.[i]

I think Tom’s presentation makes it very clear that software vulnerabilities are a big problem, and the vulnerabilities that we know about are just the tip of the iceberg. But what would you tell me to do if I told you that this was just the second-biggest problem that Tom pointed out?...Well, I’m not going to do that, but I will describe that problem in one of my next posts. Stay tuned.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

I had to miss S4 last week, but I did hear a lot about it from attendees; as usual, it was very good. There was a lot of discussion of SBOMs, but I heard a couple of ideas that supposedly met with a lot of approval, which strike me as completely wrong. I’ll address the first of those in this post, and the second in another post soon.

The first disturbing thing I heard was that a lot of people supported the idea that they don’t need SBOMs to accomplish their vulnerability management goals, but just attestations. I’ve saidthat users don’t need SBOMs (or VEXes), but that was just by way of saying that what software users do need is the risk analysis that requires SBOMs.

In other words, you don’t need to have SBOMs, but you do need to learn about exploitable vulnerabilities due to components in your software. You will be able to learn that information yourself by receiving SBOMs and VEXes and feeding them into the proper tool, but there will also be third parties that will perform that analysis for you (and many other organizations) as a service.

However, what it seems the people at S4 were saying was, “I don’t need to track component vulnerabilities at all, in the software products my organization uses. I just need to get an attestation from the supplier that the software doesn’t have vulnerabilities. Then I can show the attestation to my regulator, compliance department, or whoever’s bugging me about how safe my software is. This will make them shut up, and I can go back to doing the other 9,999 things I have to do.”

Of course, a conscientious supplier would never give you such an attestation in the first place. Even though the supplier might honestly believe it’s completely true today, tomorrow they might find out that the product has Log4j all over the place – and has had it for years.

 I’m sure there are suppliers who will give you such an attestation. But keep in mind that:

1.      The average software product has well over 100 components. Many have thousands.

2.      Each component has probably as much chance of having a serious vulnerability as does the code written by the software developer itself. So the likelihood is that there will be many times more vulnerabilities in components in your software, than are listed in the NVD for the software itself.

3.      A 2017 studyby Veracode found that “A mere 52 percent of companies reported they provide security fixes to components when new security vulnerabilities are discovered.” It also found that “..only 23 percent reported testing for vulnerabilities in components at every release.”

4.      Of course, one would hope that those numbers have improved since that time, but if you think all software suppliers are now diligently patching every vulnerability in a component in their product – or even checking for component vulnerabilities…well, I think you need to think again.

This isn’t to say that the average software developer is trying to pull a fast one on you. But the fact is that we’re all busy, and we all put off things that nobody is bothering us about. If your customers don’t have SBOMs and therefore don’t know what components are in your product, your phone isn’t going to be ringing off the hook whenever a serious vulnerability is discovered in one of those components. Of course, you’ll get around to fixing those vulnerabilities, no question. Right after you get the next release out…

Meanwhile, if a customer asks for an attestation that your software is vulnerability-free, and since the NVD doesn’t currently show any vulnerabilities for it…well, you’ll be happy to sign that, because after all…you know you’ll get to those component vulnerabilities sooner or later.

So take these attestations for what they’re worth, which is not very much. If you really want to learn about the vulnerabilities lurking in components in your software, you need SBOMs to tell you (or your chosen third party service provider) what those components are. And you need VEX documents to tell you which of those component vulnerabilities are exploitable. Accept no substitutes! 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

If you’re involved with the electric power industry and you’ve been reading this blog for a while, you’re undoubtedly familiar with Lew Folkerth. Lew is Principal Reliability Consultant with the RF region of NERC and is probably the most respected authority on the NERC CIP standards. But more importantly, he’s a great teacher on those standards and he places everything he says in the context of cybersecurity and risk management (he’s very knowledgeable about both subjects).

Lew writes a column on NERC CIP in RF’s newsletter, which is published quarterly. Since the newsletters are big files, Lew also publishes his columns separately. You can access them by going here and dropping down the menu for Standards and Compliance at the bottom. Under Outreach, you’ll see a link to every one of his columns since he started writing them in 2014. And BTW, you’ll also see the link to the slidesfor the talk I gave on SBOMs and CIP-013 compliance at RF’s March Tech Talk.

Most importantly, Lew has just put together, for the first time since 2019, a single filewith all of his columns. Here are some of my favorites, starting this year and moving back (page numbers refer to the PDF itself, not the numbers at the bottom of each page):

1.      BCSI Revisions  (page 127) – this is an excellent article (published in Q1) discussing the revisions to CIP-004 and CIP-011 to update the protections for BES Cyber System Information, including BCSI in the cloud.

2.      Using Advanced IT Technologies in an OT Environment Part 2 – Containers (page 121) – another excellent article that both gives a great introduction to containers and describes how you can utilize containers within your Electronic Security Perimeter, yet still be in compliance with the CIP requirements. I had never thought this was possible.

3.      Implied Requirements (page 117) – This is one of the endearing “features” of the NERC CIP requirements – there are so many requirements that are implicit. Because they’re implicit, you can’t receive a violation for missing them, but missing them will put you out of compliance with other requirements. I wrote about implicit requirements several times, including hereand here.

4.      Incident Response and Incident Management (page 115)

5.      CIP-012-1 In-Depth (page 104), followed by a very detailed accompanying article starting on page 106

Of course, if you get hooked on Lew, you should subscribe to the RF newsletter, which has a lot of other interesting articles besides Lew’s columns.

If you’re with a NERC entity or an IT or OT supplier to the power industry, I’d love to have a discussion with you about CIP-013 and supply chain cybersecurity. Please drop me an email.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.