Almost anybody who has been involved with software vulnerabilities in any way (even hackers!) has a love/hate relationship with the National Vulnerability Database (NVD). On the plus side, it’s by far the largest and best-supported vulnerability database in the world. But on the minus side, there are many problems that make it hard to use the NVD, and in many cases make it impossible to find vulnerability information which almost certainly is in the database somewhere or other.

A little less than a year ago, I convened an informal group of “SBOM industry” leaders to discuss why it is that SBOMs are grossly underperforming, at least when it comes to distribution to and use by organizations whose primary business isn’t software development.[i] The goal of the group was not just to discuss those issues, but to figure out how they can be addressed, and do what we can to set them on the road to being resolved. We call ourselves the SBOM Forum, and we meet weekly on Zoom.

We decided that, while there are a lot of issues that are inhibiting SBOM distribution and use, we would focus on the show-stopper issues; I personally think there are no more than three or four of these. We didn’t have a formal discussion of which issue we would address first, but within two meetings we had found it: the naming problem.

However, even we weren’t stupid enough to try to take on the entire naming problem, which has many aspects and is found to some degree in every software or vulnerability database in the world. We focused right away on the Big Daddy of vulnerability databases, which was the one we all had experience with. The NVD uses “CPE names” to identify products, and those are the source of a lot of problems; we described those problems in pages 4-6 of the proposalwe published on the OWASP site last September. Our proposal described how to fix (or at least greatly remediate) the problems with CPE, although this required involvement of a few other federal government and private sector organizations.

That proposal was meant to address the bulk of the naming problems in the NVD, and we’re hoping it can be completely implemented in 2-3 years (which of course is close to light speed when you’re dealing with the federal government). The appropriate agencies in the federal government started considering our proposal, and we were fairly sure it was on the road to implementation in our time frame.

After publishing our proposal, we had discussions on other topics and were settling in on VEX as our next topic. In my opinion, VEX and the naming problem are the two biggest show-stopper problems preventing SBOMs from being distributed and used by non-developers.

However, recently we became aware of a reason why implementation of our proposal might be delayed significantly longer than three years. We had a meeting to discuss this problem. While we received some assurances then that our immediate fears might be overblown, we ended up having a more wide-ranging discussion of the NVD, at which other issues came up. At the end of that hour-long meeting, we decided we wanted to focus on the NVD itself next, and not limit ourselves to discussing just the naming problem within the NVD.

We have representatives of some very big software and intelligent device suppliers in the SBOM Forum (as well as a number of smaller tool vendors and a few consulting firms. We only have a few end user organizations and we’d like to have more), who were surprised to hear what was said about some NVD problems that have nothing to do with naming. They wanted to hear about all the problems the other members of our group had run into.

Even more importantly, we started to have a discussion about what the NVD could be if it were allowed to move out of the narrow box it finds itself in now. For example, given that people all over the world use the NVD, yet the entire physical infrastructure is housed in the US, what might happen if the NVD could place infrastructure (perhaps through content delivery networks) on other continents – while at the same time getting support from private and public sector organizations on those other continents?

Note that I don’t for a minute blame any individuals, or even government agencies, for the NVD’s problems. Any organization that’s grown very rapidly, yet has to fulfill the obligations of being a government-controlled entity, will probably find itself in a similar box sooner or later. In fact, there’s a great example of a similar organization that was incubated in the NTIA, the same federal agency that “incubated” the Software Component Transparency Initiative, also known as “Allan’s Army”. That organization found itself in an overly box much quicker than the NVD has, and now it’s a very effective private sector organization, that gets some help from governments.

Has anyone heard of DNS? Let me put that another way: Is there anyone who uses DNS fewer than perhaps 5,000 times a day (almost always without even thinking about it, of course)? Our lives would be very different if, instead of being able to find any web site we want through a single DNS query, we had to first obtain from the operators of the site (perhaps by calling them – do you remember phone calls?) their 31-hex character IPv6 address, then enter it by hand in our browser. And woe betide you if you got one of those characters wrong; you’d have to re-enter it until you did it perfectly.

Without going into a lot of detail, the NTIA saved you from that fate by picking up an idea developed by an academic named Paul Mockapetris and turning it into a real service. In fact, NTIA itself was the first domain name registrar. But, as you can imagine, business grew very rapidly, and since the NTIA (and the federal government in general) doesn’t want to go into business doing something the private sector could probably do better and certainly less expensively, they looked for a private sector organization to take over this role.

The NTIA first made a false start when they chose a network consulting firm to handle domain registrations. After a couple years of performing well, they one day decided it would be a great idea to email organizations that requested domain names to see if they’d like some of their other services; that email set off a firestorm, and the NTIA looked for a different organization to take over domain registrations. Finally, they turned the business over to the Internet Assigned Numbers Authority (IANA), which remains in charge of assigning domain names to this day.[ii]

Our group is now in the process of enumerating both problems with the NVD as it exists today and opportunities it could have in the future, whether or not it remains a part of the US government and whether or not it retains the NVD name. We have some ideas already, but we’re looking for others. If you have anything you would like to contribute to this discussion, either with a comment or by suggesting a text edit or addition, please go here. You can contribute either using your email address or anonymously. We would prefer the former, but we want most to hear what you have to say, no matter how you say it.

Once we have our list of problems and opportunities together, we’ll make that publicly available. We’ll also start discussing how those problems and opportunities can be addressed, both in the short term and the long term. You’ll be welcome to participate in that as well.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

After I put my most recent post on LinkedIn, noted ICS security guru (and S4 founder) Dale Peterson made a good comment, which turned into a chain of comments and replies between the two of us. Because I think the conversation was valuable, I’m reproducing most of it (along with commentary of my own in italics) here.

1.      I had stated in my first post on this topic that I basically agree with the statement on page 20 of the White House cyber strategy document, which strongly implies that suppliers shouldn’t be able to “fully disclaim liability”. Or at least, I agree that any such disclaimer shouldn’t be buried in a 100-page agreement the user has to sign before they can use the software at all. However, if the supplier feels they can’t even offer the product to the public unless they disclaim or limit liability and they provide a separate agreement just addressing liability, I think they should be able to do that. If the user doesn’t want to sign away liability even if that means they can’t use the product, they’ll be free to do that. At the same time, the supplier will need to refund any money the user has already paid, if they won’t sign the separate liability provision.

2.      Soon after the second post (my most recent one) appeared, Dale made a general comment and I replied, “What's amazing is that the author of this section thinks it's even possible to ‘shift liability’. In our legal system, liability is established by a court case. The government doesn't have any power to decide liability beforehand.”

3.      Dale replied to my comment that, “I wouldn't go that far. Congress can pass laws that shift liability. The much in the news Section 230 is a great example of this. (It’s) Less clear that the Executive Branch can shift liability, except perhaps via DoJ bringing cases to court.”

4.      I replied to Dale, “I’m sure you know better than I do, but it seems to me that liability is always a question of the circumstances of each case. Here, we're talking about cyber breaches. Can even Congress pass a law that says the supplier is the one liable in every breach, or at least that the default assumption is that they're liable?” I added, “I'm sure the executive branch can't do that, although I know a lot of presidents have tried.”

5.      Dale replied, “Channeling my Schoolhouse Rock ... Congress passes the bills, Executive signs them, Judiciary judges them. Section 230 is a good example. Congress passed and Executive signed the Communications Decency Act. Section 230 provides immunity to liability for online computer services with respect to third-party content generated by its users. Now there are cases that have made it all the way up to the Supreme Court saying Congress can't free them from this liability.
Would it be wise for Congress to pass something that will be struck down by the courts? Of course not. I think your points in the article were quite good and hadn't really been discussed post Strategy announcement.”

6.      I replied, “Thanks, Dale. However, in this case the WH seems to be assigning liability to the software suppliers, not removing it. That strikes me as something almost like a bill of attainder, which was explicitly prohibited in the Constitution. The suppliers can be held liable now, of course (although it’s important to remove those buried disclaimers of liability in the usage agreements we all sign without reading, or at least declare those unenforceable), but there needs to be a trial to determine that fact. The WH seems to want to skip the trial altogether, and proceed right to the sentence.”

7.      Dale replied, “A lot of what is in the strategy requires legislative action and the Biden administration is aware of this. You see this in the text and in the briefings they have been giving. Even a lot of the regulatory things they want to do will require Congress to give the Executive branch more regulatory power.”

8.      I replied to Dale, “There's a difference between regulations and liability. Absolutely, Congress can give CISA (since that's the agency that's behind this section, I'm sure) regulatory authority. But Jen Easterly has said that CISA isn't a regulator - and I think that's a good position for them to take.
However, it seems like CISA is trying to develop a back door for themselves by saying the suppliers are by default liable for breaches, and then hoping this will scare them into good practices. The problem with this approach is that, instead of reasonable regulatory fines, now the suppliers will be subject to potentially huge damage awards for large breaches.
I'm not very worried that this will happen, of course, since it will have a huge inhibitory effect on software suppliers and Congress will never approve it. So, if CISA wants to use the coercive approach, they should just become regulators and be done with it.
But since CISA doesn't want to be a regulator, they shouldn't then turn around and be an executioner. They should figure out positive incentives for suppliers - but also for users, since they're as often the cause of breaches as the suppliers are.”

9.      The next day, I made another comment, “I do want to add that I don't object to a supplier having to pay a huge damages award, if they're found in a trial to be liable for an especially serious breach. What I do object to is the government's putting their thumb on the scales of justice, so that the liability will be determined without a trial at all. That's the ‘solution’ adopted by the cowboys in my post.”

That’s where the conversation ended. I think Dale and I are in agreement on all points, to wit:

A.     Whatever the WH wants to do in this section, it will require Congressional action. And since I think it would be hard to get Congress to name a post office after George Washington nowadays, I’d say the chance of such action is effectively nil. So this is currently a moot point.

B.     There’s a difference between regulation and assigning liability. Neither of us thinks regulation should be ruled out, but (and this is my opinion, since I haven’t discussed it with Dale) I think the only type of regulation that makes sense, when it comes to cybersecurity, is risk based: i.e., a requirement that the entity should “identify and assess” the risks they face regarding a particular domain like supply chain security, then develop a plan to mitigate the most serious risks – and follow that plan.

       Since 2013, I’ve probably written at least 4-500 posts on the NERC CIP cybersecurity requirements, which used to be oppressively prescriptive but now are entirely risk based (or at least the new requirements are risk based. Unfortunately, the prescriptive requirements are still mostly on the books). For example, CIP-013, the supply chain cybersecurity risk management standard, is entirely risk based (perhaps to a fault, since it provides far too little guidance on what it means to comply with it, IMO).

C.      But assigning liability up front, so that the supplier is assumed to be liable (and perhaps for a lot of money), unless they can prove they’re not liable, is pure overreach, and frankly Orwellian to boot.

Last August, I wrote a postthat described another bit of Executive Branch overreach, in that case rolling out some “voluntary” cybersecurity requirements for critical infrastructure, while at the same time making it perfectly clear that critical infrastructure operators had no choice but to follow those requirements (of course, since there are hundreds of ways that the federal government touches any private organization every day, this is without doubt a threat to make life difficult for them if they didn’t do what they were being told to do). I started by saying,

Bad things happen when government agencies try to take the easy route, rather than the right one, to achieve their goals. And if the agency is trying to get private organizations to do something that they just know is the right thing for them to do, they’re even more tempted than normally to take the easy route. After all, their goals are righteous! How can anybody complain if they’re just doing everything they can to achieve those goals?

This paragraph applies perfectly to what Dale and I discussed on LinkedIn.

My former economics professor Milton Friedman, when the Ford administration had just put in place wage and price controls, said in class, “Now we have ‘voluntary’ wage and price controls – meaning, ‘You’ll voluntarily do this or you’ll get your head cut off.’”

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

There’s been a lot of discussion of my most recent post on LinkedIn, as well as of a similar post by Jeff Williams of Contrast Security. Jeff is best known as the originator of the OWASP Top 10, but is now known to me as a player on a men’s over-50 basketball team that has in the past won the national championship and has also beaten the Mexican national team (although they didn’t win this year). Who would’ve thought a basketball star would have a sideline in software security?

But I digress. In the process of replying to the comments on my post (and replying to my replies), I’m now able to articulate my position on this issue more clearly:

1.      The White House is understandably concerned about the increasing volume of software supply chain cyber attacks, many of which are due to an attacker exploiting a vulnerability found in a software product used by the organization attacked.

2.      What remedy does the White House propose for this problem? In the section entitled “Strategic Objective 3.3…” of the White House cybersecurity strategy document released last Thursday, the remarkable statement is found, “We must begin to shift liability onto those entities that fail to take reasonable precautions to secure their software…” Of course, this section makes it clear that “failure to take reasonable precautions” is a fault found only in the supplier of software, not the consumer of the software, not the maker of a security tool that didn’t detect the vulnerability that caused a breach, etc. This section says nothing about apportioning liability, or about anything other than assigning 100% liability to the supplier.

3.      However, suppose that a software developer followed poor development practices and included in their product (during development) either a certain sequence of code that was known at the time to constitute a vulnerability with some non-zero likelihood of being exploited, or a third-party component that contained code that constituted such a vulnerability.

4.      Moreover, suppose that, due to ignorance or deliberate negligence, the developer didn’t patch this vulnerability (or replace the vulnerable code or component) before they shipped the product to end users. Whether or not the developer knew about the vulnerability, they should have identified it in testing before they shipped the product. After all, it was a known vulnerability. I would consider this fact to be clear evidence of negligence on the developer’s part, although that’s not the same thing as liability, of course.

5.      Now, let’s suppose that, some years after the supplier shipped this vulnerable product to customers, one of those customers (“Customer A”) was breached because the vulnerability in question had been exploited by a hacker. The breach caused serious damage to Customer A, perhaps because the attacker used the breach to plant ransomware in the customer’s network.

6.      Sometime after they were breached, Customer A filed a lawsuit against the developer, saying their poor development practices were the cause of the breach; therefore, liability for the consequences of the breach (perhaps a big disruption to their business, caused by the ransomware incident) rests 100% with the developer. They point to this section of the 2023 White House cybersecurity strategy document in support of their assertion that the developer is 100% liable. Sounds like an open-and-shut case, right?

7.      If I were the judge in this case (say it was a bench trial), would I rule that the developer was indeed 100% liable for the damages suffered by Customer A? I would probably do that, as long as no convincing evidence to the contrary were presented by the developer.

8.      However, suppose that the developer defends itself, because they don’t believe they’re liable at all (of course, if they thought they were liable, they would presumably have settled the lawsuit before it ever went to trial). They admit they erred in not doing adequate testing for vulnerabilities before they shipped the version of the product that contained the vulnerability (i.e. the version that Customer A purchased).

9.      However, they point out that, a year after they shipped the product to Customer A, they discovered the vulnerability in the product and immediately produced a patch for it. They notified all their customers that the patch was available, including Customer A.

10.   Customer A never applied the patch that addressed this vulnerability. In fact, they didn’t apply any patches to the product for the next four years. Since the developer produces what are called “cumulative” patches – which include the contents of previous patches – all Customer A would have had to do, in order to be protected against the vulnerability that ultimately brought them low, was to apply any oneof the patches released in those four years. But they never applied a single one.

11.   Of course, four years after the developer discovered the vulnerability and issued a patch for it, the customer’s instance of the product was breached, and the rest is history. The developer, since the customer is suing them for a lot of money, has developed evidence showing that many of their other customers, who had applied the patch, were also attacked by the same ransomware group that breached Customer A. However, none of those other customers were breached, because the vulnerability was patched in their instances of the product.

12.   Given this evidence, would I, the judge, now a) dismiss the lawsuit with prejudice (meaning the suit couldn’t be refiled), since in my mind, the developer is clearly not liable at all (perhaps I would also require the plaintiff to pay all the defendant’s legal costs)? Or would I b) find for the plaintiff, because in my mind none of the evidence the developer introduced matters – i.e., the developer is in fact 100% liable (perhaps the strategy document from 2023 would influence my judgment)? Or would I c) dismiss the lawsuit without prejudice, meaning that the plaintiff could still refile the suit if they discovered stronger evidence of the developer’s liability?

13.   Frankly, any of these three outcomes is possible, and there are plenty of other possible outcomes as well. The point is that it would be quite unusual if the case were so open-and-shut that the suit would either be completely successful (option b) or a complete washout (option a).

In fact, if assignment of liability were as clear-cut as the White House strategy document makes it seem, there would never be a need for any trial regarding liability, at least in software breach cases. All Customer A would have to do is file some sort of notice that they use the developer’s software and they were breached. At that point, the developer wouldn’t be able to do anything other than open their checkbook and ask how big a check they need to write.[i]

However, it now seems the White House (and perhaps DHS) thinks there’s no need at all for court cases to determine liability for software breaches. As far as they’re concerned, the outcome of these cases is determined from the start.

This is like the story of two cowboys that captured a cattle rustler and were starting to hang him. One of the cowboys had a sudden attack of conscience and asked, “Shouldn’t we give this man a fair trial?” The other cowboy exclaimed, “You’re right! We’ll give this man a fair trial…”

“…and then we’ll hang him.”

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

The White House published the National Cybersecurity Strategy on Thursday and it understandably received a lot of attention. I think it’s an excellent document and very well thought out (I especially liked the statement that the government will “promote the further development of SBOMs”). However, there’s one section I find to be…well, let’s say I find it lacking in sound reasoning. That’s the one on pages 24 and 25 entitled “Strategic Objective 3.3: Shift Liability for Insecure Software Products and Services”. The contents of this section were essentially given a road tryout by Jen Easterly, Director of CISA, in a speech at Carnegie-Mellon University on Monday; the speech was reported on in thisarticle in the Washington Post on Wednesday.

To be honest, I don’t even object to all the language in this section. For example, I completely agree that software suppliers should not be able to disclaim all liability for defects in those 100-page “contracts” that nobody reads but everybody has to sign, if they want to use software at all.

However, what I do object to is the implication in that section (as well as in Ms. Easterly’s comments on Monday) that when there’s a cyber breach, the default assumption will always be that it’s the software developer’s fault. This is because many developers “ignore best practices for secure development”, “ship products with…known vulnerabilities”, or “integrate third-party software of unvetted or unknown provenance”.

The best articulation of this attitude is found in this WaPo articlepublished after the strategy was released, which says a senior administration official (who of course needs to remain anonymous, since otherwise they’ll be punished for telling the truth) “told reporters Wednesday that the proposal would be to place liability ‘where it would do the most good,’ primarily ‘the company that is building and selling the software.’”

Let me rephrase this to make clearer the meaning of this remarkable statement: ‘We’re not concerned with determining where to place liability in the sense of “Whose fault is this?”; that’s so 20^th Century. Instead, what we’re really concerned with is ‘Who will be most susceptible to being pressured to take the fall for the breach, since we’ve already said up front that they’re the culprits?’ I commend this candor, although it would have been even better if the official had identified his or herself.

Of course, I certainly don’t deny that there are developers who violate best practices. But it surprised me that, given that whoever wrote this section of the strategy was clearly quite concerned about assigning liability for future cyber breaches, there was no mention of other entities that might share liability. First and foremost, how about users that aren’t applying patches, configuring their firewalls properly, or investing a nickel in training their security staff (which may consist of the president’s teenage son, who comes over when he’s done with soccer practice)?

But let’s not stop there. There was also no mention of cloud providers who make their customers be responsible for their own security, even though many of them clearly don’t understand how security works in the cloud (here’s Exhibit A in that regard). Or of cloud providers who gladly allow new, untested companies to sell access to apps on their cloud without paying too much attention to pesky little questions like...you know...do these guys, who were peddling mortgages a year ago, have the slightest idea how to do this securely?

Finally, there was no mention of a national government (I won’t tell you which national government I’m talking about, but it’s not Estonia’s) that invested huge amounts in Project Einstein, the 21^st Century version of the Maginot Line. Project Einstein was designed to secure the US from foreign cyberattacks. And to that government’s credit, it succeeded in protecting the US from Russian cyberattacks about as effectively as the Maginot Line protected France from the German armies in 1940 – that is, not at all.

Undeterred by press releases, the Russians bypassed Project Einstein by setting up all the servers they needed to carry out their tremendously well-executed attack on SolarWinds at US-based cloud providers, all completely within the borders of the good old US of A! At least the French can use the Maginot Line fortifications as a tourist attraction today, which is more than we can say for Project Einstein.

I could go on, but you get the idea: The liability for almost any cyber breach can be traced to thousands of clueless individuals in all walks of life. If you wanted to assign liability properly, you’d have to trace down all these many individuals and spend a year or two figuring out exactly how much of the bill each of those parties is responsible for. Then, you’d have to get each of them to pay their fair share.

But it’s so much easier if you just say the software developer is responsible. That way, you can be home in time for dinner with the family.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

In my last post, I described an important issue regarding VEX that hasn’t yet been discussed in any published document (but should be): the question of how to handle the fact that how a user interprets a statement that a vulnerability is exploitable in a product or not (which of course is the purpose of VEX) is going to depend heavily on what kind of attacker they think they face. This is important, since if the user’s assumption on hacker skill level diverges substantially from the supplier’s assumption, they may draw exactly the wrong conclusion from what the supplier intended, when they designated a vulnerability as exploitable or not in their product.

There is another issue, which like the first hasn’t been discussed in print at all but is also important (although it was discussed a good deal within the NTIA-now-CISA VEX working group, as was the hacker skill level issue): the question of how the supplier, in preparing a VEX statement, should account for the environment in which it’s installed, especially any security controls. Here are some forms in which this question comes up:

1.      Say a vulnerability is normally not exploitable unless a default configuration setting is changed. If a user makes that change (either deliberately or inadvertently), does that make the vulnerability is exploitable in the product, meaning its status in a VEX statement should be “affected”?

2.      Now let’s say a vulnerability, which would otherwise be exploitable, will be rendered not exploitable by a product configuration setting recommended in writing by the supplier (but not made part of the default configuration). Should the supplier list the VEX status of that vulnerability in the product as “affected”? After all, the user might not follow their recommendation.

3.      If the supplier recommends their product not be connected to the internet and says a particular vulnerability isn’t exploitable, yet a user connects the product to the internet anyway and gets compromised through an exploit of that vulnerability, does this mean the vulnerability was in fact exploitable?

4.      What if the supplier says a vulnerability isn’t exploitable if their product isn’t connected to the internet, yet because of how the product operates, it would be difficult to use the product efficiently if it weren’t connected? If the vulnerability is exploited and the user suffers some loss because of that, is this the supplier’s fault or the user’s? After all, the supplier said the vulnerability isn’t exploitable if the product isn’t connected to the internet.

5.      What if a supplier recommends deploying their product securely but doesn’t describe what that means, and a user mistakenly doesn’t put the product behind a firewall? If the product gets compromised because of the lack of a firewall, is this the supplier’s fault for not explicitly stating the need for one?

6.      What if a supplier assumes that users will normally install a web application firewall, yet doesn’t state that explicitly in their installation instructions? A user has a normal stateful inspection firewall but not a WAF, and they get compromised through a vulnerability that the WAF would have protected against. Is this their fault or the supplier’s? Or is it neither’s fault?

7.      What if the supplier has issued a patch for a vulnerability? Is the vulnerability exploitable or not?

Here’s what I think the rule should be for cases like this (and they come up all the time): The supplier should assume that a) The user has followed any configuration or environment settings the supplier has recommended in writing, and b) the user hasn’t changed any default settings, and c) The user has followed basic security best practices (although the supplier doesn’t have to state exactly what they think those best practices are).

However, the supplier should, whenever they think there might be any question in the user’s mind about how the product should be configured, or what would constitute a security best practice in a particular case, always take an extra step and state explicitly what they have in mind. Here is how I would answer each question:

1.      A default configuration setting constitutes a very strong recommendation from the supplier that they want the user to leave it in place, meaning the supplier is justified in believing that the user won’t change the setting. Thus, the vulnerability would have a “not affected” (i.e., not exploitable) VEX status. However, it certainly might save both the supplier and their customers a lot of headaches if the supplier points out explicitly in their documentation that the configuration setting is required to prevent exploitation of that vulnerability, and if the user changes it, they risk being compromised because of their action.

2.      This is a lot like the first question, but the difference is that in this case, the configuration setting is just recommended, not made part of the default configuration. Nevertheless, in my opinion, the supplier is justified in assuming the user has followed the recommendation. Therefore, the VEX status should be “not affected”. Again, it would help if the supplier explicitly pointed out in their documentation that changing this default setting might make them vulnerable to an attack.

3.      This is in effect the same as the second question. The supplier has made an installation recommendation and is justified in assuming it will be followed. Again, the status for the vulnerability should be “not affected”.

4.      This is superficially the same as the third question, but is in effect very different. I would hope that no supplier would take such a cynical approach as to require the user do something that makes their product unusable, but I’m sure there are a few that can and will do this. In this case, the supplier is assigning a “not affected” status, but they’re basing that on the assumption that the user won't use the product as it's intended to be used - which is obviously horse manure. A customer who sees something like this happen more than once might want to re-evaluate whether they wish to stay with this supplier. Who knows what other shady steps they’re taking?

5.      In my opinion, a firewall is basic security best practice. No supplier should have to tell their users to put in a firewall. Therefore, the supplier should say the vulnerability isn’t exploitable (“not affected”).

6.      I don’t consider a WAF to be a basic security best practice. If the supplier thinks one is needed to protect their product, they should state it explicitly, not just assume it. Therefore, the supplier should list the status of the vulnerability as “affected” and list the WAF as a mitigation for the vulnerability.

7.      It’s important to keep in mind that the question is never whether the product itself is vulnerable, but whether a particular version of the product is vulnerable. If Product A version 2.0.0 is vulnerable to CVE-2023-12345 but a patch has been issued for that vulnerability, the patched version must have a new version string, say v2.0.1. Moreover, if the user applies the patch to v2.0.0, the version of their instance should change to 2.0.1. This is required both for VEX to work and for support reasons, since if multiple codebases have the same version number, the product becomes unsupportable. So, the status of v2.0.0 should be “affected”, but the status of v2.0.1 should be “fixed” or “not affected”. The mitigation that goes with the “affected” status for v2.0.0 might be “upgrade to v2.0.1” or “Apply patch XYZ, available at (URL)”, or perhaps both.

This issue, as well as the hacker skill level issue that I discussed in my previous post, is one that needs to be discussed and documented. With both issues, there isn’t one “right” answer, but there does need to be some articulated position, so that both suppliers and users will interpret the status settings correctly. It’s to address issues like this that a VEX playbook is needed. It might be developed by the CISA VEX working group; it might be developed by the SBOM Forum; or it might be developed by some other group where people discuss issues with SBOM and VEX. But it needs to be developed. 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

The informal group I formed last spring to discuss problems that are holding back widespread (or even narrowspread) distribution and use of SBOMs has been focusing on VEX recently. This isn’t an accident, since I (and many others) now believe that the biggest factor inhibiting the success of SBOMs is the fact that VEX is such a poorly understood concept – mainly because there’s so little documentation of VEX, and what little there is contradicts itself. For example, the recently approved, but not yet published, VEX “Minimum Requirements” document directly contradicts the previous CISA Use Cases document on several key points, without even mentioning that fact. The user is left to discover this contradiction on their own.

The idea of VEX came about to address the case in which a user has received an SBOM and identified a number of vulnerabilities in the components listed in the SBOM, by looking them up in the NVD or another vulnerability database. Since there’s broad agreement in the software security community that over 90% of vulnerabilities in components of a product aren’t exploitable in the product itself, the supplier needs to have a means to let their customers know about those non-exploitable vulnerabilities, before they inundate the help desk with calls about both types of component vulnerabilities. VEX is a tool they can use to do that (although I and other members of the SBOM Forum now believe that VEX, as currently conceived, will never be able to accomplish this purpose. We’re now discussing how the VEX “delivery vehicle” can be improved to make it an effective one, although we’re not trying to change the basic use case for VEX itself).

One of the big issues that has come up multiple times in the SBOM Forum discussions (which also was discussed multiple times by the NTIA VEX Working Group in 2021. We thought we’d resolved the question then, but we didn’t put it into a document. That turns out to have been a big mistake) is how the supplier accounts for the attacker’s skill level in a VEX document.

The problem is that the supplier’s decision whether a particular component vulnerability is exploitable or not within one or more versions of one of their products will depend heavily on what they assume to be the likely skill level of the attacker. Consider three cases:

1.      The attacker has my skill level, which is somewhere around that of a (poorly) trained monkey. Using this assumption, almost no vulnerability should be considered exploitable.

2.      The attacker has a moderate skill level, meaning that vulnerabilities whose exploitation doesn’t require a “super hacker” should be considered exploitable, while others are not exploitable.

3.      The attacker is a super hacker, who could figure out a way to compromise a brick and make it do back flips.

Ruling out case number 1 as not at all realistic leaves cases 2 and 3. Which one of these should the supplier assume? The decision the VEX workgroup made in 2021 was that it doesn’t make sense to assume anything other than an attacker of average skills (with the understanding that none of these three levels is defined exactly and it would be impossible to do so). The problem with assuming a super hacker is that almost any vulnerability will be exploitable, so instead of saying that 90% of component vulnerabilities aren’t exploitable, we’ll have to say that literally all of them are exploitable. The supplier will have to patch every component vulnerability and their customers will have to apply all of them.

So, assuming case 3, there’s no need for VEX. Plus, this means that all the software security professionals who say that more than 90% of component vulnerabilities aren’t exploitable are grossly mistaken (and Chris Wysopal of Veracode was very mistaken when he said at RSA in 2017 that 97% of component vulnerabilities in Java apps aren’t exploitable in the app itself). Obviously, one should never rule out mass delusion, but I’d like more evidence of that before I make this assumption.

Thus, it seems that suppliers should assume that attackers will be of medium skill level, when they’re deciding whether or not to rule that a component vulnerability is exploitable in one or more versions of one of their products.

But here’s the problem: There are a number of users who assume, with a lot of justification, that they actually face the super-hackers. This can’t be dismissed as paranoia. After all, you’re not paranoid if everybody really is out to get you!

It’s not hard to find examples of this. Tim Walsh of the Mayo Clinic regularly states in the SBOM Forum meetings that he and his team face the most skilled hackers. There’s a lot of justification for this assumption, since in the last couple of years, the ransomware gangs have figured out that they’re most likely to get a serious payout if they compromise a hospital, and the bigger the better (although I haven’t heard about big hospital attacks lately, so maybe their defenses are getting better. Sure hope so). Other obvious targets are electric utilities, oil refineries and military contractors (plus the military itself, of course). All these organizations would be quite justified in wanting suppliers to assume a much higher than average attacker skill level when they designate a vulnerability’s exploitability status in a VEX document.

However, it’s out of the question to require all suppliers to assume that all attackers are of the super-hacker level, unless the supplier only sells into one of the high security markets like hospital organizations (officially called “healthcare delivery organizations”, or HDOs, nowadays). After all, if 90% of component vulnerabilities aren’t exploitable by a hacker of average skill level, requiring the supplier to assume that all attackers are super-hackers means they will have to develop ten times as many patches as they would otherwise; and it will require their customers to invest ten times as much time to apply patches to the supplier’s products. Security always comes with a cost, and we shouldn’t require users and suppliers to invest ten times as much time (and therefore money, of course) in security measures, if we’re not convinced this is necessary.

How can a supplier who sells into all markets accommodate both the Mayo Clinics of the world and my local dry cleaners, which is – I’m just guessing here – perhaps not at the top of Vladimir Putin’s target list?

Fortunately, there is a vehicle to address this issue in VEX. It’s been available in CycloneDX for a couple of years, and in the spring of 2012 the CISA VEX working group published a document describing their idea for it. The document is called “VEX Status Justifications” (the CycloneDX capability is just called “justifications”).

Justifications are short, machine-readable explanations of why the supplier believes a particular vulnerability isn’t exploitable in a product/version they use; a justification is only used when the status of the vulnerability in the product is “not affected”, meaning it’s not exploitable. The justifications were developed based on the realization that some software users will not be satisfied with just learning that a vulnerability, in the supplier’s estimation, is not exploitable. Instead, they will want to know why the supplier believes this to be the case, so they can take a more conservative approach to determining exploitability than did the supplier.

For example, if a supplier is producing a CycloneDX VEX document for a version of one of their products and they list the status of CVE-2023-12345 in the product as “not affected”, they should examine the nine options for “justification”:

• code_not_present = the code has been removed or tree-shaked.
• code_not_reachable = the vulnerable code is not invoked at runtime.
• requires_configuration = exploitability requires a configurable option to be set/unset.
• requires_dependency = exploitability requires a dependency that is not present.
• requires_environment = exploitability requires a certain environment which is not present.
• protected_by_compiler = exploitability requires a compiler flag to be set/unset.
• protected_at_runtime = exploits are prevented at runtime.
• protected_at_perimeter = attacks are blocked at physical, logical, or network perimeter.
• protected_by_mitigating_control = preventative measures have been implemented that reduce the likelihood and/or impact of the vulnerability.

If the reason the supplier chose the “not affected” status for CVE-2023-12345 corresponds to one of these justifications, they should include that in the VEX document. However, if none of these describes their reason well, the supplier needs to include a textual description of their reason in the “details” field.

Of course, for my friend Tim Walsh of Mayo Clinic, just seeing any justification for a “not affected” vulnerability status isn’t enough to make him feel good. For example, I know that Tim assumes that Mayo’s networks are already compromised (of course, that’s just a conservative approach. It doesn’t mean he really thinks Mayo Clinic’s networks are any more likely to be compromised than those of any other high-security organization).

If the supplier lists the “protected_at_perimeter” option, my guess is Tim will laugh and say something like “Yeah, right. I’ve got a Russian ransomware gang already inside my network, and I’m going to prevent them from exploiting this vulnerability through perimeter protections!” So, Tim will modify the tool that interprets the VEX document to make it change the status of any vulnerability, for which the “protected_at_perimeter” justification is listed, from “not affected” to “affected”.

Tim might do the same with most of the other justifications as well. However, there is probably at least one justification that he would be willing to accept: “requires_dependency”. This means that the component that is the source of this vulnerability isn’t even present in the product (I assume this would happen when the tool that generates the SBOM mis-identifies a component altogether). He might do the same for the “code_not_present” justification, which means the component is included in the product, but the code that’s the source of the vulnerability isn’t included in the component.

Let’s say that Tim decides to accept these two justifications but reject the other seven. He’ll modify the tooling to change the status of a vulnerability from “not_affected” to “affected” if any of the seven unacceptable justifications is used, while leaving the status at “not_affected” when one of the two acceptable justifications is used.[1]

So, boys and girls, our story has a good ending after all: Tim is content, because he knows he can control the status designations of the vulnerabilities in the VEX documents he receives; they will now more closely reflect his understanding of those designations.[2] Meanwhile, the dry cleaners is content, since they don’t have to lay our serious cash to pay some security person to do ten times as much patching work on their product than would otherwise be necessary.

And they all lived happily ever after.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

My previous post described how the proposed OpenVEX format doesn’t have the capability to address the most important VEX use case, since it doesn’t support version ranges. However, OpenVEX has an even bigger problem with regard to that use case: It doesn’t address it at all. In fact, it seems that Chainguard, the organization that led the development of OpenVEX, doesn’t understand what VEX was meant to do, period.

However, the fact that Chainguard doesn’t understand VEX does not mean that OpenVEX isn’t a worthwhile endeavor; in fact, it seems to me that it fulfills an important need. It’s just that the need it fulfills has nothing to so with either SBOM or VEX.

What is the need that VEX was designed to fulfill? There’s no better description of that than this (unfortunately unpublished) document, which was drafted by Dr. Allan Friedman in 2021. At the time, he was part of the NTIA, although now he’s part of CISA. Here’s my summary of that use case:

1.      When a supplier provides a new SBOM for a version of a software product or intelligent device (hereinafter referred to collectively as “product”) to the customers of that product, the users will probably look in the National Vulnerability Database (NVD) for vulnerabilities that apply to product components listed in the SBOM (there are a number of tools that do this, although the leader is the open source Dependency-Track).

2.      Let’s say a user identifies 100 component vulnerabilities of varying levels of severity (indicated by the CVSS score) and exploitability (indicated by the EPSS score, although this sense of “exploitable” is very different from the VEX sense of the word). Furthermore, let’s say that the user decides that 20 of these have a high enough CVSS and/or EPSS score that they want to pursue them right away. They will jump on the phone and start pressing the supplier on all 20 of these: When will the supplier fix them?

3.      The user will find that the supplier’s help line has a long wait, since a lot of other users of the same product were able to identify the same vulnerabilities in the NVD, and also concluded that they needed to contact the supplier about these vulnerabilities.

4.      When the user finally reaches the help desk, they will already be hot under the collar (a result of their 25-minute wait, no doubt), and will be a little impolite as a result. They’ll demand to know when the supplier will fix each of these vulnerabilities, with “tomorrow” or “the day after tomorrow” being the only acceptable answers.

5.      However, instead of giving them the answer they want, the somewhat weary help desk person will answer – for the 57^th time that day – that there’s no need to worry about 19 of those 20 vulnerabilities, since they’re not “exploitable” in the product itself. This means that, if you consider the component as a standalone product, it’s subject to the vulnerability, but because of how the component was installed in the product (or sometimes because of the fact that it was notinstalled, yet showed up in the SBOM for another reason), the product itself isn’t vulnerable. And since a hacker is seldom able to directly attack just one component of a product, rather than the entire product, they will never be able to take advantage of this vulnerability. In other words, the 19 vulnerabilities the user believed to be exploitable were in fact false positives.

6.      Of course, the user will be glad to hear that the problem is only 1/20 the size of what they thought it was; on the other hand, they’ll be miffed that the supplier didn’t proactively inform them that 19/20 of the vulnerabilities in this product and version weren’t exploitable.

7.      The next day, the user receives an SBOM for a different product from a different supplier. However, the experience is roughly the same: The user identifies around ten important vulnerabilities attributable to components in the SBOM, but when they contact the supplier, they’re told that all but one or two of them aren’t exploitable in the product.

8.      Again, this is good news, but now the user starts to get angry that the suppliers are sending them SBOMs, but not warning them that most of the vulnerabilities they learn about aren’t exploitable. This is wasting a lot of their time, as well as the help desk’s time. Why can’t the supplier let them know about these unexploitable vulnerabilities, without their having to waste all this time to learn about them?

9.      This goes on for another few weeks. Finally, the user decides it’s not worth it to receive SBOMs anymore. Sure, it’s nice that there are so few exploitable component vulnerabilities, but the value of the time they waste in learning this outweighs the advantage of learning about the small number of exploitable component vulnerabilities. The next time a supplier offers to start sending them SBOMs, they tell the supplier what they can do with their SBOMs (it’s not nice, by the way).

When the NTIA Software Component Transparency Initiative began in 2018, the suppliers involved in it (mostly very large organizations) soon came to realize that simply distributing SBOMs to their customers, without at the same time implementing some means to inform them of non-exploitable component vulnerabilities that they were likely to find, would literally be counterproductive. The group decided that the best way to address this problem was to develop a machine-readable document format that could be utilized in conjunction with the machine-readable SBOM format. That format was called VEX – and once the acronym was decided on, the name Vulnerability Exploitability eXchange was developed to go with the acronym.

This is the use case that VEX was designed for; it is still by far the most important one. As you can see, it’s very tied to the idea of software component transparency. VEX only becomes necessary when suppliers start to distribute SBOMs; moreover, suppliers are unlikely to want to distribute SBOMs without a mechanism like VEX being in place, for fear of being overwhelmed with completely unnecessary support calls. Even worse, the suppliers are worried that their customers will waste huge amounts of their own time in calling the help desk, and they will blame the suppliers for this. There’s no doubt in my mind that one of the two or three primary reasons why SBOMs aren’t being widely (or even narrowly) distributed today is that there’s so much uncertainty around VEX.

Is this the use case that OpenVEX addresses? Let’s find out. First, we’ll look at the spec published on GitHub. However, the spec says nothing about the purpose of VEX, only that this spec meets “all requirements of a valid VEX implementation as defined in the Minimum Requirements for Vulnerability Exploitability eXchange (VEX) document published on XXX (sic) by the VEX working group coordinated by the Cybersecurity & Infrastructure Security Agency (CISA).”

This is an interesting statement, since the document they refer to has not only not been published, but the working group only finished with it yesterday, February 13 (the OpenVEX spec was created at least a month ago). And when the working group has finished it, it will probably be reviewed for a couple of months by CISA lawyers and others, so it will undoubtedly be changed further. Since the link provided by Chainguard, http://example.com, goes nowhere, here’s the link to the Google Docs version of the document (I wouldn’t normally link to an in-progress document, even though there are no restrictions on who can view it. However, since Chainguard seems to think the document is finished and published, I figured you at least ought to be able to see it).

Since the OpenVEX spec doesn’t say anything about the purpose of this format, we have to be satisfied with the single “Sample Scenario” that it provides:

“As an example, consider the following evolution of a hypothetical impact analysis:

1. A software author becomes aware of a new CVE related to their product. Immediately, the author starts to check if it affects them.
2. The investigation determines the product is affected.
3. To protect their users, the author mitigates the CVE impact via a patch or other method before the vulnerable component issues a patch.

Without VEX data, users scanning the author's software will simply get a third party alert with no details on how the status is evolving. Most critically, when the product is patched (in #3), the alert becomes a false positive.

To inform consumers downstream of the vulnerability evolution, the author can issue a VEX document (in #1) when the CVE is published to let their users know it is under investigation. In #2, when the product is known to be affected, the author can ship a new VEX document, stating the product is affected and possibly some additional advice, like temporary mitigation instructions. Finally when the product is patched, its SBOM can be complemented with a new VEX document informing it is no longer affected by the CVE. Scanners could consume this document and stop alerting about the CVE as it no longer impacts the product.”

(Audible sigh) Where to begin? Here are my questions:

1.      Why would a supplier advertise that their product might be affected by a vulnerability before they have a patch for the vulnerability? After all, when any document is published, even to a small group of customers that have all signed NDAs, you have to assume it will make it to the hacker community within hours. Wouldn’t it be better to…you know…say nothing at all until the patch (or an effective mitigation) is ready?

2.      The penultimate sentence says, “..when the product is patched, its SBOM can be complemented with a new VEX document informing it is no longer affected by the CVE.” In the world of commercial software that I’m most familiar with, and which I would hope the contributors to this document should know something about, that sentence simply makes no sense. The “product” is never “affected by the CVE”, and the “product” is never “patched”. Instead, one or more versions of the product is affected by the CVE, and one or more new versions contain the patch.

3.      Usually, the user of a supported version is given the option of upgrading to a patched version or applying a patch themselves. If they choose the latter option, applying the patch should increment the version number on the user’s instance, so that if the user needs to contact the help desk later and they’re asked what version of the software they have, the friendly help desk person will immediately know whether or not they’ve applied the patch. 

      The VEX document should list the status of each version that’s supported – whether it’s affected, not affected, or fixed (i.e. patched), or perhaps still under investigation. What good does it do to say “the product” is affected or not affected, unless it doesn’t have versions? And if it doesn’t have versions, then, to repeat my quote in the previous post from Bruce Lowenthal, Senior Director of Product Security at Oracle (and one of the originators of the VEX concept), any time there are multiple code bases that have the same version number (or no versioning at all), the product becomes unsupportable. I’m not sure I’ve ever seen a commercial product that wasn’t versioned.

Most importantly, what does this scenario have to do with the paradigmatic VEX use case above? That use case arises because an SBOM is distributed. However, in this scenario an SBOM is only mentioned in the last step, after the product has been patched. Even though the scenario indicates that three VEXes are distributed (my, this supplier seems to love distributing VEXes! I won’t ask exactly how they’re going to get them to all their customers, but it must be really easy, since they like to produce them so often), only the last one seems to have anything at all to do with the VEX use case I described.

But why, in step 3, would the scanner indicate that the vulnerability is present in the product, when in fact it has been patched? It seems this is because scanners often throw off false positives. When that happens, an OpenVEX document that says the product is “fixed” will certainly be helpful, since it will override this false positive designation. I have no argument with that idea.

And indeed, this seems to be the main use case for OpenVEX: Given that scanners throw off a lot of false positives, receiving documents stating that particular vulnerabilities aren’t in fact exploitable will reduce the amount of time that users waste chasing these false positives (as well as bothering the help desk about them). Indeed, this is a fairly good summary of the use case described by Dan Lorenc, the CEO of Chainguard, in a web presentation to an Open SSF working group on January 25 (more on that presentation, and the misstatements that were made by Dan and a couple of guest speakers, in another post soon).

This is a use case that I’d never thought of, and which – although I don’t know a lot about scanners – seems to be a very useful one. While it has nothing particularly to do with SBOMs or with the use case that VEX was developed to address, I certainly wish Chainguard good luck with whatever business they expect to generate out of OpenVEX (if nothing else, this has probably already generated them a lot of good publicity. Nothing wrong with good publicity, of course).

However, Chainguard is wrong in claiming that OpenVEX is in any way superior to, or even competing in the same league as, the two existing VEX specs: one based on CSAF, the other on CycloneDX VEX. It’s something like an up-and-coming hockey team claiming to be competitive in football with two NFL teams (of course, a good hockey team might have a decent chance against the alleged professional football team in my town, Chicago. Fortunately, they’ll be decamping for the suburbs in the next 4-5 years. Couldn’t come soon enough for me). Guys, you’ve defined a whole new product category: false positive reduction for scanning tools. This is a good opportunity. Don’t waste it by pretending to be something you’re not, which is a VEX format that’s comparable to the two existing formats. It’s not going to end well.

On the other hand, I have no problem if Chainguard wants to retain the “VEX” name. But I’m also fine if they want to name the format “Larry”, “Sally”, the “Magna Carta” or the “Tibetan Book of the Dead”. None of these names are trademarked, as far as I know.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Last week, a big announcement was made by Chainguard, a software supply chain security tools vendor, that they had developed, along with six other organizations, a specification for a new VEX format called OpenVEX. This was followed within a couple of days by a web presentation to the Open Source Software Foundation’s Vulnerability Disclosures Working Group, at which Chainguard’s founder Dan Lorenc, and the two leaders of the CISA VEX working group (one a CISA employee (!), the other now a private consultant) praised the new spec and made a (completely erroneous) derogatory comment about one of the two existing VEX formats. It was also followed by publication of a Security Week interview with Dan Lorenc, which contained at least four total falsehoods, although I doubt Dan even knew they were false.

There are two big problems with the OpenVEX spec:

1.      It’s not “open”, since it purports to be the only VEX format that’s completely “compliant” with the contents of an unfinished document being developed by the CISA VEX working group. Even when it’s finished and published, the document will not in any way be a “standard”, “requirement”, “specification”, “papal encyclical”, or anything else that’s compulsory. I’ll elaborate on this statement in another post, coming soon to an inbox near you.

2.      It’s not VEX, as I’ll discuss in this post and (hopefully) the next one.

If OpenVEX isn’t VEX, what is VEX? It would be nice if there were a published VEX definition or specification, but unfortunately, there isn’t one[i] – and the document that’s still unfinished won’t be anything close to that[ii]. The closest thing to an “official” VEX definition is what’s found on pages 1-3 of the VEX Use Cases document. But unofficially, by far the best “definition” of VEX is the document drafted by Dr. Allan Friedman in 2021, when the VEX working group was still under the auspices of the NTIA. Unfortunately, that document was never published (although I recommended last year to the working group – then as now under CISA – that it be updated and published. It would still be worthwhile to do that, but I’m sure it won’t happen).

Here's my VEX “definition”: VEX is a concept that was developed by members of the NTIA Software Component Transparency Initiative to address a specific problem that was identified as part of that initiative. The problem was that a huge percentage of vulnerabilities (almost certainly over 90%), that have been separately identified in components that are included in an SBOM for a software product or an intelligent device (collectively, a “product”), and which are publicly reported (usually to CVE.org), aren’t in fact exploitable in the product itself.

Therefore, if end users, after receiving an SBOM and identifying vulnerabilities for components, start searching for those vulnerabilities on their networks and contacting the product supplier about them, a huge amount of time and effort will be wasted by both software users and suppliers’ help desk employees in discussing these false positive observations (i.e., more than 9/10 of the time and effort will be wasted). This isn’t a sustainable situation, of course. Suppliers need to have a way of reliably and efficiently notifying their customers of non-exploitable component vulnerabilities in the products they operate. Until they have that, they won’t regularly distribute SBOMs to their customers.

It wasn’t foreordained that the solution to this problem would be a format. Yet, since the issue arose due to the imminence of distributing formatted SBOMs in volume to software customers[iii], it seemed natural that the solution to the problem would also be a format. While neither I nor anyone else questioned this at the time (I attended the second weekly meeting of the VEX working group in the summer of 2020 and have missed few meetings since then), it’s clear to me now that focusing immediately on documents was a mistake. I now think my real-time VEX API proposal is the best way to address the most important VEX use, which I just described. There are still other use cases for VEX documents, but if we can address by far the most important use case – and the one that VEX was designed to address – this year, we will have made a huge leap forward.

However, not only does OpenVEX not address the most important VEX use case at all (or even understand what it is), but it would actually set users back. This is because it will take away what will be a capability needed by probably a majority of suppliers. This is the ability to specify the exploitability status of a vulnerability in multiple versions of a product, not just the product itself – and especially in a range of versions. The reason this is necessary is simple: Almost any commercial product has versions. If there weren’t versions, the product would literally be unsupportable (as Bruce Lowenthal, Oracle’s Senior Director of Product Security, has pointed out regularly in past VEX working group meetings).

True, there are some open source projects that don’t have versions, but both private sector and government organizations run primarily on commercial software (which includes open source software that is supported by a commercial organization like Red Hat™). Essentially, private sector and government organizations want one throat to choke when there are problems; having to beg an open source project team to invest some of their own time to fix a problem just doesn’t appeal to most organizations. Since it’s almost never the case that all versions of a software product have the same status with respect to one vulnerability – i.e., they’re all affected or they’re all not affected – a VEX document that doesn’t support versions is literally useless for the great majority of private and public sector organizations.

However, not only must a VEX format support product versions, but it must support version ranges. After all, an exploitable vulnerability almost always is identified in a product at some point in its lifecycle – very seldom was it "baked into” the product when it was first developed. While it’s essential to make VEX statements like “Versions 2.1 and 2.2 are affected by CVE-2023-12345. All other versions are not affected”, that’s not sufficient. It’s also essential to be able to say, “All versions of Product A before version 6.5 are not affected…”

But OpenVEX doesn’t have a separate field for version numbers. To specify a specific version of a product, you need to construct a synthetic “product”, consisting of the product name and the version number.[iv]That is, in order to say “Version 3.7 of Product B”, you need to create a “product” named “B v3.7” or something like that, and then state the status of the vulnerability with respect to that “product”. Since OpenVEX allows only one product per statement, there needs to be a separate statement for each version.

For example, if you wanted to say that Product C Versions 3.1.5 and 3.2.0 are subject to a particular vulnerability, you would in effect need to write, “C 3.1.5 is affected by CVE-2021-54321” and “C 3.2.0 is affected by CVE-2021-54321”. Annoying, right? Well, now try to say that all versions before version 10.5.0 are subject to CVE-2022-98765. You would have to create a statement for every version.

How hard can that be? Quite hard, as it turns out. Consider that, for VEX purposes, every patch version, every new build, etc. needs to be a separate version number, along with of course every major and minor version. It’s not out of the question that a version range would include more than 1,000 versions. Guess how many statements that would take in OpenVEX? You got it…1,000. Now, how many statements would this take in CycloneDX VEX? One. You would have to write, in effect, “vers:generic/>=1.0.0|<=10.5.0 is not affected by CVE-2022-98765”.[v]

In my (hopefully) next post, I will discuss the second reason why OpenVEX isn’t VEX: the fact that it doesn’t address the main VEX use case.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

The VEX working group is struggling with a document that makes a number of statements about what should be available in a VEX format (“should” isn’t meant in a regulatory sense, although that fact seems to have been lost on some people). In one section, we describe how a product or products can be identified in the format. One of the options that is currently included in the document is the ability to indicate a supplier name (say, ‘Supplier A’), which would be interpreted to mean, “the set of all products or components produced or sold by Supplier A”. Of course, Supplier A is usually the supplier that created the VEX.

The main reason usually cited for including this capability in a VEX format is because suppliers may want to make a statement like, “None of our products is affected by the log4shell vulnerabilities”. Of course, many suppliers said this in human-readable (or human-audible) form at the end of 2021 and the beginning of 2022. I’m sure it was very comforting for customers of these suppliers to hear they didn’t have to spend nights and weekends trying to figure out whether that supplier’s products, which were installed on their network, contained the Log4shell vulnerabilities.

So, isn’t it reasonable to make that same statement in machine-readable form as well? The point of VEX is that it’s a machine-readable document, even though there are few tools now available to read a VEX, and even fewer that can incorporate the information extracted from a VEX into an ongoing vulnerability management toolchain.

However, here’s the SBOM industry’s dirty little secret (well, one of them, anyway): The big problem with machine-readable formats is that they have to be read by…a machine. Unlike humans, machines take everything you feed them literally; common sense is foreign to them (although that’s also true of a lot of people I know). When making any statement in a machine-readable format, you always need to ask, “How will the user’s tooling interpret this statement?” along with its corollary, “Will the user tool arrive at the same interpretation as would an average human reading the document?

After all, just because a statement seems perfectly understandable when written in English, you should never assume it will be “understood” in the same way by a user tool that interprets a VEX document written in JSON.

Upon scanning the VEX statement “None of Supplier A’s products is affected by log4shell”, a user tool that ingests VEX documents will probably:

1.      Set up a rule to scan the supplier name of any product on the user’s network, to determine whether it matches the supplier name listed for the above VEX statement. Of course, it’s important to note there can be a variety of supplier names, even though the person making the above statement has elided that point in their announcement. For example, someone in the NTIA SBOM initiative once counted 27 different names for Microsoft, that were used by Microsoft employees to identify the company they work for.

2.      Apply whatever rule is set out in the VEX statement to each product whose name matches the supplier name in the statement. In this example, assuming the name of the supplier enunciating the VEX rule was “Supplier A”, the VEX statement might be, “No Version of any Product developed or sold by Supplier A is subject to CVE-2021-44228 or CVE-2021-45046”. In practice, the tool that interprets the VEX statement is likely to treat this as meaning that the status of every product/version pair, whose Supplier is Supplier A, is “not affected”.

Sounds simple, right? Yes, it is - to the computer. But think what it means in practice: This condition will apply to every version of every product whose supplier is Supplier A, no matter how long ago the product/version pair was developed. Moreover, it will apply to every version of every product developed by Supplier A that is acquired in the future – until this VEX statement is overruled by a future statement.

Of course, it may be that the supplier that made this statement intended for it to mean exactly that: Every version of every product they have every produced in the past, or will produce in the future, is not subject to the log4shell vulnerabilities.

However, my guess is that if whoever thought this statement is a great idea goes to their legal department and asks if they concur with that assessment, they may hear something different. They’re likely to be asked questions like:

1.      Have you assessed every product we make or have made in the past, and confirmed that it doesn’t contain any of the vulnerable versions of log4j?

2.      Of course, this also assumes that no component on any “level” of the product contains one of those versions…yea, unto the farthest levels. Have you confirmed this for every one of these products and versions, both past and present?

3.      Have you put in place sufficient controls on product development and product acquisition to ensure that no version of any product we develop or acquire in the future will ever contain one of the vulnerable log4j versions?

4.      Similarly, do you have controls in place to ensure that no component of any version of any product we develop or acquire in the future will ever contain one of the vulnerable log4j versions?

Finally, if I were one of these lawyers, I would ask,

1.      Will you personally indemnify the organization if it turns out that any current, past or future product of ours contains a vulnerable version of log4j?

2.      If you answer yes to the above question, is this to indicate you have a good sense of humor?

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Last April, I put up a postabout the fact that Steve Springett’s Dependency-Trackwas being used 200 million times a month to look up components listed on an SBOM in the OSS Index open source vulnerability database; last fall, I updated that number to 270 million. So I was quite pleased today to hear from Steve that the number is now a nice round 300 million per month. In other words, use of Dependency-Track is growing more than 50% a year.

Of course, the point of my post wasn’t so much to laud Dependency-Track (although I do want to point out that D-T just celebrated its tenth birthday. This is quite interesting, because just about nobody was talking about SBOMs ten years ago). Rather, it was to show that SBOMs, even though they’re currently being used almost exclusively by software developers for product vulnerability management purposes (and there are many other tools like Dependency-Track, that are also being heavily used), clearly are needed just as much by their customers.

But the customers aren’t asking for them. Why aren’t they?

Good question…

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

In the SBOM Forum meeting last Friday, we had a lively discussion – nay, argument – on a question that frankly surprised me: Will it take regulation to make SBOMs widely distributed and widely used by private and government al organizations whose main business isn’t software?

It’s quite clear that SBOMs are being widely used by software developers for their own product security purposes. However, it’s also clear that SBOMs are not being widely – or even narrowly – distributed to non-developers[i]. And, while there are certainly a lot of suppliers who don’t want to distribute SBOMs, there are also a lot of users who would like to utilize the information that SBOMs provide, but have no idea how they will be able to get that information from an SBOM when they receive it. This is because there aren’t any low-cost, commercially supported tools (or really any tools at all) that ingest both SBOMs and VEX information and output lists of exploitable component vulnerabilities in a particular product and version.

In the meeting on Friday, a number of well-known, very experienced people in the SBOM space were telling me that SBOMs won’t really be distributed until suppliers are forced to do that by regulations. However, those people say that regulations are right around the corner, so SBOMs are also surely right around the corner.

I frankly don’t know what these people are talking about. I’m optimistic that within two years there will be substantial distribution and use of SBOMs. However, it won’t be due to regulation. It will be due to – dare I say it? – the operation of the free market. Simply put, it will be due to organizations that use SBOMs deciding that a) they want to have the information that can be gained from SBOMs, and b) the tools and/or services needed to obtain that information are readily available to them (which they aren’t now).

Here is why regulations aren’t coming (or at least not in anywhere near the volume or strength required to make a difference), and also why they wouldn’t be needed, even if they were coming:

1.      I, like everyone else involved with SBOMs, thought Executive Order 14028 would be a game changer; it certainly did change the game in terms of awareness of SBOMs. But the date for compliance with Section 4(e) of the EO (which includes the SBOM provisions) was set by OMB for August 10, 2022. On that date, government agencies were supposed to start requiring SBOMs from their suppliers. I haven’t heard of any flood of SBOMs after that date, have you?

2.      After that date passed, expectation grew for OMB’s EO implementation memo in September. But that memo, when it appeared, required every agency to…decide for themselves what if anything to do about SBOMs. Not exactly a game-changer, IMHO.

3.      The memo does require an attestation from the supplier about their software development practices (including SBOMs), but if the supplier attests that they will produce an SBOM and don’t do it, or if they just attest that they won’t produce one at all, there’s no mechanism for an agency to force the supplier to do this. And if an agency were to even consider terminating a relationship with a supplier due to failure to produce an SBOM, the supplier will simply say, “Please show me where in our current contract it says we’re required to produce SBOMs.”

4.      Indeed, the EO did call for changes in the Federal Acquisition Regulation (FAR), which would be required in order to change new contracts. I haven’t heard anything about any changes in FAR being implemented (and I don’t know whether they require Congressional approval if so). And even when the FAR is changed, remember that would only apply to new acquisitions, not to any current contracts. Of course, federal contracts with suppliers are usually multiyear, meaning it will be years before any changes will be implemented due to the EO.

5.      So – and I should have realized this initially – there’s simply no way that the EO is going to make any real difference in SBOMs, unless federal agencies decide they really need SBOMs (or at least data from them) and do more than just ask meekly for an SBOM – then move to the next topic when the supplier says no. In other words, it will be demand from consumers – the agencies – that will drive federal use of SBOM data. The EO certainly will have played a big role in inspiring that demand, but it won’t be the reason that SBOMs are being distributed and used.

6.      If not from the EO, where is federal regulation going to come from, which will force suppliers to provide SBOMs to their users? Of course, there’s the FDA, which was empowered by Congress in December to require medical device manufacturers to meet cybersecurity requirements – including one for SBOMs – in order to be approved to market their devices in the US. In the SBOM Forum meeting, several people pointed to this as a reason why there would soon be a lot of regulations – i.e., other agencies would rush to jump on the FDA’s bandwagon.

7.      But this development isn’t new news. In 2017, the FDA put out a memo saying they intended to require SBOMs (and other cybersecurity measures, to be sure) in the future, without giving a date. The December Omnibus bill finally gave them the authorization to do that. But if other agencies were going to jump on that bandwagon, they would have done it years ago.

8.      However, what other government agency couldjump on this bandwagon? It would have to be one that currently has the power to regulate suppliers, for anything other than product safety. The Nuclear Regulatory Commission (or one of the other nuclear power agencies – there are a few) is one. The military and intelligence agencies are certainly another. Is there any other? Not that I know of. In other words, currently there’s no authority for any government agency to require anything besides product safety of the suppliers of any industry outside of nuclear power or the military/three-letter agencies. Where are all these requirements for SBOMs going to come from?

9.      A supplier to critical infrastructure brought up that sector (or sectors) as a likely subject of mandatory SBOM requirements soon. But what agency is going to promulgate those requirements? Remember, the agency would need to have responsibility for the suppliers to an industry, not just the members of the industry itself. A good illustration is the NERC CIP-013 supply chain cybersecurity standard, which came into effect in 2021 and is the only supply chain security standard that I know of outside of the FDA, military and nuclear power. But CIP-013 doesn’t regulate power industry suppliers, since FERC (which provides the enforcement power behind NERC standards) has no authority at all over suppliers. The utilities are supposed to develop supply chain cyber risk management plans, a big component of which is managing vendor risk, and they can face penalties if they develop a slapdash plan, or if they don’t follow the plan they developed.

10.   Sure, the power industry suppliers have felt a lot of pressure from their utility customers due to CIP-013, and are getting their collective cybersecurity acts together; but they don’t face any direct penalties if they refuse to cooperate at all, unless the utilities take action against them like dropping them as suppliers. As with the EO, CIP-013 will only be effective if the regulated entities (government agencies in the case of the EO, and electric utilities in the case of CIP-013) demand security measures of the suppliers (of course, CIP-013 doesn’t say anything directly about SBOMs, but a utility can certainly require them if they think they’re important. To be honest, I doubt too many are doing that now, although the Edison Electric Institute did put out model contract language recently that includes a modest SBOM provision).

11.   Bottom line: If there’s going to be regulation of suppliers in the US – other than safety regulations – it will have to be from some new agency still to be created by Congress. I haven’t heard the slightest peep about that happening, have you?

However, I am quite optimistic that SBOMs will be widely used (or more importantly, that the data to be derived from SBOMs and VEX information will be widely used) within the next two years - although “widely” is relative. The total “market” for SBOMs is every organization, public or private, on the planet. Ultimately, that’s what “widely” will mean, but I won’t even guess on when that will come into existence. Fortunately, there’s lots of growth opportunity long before that outcome is realized.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

The NTIA Minimum Elements document contains this short paragraph (on page 12, for those keeping score at home):

An SBOM should contain all primary (top level) components, with all their transitive dependencies listed. At a minimum, all top-level dependencies must be listed with enough detail to seek out the transitive dependencies recursively.

Translated into English, this says that an SBOM “must” list all the “first level” components – meaning those that are direct dependencies of the product itself (although remember, this document is just supposed to contain suggested guidelines[i] for the federal government. Exactly what status the word “must” has in that context is debatable).

However, if we can now step into the real world, let me ask how many readers are currently receiving SBOMs from even one of their software or intelligent device suppliers, which list every one of the “first level” components (I put that phrase in quotes, since technically there’s no such thing as levels in an SBOM, but in practice everybody talks about them, including me)? And, even if the supplier swears on a stack of Bibles that every first-level component is listed, are you sure they’re telling the truth?

I didn’t think I’d see any hands, although I’ll point out that’s a trick question. After all, how many users are going to audit their suppliers to make sure they’re including all the “first-level” components in an SBOM? That’s right, nobody is – nor would it make any sense to do that. The only way you could audit the supplier would be to ask them to produce another SBOM – and then you’d have to audit that one, and then…turtles, all the way down.

But the biggest problem with this suggestion (or whatever it is), is that it’s likely to prevent many suppliers from producing SBOMs for their customers at all. After all there are some perfectly good reasons why the suppliers won’t be able to “comply” with this suggestion. Perhaps the three most important are:

First, the contents of the SBOM – including the number and identities of components - will vary depending on when the SBOM is produced: whether it comes from the source code, whether it was created during the build process, whether it was created with the final build, whether it was developed from the actual binary files distributed with the software, etc.

Because components are added, replaced or removed at each stage – and since the components in the SBOM on one day will often be different from the components you’d find the next day, if the software were still being built – the only way the suggestion could possibly have any rigorous (i.e. auditable) meaning would be if it specified exactly the circumstances under which the SBOM was created. And, if there were some way to come back and recreate those exact circumstances later during an audit, which of course there isn’t.

Second, components are notoriously hard to identify with any certainty, because of the naming problem. For vulnerability management purposes, it does the software user no good to learn that a component is found in a product they use, but at the same time not know of an identifier with which they can find that component in a vulnerability database like the NVD.

If the supplier decides to leave that component out of the SBOM (or more correctly, leave it in, but just enter “NOASSERTION” for each field describing the component. NOASSERTION is the SPDX term meaning “I have nothing to say about this field for this component”. In CycloneDX, it’s just a blank line), IMO that is perfectly understandable, and shouldn’t be held against them.

Finally, there are some suppliers (I know auto industry suppliers are a good example of this) who regard the identities of some components (even open source components) in their products as their IP, which would be jeopardized if they revealed it.

Are they correct in holding this belief? Who knows, other than the suppliers themselves?  But here’s the punch line: Would you rather receive “incomplete” SBOMs from a few suppliers or receive exactly what you’re receiving currently - zero SBOMs from zero suppliers? If you prefer the latter, then I recommend you put ironclad language in your contracts requiring complete SBOMs, as described in the Minimum Elements. And give yourself the title, “Director of SBOM Prevention”.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

In 2019, the NERC Supply Chain Working Group published six guidelines on supply chain cybersecurity for systems used for the reliable operation of the North American Bulk Electric System (BES). The papers were developed by separate working groups. I led two of those groups, which produced two of the guidelines.

Last year, we updated (or started to update) all six guidelines, and I led the groups that updated both of those documents. I’m pleased to announce that one of the two documents, Supply Chain Cyber Security Risk Management Lifecycle, was just published. The other one, Vendor Risk Management Lifecycle, is finished, but has to wait another three months before it’s officially approved. In the meantime, I will be glad to send anyone who wants to see it the final draft of the document; just email me.

Two other guidelines were just published: Supply Chain Secure Equipment Delivery, led by Wally Magda of WallyDotBiz LLC) and Risk Considerations for Open Source Software, led by George Masters of Schweitzer Engineering Labs. I want to point out that George is a real master of the subject of securing open source software. I know the guidelines I led are applicable to many industries, not just electric power; I’m sure this applies to the other two documents as well. So you don’t have to work for say Duke Energy to find these helpful. I can assure you a lot of work went into them!

I also want to point out that Steve Springett, Chair of the CycloneDX Core Working Group and one of the most creative (and impactful) people in the world of software supply chain security (including SBOMs, but in no ways limited to them!), will be presenting a webinar on February 1 titled Understanding and Using the CycloneDX SBOM Standard. There’s a lot going on with CycloneDX nowadays, including support for both VEX and VDR (Vulnerability Disclosure Report) – with a new version of the format coming out very soon. I’m looking forward to the webinar!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

One of the fundamental tenets of the SBOM faith is that the developer must produce a new SBOM whenever anything in the code of the product has changed; even if the change is just a patch being applied, there needs to be a new SBOM. I myself have repeated this teaching many times. Of course, it’s also enshrined in the Minimum Elements document.

However, a recent conversation with the Director of Project Security of one of the largest software developers in the US has changed my thinking. There’s something special about patches that makes a new SBOM almost impossible in most cases. The only real solution to the problem is for the developer to do a complete version upgrade whenever there’s been a new patch issued (or even better, they don’t issue a patch; they just require all users to upgrade. But that would mean a lot more work for both the developer and their customers).

Of course, given that nowadays there are very few developers that are issuing a new SBOM with every major version upgrade, let alone with every patch, this isn’t currently a problem. When it becomes a problem, I’ll just count it as an indication that SBOMs have truly arrived – somewhat like the “problem” that too many wealth managers will call you if you just won the lottery. We should all have such problems.

Here is what my friend said:

1.      Typically, a developer will issue multiple patches between version upgrades. The patches usually aren’t cumulative. This means that, if there have been five patches since the last upgrade, applying the fifth patch won’t also get you the previous four patches. If you want all five, you need to apply each one. Of course, when the next version upgrade comes out, you’ll receive all five patches “baked in” to the upgrade. But it’s certainly not a good idea not to patch and to just wait for the next upgrade!

2.      If the developer wants to issue a new SBOM after say the fourth patch, they need to make an assumption about which of the previous three patches were applied by the user. Of course, assuming those patches weren’t automatically applied, it’s certain that different combinations of the previous three patches may have been applied by different customers (and some customers might not have applied any of them); moreover, it would be a huge waste of time for the developer to poll their customers whenever they release a new patch, to find out which of the previous patches they’ve applied.

3.      What should the developer assume when preparing the SBOM? The only thing they can assume is that all possible combinations of the three previous patches have been applied by at least one of their customers. So, completeness requires preparing an SBOM for each possible combination of patches that might have been applied.

4.      The number of ways you can group any number of items is the factorial of the number; three factorial is 3 X 2 = 6. Since there’s also a possibility that a user didn’t apply any of the three patches, this means the developer would have to create seven SBOMs, in order to cover all the possible cases. This will be a lot of work, especially when you consider that the developer will have to develop multiple SBOMs with each new patch that they issue (except for the first patch issued after an upgrade. That’s because there are zero previous patches, and zero factorial is zero).

5.      Seven SBOMs might sound doable to some people; let’s not spend time discussing that question. How many SBOM’s does the developer need to prepare if there have been five previous patches? That’s 121 SBOMs. Seven previous patches? 5,041 SBOMs. And how about ten previous patches? That’s about 3.6 million. Producing that many SBOMs will probably take more than a few weekends, I’m afraid.

Clearly, the developer can’t be expected to produce a new SBOM whenever they release a patch, except perhaps an SBOM that assumes all previous patches were applied. So, please don’t try to force your developer to follow SBOM dogma and produce a new SBOM whenever the software changes. To start, count yourself quite lucky if the developer promises to provide one SBOM with every major version upgrade. Just that will be a lot better than what you have now. Here’s the mathematical rule that allows me to say this: Any positive integer is greater than zero.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Starting in 2017, the FDA has said they intend to require SBOMs for medical devices. Early this year, they released a draft requirement for comment, saying they planned to finalize a requirement and make it mandatory by the end of the year.

However, a few weeks ago, I heard in one of the bi-weekly meetings for the Healthcare SBOM Proof of Concept (which is now conducted under the auspices of the Healthcare ISAC) that the FDA had just announced they were going to postpone enforcement of the requirement until late next year (and moreover, that they weren’t going to allow any changes to the draft requirements, even though they…ahem!...left a lot to be desired). Of course, this was disappointing, since there was a lot of agreement (including among medical device makers) that it’s important to have the SBOM requirement.

However, in the CISA SBOM Onramps and Adoption workgroup meeting this week, Josh Corman announced that the Patch Act had been passed last week in the year-end rush of legislation. This is a piece of legislation that requires the FDA to mandate cyber protections for medical devices. It had initially shown promise of being passed, but was withdrawn without a vote earlier this year. The FDA has 30 days to complete the rulemaking for this (and I hope they’ll change what they proposed last April), which seems to indicate it’s on a fast track to implementation. While the Act doesn’t specifically mandate SBOMs, it lists them as one of the measures that might be required, and the word is they will be required.

This is important for two reasons. One is that this will be the first mandatory requirement for SBOMs anywhere in the world (at least as far as I know). Of course, nobody is advocating such a mandatory requirement for every software product and device in every industry, but there are certain cases where mandatory SBOMs are justified. I would say that medical devices, which literally keep people alive in hospitals, are one of the best of those cases.

However, in my opinion the second reason is even more compelling: It’s that intelligent devices in general, not just medical devices, pose a unique cybersecurity challenge; SBOMs can play a key role in addressing that challenge.

I had never stumbled upon that challenge before I and Isaac Dangana, who works for my client Red Alert Labs(a company based in Paris that focuses on security and compliance for IoT devices), published this article in the Journal of Innovation last summer. While that article was primarily intended to be an introduction to SBOMs for people in the IoT and IIoT worlds who weren’t familiar with the concept, Isaac and I ended up pointing to a serious issue that affects the ability of a user to secure an IoT or IIoT device, or even to learn about vulnerabilities that may affect it. In fact, we ended up making the case – without intending to – that IoT and IIoT devices may currently be less secure, and are certainly much less transparent to users, than are “user-managed” software products (i.e. standard software running on Intel-standard servers).

The problem begins with the fact that IoT and IIoT devices are intentionally sealed. The results of this include the fact that users can’t identify by themselves the software products installed in the device, and they can’t update or patch any of the individual software packages in the device.

Instead, all the software in the device is treated as a single “lump” (to quote a friend who manages cybersecurity for the thousands of medical devices installed at a large hospital chain). The device comes with all the required software installed, and patches or updates to individual software products in the device are held until the next scheduled device update – at which point the entire “lump” is replaced as a unit, often at night via an internet connection.

Of course, for a network administrator who runs him or herself ragged, trying to keep up with vulnerabilities found in the hodgepodge of software products running on the servers they’re responsible for - as well as trying to apply the never-ending stream of patches to software products installed on those servers - the idea that all responsibility for vulnerability management and patching would be completely taken out of their hands and assigned to some gremlins that expect neither pay nor recognition must seem like heaven on earth.

However, this is only heaven on earth if the device user has complete confidence that the device manufacturer will:

a)      Monitor each software product installed in the device for vulnerabilities and “coordinate with” (i.e. bug the hell out of) the software product’s supplier to make sure they expeditiously patch any serious vulnerabilities in it.

b)     Require the supplier of at least every “top level” software product installed in the device to provide an SBOM for their software and update the SBOM whenever they release an update or patch for that software.

c)      Using the SBOMs, track components in each of the software products installed in the device and identify vulnerabilities applicable to each component.

d)     Require each software supplier to provide VEX notifications, which will let them know about component vulnerabilities that aren’t in fact exploitable in the product and prevent their wasting a lot of time chasing down non-existent vulnerabilities.

e)     Whenever a serious vulnerability can’t be patched immediately, provide a notification to all the customers of the device regarding mitigations they should apply to make it less likely that the vulnerability will be used to compromise the device.

f)       If one of the software products installed in the device has proved to be a “problem child”, with multiple longstanding vulnerabilities and no commitment by the software’s supplier to fix those in anything less than the remaining life of the universe, commit to replacing that software within say three months.

g)      When a software product has reached end of life status, commit to replacing it within say six months.

h)     When an open source software product hasn’t been patched or updated for say six months, admit that the product is effectively in end of life status and commit to replacing it within six months - although sooner if there are one or more serious unpatched vulnerabilities in the product.

i)       Require that the supplier of each software product installed in the device either attest they have followed every provision of the NIST Secure Software Development Framework (SSDF), or identify any provisions they haven’t followed and provide an explanation of why they haven’t done so.

j)       Etc., etc.

Of course, the device manufacturer is likely to complain lustily that they can’t stay in business if they’re really forced to do all of the above, with respect to every software or firmware product that’s installed in their device. And believe it or not, I agree that this is too much to ask of them.

However, by the same token, I say it’s too much to ask of their customers that they stay in Fat, Dumb and Happy mode, in total ignorance of any security measures that the device manufacturer took or didn’t take to secure the software within their device. If the manufacturer isn’t willing to do literally everything possible to secure the device (i.e., all of the items listed above), they should at least give their customers the information they need to learn about the measures they have taken, and express willingness to have a discussion with any customer who thinks there are measures the manufacturer should take but hasn’t.

What is the information the end user should have about the device? First, they need at least a “one-level” SBOM, showing every software or firmware product installed directly in the device, along with whatever identifiers are required to look up the software in the NVD or another vulnerability database like OSS Index. If the customer has that much, they’ll be at the same point that most users of “user-managed” software find themselves at, with respect to the software installed on the servers they operate: They’ll know what software products are directly installed on each device (or on each server, in the case of user-managed software), and they’ll be able to investigate vulnerabilities found in those products.

But is that enough? After all, any server administrator today can identify every software product that’s installed on their servers using a tool like PowerShell or even Windows™ Explorer; then they can learn about vulnerabilities in those products by looking them up in the NVD or another vulnerability database. But they won’t learn about vulnerabilities caused by components of those software products if they don’t also have an SBOM for each product; then they can look up each component in the NVD, to learn about its vulnerabilities.

How will the server administrator get an SBOM for one of the user-managed software products? They’ll probably ask the software’s supplier for it, and they’re likely to get it, too (at least one SBOM, anyway). Why is the product’s supplier likely to give it to them? Because they’re a customer, of course.

However, if an IoT or IIoT device manufacturer gives a customer a first level SBOM for the device, the customer will just know the names of the software products installed in the device – the equivalent of what they could learn from running Windows Explorer on a standalone server. The IoT customer won’t know anything about components of those products, unless they have an SBOM from the supplier of each software product installed in the device.

At least, having a first level SBOM, the device customer will know the names of the software products installed in the device and their suppliers’ names. What will they hear when they contact those suppliers to ask for an SBOM? Most likely nothing at all, or perhaps “Get lost”. The problem is that the device customer isn’t a customer of the suppliers of the software installed in the device; the device manufacturer is. It’s very likely that only the manufacturer will be able to get the SBOMs, and perhaps not even them..

Thus, it isn’t likely that IoT or IIoT device customers will be able to obtain SBOMs for software products installed in intelligent devices they own or utilize. Of course, they can beg the device’s manufacturer to please, pretty please get the SBOMs for them – but I doubt that will be a very productive exercise.

But more generally, why should analysis of cyber risks found in intelligent devices be exclusively the responsibility of end users? After all, the manufacturer chose the “first level” software and firmware products that are installed in the device. It’s their job to make sure they’re secure; if they’re not, it’s the manufacturer’s responsibility either to get the supplier to fix vulnerabilities in their software, or to replace the software with a more reliable product.

But how can the device customer even “audit” the manufacturer’s performance in vulnerability management? If the customer receives an SBOM that lists the first-level software and firmware products installed in the device (which will hopefully start happening soon with medical devices), that at least puts the device customer on the same level as the server administrator that identifies each software product installed on their server using Windows Explorer: The device customer will usually be able to learn about vulnerabilities in those products, although they won’t learn about vulnerabilities found in the immediate components (dependencies) of those products.

However, why should the device customer even have to look up vulnerabilities in each of the “first-level” software and firmware products installed in the device? Each customer presumably has exactly the same set of software products installed in their device as any other customer does. If there are say 10,000 customers of a device, why should every one of them have to perform exactly the same set of steps to learn about vulnerabilities in the device, when the manufacturer can perform those steps for all 10,000 customers at once?

And not only can the manufacturer perform those steps themselves, but they should be performing them already. That is, they should be doing it, assuming they care about securing the devices they sell. For example, if there are say 15 software and firmware products directly installed in the device (there may be a lot more, of course), why can’t the manufacturer share with their customers every exploitable vulnerability associated with those products?

Besides reporting each vulnerability to their customers, the manufacturer also needs to register the device itself in the National Vulnerability Database (NVD) and report all exploitable vulnerabilities they identify in the device. This includes both vulnerabilities due to code written by the manufacturer and vulnerabilities due to code written by the suppliers of the third party software and firmware products installed in the device.

When device manufacturers start doing this, vulnerability management for intelligent devices, including medical devices, will be much easier. The customer will just have to search on the device’s CPE name in the NVD, in order to learn about all exploitable vulnerabilities in the device. This is what software users are supposed to be able to do now. Why shouldn’t device users be able to do the same thing?

I hope the FDA includes this in their device security requirements, along with requiring an SBOM for the device.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

The NTIA Minimum Elements Report describes a number of practices regarding SBOMs that might qualify as “best practices" in a lot of people’s lexicons. However, it seems both NTIA and CISA are forbidden from using the superlative form of any adjective, so the report’s title uses the less imposing “minimum elements” phrase.

Despite the fact that the Report states, “The minimum elements should not be interpreted to create new federal requirements”, some software security practitioners – and not just those at federal agencies - have glommed onto them as de facto requirements they need to include in every software contract they write. And indeed, the document makes it more likely that people will make this mistake by the fact that the word “must” (underlined, no less!) is used at various places in the document.

But, dismissing the question whether or not the report really provides “binding requirements”, the more important question is whether the report in fact describes best practices, which might be suited to ask for in software contract negotiations. That is, if the Report says a supplier must do something, should you really try to force your supplier to do it? 

My answer to that question is, “Sure, if you don’t care whether the supplier tells you to go pound sand and find another supplier that can provide as functional a product, with as good support, at an equivalent price.” I say that because, in requiring your supplier to follow all of the provisions in the Minimum Elements Report, you’re really asking the supplier to commit to performing impossible tasks. Here’s one of the more egregious examples, which I know has caused a lot of suppliers heartburn.

Let’s look at the “minimum data fields” in the table on page 9 of the report. Most software security practitioners would probably assume this phrase means that each of these fields must be included in the SBOM, for every component. For three of the fields (the first, sixth and seventh), compliance is trivial, since they only require one response that applies to the entire SBOM. However, for the second through fifth fields, it’s just about certain that the supplier will be way out of compliance if you take the provision literally and require them to provide a response for every field.

Fields 2 to 5 all have to be filled in for every component in the product. Let’s look at the fifth field, Dependency Relationship. Essentially, this refers to a possible “parent/child” relationship between two components in the product (and the product itself is a component, bearing the august title of “primary component”); of course, the “child” component is itself a component of the “parent”. Let’s do some arithmetic regarding this field:

1.      The average software product includes around 150 components, although there are many products that have thousands or even tens of thousands of components.

2.      You may wonder why a supplier wouldn’t be able to provide a response in this field, for any component of their product. After all, isn’t the supplier supposed to know every component in their product? That means they should also know the relationship of every component to every other component – with the three possibilities being child, parent, and not applicable (when neither component is a parent of the other).[i]

3.      Let’s look at “Dependency Relationship”. That sounds simple, right? If a component is included in the software your organization buys, you should not only know how the component is identified (name, version string, etc.), but you should also know what relationship that component has to other components. However, keep in mind that a Software Composition Analysis (SCA) tool will list components in a product, without providing a “dependency graph” that describes their relationships. In order to truly understand all the relationships between components in their software product, a supplier needs to have an SBOM for every component.

4.      To make it easier to describe what we’re working with when it comes to component relationships, people often talk about “tiers” of components within a product. The first tier consists of components that are direct dependencies of the product itself. These are the components the supplier has themselves included in the product. The product is the parent, and each first tier component is a child.

5.      The remaining tiers – second, third, etc. – all consist of components that are children of the components in the previous tier. For example, the second tier consists of components that are children of first tier components. In fact, this isn’t an exact description, since sometimes Component A is a child of Component B, while B is also a child of A – so there’s no good way to define tiers. But, the concept does help in making high-level statements.

6.      In practice, it will usually be extremely difficult to obtain an SBOM for any component that isn’t a direct dependency of the product itself. In other words, while it will be difficult (at least currently) to obtain an SBOM for any component other than the ones in the “first tier” of components, it will be well-nigh impossible to obtain an SBOM for any component on a tier lower than the first one. The main reason why this is the case is that the supplier of the product, even though they’re the direct “customer” of every supplier of a first-tier component, has no direct relationship to any of the lower-tier[ii] suppliers.

7.      How do we estimate the number of components in the product? That’s simple: Every component can be assumed to itself have 150 components, since a component can often be a standalone product on its own. Thus, the product contains 150 X 150 components, or 22,500 in total.

8.      Now, how do we estimate the number of dependency relationships among those components, since that’s what this field is calling for? That’s also simple: Since every component has one “parent” and 150 “child” components, there will be 22,500 * 151 relationships, which is approximately 3.4 million. However, since you can’t count both the parent/child and the child/parent relationship of the same two components as different, this means we have to divide this number by two, yielding 1.7 million relationships.

9.      Thus, in order to completely fill in this field as “required” by the Minimum Elements Report, the supplier of a typical software product or intelligent device has to obtain 22,500 SBOMs and use them to identify 1.7 million dependency relationships. If we assume that obtaining each component’s SBOM (or perhaps creating it using a software composition analysis tool) takes just one hour (a wild underestimate, of course), and if we assume that identifying each relationship takes just one minute (also not at all realistic), this will require 22,500 + 28,050 hours, which equals about four years. Of course, that’s for one SBOM with 150 components, not one SBOM with 10,000 components; using rough calculations, the latter will take…a lot longer.

The moral of this story is that, if a supplier of a typical software product with 150 components commits (perhaps in a purchase agreement) to listing all Dependency Relationships of each component, they’ll be committing to do the impossible. In fact, of six “must” items that I count in the Minimum Elements Report, five of them will almost always be impossible to fulfill completely.

What this means is that, if your organization wishes to start obtaining SBOMs from your software suppliers, you will be shooting yourselves in the foot if you try to require the suppliers to comply with specific rules, including what’s in the Minimum Elements Report.

Instead, if the supplier doesn’t already have a specific program for producing and distributing SBOMs to customers, you need to have a discussion with the supplier about what they know they cando at this point. You can ask questions like,

·        How often can they provide an SBOM? If they can provide one with every major new version, that would at least be a start.

·        Do they have a rough idea of the number of minimum data fields they’ll be able to fill in? Even if they say they can only provide information on a small number of first-tier components, that’s a lot better than not providing any information on any components. Call it a win for now.

·        What format will they use for the SBOM? If they can provide it in one of the two primary machine-readable formats, that is good. However, machine-readable SBOMs, require a machine (consisting of hardware and software) to utilize them. There are literally no examples of what I call a “complete” SBOM consumption tool currently available, although there are a few open source products like Dependency-Track and Daggerboardthat do at least part of what a complete tool would do. Since there are vendors that provide services that utilize SBOMs for vulnerability management purposes, you should explore what you can get from them. You would just need to have the supplier provide their SBOM (and any accompanying VEX documents) to the service vendor.

·        What fields will the supplier include in their SBOM, besides the minimum ones? An SBOM with just the seven minimum fields won’t be terribly useful; there are a number of fields that will be needed for specific use cases. You and the supplier need to discuss what specific fields they will add to the minimum ones. Thiswhite paper from Fortress Information Security (a company for which I consult) contains good suggestions on additional fields you may want to ask for.

A supplier may tell you they can’t provide SBOMs at all. Especially if you’re part of a federal agency, you shouldn’t accept such a statement. If the supplier is concerned about software security, they should be producing SBOMs for their own use, meaning they can’t plead some sort of technical challenge. And if they aren’t using SBOMs now to learn about vulnerabilities in their products, why is your organization still buying from them? There’s no real question in the software developer community nowadays that it’s essential to track component vulnerabilities using SBOMs.

I do want to point out that a number of suppliers are worried about providing SBOMs, since they think they will give away IP if they do. This probably isn’t a valid objection in most cases, but at this point, the last thing you want to do is try to force the supplier to do something they think will jeopardize their business. You should tell them they don’t need to fill in any field for which their answer might contain IP.

At this point, with few suppliers regularly distributing SBOMs to customers, the important thing is to get your supplier to distribute something, even if it consists mostly of empty fields. We can save perfection for later.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Recently, my friend Kevin Perry sent me this link to an article in CyberScoop. It gives a fairly good summary of where SBOMs currently stand: End user organizations (i.e. organizations whose primary business isn’t developing software, which even includes the business side of software developers) aren’t using SBOMs in any substantial way at all. But developers are using them very heavily to manage vulnerabilities found in components of the software they’re developing (the person who wrote the article didn’t know how successful developers have been in using them, since that’s not commonly known. However, it’s very possible that the majority of software developers still don’t utilize SBOMs to secure their products at all).

I’ve compared current SBOM utilization to a string which is lying flat on a table. Many developers, seeing how useful SBOMs have been for them, would like to distribute them to their end users, but the end users aren’t asking for them because they have no idea what they’ll do with SBOMs if they receive them. The developers are pushing on their end of the string, but without the end users pulling on their end, the string is going nowhere.

What will get end users to change their minds about SBOMs, so they start seeing some real value in them? It’s not just persuading end users that SBOMs will help them secure their networks; there’s been enough written about that subject – and attention paid to it - that this can no longer be the primary obstacle for end users, or even a major one.

One real obstacle for end users is the lack of easy-to-use and low cost tools and third party services to “process” SBOMs for vulnerability management purposes. The small number of tools that are available today address parts of the software component vulnerability management process, but what I consider the most important part – passing vulnerability data to the vulnerability and configuration management tools that are now deployed by end users – is currently being addressed by nobody.

However, this problem isn’t a fundamental one. The tools will come when there’s clarity about how SBOMs can be successfully utilized by end users. However, there are still some big obstacles standing in the way of that clarity. The biggest of these obstacles has to do with the fact that, even though there are lots of tools (including open source tools like Dependency-Track) that will look up vulnerabilities for components listed in an SBOM, around 95% of those component vulnerabilities won’t in fact be exploitable in the product itself.

This means that, if the user takes every one of the component vulnerabilities it finds seriously and contacts the supplier’s help desk to find out when they will be patched (as well as looks for those vulnerabilities in the products where they’re supposedly found), 95% of their time (and the help desk’s time) in this effort will be wasted. As Tony Turner pointed out recently, given that security staff are already overwhelmed by the real vulnerabilities their scanners find on their networks, the last thing they need to do is waste time tracking down false positives.

Of course, this problem is precisely the one that VEX documents were supposed to solve. But VEXes are currently going nowhere. No supplier is regularly distributing them to users, whereas, if they were to work as intended, oceans of VEX documents would need to be coming out daily. There are specific reasons why this isn’t happening (like a lack of guidance on how to put a VEX together in either the CSAF or CycloneDX VEX formats). Those reasons do need to be addressed, since there are still important use cases for VEX documents.

However, I now realize – after having participated in just about every NTIA and CISA VEX workgroup meeting since they began in mid-2020 – that VEX as currently conceived (and even including my real-time VEX suggestion) will never achieve the primary purpose for which it was intended: shrinking the user’s list of component vulnerabilities, so that only the 3-5% that actually are exploitable will remain. Given the huge number of VEX documents that would need to be issued in order to achieve that goal (perhaps hundreds or even thousands, for a single SBOM), and the fact that the user will never be able to be sure there aren’t still some component vulnerabilities that aren’t exploitable - but for which a VEX statement has yet to be issued - this approach is simply not a good one.[i]

But the current VEX idea, which I now think is mistaken, didn’t spring out of nowhere; there are two assumptions underlying it that I now believe to be wrong. Only by removing those assumptions will we be able to identify an effective solution to the component vulnerability exploitability problem.

The more important of the two assumptions is that this is fundamentally a user problem. By that, I mean the assumption that managing vulnerabilities due to components in a software product they utilize, is the responsibility of the end user, not the supplier. In this view, it’s up to the users to look up component vulnerabilities after they receive an SBOM, then use the VEX information to winnow them down to just the exploitable vulnerabilities.

However, I no longer believe this is fundamentally a user problem. Users need the information found in SBOMs and VEXes, because suppliers haven’t previously been doing a good job of fixing vulnerabilities due to components in their products. A 2017 study by Veracode said “A mere 52 percent of companies reported they provide security fixes to components when new security vulnerabilities are discovered. Despite an average of 71 vulnerabilities per application introduced through the use of third-party components, only 23 percent reported testing for vulnerabilities in components at every release.” One hopes that this situation has improved in five years, but it’s certainly possible that not even a bare majority of suppliers are doing a good job in this regard now.

Who introduced the components into the products? It wasn’t the users. Yes, the users benefit from lower software costs and improved functionality, achieved through use of third-party components. But the suppliers certainly reap the lion’s share of the benefits of using components. Frankly, they should provide the lion’s share of the effort required to identify and fix exploitable vulnerabilities in those components, both before the software is distributed to customers and afterwards.

Moreover, the supplier’s costs for identifying component vulnerabilities should be close to zero, since they’re already identifying them now. And if they’re not, shame on them – they should be. By instead putting the burden of this on the user, the suppliers are essentially saying, “It’s more efficient to have each of our customers, even if there are thousands of them, perform exactly the same analysis that we’re already performing, than it is for us just to distribute the information we already have.”

Think about it: If done right, each user’s effort to identify component vulnerabilities in a product will yield exactly the same result as would the supplier’s effort, which they should already be performing. Why is it that, if there are 10,000 customers for a product, it’s expected that they should each perform a set of steps that are the same as what the supplier is performing now? Moreover, by intention all those users will at best achieve exactly the same results as the supplier has already achieved on their own. What’s wrong with this picture?

But, as I said earlier, around 95% of the component vulnerabilities that a user or supplier identifies (by searching for the component’s name in a vulnerability database like the NVD) will be false positives. VEX documents are supposed to fix that problem, since they’re intended to let the user (or specifically their component vulnerability management tool, no examples of which exist in complete form at the current time) separate the 5% wheat from the 95% chaff.

However, beside the fact that it’s highly unlikely that VEX documents will ever be produced in the volume that’s required to accomplish that purpose, the bigger question is why there needs to be a document at all. After all, it’s the supplier who knows how their software is put together and in most cases needs to make the judgment whether or not a particular vulnerability is exploitable in their product. If the supplier takes responsibility for identifying component vulnerabilities in their products, they can also easily take responsibility for removing the 95% of component vulnerabilities that aren’t exploitable, leaving just the exploitable 5%. Since the exploitability information will never have to leave the supplier (except to go to a service provider who may be performing this service on behalf of the supplier), it can be communicated using any secure internal method.

In other words, it would be far more efficient, and immensely more effective, if the supplier looks up the component vulnerabilities themselves (or outsources this effort to a third party services provider) and then removes the 95% that aren’t exploitable. Then, they would provide this list of exploitable vulnerabilities to their customers. The customers could search diligently for every vulnerability on the list, knowing that – to the supplier’s current knowledge – every one of them is a real vulnerability in the product.

Of course, anyone who knows anything about cybersecurity will now scream out, “Are you crazy? You want to have each supplier send out documents that list each exploitable component vulnerability in each of their products to every one of their customers? How many nanoseconds do you think it will take before each of these lists shows up on Vladimir Putin’s desk?”

And here’s where the second incorrect assumption appears: It’s the assumption that the only good way to provide this information to customers is to send out a document, even a machine-readable one. Instead, I think this information needs to be provided in a strongly authenticated, machine-to-machine transaction that is initiated by the customer, not the supplier. Moreover, there needs to be strong encryption of the transaction.

And if that isn’t good enough, there are certainly ways to add security to the transaction. For example, the supplier might send – through a different encrypted, customer initiated transaction, perhaps to a different device on the customer end – a list of the component vulnerabilities that can currently be found in the NVD (or other vulnerability database, like OSS Index) for the version of a product that is described in a recent SBOM. This list would consist of the CVE identifier for each vulnerability, as well as a unique short identifier for the vulnerability, consisting of randomly selected alphanumerics.

The list of exploitable vulnerabilities would be sent separately from the above list. It wouldn’t show the actual CVEs, but just the short identifiers for them, along with their status (exploitable or not); of course, this list could still be encrypted. And if that isn’t enough security, I’m sure there are more security layers that could be placed on top of that, when required by the nature or use of the software product.

There’s a lot more that can be said about this proposal, as well as many loose ends that will need to be tied up. I’d appreciate hearing anybody’s comments on this. This idea will be discussed in a more participatory context at some point. In the meantime, I’m not saying we need to drop the idea of VEX documents or of real-time VEX, since there are use cases for both types of notifications.

But I don’t want to play down the magnitude of the problem I’m trying to address. Without at least partial resolution of the vulnerability exploitability problem (whether it’s VEX or something else), SBOMs will never be widely (or probably even narrowly) used.

Period.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

In October, I postedabout the fact that my client Red Alert Labs had become certified as an assessor by the ioXt Alliance, the global standard for IoT security; they’re only the eighth organization to receive this honor. Now, the Alliance has put out a press release to announce this fact:

NEWPORT BEACH, Calif. — Dec. 15, 2022 — The ioXt Alliance, the global standard for IoT security, today announced the addition of Red Alert Labs, a Europe-based Cybersecurity Lab specialized in IoT, to the ioXt Authorized Labs certification program. Authorized labs are the exclusive test providers for the ioXt Alliance and perform all testing required for devices to be certified by ioXt and to bear the ioXt SmartCert label, which provides security assurance to consumers and enterprises.

Red Alert Labs (RAL) is an IoT security provider helping organizations trust IoT solutions throughout their lifecycle. RAL provides comprehensive IoT security by design, risk management, consulting, audit and certification services supported by automated processes. RAL provides assessments and certifications of connected devices based on multiple standards, including IEC 62443, Common Criteria, ETSI 303 645, and NIST 8425. RAL is also involved with the European Union Agency for Cybersecurity (ENISA) to develop the EUCC scheme for ICT products and EUCS scheme for cloud services in the context of the Cybersecurity Act in Europe.

Ayman Khalil, managing partner and COO of Red Alert Labs, said, “Given our experience performing IoT device evaluations and certifications for various standards like ETSI 303 645, we are quite pleased to be working with ioXt Alliance, both for SmartCert certifications and for the upcoming U.S. IoT device security labeling program. IoXt is working closely with NIST, in accordance with the executive order given by the White House, in supporting the development of that program.”

“Authorized labs are important organizations in the ioXt Alliance as they provide ioXt certification testing to ensure devices are secure for consumers and businesses to use,” said Jan Bondoc, vice president of information technology at the ioXt Alliance. “We’re very pleased to welcome Red Alert Labs as an Authorized Labs partner to work with us to advance security in the IoT industry.”

With profile creation by top-tier companies in technology and device manufacturing, the ioXt Alliance is the only industry-led, global IoT device security and certification program in the world. Devices with the ioXt SmartCert label give consumers and retailers greater confidence in a highly connected world.

ioXt certification includes both security controls implemented in a connected device and the manufacturer’s security practices. An example of the former is whether security updates are applied automatically when possible. An example of the latter is whether the manufacturer published a policy to notify customers when support will end for their product.

Besides assessing and certifying connected devices and their manufacturers, RAL helps end-user organizations assess the cybersecurity risks they face from devices they are considering for procurement. After procurement, RAL helps those organizations assess and mitigate security issues identified in devices they use. For example, RAL will soon provide services based on the NIST.IR 8425 cybersecurity framework for connected devices, developed by the U.S. National Institute of Standards and Technology (NIST).

About the ioXt Alliance

The ioXt Alliance is the Global Standard for IoT Security. Founded by leading technology and product manufacturing firms, ioXt is the only industry led, global IoT product security and certification program in the world. Products with the ioXt SmartCert give consumers and retailers greater confidence in a highly connected world. Learn more at ioxtalliance.org.

About Red Alert Labs

Red Alert Labs is an IoT security provider helping organizations trust IoT solutions. An independent cybersecurity lab with a disruptive business offer to solve the technical and commercial challenges in IoT. Its expertise has been recognized by numerous awards. Red Alert Labs is a valued member of IoXt Alliance, EUROSMART, IoTSF, CCC, ACN, SYSTEMATIC, CEN-CENELEC, and ECSO.

I’ve been working with Red Alert Labs for a year and a half, and I can attest that they’re a high quality organization. Note they work with both IoT device manufacturers and end users. In fact, I and Isaac Dangana of RAL wrote an articlethat was published this summer, on why IoT manufacturers need to follow different practices with respect to SBOMs and software component vulnerability management than suppliers of “stand-alone” software. Intelligent devices (especially medical devices) introduce a lot of unique security concerns. Given the rate at which devices are proliferating, there’s lots of work to be done!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

There haven’t been a huge number of startup companies providing services and/or tools to the SBOM community (at least I’ve heard of fewer than I expected to by now, although I’m certainly not in a position to learn about all or even most of them).  However, I understand from what a lot of these entrepreneurs have said that they’re expecting to get a big push from regulations, in some way.

What are these regulations? Some of the ideas I’ve heard are:

1.      Some entrepreneurs think they’re helping software suppliers comply with “NTIA regulations” for SBOMs. However, the NTIA doesn’t do regulations, period. Never has, never will. NTIA’s goal is to get members of an industry together to discuss guidelines that will enable a new technology to take off (one technology they did this for – without writing a single regulation – was DNS, for which NTIA was the first domain registrar. The last time I checked, perhaps five seconds ago, DNS is up and running quite well, thank you). The NTIA Software Component Transparency Initiative ran from 2018 through the end of 2021, and it produced a number of documents of mostly good quality. However, they were produced by different groups at different times, and they focused almost entirely on the needs of software and intelligent device suppliers (which isn’t surprising, given that over 95% of the participants, as far as I could see, were from that side of the industry). The Initiative was quite successfulin spurring production and use of SBOMs by suppliers for their own component vulnerability management purposes, but everyone agrees the focus now has to be on the consumer side – i.e. the 99+ percent of organizations (public and private) whose primary business isn’t developing software but…you know, using software to provide insurance, build cars, provide cybersecurity consulting services, etc. That is, the rest of us.

2.      Other entrepreneurs have pointed to the SBOM provisions in Section 4(e) of Executive Order 14028. Those entrepreneurs say they’re helping federal agencies comply with the EO (although they’re all sure that whatever they do for the agencies will end up being demanded by private industry as well). However, all the EO requires is that agencies start asking for an SBOM from each supplier of “critical software”. The supplier could tell them to go pound sand, but the agency would still be in compliance.

3.      The EO called for NIST (really the NTIA) to develop a list of “minimum elements” for an SBOM. The NTIA did that, but most of the statements in that document are written as “may” or “should”. There are some that use “must”, including this one: “..all top-level dependencies must be listed with enough detail to seek out the transitive dependencies recursively..” This means that all “top-level” components in an SBOM need to have detail on each of their components. But guess what? I doubt any supplier will do this for a while. For example:

a.      There are some products with thousands or tens of thousands of “top-level” components (and BTW, the concept of levels doesn’t have any formal meaning in an SBOM, since Component A can be a dependency of Component B, while at the same time Component B can be a dependency of Component A in the same product. However, there’s a general understanding of what it means). Let’s assume a product contains 1,000 top-level components (dependencies). Each of those will normally have at least 140 sub-components (which is what’s meant by transitive dependencies, although that term applies to all the levels “below” sub-component). This means the supplier of this product will have to contact 140,000 suppliers of sub-components (90% of which will be open source projects, from which the supplier will usually not get any response) to ask for an SBOM, for just one of the supplier’s products. How many of these “second-tier” SBOMs are they likely to get? My guess is they’ll be lucky to get five. So that means our luckless supplier will have 139,995 instances of non-compliance. Talk about a bad day!

b.      For probably at least 95% of the components the supplier lists in their SBOM, the component name originally produced by the tool that created the SBOM won’t be searchable in any major vulnerability database, meaning the supplier will have to use AI, fuzzy logic and other tricks to find it – and in many cases, they may never find the component in a vulnerability database. This is the famous “naming problem”. Thus, just producing a single usable SBOM could be a major project, again for a not-unusual product with hundreds or thousands of just “top-level” components.

4.      The EO required NIST to develop “guidelines” for SBOM production and use by February 6 of this year. NIST did put out a document on that date, but it mainly referred to other documents, produced by NIST and others. There was no set of guidelines in NIST’s document. This wasn’t a surprise to people who understand that NIST doesn’t produce guidelines (let alone regulations) for anything having to do with cybersecurity. NIST – very properly, in my opinion – understands that cybersecurity is a risk management exercise. 

All NIST’s cybersecurity documents provide a framework for managing a particular type of cyber risk, but none of them have ever prescribed certain actions – nor will they ever do that. Sure enough, the documents NIST put out early this year – SP800-161r1 and NIST800-218 (aka the Secure Software Development Framework) - had nice things to say about how SBOMs fit into their framework, but nothing else in terms of prescriptive requirements, or even guidance. I’ve just learned that NIST asked to be removed from EO 14028 compliance responsibilities earlier this year, so perhaps they found this episode a little too traumatizing for their taste.

5.      The most recent dashed regulation hope was the US House version of the National Defense Authorization Act (NDAA) of 2023. A stiff requirement that software suppliers to the US armed services provide SBOMs was removed last week, right before the bill was passed by the House (the bill now needs to be passed by the Senate, which of course is very unlikely to add SBOMs back into it).

There was one more hope of the regulation buffs: The Office of Management and Budget (OMB), which was charged with implementing the EO, had set August 10, 2022 as the date when federal agencies would need to start requesting SBOMs from their suppliers. That date passed without any formal guidance being issued, but “guidance” did come on September 14, in the form of Memorandum M-22-18. Surely, that document set out some concrete requirements for SBOMs, right?

‘Fraid not. The OMB simply said:

A Software Bill of Materials (SBOMs) may be required by the agency in solicitation requirements, based on the criticality of the software as defined in M-21-30, or as determined by the agency. (my emphasis)

 It followed that up by saying:

SBOMs must be generated in one of the data formats defined in the National Telecommunications and Information Administration (NTIA) report “The Minimum Elements for a Software Bill of Materials (SBOM)”. (since just above every SBOM I’ve seen in the past couple of years is in either SPDX or CycloneDX format, it’s clear the market is already requiring this. No government statement was needed).

In other words, any entrepreneur who believes that existing regulations, or coming regulations, will make them rich in the SBOM “marketplace” had better recalibrate their business plan, so it doesn’t depend on regulations at all.

But guess what? I’m quite optimistic that it won’t be hard for these entrepreneurs to recalibrate their plans, since I think an initial wave of SBOM use – which I’ll admit I thought, until this spring, would occur when Section 4(e) of the EO became mandatory on August 10 – is definitely coming. But I’d say the ETA for this initial wave is 2-4 years from now (although I’m sure that at least a few major suppliers will start producing SBOMs with some regularity next year).

My optimism is mainly due to one piece of evidence: In March, Steve Springett, leader of the team that created and maintains the Dependency-Track open source SBOM analysis tool (which is now ten years old) as well as co-leader of the CycloneDX team, learned to his surprise that DT was being used 202 million times a month to look up the components in an SBOM in the OSS Indexopen source software vulnerability database. Moreover, in September he found out that number had jumped to 270 million (he estimates it’s at least 300 million now, which I don’t doubt). This amounts to at least a 50% annual growth rate. In case you don’t know, that’s a lot (and this isn’t individual component lookups – this is just looking up all the components in one SBOM. If you assume that the average SBOM has 100 components - a low estimate - this means there are now 30 billion monthly queries to OSS Index, which are produced by just one software tool (there are a number of other tools that do what DT does).

Dependency-Track has always been aimed primarily at software developers, and while Steve Springett is sure that at least some non-developer software consuming organizations (i.e., any organization whose primary purpose isn’t developing software) are using it to manage component vulnerabilities in third-party software that they operate, he’s also sure there aren’t many such cases.

However, given the rate at which developers are increasing their use of Dependency-Track, it seems to me inevitable that their customers (i.e. users) will want to get into the action as well. After all, if the software gets attacked, they’ll be the ones that pay the price (and speaking of price, what do you think the SolarWinds attacks, which resulted in a number of large federal departments and agencies being compromised for many months, cost the US economy? I have no idea either, but it has to be huge – and we probably still don’t know everything that happened during the many months that the agencies were essentially owned by the Russians).

My feeling is that if Dependency-Track, a supplier tool that could fairly easily be turned into an end user tool, is experiencing such heavy – and rapidly-growing – usage now, it’s inevitable that users will soon say, “Hey, I want some of that.” They’ll start asking for SBOMs from their suppliers (which they’re not usually doing now, unless they’re subject to the EO). When some suppliers start providing SBOMs to their customers, those same customers will ask all their software suppliers to do the same. Soon, we’ll have what’s known in the military as a “self-licking ice cream cone”: a process that sustains itself by its very existence.

When that happens, do I think that SBOMs will start flowing out to end users like water on a desert? They will flow out, but mostly not to end users. I don’t know of a single complete end user tool for consuming SBOMs today - and certainly not one that’s free or low-cost. I doubt this situation will change in the next few years, either. Since I’m sure that end users will need an almost completely automated tool to perform all the steps required for software component vulnerability management, I don’t see them being at all interested in performing these steps themselves (or even stringing together a bunch of open source tools), until such an automated tool is available to them.

But this doesn’t mean that software consumers won’t want to learn about exploitable vulnerabilities in the software they depend on, or that they have to wait for a tool to do this. Rather, I think the initial 2-3 years of SBOM utilization by end-user organizations will occur mostly through engagement of third party service providers, who will provide subscription-based services for SBOM/VEX-based software component vulnerability management. Those few providers will be able to spread out the cost of developing these services over a large number of customers, just like Henry Ford was able to spread the cost of building cars over a much larger number of customers than his competitors could, by implementing efficient assembly lines.

As a matter of fact, I really don’t think that end users should even have to pay the service providers’ bills for identifying and tracking exploitable software component vulnerabilities; this should be the supplier’s responsibility. After all, they’re the primary beneficiaries of the huge increase in use of software components in the last maybe 20 years. Why shouldn’t they also be primarily responsible for mitigating the additional risk introduced by this practice? More importantly, why should every one of a supplier’s say 10,000 customers have to perform exactly the same analysis, to learn about exploitable vulnerabilities in the software they use, when the supplier has already performed this analysis and could easily share the results with all of them?

There’s another big advantage to having third party service providers perform this analysis: the need for the biggest use case for VEX – telling software end users that component vulnerabilities they’ve identified through having an SBOM aren’t exploitable in the product itself – goes away. The supplier and the service provider can make their own arrangements for communicating whether or not component vulnerabilities are exploitable in the product itself; they don’t need to follow any particular format to do that. Sounds good to me!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Today, I was emailing with a reporter about my poston the North Carolina substation attacks, when I saw this articlethat had been linked in the Utility Dive newsletter (which I normally open as soon as it hits my inbox). It seems that NC might not have been an isolated incident after all. You should read the whole article, but IMHO the executive summary is these two paragraphs:

“Power companies in Oregon and Washington have reported physical attacks on substations using handtools, arson, firearms and metal chains possibly in response to an online call for attacks on critical infrastructure,” the memo states.

The aim, according to the memo, is “violent anti-government criminal activity.”

Another:

The department wrote that attackers would be unlikely to produce widespread, multistate outages without inside help. But its report cautioned that an attack could still do damage and cause injuries.

Of course, we’re not talking about multistate outages. A multi-day, multistate outage might be a catastrophe with loss of life, especially if there were a big city in one of those states (see Ted Koppel’s Lights Out, which very eloquently describes what would happen if there were a multistate outage that lasted more than a few days. What’s unfortunate is that Ted let someone persuade him that he should sell the book as being about the effects of a cyberattack on the grid, when exactly the same results would occur, no matter what the cause. The book is an easy read and still definitely worth it, years after it came out).

But an attack that could “do damage and cause injuries” is a good description of what just happened in NC. It certainly caused damage, and people were injured in car crashes, if for no other reason. We may hear later about people on oxygen at home, etc. that were victims as well. An extended power outage is always a big problem.

Finally:

The targets also present an increasing challenge to secure because attackers don’t always have to get as close as they did in North Carolina in order to do damage, Southers said. With the right rifle, skill and line of sight a sniper could take a shot from as far as 1,500 meters (about 4,900 feet) away.

That’s quite interesting. If line of sight is a problem (which it definitely was with the Metcalf attack), then that will require fairly big, expensive fences.

Unfortunately, as I told the reporter today, it will be impossible to prevent attacks like this without huge expenditures (unless there’s a good way to triage substations for degree of risk, which I’m not sure is the case here). One thing I suggested is that, since this is obviously a national problem, the feds should finally step in and pay for the mitigations themselves – rather than dump all the cost on the utilities and especially their ratepayers. This has been for the most part the practice so far, when it comes to both physical and cybersecurity, but it’s time to acknowledge this is a national problem.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.