I’m thinking of renaming this “Mariam Baksh’s Blog”, since once again the NextGov reporter has written a really interesting storyon cyber goings-on in Washington (my last postwas based on one of her articles, and I’ve written a few others before that). Mariam isn’t the kind of reporter whose goal is to write the best story about something that everybody else is also writing about. Instead, she is constantly digging up interesting stories where I would have never even dreamed to look.

The story this time is an outgrowth of the Colonial Pipeline attack. Once the TSA issued a cyber directive to the pipeline industry a few months after the attack, most reporters assumed the problems were solved. After all, you solve cyber problems with regulations, right?

I might have assumed the same thing, except a kind person sent me a redacted[i] copy of the directive, which I wrote about in thispost. What did I think of the directive? Well, I thought the font it was written in was attractive, but it was all downhill from there. However, I have found a great use for the directive: In February, I’ve been invited to speak on cyber regulation for a seminar at Case Western Reserve University. At first, I was going to talk about lessons learned from various compliance regimes: NERC CIP (which I know most about), PCI, HIPAA, CMMC, etc. But after reviewing my post on the TSA directive, I realized I hit the gold mine in that one: Just about everything the TSA could have done wrong, they did. They should stick to telling people to take their shoes off in airports. It’s the perfect pedagogical example of how not to develop cybersecurity regulations!

And maybe they will, since Mariam’s article points out that Rep. Bobby Rush[ii] of Illinois has introduced legislation that would take the job of cyber regulation for pipelines away from the TSA and vest it in a new “Energy Product Reliability Organization”. This would be modeled on the North American Electric Reliability Corporation (NERC), which develops and audits reliability standards for the North American electric power industry, under the supervision of the Federal Energy Regulatory Commission (FERC). NERC is referred to as the “Electric Reliability Organization” in the Electric Power Act of 2005, which set up this unusual regulatory structure.

The best known of NERC’s standards are the 12 cybersecurity standards in the CIP family. And clearly, these standards have a good reputation on Capitol Hill. This was reinforced by FERC Chairman Richard Glick’s testimony at a hearing on Wednesday, when he was asked by Rep. Frank Pallone, “…do you think that the industry led stakeholder process established by Chairman Rush's legislation would likewise be a successful mechanism for protecting the reliability of the oil and gas infrastructure?” Glick replied, “I believe so. The electricity model has worked very well … and I believe a similar model will work with pipeline reliability.”

I don’t deny that the NERC CIP standards have made the North American electric grid much more secure than it would be without the standards. On the other hand, there are some serious problems with the CIP compliance regime, which I wouldn’t want to see replicated for the pipeline industry:

1.      The standards should be risk-based, like CIP-013, CIP-012, CIP-003 R2, CIP-010 R4 and CIP-011 R1. This means they should not prescribe particular actions. Instead, they should require the entity to develop a plan to manage the risks posed by a certain set of threats - e.g. supply chain cyber threats or threats due to software vulnerabilities. Then the entity needs to implement that plan. In drawing up the plan, it should be up to the entity to decide the best way to manage the risks, but there will be a single set of approved guidelines for what should be in the plan (something that is missing with CIP-013). Prescriptive requirements, like CIP-007 R2 and CIP-010 R1, are tremendously expensive to comply with, relative to the risk that is mitigated. Explicitly risk-based requirements are much more efficient, and are probably more effective, since the entity doesn’t have to spend so much money and time on activities that do very little to improve security.

2.      Auditing should be based on how well the plan followed the guidelines. Of course, this isn’t the up-or-down, black-or-white criterion that some people (including some NERC CIP auditors, although I believe that sort of thinking is disappearing, thank goodness) think should be the basis for all auditing. If an entity has missed something in the guidelines, but it seems to be an honest mistake, the auditor should work with them to correct the problem (in fact, the auditor should work with them in advance to make sure the plan is good to begin with. This is currently not officially allowed under NERC, due to the supposed risk to “auditor independence”, a term that’s found nowhere in the NERC Rules of Procedure or GAGAS).

3.      In other words, auditors should be partners. They actually are partners nowadays, but when they do this, they’re officially violating the rules – and note that they’ll still never write down any compliance advice they may give and they’ll always say it’s their personal opinion - meaning you can’t count on the next auditor saying the same thing).

4.      Auditing should also be based on how well the plan was implemented. This is where I think auditing actually should be black & white. Once the entity has created the plan, they need to follow it. If they decide something needs to be changed, they should change the plan and document why they made the change. But they shouldn’t just deviate from the plan as it’s currently written (I believe this is how CIP-013 is audited, as far as I know. The entity can make a change to the plan whenever they want, but they need to document the change, and then follow it).

5.      Identification of new risks to address in the standards needs to be divorced from the standards development process. When a new area of risk is identified as important, entities should immediately be required to develop a plan to mitigate those risks and follow that plan – this shouldn’t wait literally years for a new standard to be developed and implemented.

6.      NERC standards development proceeds in geologic time – and because of that, a number of important areas of cyber risk have never been addressed by CIP, since nobody wants to go through the process of developing a new standard. For example, where are the CIP standards that address ransomware, phishing, and APTs? These risks have been around for at least a decade, yet a new standard has never even been proposed for any of them, let alone written. And how long does it take for a new standard to appear after the risk first appears? The risk caused by use of “visiting” laptops on corporate networks has been well known since the late 1990s. When did a requirement for “Transient Cyber Assets” take affect? 2017.

7.      There needs to be a representative body – with representatives from industry stakeholders, the regulators, and perhaps even Congress and the general public – that meets maybe twice a year to identify important new risks that have arisen, as well as to identify risks that are no longer serious. If the body decides a new risk needs to be addressed, a new standard should be created, the mechanics of which would be exactly the same as in the other standards. Only the name of the risk and the guidelines for the plan would differ from one standard to another. So no new requirements need to be developed.

8.      The unit of compliance for the standards shouldn’t be a device (physical or virtual) – specifically, a Cyber Asset, as it is in the current CIP standards. Instead, it should be a system. As I have pointed out several times before, putting BES Cyber Systems in the cloud (e.g. outsourced SCADA) is now impossible, since doing so would put the entity in violation of twenty or so CIP requirements – simply because they would never be able to furnish the evidence required for compliance. Basing the standards on systems (and a BCS is defined just as an aggregation of devices, nothing more) would allow cloud-based systems, containers, and more to be in scope for CIP.

So I wish the new Energy Product Reliability Organization good luck. I think having stakeholder involvement is crucial to having successful cyber standards for critical infrastructure. But don’t spoil it all by not taking account of the lessons learned from the NERC CIP experience.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they necessarily shared by CISA’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Mariam Baksh of NextGov published – as usual – a very interesting articleon Jan. 16, which begins with this paragraph:

A senior administration official put questionable timing aside and commended the Kremlin’s arrest Friday of individuals Russian officials say comprise the notorious REvil ransomware group, which U.S. officials have attributed to attacks on critical infrastructure.  

“Questionable timing” indeed! Putin is poised with a knife to the Ukraine’s throat and threatening to send troops to Venezuela and Cuba to threaten the US – so of course this is a great time to thank him for his noble efforts against REvil.

Let me suggest that the real question is this: Seven months ago, Biden - after the Kaseya attacks, which were instigated by REvil) – said (quoting from WaPo) “Putin must put an immediate stop to this activity, or Biden’s administration will take ‘any necessary action’ to stop it.” Why is the administration now taking credit for the fact that Putin finally acted, when Putin’s people certainly knew all along who needed to be arrested (because the Russian intelligence services collaborate with those people all the time, and the US intelligence services had given them a list of names)?

And why, after calling for an immediatestop to “this activity” (which, in case you hadn’t noticed, didn’t bring Russian ransomware activity to a hard stop last July), didn’t Biden keep the pressure on Putin all this time? And given that Putin obviously didn’t pay any attention at all to Biden’s order last July, why doesn’t this “senior administration official” even think, “Hey, the fact that he’s finally arresting the REvil guys now is probably not because he’s been listening to us. It’s because he wants to look as good as he can in other areas, while he’s issuing a new ultimatum to Biden to abandon the Ukraine to him”?

Perhaps the senior official is Secretary of State Blinken, who in September assertedthat “no one in the U.S. government expected the Afghan government to fall as quickly as it did.” Of course, there was no way the administration could possibly have known that the Taliban wouldn’t keep their promise of a cease-fire. After all, the Taliban are honorable men. Why Blinken wasn’t fired after that debacle would be a mystery to me, were it not clear that there are lots of others in the administration who also think their job is to claim credit for successes, rather than actually be successful.

Of course, the senior administration official’s comments really aren’t about Putin at all. They’re a vain attempt to at least get some good news out about the administration, since lately all the news has been bad. But frankly, the fact that this clown – whoever he or she is – is trying to turn the fact that Putin completely stiffed Biden for seven months and is now doing what Biden ordered only because he’s trying to divert attention from a much bigger transgression he wants to commit, only shows how weak and clueless they are. And unfortunately, that’s not news.

A much better thing to say – and not through an anonymous spokesperson – would have been “We wish to ‘congratulate’ Mr. Putin on finally taking an itty bitty step to combat one of the many evils Russia has inflicted on the world in recent years. Now here are some more steps Mr. Putin must take, and the consequences that will follow if he doesn’t (BTW, this time we mean it about the consequences):

1.      Adequately compensate the families of victims and governments for their losses in the shooting down of flight MH17 in 2014 or face a ban on all Russian aircraft in international airspace.

2.      Compensate Maersk and the other companies worldwide that lost an estimated $10 billion in the NotPetya attack, or risk being cut off from the SWIFT international funds transfer system.

3.      Compensate the victims (especially government agencies) of the SolarWinds and Kaseya attacks and arrest the perpetrators of both of these (who are either Russian government employees or well known to them) or face an order to US financial institutions and citizens to divest themselves of their Russian bonds and not own them in the future.

And speaking of Russian attacks, here’s another idea: Why don’t we investigate the assertionsmade by the CIA and FBI in the last “annual” Worldwide Threat Assessment in 2019, to the effect that the Russians had penetrated the US power grid and could cause outages at any time? There’s never even been an investigationof those statements. And there haven’t been any more WTAs since that one.

I guess that’s one way to solve the problem of bad press.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they necessarily shared by CISA’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Do you remember the NERC Supply Chain Working Group (SCWG)? We’ve been in operation since 2018 and are still going strong. Our current project is to update the set of supply chain guidelines that we drew up in 2019. Drafting them was a great experience, since we had a lot of people participating on the calls. I anticipate that re-drafting them (or perhaps adding to them, since my guess is most of them don’t need to be amended very much) will be equally interesting.

One important goal of the guidelines is to keep them short enough to be read in maybe 20 minutes. The original guidelines were all 3-5 pages long, and I anticipate the new ones will be 5-8 pages. I’ll point out something you may already know: It’s a lot harder to write a short paper than a long one. That may be why the drafting meetings were so interesting – when every word has to count, you need to decide what has to be said and say it as economically as possible (which of course is why blog posts sometimes go on and on – since the blogger knows there’s no limit imposed and he doesn’t have to be careful about his words. Of course, I don’t personally know any bloggers like that, but I’m told they’re out there).

I ended up leading the drafting of two of those papers, and I’ll be leading it for the new versions of both papers (unless someone else would like to take the lead on one of them and I’ll just participate. That would be fine with me). We need to get the drafts done by I believe mid-March, so we won’t have many meetings to draft them. I will have 3-4 meetings for each paper, and I’ll alternate weeks, so there will be a meting for one paper the first week and the other the second, etc. Of course, you can come to as many meetings as you want – although of course there won’t be any recordings made.

The two papers for which I’ll lead the drafting are both found here, along with some slides from presentations we did in Orlando in June 2019. There are also recordings of webinars we did (one for each paper) in 2020, which were well attended (since people weren’t going to a lot of in-person meetings in April through June of 2020). My two papers are Cyber Security Risk Management Lifecycle (which should really be called Supply Chain Cyber Risk Management Lifecyle – we’re not trying to tackle the entire field of cybersecurity in five pages!) and Vendor Risk Management Lifecycle.

You’re welcome to attend any or all of the meetings; I’m not going to keep attendance. You don’t have to be a member of the SCWG, although we’ll probably enroll you anyway. This will entitle you to all the benefits and emoluments of membership - priceless. We’re even waiving the normal $1,000 signup fee…😊

Also, even though these meetings are mostly populated with electric power industry types, I can assure you there’s nothing we’ll be talking about that’s specific to the power industry. So anyone is welcome to participate, both suppliers and end users. We’ve put out Doodle polls to find the best time for both series of meetings. The poll for the Cyber Risk Management meetings is hereand for Vendor Risk Management is here.[i]

I hate to pressure you, but for the Cyber Risk Management meetings, we’ll have to decide the time by tomorrow afternoon, since we want to have the first meeting this week. So if you’re interested in that, please sign up asap (note we won’t meet on Monday the 17^th, although if Monday is the best for someone, we could later move the meeting to Monday if the rest agree). For Vendor Risk Management, we’ll meet next week (the week of the 24^th), so we’ll wait a few days before we set that time. We’ll send everybody who’s participated in the poll an invitation for the series.

I hope you can help us out!

Need CIP-013 compliance help, either from the NERC entity or the vendor side? I’ve worked with a number of electric utilities on CIP-013 compliance, and I’m currently working with two vendors to the industry. Drop me an email and we can talk!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

On Sunday, I put up a post in which I included this passage:

…CycloneDX isn’t standing still, either. That project will soon announce (I think imminently, but definitely by the end of January) its new v1.4, which will significantly enhance the vulnerability management capabilities in v1.3 – and I thought those were already pretty good. While I haven’t seen any specs for v1.4 yet, I know that one addition will be a VEX-like capability.

Even though I used the word “imminently”, I was still surprised when I received an email announcement on Wednesday from Patrick Dwyer, co-leader of the OWASP CycloneDX SBOM format project (with Steve Springett, who has the dubious distinction of being mentioned in four of my last five posts, including this one), announcing that CDX 1.4 is now available.

While Patrick’s email listed four or five new capabilities in CDX 1.4, the one he promoted the most was the fact that it includes a VEXcapability, which is of course something that I’ve written a lot about. Patrick also mentioned the new SaaSBOM capability, which I think begins to address the biggest new challenge for the CISA SBOM community – properly identifying components of cloud software, as well as cloud-based components of locally-installed software (when the CISA-sponsored meetings start up in February, SaaS - i.e. cloud-based – components will be the entire focus of one of the four working groups, although there are also old challenges that still need to be addressed as well. Hopefully, they will be addressed by the other working groups. BTW, anyone is welcome to join those working groups. Drop an email to sbom@cisa.dhs.gov).

But since VEX is something I know about, and since Patrick made it the big focus of his announcement, I’d like to ask – and answer – the question, “Will the CDX 1.4 implementation of the VEX concept be very useful to organizations trying to manage risks that arise from components found in software they use?”

First I want to describe how I view the VEX concept:

1.      The idea for VEX arose a few years ago in discussions among participants in the NTIA Software Component Transparency Initiative, which is now under CISA. The participants realized that the majority – and perhaps the great majority, like 90% or more – of vulnerabilities that can be identified in software components (i.e. by looking up the component’s CPE name in the NVD) are in fact not exploitable in the product itself. In other words, if an attacker looks up a particular software product’s component vulnerabilities in the NVD and then tries to hack into the product by using them one-by-one, probably at most he’ll succeed one in ten times.

2.      That’s great news, right? Unfortunately, this is a good news/bad news situation, with the latter being more important. The flip side of the fact that at least 90% of software component vulnerabilities aren’t exploitable in the product itself is that, when the customer of a product discovers a component vulnerability in the NVD and contacts their supplier to ask when it will be patched, in at least nine cases out of ten, they will hear that no patch is needed, since the vulnerability isn’t exploitable in the product itself, often because of how the vulnerable component was implemented in the product. This will be a source of much frustration to customers and it will cause a lot of unneeded expense for software developers, since they’ll have to tie up some expensive software developers, explaining to hundreds of callers every day why they don’t have to worry about a particular vulnerability, even though it’s technically present in one component of their product.

3.      Because of this, many large suppliers (including at least a few who are actively participating in the SBOM Initiative) are holding back from releasing SBOMs to their customers, for fear their support lines will be overwhelmed with literally thousands of “false positive” calls like this. In my opinion, in order for those suppliers to start distributing SBOMs, they will need to be assured that there’s some way for them to get the word out to users, when a component vulnerability isn’t exploitable. One large supplier said in a VEX meeting that they thought they would get 2,000 false positive calls to their help lines, if they started distributing SBOMs without also distributing VEXes.

4.      The solution that was decided on a few years ago (and work started on the solution in the summer of 2020) was a document that would state that e.g. CVE-2022-12345 is not exploitable in product X version 2.4. Like the SBOM itself, the VEX would be machine-readable, and it would be distributed through the same channels as the SBOM – since in many if not most organizations, the staff members who need to see SBOMs will also need to see VEXes.

However, I use the word “see” loosely. SBOMs, and especially VEXes, were never meant to be human-readable, although even the explicitly machine-readable versions can be read by humans, with some understanding of what they say. But, given that some software products can contain literally thousands of components, there’s simply no way that machine readability can ever be optional. No non-machine-readable format will ever scale.

Now let’s discuss use cases for VEX, although we need to start with SBOMs. By far the largest use case for SBOMs is vulnerability management. Specifically, the customer wants to learn, for every component listed in an SBOM, what vulnerabilities are currently listed in the NVD for it. To do this, the customer needs a tool like Dependency Track, which I discussed in this post. D-T ingests a CycloneDX SBOM and identifies all vulnerabilities (CVEs) listed in the NVD for each component of the product. Moreover, it will keep updating that list as often, and for as long a period, as desired by the user.

That’s good, but the fact is that at least 90% of the vulnerabilities identified by D-T won’t actually be exploitable in the product itself. That’s where VEXes come in, but D-T doesn’t today ingest VEXes; and since the VEX format is about three months away from being finalized (even though some of us had thought it was finalized last summer), neither D-T nor any other tool can ingest it now. In other words, with Dependency Track or similar tools (including software composition analysis tools), you can see a list of component vulnerabilities applicable to a software product. However, you can’t see the much smaller, but much more useful, list of exploitable component vulnerabilities in the product.

Obviously, D-T won’t be able to ingest VEXes until the format is finalized, and I have no idea if VEX ingestion is planned or not.[i] But if we did have a tool that a) ingested SBOMs and VEXes, and b) output a list of exploitable component vulnerabilities in the product (hey, a guy can dream, can’t he?), how would it work? The important point about that tool is it needs to create a database. Even if it the tool just analyzes a single SBOM and identifies vulnerabilities applicable to components found in that SBOM, the fact is that VEXes that apply to that SBOM can appear at any time in the future. In fact, it’s very likely that, when the SBOM for a new product version is first released, there will be few VEXes that are applicable to the specific components that are called out in the SBOM. An SBOM is released at a point in time, whereas VEXes that apply to that SBOM (and sometimes to other SBOMs as well, usually for different versions of the same product) will dribble in over time.

This means that the tool will need to maintain a database that lists different versions of a product, each of which should have its own SBOM, since any change at all in the software requires a new SBOM. For each of those versions, the database should list the components (including version numbers) found in the SBOM, as well as any vulnerabilities found in the NVD for those components. Note that, even though the component vulnerabilities are of interest due to the fact that they apply to a component of the product, the vulnerabilities are “in” the product itself – since there’s in general no way that an attacker could attack just a component, not the whole product. Once the vulnerability has been identified in the NVD and linked to the product that contains the component, there’s no further reason to link the vulnerability with the component.

In other words, while it might be interesting to know which component is “responsible” for a particular vulnerability being present in a product, the only information that’s essential is the list of exploitable vulnerabilities in the product itself, regardless of which component they might apply to (or even whether they apply to any component at all. See items 2-7 in the list of VEX use cases below, none of which even mentions a component).

As I’ve described it so far, the database just lists vulnerabilities applicable to a product, not the much smaller (but much more useful) set of exploitable vulnerabilities. The latter set, a subset of the full vulnerability set, is arrived at by applying information retrieved from VEX documents to the lists. That is, if a VEX says that CVE-2022-12345 isn’t exploitable in Product A, then the tool will remove that CVE from the list. Even better, the tool will flag that CVE as not exploitable and put it in a separate list, but it won’t delete it from the database altogether. I’ll discuss why this is a good idea in a future post.

Thus, the database will maintain, for each version of each product in the database, a list of exploitable component vulnerabilities; that list might change every day (and for a product like the AWS client, which has 5600 components, it might change hourly). In theory, the user will be able to look in the database at any time to find the most up-to-date list of exploitable component vulnerabilities, for each product and version that they’ve stored in the database (which hopefully will correspond to every product and version currently in use in their environment – although that might not be possible for a while).

Now, let’s look at some use cases for VEXes. I’ve been participating in the NTIA/CISA VEX meetings since the second one in September 2020 and I’ve heard lots of discussion of use cases, so I’m not inventing any of these:

1.      The simplest use case is: A VEX states that CVE-2022-12345 is not exploitable in Product A version 2.4, even though it’s listed in the NVD as a vulnerability of  X, and X is a component of A v2.4.[ii]

2.      A VEX states that CVE-2022-12345 is not exploitable in versions 2.2 through 2.4 of X.

3.      A VEX states that CVE-2022-12345 is not exploitable in any current version of a family of products made by a particular manufacturer (e.g. routers made by Cisco™).

4.      A VEX states that CVE-2022-12345 is not exploitable in any current version of any product made by a particular software developer.

5.      A VEX states that CVE-2022-12345 is not exploitable in the current version of a software product, but it was exploitable in all previous versions after v1.5. In other words, the supplier has just patched a vulnerability that had been present in their product for a while, but which they didn’t know about until very recently (this is a very common scenario, BTW, since usually nobody knows about a vulnerability until it’s discovered by some researcher or software developer – or maybe by one of the bad guys, heaven forbid).

6.      A VEX states that CVE-2022-12345 is not exploitable in versions 2.0-2.7, 3.0-3.2, 3.8-4.1, and 5.9 of product X. It should be assumed to be exploitable in all other versions.

7.      A VEX states that none of the collection of vulnerabilities known as Ripple20 is exploitable in any of a supplier’s current product versions.

So here’s the big question: Which of these use cases would the VEX capability in CycloneDX 1.4 be able to address? As far as I can see, it only addresses the first case. I believe the CDX VEX is intended to refer to (or even be part of) a particular SBOM; this means it can refer to only one version of one product.[iii] Since all of the other use cases deal with multiple versions of one product or with multiple products, I don’t think the CDX 1.4 VEX capability can address them.

While it is certainly valuable for CDX 1.4 to offer the capability it does, the VEX concept itself – as outlined by the NTIA in this one-page paper – encompasses all of the above use cases (some people think it should handle even more complex use cases, although I think the cases listed above are complex enough, thank you very much). Suppliers (who will be the primary entities preparing VEXes) will need to have another VEX solution along with CDX 1.4, if they want to address use cases 2-7 above (as well as others not listed).

But there’s one other point: If a supplier wants to convey information about vulnerabilities in their SBOMs (which is only possible in CycloneDX at the moment. SPDX doesn’t now have that capability, although I believe that SPDX 3.0 will, when that version is available), they have to keep in mind that they will need to update their SBOMs much more regularly than if they don’t include vulnerability information.

This is because applicable vulnerabilities change much more rapidly than do product components. A new SBOM should be issued whenever any code in the product has changed, and especially when there has been some change in a component. However, those changes will undoubtedly be far less frequent than changes in the vulnerabilities applicable to the product. As I mentioned above, a product that has thousands of components might literally require a new SBOM every hour, if the SBOM includes information on vulnerabilities. This is because it will need to be updated whenever a new component vulnerability has appeared in, or been removed from, the NVD – as well as in the VEX case, where a component vulnerability that appears in the NVD has been determined not to be exploitable.

Of course, pushing a new SBOM out to customers multiple times in a day, or even daily or every couple of days, will be a big chore, even if the whole process is ultimately fully automated. But there’s another way to make vulnerability information available to customers: the supplier can post SBOMS for all versions of all products online (in a portal with controlled access). Moreover, they can constantly look up component vulnerability information in the NVD and post that information with the SBOMs (whether or not the information is included inside the SBOMs themselves. I would recommend it not be, but I don’t think it will make a lot of difference either way). Finally, they can use VEX information (which of course comes from the supplier anyway) to remove vulnerabilities that aren’t exploitable from the list for each product.

In other words, the supplier can provide a constantly-updated list of exploitable product vulnerabilities online, without having to send a single SBOM or VEX to a customer (the SBOMs and VEXes will of course be available for download, for any customer that wants to check the supplier’s work in identifying vulnerabilities). And if they don’t want to do this themselves, I’m sure there will be third parties that can provide this service to their customers in their place – the supplier will just need to provide the SBOMs and VEXes to the third party.

In this post, I pointed out that it should be the supplier’s responsibility – or that of a third party service provider, either chosen by the supplier or by an end user organization - to develop and maintain a list of exploitable component vulnerabilities for each of their software products (and in every version of every product, at least those that are still being supported). I gave reasons for saying this in that post, but I didn’t emphasize the most important reason: that the lack of a consumer tool that will ingest SBOMs and VEXes and constantly list exploitable vulnerabilities in a product currently serves as a roadblock to SBOM adoption, let alone VEX adoption[iv]. If no tool is needed and this service is provided online by the supplier or a third party, the roadblock disappears.

To get back to CycloneDX 1.4, I think it’s a great product, and for certain lightweight VEX use cases it’s quite appropriate. But I think Patrick and Steve are making a mistake by emphasizing that capability so much. I suspect that 6-12 months from now, the SaaSBOM capability will be seen as the huge innovation at the heart of CDX 1.4, with the VEX capability being a nice-to-have but not the primary reason for users to adopt v1.4. The CDX team might want to imagine the use cases that SaaSBOM opens up and make those the heart of their efforts to promote the new version.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they necessarily shared by CISA’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

[iv] There are at least two other such roadblocks. All three would be bypassed and cease to be roadblocks, were my proposal to be adopted. 

Note from Tom: There may have been a glitch in the email feed that caused some people (including me) not to receive the email containing my most recent post. If you were one of those unfortunate people who didn't receive it, you may want to read it now. On the other hand, you may consider yourself fortunate when you don't receive my posts...I know there are at least a few people that fall into that bucket.

When they first start learning about SBOMs, some people are dismayed by the fact that there are several SBOM formats, and that the NTIA/CISA Software Component Transparency Initiative so far hasn’t anointed one of these as the Chosen One. In fact, the Initiative makes a point of saying that there is no reason for the software world to standardize on one SBOM format.

Moreover, neither Executive Order 14028(which mandated that all federal agencies start requiring SBOMs from their suppliers. This will be required starting in August 2022), nor the two supporting documents that were mandated by the EO (the NTIA Minimum Elements document and NIST’s implementation guidance for the SBOM provisions, which is due on February 6 and for which draft language was posted in November) even hints that there will ever be a single “standard” SBOM format. I won’t say the day will never come when there’s a single universally accepted format, but I will say that I don’t think it should be imposed, whether by government regulations or even by some broad consensus of organizations that develop software and organizations that primarily use software (which pretty well means every public or private organization in the world).

Nevertheless, there’s a lot that can be said about formats:

1.      While the Initiative officially recognizes three formats - SPDX, CycloneDX and SWID – only SPDX and CycloneDX should be considered true SBOM formats. SWID was developed as a software identifier, and that is its primary use. While it does contain component information including the seven Minimum Elements, it doesn’t offer the richness of the other two formats.

2.      SPDX and CycloneDX differ in their original use cases. SPDX was originally developed (around 12 years ago) for open source license management – a use case that is very important for software developers, but not very important for organizations that do little or no software development. CycloneDX (CDX) was developed in 2017 for purposes of vulnerability management (it was developed for use with Dependency Track, which I discussed in my last post. Both CycloneDX and D-T are OWASP projects; in fact, CDX was designated an OWASP Flagship Project last June. Steve Springett is lead developer for D-T and co-lead for CDX).

3.      Because of their original use cases, SPDX is probably better at license management, while CDX is probably better at vulnerability management. However, one big advantage of having two major formats is that they compete with each other. The current version of SPDX is 2.2 (this version was designated an ISO standard last year). But the next version 3.0 will be a total rewrite, aimed squarely at vulnerability management.

4.      I have been hearing about v3.0 for a while. It sounds good, but since it has been under development for many years, and since I assumed it was going to be hobbled by the need to maintain compatibility with v2.2, I was not expecting it to be available soon (definitely not this year), or to be very useful when it arrived.

5.      So I was pleasantly surprised to hear Bob Martin of MITRE Corporation - who has been very involved with v3.0 development, and who is someone you run into every time you turn a corner in the world of software supply chain risk management – say last week that a) SPDX v3.0 will be available for general use this March, but b) it will not be compatible with v2.2. I interpret the latter to mean that a v2.2 SBOM will not be comprehensible to a tool that reads v3.0 SBOMs (of course, I’m sure there will be a conversion utility between the two formats).

6.      The latter item was good news. Trying to aim for compatibility with v2.2 would have severely limited how well v3.0 could accomplish its purpose of aiding the vulnerability management process. The fact that v3.0 “cuts the cord” to previous versions means it might well be a serious competitor to CDX in the vulnerability management space.

7.      However, CycloneDX isn’t standing still, either. That project will soon announce (I think imminently, but definitely by the end of January) its new v1.4, which will significantly enhance the vulnerability management capabilities in v1.3 – and I thought those were already pretty good. While I haven’t seen any specs for v1.4 yet, I know that one addition will be a VEX-like capability. That might in itself be a very significant development for several reasons, although the practice of using SBOMs for vulnerability management – as currently envisioned – will need to change, in order to take advantage of this capability. But if it’s going to change, now’s the time to do it, before anything is set in stone. The processes for using SBOMs for vulnerability management – for end users who aren’t also significant developers – is somewhere between infancy and toddlerhood.

But speaking of change, as I described recently, I no longer think the primary responsibility for identifying exploitable component vulnerabilities should rest with the end user. It should rest with the supplier, or perhaps with a third party service provider that has been engaged by the supplier or by the user. I have several reasons for saying this, but the most important is: Why should the 10,000 users of a software product all have to perform the exact same set of steps to identify exploitable vulnerabilities – and each acquire a so-far-nonexistent tool to help them do that – when it can be done much more efficiently, and of course at a significantly lower cost per user (in both money and time), by one or a few central organizations (either the supplier or a service provider)? Especially when the supplier should already be doing this work anyway?

In other words, in the long run I believe the biggest use for SBOMs, for software users that aren’t also significant software suppliers, will be checking the supplier’s (or the third party service provider’s) work of identifying exploitable component vulnerabilities. Some organizations will want to do this and will invest in the required tooling and expertise. Other organizations will decide to leave this work to the experts and just focus on the more important part of vulnerability management: given the current lists of exploitable component vulnerabilities in the software products the organization uses, figuring out where and how the vulnerabilities can affect the organization, and taking mitigation steps (primarily, bugging the developers to patch the serious vulnerabilities ASAP) based on the degree of risk posed to the organization by each vulnerability.

Even in the ideal world I’m outlining, I think most users will still want to receive SBOMs and VEXes from their software suppliers, even though they won’t bear the primarily responsibility for identifying component vulnerabilities. But the decision which SBOM format to request from suppliers (or even whether to ask for a specific format at all, as long as the supplier uses either SPDX or CycloneDX) won’t be quite as crucial as I had thought previously.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the CISA’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. 

This morning, I had a conversation with Tim Walsh of the Mayo Clinic, one of my “TEAMS friends” from the SBOM community (Tim was one of the original members of the Software Component Transparency Initiative and is still quite active in the group).

He mentioned that the Mayo Clinic has 400,000 devices on their network (the average hospital room has 13 networked devices), and he would love to have SBOMs for all of them. But he’s not just talking about infusion pumps and heart monitors. He’s talking about coffee makers and vending machines, since they’re all network connected now. He pointed out that a huge percentage of devices have at least some code written in Java, and that those are all very likely to have log4j, since that’s the fastest and easiest way to implement logging in that environment (it’s also free, of course).

So even though a hack of an infusion pump poses a direct threat to patient safety and a hack of a coffee maker doesn’t, hacking into a coffee maker can give the attacker a platform from which to attack lots of other devices that can affect patient safety (or the financial health of the hospital, of course). I don’t know if Tim has started patching the coffee makers yet (I think they’re still working on the safety-critical devices, of which of course they have thousands or even tens of thousands), but he’s at least going to watch them for signs of compromise (note to Tim: A coffee maker or vending machine shouldn’t be talking to a critical ER device. Just some friendly advice from Dr. Tom).

Of course, widespread availability of SBOMs would have saved a lot of IT people a lot of time since log4j came out. Since they’re not widely available now, this means that users have to hear from the software suppliers about whether or not they have any log4j in their product. And Tim pointed out that Mayo didn’t wait to hear from the suppliers; they’re patching all of the devices now, on the assumption that they’re all affected until proven otherwise.

This means that what’s just as important as a supplier notification that their product is affected by log4j is a notification that their product isn’t affected by it – so people like Tim don’t have to preemptively patch it. This is why I was quite pleased to see the notificationtoday from Fortress Information Security that their three main software products aren’t affected by either of the log4j vulnerabilities.

Not only did they provide this notification in a blog post, but they also attached a VEXfor each of the products. While these won’t do you any direct good at the moment, since tools aren’t available to read the VEX format (and the format is still being worked out), it’s interesting to see them, nonetheless.

I certainly recommend that other software suppliers provide a similar notification to their customers for every product of theirs that isn’t affected by log4j. You’ll be saving them a lot of unnecessary work. In fact, you might go further than that and point out the module that’s affected (log4j-core). So if that module isn’t present in the product, they don’t need to worry.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they necessarily shared by CISA’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

I had a Road to Damascus-type incident recently, except that, unlike in the original incident, I wasn’t blinded and I didn’t fall off my horse.

What led to my incident? I’ve become increasingly concerned of late about the prospects for consumption of software bills of materials (affectionately known as SBOMs). I’m not worried about production: software suppliers are already producing lots of SBOMs for their products and reaping a lot of benefits from doing so. But those benefits are strictly internal; few suppliers are distributing their SBOMs to their customers, and close to none are doing it with any regularity (in general, a new SBOM should be released whenever there has been any change at all in a software product).

Given that they’re producing lots of SBOMs, why aren’t suppliers distributing them to their customers? Is it because they’re all meanies and they don’t want to share the benefits of using SBOMs with their customers? No, it’s much simpler than that: They’re not sharing them because the customers aren’t asking for them.

So why aren’t the customers asking for SBOMs? There are two simple reasons:

First, a lot of software-using organizations don’t feel like making the (admittedly substantial) investment in time required to start using SBOMs intelligently, at least currently. The people inside these organizations who would be called on to find some use for the SBOMs (i.e. currently overworked security people, fresh from losing their holiday break to log4j, thank you very much) would have to invest a lot of time learning about SBOMs and trying to think of ways their organizations could use them (since it seems the end users within their organizations aren’t asking for them either).

Currently, anyone who’s starting with no knowledge of SBOMs needs to read about five or six NTIA documents and synthesize the sometimes conflicting statements in them into a single narrative (which even then will include a number of gaps that haven’t been addressed by the group yet). Until either the learning burden diminishes, or someone can do all of this learning for them (and maybe transfer the knowledge to their brains via a head-to-head USB cable – have humans evolved USB ports yet?), these people – and the organizations they work for – aren’t going to be interested in SBOMs.

Second, customers who do see benefit in having SBOMs have done enough reading to know that the two major SBOM formats – SPDX and CycloneDX – are both proudly machine-readable. Yes, you can get them in non-machine-readable formats like XLS, but given that for example the AWS client has 5600 components (as Dick Brooks of Reliable Energy Analytics has pointed out), do you really want to try to deal with all of those in a spreadsheet?

But what happens when this second group of customers looks around for easy-to-use low-cost or open source vulnerability management tools that can ingest SBOMs (and later VEXes, since the two need to go hand in hand)? They don’t find them. I believe the best SBOM consumption tool for vulnerability management purposes is Dependency Track, an open source tool developed under OWASP. It was originally developed in 2012, about five years before the term SBOM started being widely used in the software community.[i]

Dependency Track does all the basics required for software component vulnerability management and is widely used by developers. It just requires that an SBOM be fed into it (or it will create one from the source code). Then it will (among other tasks) identify all vulnerabilities (CVEs) in the NVD that apply to components and update this list as often as required. It does suffer from the limitation of not being able to ingest VEXes – but VEX is so new (and still undergoing modification) that no other product currently supports this format, either.

But since Dependency Track is an open source tool that requires more user involvement and knowledge than just pushing a Download button and then hitting Yes or Next a few times, there will always be a lot of users who won’t want to get involved with it. This despite the fact that IMO it’s as of now the only show in town (but that being said, I think there could be a lot more non-developers who would start using D-T for component vulnerability management purposes, if they were informed about easy it is to use the tool once it’s installed. Steve says there are a large number of these non-developer users now; in fact, OWASP may sponsor a webinar soon, focused on exactly this use case. If and when that happens, I’ll be sure to let you know).

To sum up what I’ve said so far, I don’t see demand for SBOMs jumping significantly until two things happen. First, there needs to be a single document that walks a technically-oriented reader, who has no previous knowledge of or experience with software development or SBOMs, through the entire process of using SBOMs for vulnerability management purposes.

Second, there needs to be one of these two items (and hopefully both):

·        An easy-to-install open source (or low cost) tool that at a minimum ingests an SBOM for a product, regularly (hopefully daily) searches the NVD for vulnerabilities applicable to components identified in the SBOM, and removes from that list any vulnerability that has been identified by the supplier of the product as not exploitable in the product itself (the latter information may someday be communicated in a VEX document, but it might be communicated in other ways as well).

·        A third-party service that processes SBOMs and VEX information and provides to the customer the same list of exploitable component vulnerabilities provided by the hypothetical tool described above (the SBOM and VEX documents will be provided by a software customer, since customers will usually be the only organizations allowed to see SBOMs and VEXes for a product. So the service will only provide the information that’s based on an SBOM for a product to a customer of that product). Of course, since there are other sources of component risk besides just vulnerabilities listed in the NVD (such as the “Nebraska problem” and three others, mentioned in thispost), the service will probably address these risks as well. It might also address risks due to vulnerabilities not listed in the NVD, but identified in other databases or other non-database sources.

Of course, neither of these items is available today – otherwise I wouldn’t have written this post. I’m sure at least the service will be available by the August deadline for federal agencies to start requesting SBOMs from their software suppliers. I’m not sure about the tool, mainly because VEX isn’t finalized yet, and SBOMs without VEX information just aren’t going to be very useful. A third party service provider would hopefully be able to get VEX-type information directly from the suppliers – since they should be quite interested in getting the word out about unexploitable component vulnerabilities.

So is what I’ve just described my Road to Damascus moment? No, this is something I’ve come to realize over the last four months. My RtD moment occurred when I asked the question – prodded in part by a suggestion from a friend and client who I won’t name, since I haven’t had a chance to ask his permission for this – why software users should have to bear the responsibility for identifying exploitable software component vulnerabilities. I now think the software suppliers should bear that responsibility. I have three reasons for saying this:

1.      Currently, you usually learn about vulnerabilities in a software product that you operate from the supplier. You receive a notice directly from the supplier, or at least the supplier reports the vulnerability to the NVD, where you discover the vulnerability (hopefully, both will happen). But a lot of exploitable component vulnerabilities aren’t currently reported by suppliers, using either method. What is it that makes component vulnerabilities different from vulnerabilities identified in the supplier’s own code, other than the fact that you won’t normally be able to find out about component vulnerabilities without…envelope, please…an SBOM? None that I know of.

2.      Does it really make sense to say to each customer of a software product, “You’re responsible for finding component vulnerabilities in Product A and maintaining those lists day in and day out”? This even though the supplier is already gathering this information (or at least they should be)? If the suppliers provide this information to their customers, the latter will only need SBOMs and VEXes (and a tool or service to process them) as a way of checking to make sure the supplier hasn’t left any exploitable component vulnerabilities off the list (of course, this assumes that all suppliers immediately accept this responsibility. Nice idea, but ain’t gonna happen). But they won’t bear the responsibility for learning about the vulnerabilities in the first place.

3.      The third reason makes a lot of sense to me from an economic point of view: The party that introduces a risk for their own benefit should bear the burden of remediating that risk. As everyone knows, developers’ use of third-party components (both open source and proprietary, but mostly the former) has ballooned in recent years, but so has component risk. Log4j, Ripple20, Heartbleed, Apache Struts/Equifax and other disasters have been the result. In fact, having the suppliers be responsible for identifying and tracking component vulnerabilities isn’t a big increase in their current responsibilities, since they should already be doing the hard part now – patching those component vulnerabilities after they identify them and determine they’re exploitable (which, fortunately for the suppliers, is usually not the case).

But it’s not like the users won’t have to do anything at all about component vulnerabilities. They’ll still need to track down and identify – using configuration and vulnerability management tools – the vulnerable software on their network and apply the patches for exploitable component vulnerabilities, which will hopefully be quickly forthcoming from the suppliers. But instead of waiting for the vendors of those tools to develop the capability to ingest SBOMs and VEXes in order to obtain information on component vulnerabilities (something it seems not a single major vendor has done so far), the supplier of the software (or again, a third party they’ve engaged for this work) could provide a feed that follows the vendor’s API – meaning the vendor will have to expend exactly zero effort in order to become “SBOM-ready”. This alone is a huge benefit to users, since waiting for the tool vendors to become SBOM-ready seems like Waiting for Godot: He can’t come today, but for sure he’ll come tomorrow…

And there’s another huge advantage to the idea of making the suppliers responsible for component vulnerability management: the need for VEXes largely goes away. This is for a simple reason: VEX was designed primarily for a supplier to communicate to customers – who have SBOMs and are looking up component vulnerabilities in the NVD – that certain component vulnerabilities aren’t exploitable in the product itself. The user needs VEXes, because if they start looking in the product for every component vulnerability listed in the NVD, some huge percentage of the time they spend doing this (as well as the time they spend calling their supplier about component vulnerabilities they find in the NVD) will be wasted. The customer needs the VEX in order to winnow the list of component vulnerabilities down to only the small percentage that are exploitable.

But if the supplier itself is responsible for producing the final list of exploitable vulnerabilities, the communication that would otherwise require a VEX would be completely internal: Whoever determines whether a vulnerability is exploitable or not would send an email to the person who is responsible for the final list of exploitable vulnerabilities for the product (in fact, they may be the same person). The email would say something like, “Remove CVE-2021-12345 from the list of exploitable vulnerabilities in Product X. Even though the NVD shows that vulnerability is found in Component A, and even though Component A is included in X, CVE-2021-12345 isn’t exploitable in X. This is because A is a library, and we never included the vulnerable module in X in the first place.”[ii]

There will still be some need for VEXes, including what may become the main need for SBOMs – to make sure the supplier is properly identifying exploitable component vulnerabilities in their products. But VEXes won’t be a gating factor for the use of SBOMs at all, which is what they are today.

My unofficial slogan is “Often wrong, but never in doubt.” I have to admit that it seems too good to be true, that having the supplier take responsibility for component vulnerability management would be a win-win in so many ways. If you think I’m full of ____, please let me know. But I will interpret your silence to indicate total agreement (unless you fell asleep before you read this).

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the CISA’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

There have been a number of good email discussions on log4j in the mailing lists of the NTIA Software Component Transparency Initiative (these will be replaced by another list under CISA’s auspices soon). Last week, the discussion in one list veered into how the National Vulnerability Database (NVD) doesn’t provide a lot of help on the log4j problem, because the vulnerability was just located in one component module of Log4j, called log4j-core.

There are potentially lots of components in log4j[i] (here is a list of them all), but they’re never all installed at once. What’s installed will vary according to the version of log4j, as well as the needs of the software developer that included log4j in the product they were developing. To quote Steve Springett of OWASP, co-leader of the CycloneDX SBOM format project, and leader of the Dependency-Track project:

Not all modules will be vulnerable. Take a common scenario that occurred over the last week. Many organizations that rely solely on the log4j-api module needlessly upgraded even though they were not affected. Only log4j-core was impacted. Log4j-core has a dependency on log4-api, but not the other way around. So if you’re only using the API and not the core logging functionality, there was no reason to upgrade.

In other words, if it had been possible to learn from the NVD that only the log4j-core module of log4j was affected, that would have saved a lot of people a lot of time (and heartburn) searching for and patching log4j in products that didn’t include log4j-core at all. How would it have been possible to learn this? To simplify things greatly, if the NVD could ingest SBOMs and associate the components of a software product with the product itself, then vulnerabilities that apply to a component of a product could be seen as applying to the product itself.[ii]

However, the logistics of implementing this capability would be formidable, in no small part because the number of entries in the NVD would increase at least 20-50-fold, and the possible relationships among entries would increase exponentially. So don’t look for this to happen very soon[iii]. Until it does, you’ll have to rely on the supplier that developed the software product you run, to tell you whether or not that product is affected by a vulnerability, no matter what “tier” of components it’s on.

And if the supplier tells you they just don’t know this and you’ll have to figure it out for yourself, well…you might want to look for a different supplier. This isn’t an acceptable answer (more on this topic coming soon to a blog near you).

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the CISA’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

There’s been a lot written about the role of SBOMs (software bills of materials) in addressing widespread vulnerabilities like log4j (not that there have been many vulnerabilities as serious as log4j). It runs the gamut from “If you had SBOMs, you’d know in ten seconds whether any of the 10,000 software products running on your network contains log4j, as well as any of their components, components of components, components of components of components…yea verily, unto the 20^th degree of component” to “SBOMs will be quite helpful in finding log4j. An SBOM and $4.70 will get you a tall no-foam latte. That will keep you awake as you look for instances of log4j.”

What’s common to just about these statements is they’re written in the subjunctive voice: “IF we had SBOMs….” Because we simply don’t have SBOMs now, to any degree that would be very helpful in researching the simplest vulnerability, let alone as widespread a vulnerability as log4j.

I’d like to suggest that it’s not useful to speculate about what SBOMs could or could not do in this or any other case, any more than it’s useful to speculate about whether, if there is a multiverse, there’s a universe in which the Pope is Jewish.[i] What’s much more useful is to ask a question like, “What are the use cases in which SBOMs will help me, and what do I need in order to make those use cases real?”

SBOMs aren’t an all-or-nothing proposition, where you either have SBOMs for all of your software, or you have nothing. It would of course be nice to have a current SBOM for all software that you own (or rent in the cloud), but right now, you’ll be way ahead of the game if you have an up-to-date SBOM for even five software products on your network. After all, that’s probably five more SBOMs than you had last year.

What can you expect to have at the end of 2022?  Executive Order 14028’s mandate for federal government agencies to require SBOMs from their “critical software” suppliers kicks in next August. Even if you’re not a government agency, it’s likely your suppliers will be able to provide SBOMs, since they’re already producing them for the agencies. You should at the minimum start asking for them from your suppliers. And you should especially start asking for a recent SBOM whenever you’re purchasing software.

But the bigger question is what you do with the SBOMs when you get them. While there is a use case for SBOMs for software license management (in fact, that was the first use case), that is mostly applicable to software developers. The SBOM use case for the overwhelming number of users (and the only use case addressed in the EO) is software vulnerability management. I break that down into three types:

The first is responding to emergency vulnerability notices like log4j and Ripple 20, in which the vulnerability (or vulnerabilities) isn’t applicable to particular final products, but to components of final products. While it would be good for you to have an SBOM for each product so that you know which ones contain log4j, what’s even more important is that you lean on your supplier to tell you whether or not log4j is a “first-level” component of their software, and also whether it’s a component of any of their first-level components (i.e. it’s a second-level component). But they shouldn’t just tell you this; they should issue a patch (or patches) to fix the problem.

Of course, if you had an SBOM that included both the first-level components and SBOMs for each of those components, you would then know whether the product is affected by log4j, down to the second level of components. That would be quite helpful, but it’s also unlikely that the supplier will give you an SBOM for each of the first-level components, since they often won’t be available from the second-level suppliers (most of which are probably open source communities). Again, something is better than nothing. Having the first-level SBOM and even a few second-level SBOMs is better than what you have now for almost all of your software: no SBOMs at all.

The second use case is reducing procurement risk – i.e. reducing the risk that you will purchase a product whose supplier doesn’t have a good program for vulnerability management. Notice that I don’t say “a product that is free of vulnerabilities”. Yes, it’s a good idea to ask for an SBOM for a product you’re considering for purchase, then check each of the components in the NVD, to see if there are any serious open vulnerabilities applicable to them (you can define “serious” as you like, but a CVSS score of 7.0 or higher counts as serious in my book. And there could be lower scores that still might be considered serious, if accompanied by other risk factors. If you decide to purchase the product, you should require the vendor, in the purchase contract, to patch any serious component vulnerabilities, unless they say that the vulnerability isn’t exploitable.

However, vulnerabilities appear all the time. Just because you buy a product that’s free of serious vulnerabilities on Monday, this doesn’t mean that on Tuesday, five serious vulnerabilities in components won’t be identified. Often, this is because a researcher just discovered that a certain string of code, previously thought to be perfectly safe, can be exploited in a particular way so that bad things happen – and five of the components of the product happen to have that same code.

What’s most important for procuring secure software is the track record of the supplier. If you can get a current SBOM for the product (and hopefully at least a few previous SBOMs, say at least one for every six months), you can answer questions like:

1.      Does this supplier let components in their product get very long in the tooth before replacing them? If there are a lot of components that are 2-3 years old, or maybe five or six versions behind the current version, this should be cause for concern.

2.      If a serious vulnerability is identified in a component, on average how long does it take the supplier to replace it (this question might be hard to answer, unless you have an SBOM for every 2-3 months in the past. You may not get those, but it certainly wouldn’t hurt to ask for them)?

3.      Are there any open source components that are effectively end of life, since the community that supports the component is no longer providing patches or updates? This is a big problem. Unfortunately, there’s no gun that goes off to let you know that an open source component is EOL. To learn this, somebody (and there are services that do this) needs to watch activity in the project and raise a flag when there’s been no new activity for say six months. Another indicator is that there are serious vulnerabilities that haven’t been patched, months after they were identified.

4.      Does an important open source component suffer from the “Nebraska problem”, described by this famous cartoon? The cartoon illustrates the risk posed by an important open source project that is now dependent on a “single maintainer”. Again, there are services that will do this, but you can also check for “commits” to the project on GitHub. If only one person has committed for say six months or more, you should at least have a conversation with the supplier about this. You might also put a term in the contract that they will replace this component within say six months.

There are lots of other questions you can ask as well, that having SBOMs will help you answer when doing your due diligence for procurement.

The third use case is probably the most important: ongoing vulnerability management for software operated by your organization. This includes a) getting regular SBOMs from your suppliers (say, at least for every major new version, and hopefully every time there has been any change in the software, including just applying a patch); b) keeping a running list of the components in each product; c) regularly checking in the NVD for vulnerabilities that apply to those components; d) adjusting the list of vulnerabilities for any notifications received from a supplier that a component vulnerability isn’t exploitable in the product itself (i.e. the type of notification that a VEX would provide); e) asking the supplier when they’re going to patch any serious component vulnerabilities that are exploitable in the product; and f) confirming that the supplier keeps any promises they make regarding patches.

This is obviously the most time-consuming use case, since it needs to be conducted day in and day out, for as long as your organization operates software (i.e. up until the day you decide computers aren’t worth it, and just buy a bunch of typewriters and postage stamps. At that point - but only then - you can bid farewell to SBOMs). However, this is also the use case for which you’ll be able to find service vendors that will perform these tasks for you, as well as perhaps other risk management tasks that you hadn’t thought of.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the CISA’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

NIST is required by last May’s Executive Order 14028 to develop guidelines for federal agencies to comply with the software supply chain provisions in Section 4 (e) by February 6, 2022. NIST was also required to draft these guidelines by November. Rather than prepare this draft as a separate document, NIST decided to kill two birds with one stone and include its contents inside the draft R2 of the venerable SP 800-161 publication (this is the first revision since it was developed in 2016).

When I learned that NIST had done this, I immediately downloaded the draft. My only concern at this point was the draft guidelines on software bills of materials (SBOMs), which are found on pages 242-246. I submitted comments to NIST on the SBOM guidelines. I’m reproducing them here, along with some other material that I didn’t submit with the comments. I’ve reproduced short passages from the NIST document in italics, with my comments on each passage immediately below it. Keep in mind that the EO only applies directly to federal agencies, so I’ve confined my comments to concerns that affect them (although in practice, I doubt there’s much if any difference between what federal agencies will have to do regarding SBOMs and what private sector organizations would need to do).

Ensure that comprehensive and current SBOMs are available for all classes of software including purchased software, open source software, and in-house software, by requiring subtier software suppliers to produce, maintain, and provide SBOMs (p. 243)

It is certainly a worthy goal that SBOMs be available for more than purchased software. That is, whether an agency a) purchases software from a commercial vendor, b) downloads an open source product to run on its network, or c) develops software in-house, it should certainly require an SBOM. But the only one of those items that is directly covered by the EO is software from commercial vendors.

Another category that may or may not be covered by the EO is intelligent devices (i.e. IoT and IIoT devices) and embedded systems. These aren’t included in the NIST definition of “critical software” (which was required by the EO), but I think they should be. When you think of infusion pumps in hospitals, electronic relays that operate the power grid, and PLCs that control operations in oil refineries, among lots of other devices, these are simply too important to ignore. In any case, there should be SBOMs for devices purchased by federal agencies, given their importance.

It would also be wonderful to have SBOMs for open source software, but a federal agency has close to zero control over the online communities that develop it. If they feel like developing an SBOM, they’ll do it, and if they don’t, they won’t. Plus there’s little likelihood that an open source community will develop a new SBOM whenever the software changes in any way, as advised by the NTIA documents.

An agency shouldn’t be found non-compliant with the EO because it failed to require an SBOM for an open source product it downloaded for its own use. What would be more constructive would be a program by NIST or another government agency to develop SBOMs for widely used open source software, since currently I know of no entity that does that, other than perhaps as a consulting project.

Also, this sentence seems to be saying that the way to get SBOMs for all classes of software is to require the suppliers to lean on the first-level component suppliers (which I assume is what is meant by “subtier software suppliers”) to provide SBOMs for the components. The two issues have nothing to do with each other, so this sentence doesn’t make sense overall.

However, the question of component tiers comes up regularly. How many component tiers should you require the supplier to provide in their SBOM? In general, my feeling is that, as long as a supplier can at least include most of the “first-tier” components in their SBOM, that in itself is an achievement (and since there can often be thousands of first-level components, that in itself can be a big challenge).

Since the risk posed by each tier diminishes the farther away it is from the first tier (i.e. a component vulnerability is less likely to be exploitable in the product itself, the more tiers there are between the component and the first tier), it’s certainly no tragedy if suppliers just list the first tier of components. Getting SBOMs for second or greater tier components is notoriously difficult (in part because 90% of components are open source, so there’s no supplier to lean on for an SBOM – but also because the supplier of the primary product isn’t the customer of say a second-tier component supplier. Therefore, the primary supplier has to require the first-tier component supplier to lean on the second-tier suppliers, etc. That becomes very difficult and time-consuming, and won’t work at all in the case of open source components). Requiring suppliers just to obtain even the second tier will place a huge burden on them, yielding minimal security benefit.

Maintain readily accessible SBOM repositories, posting publicly when required (page 244)

I’m pleased to see NIST requiring this. For several reasons, I think posting SBOMs is a lot better way of making them available, than trying to push a copy of every SBOM to every customer. Of course, many or most suppliers may want to post the SBOMs on an authenticated portal, rather than make them truly publicly available; other suppliers may be willing to post them publicly. That decision is up to the supplier to make.

Incorporate artificial intelligence and machine learning (AI/ML) considerations into SBOMs to monitor risks relating to the testing and training of datasets for ML models (page 244)

There are certainly ways in which AI can help in creating SBOMs, especially in addressing the “naming problem”. However, not much work has been done so far on this idea, and none at all by the NTIA Software Component Transparency Initiative. Requiring it for EO compliance is unhelpful and places an unnecessary burden on the suppliers.

Develop risk monitoring and scoring components to dynamically monitor the impact of SBOMs’ vulnerability disclosures to the acquiring organization. Align with asset inventories for further risk exposure and criticality calculations. (page 244)

This is an important requirement. However, this is part of a section that starts with “Departments and agencies, where possible and applicable, should require their suppliers to demonstrate they are implementing, or have implemented, these foundational SBOM components and functionality along with the following capabilities” (the three passages above are also part of that section).

The problem is that this passage isn’t a requirement for suppliers, it’s a requirement for the “departments and agencies” themselves. The supplier can’t help with these steps, and the burden of executing them shouldn’t fall on the suppliers. So this requirement needs to be put in a different section.

That being said, I wish to point out here that this is the onlyrequirement in the entire SBOM section that relates to the agencies using the SBOMs. Every other requirement specifies something that the agency needs to ask their suppliers to do or provide. I hope NIST doesn’t expect the agency to simply drop an SBOM in the bit bucket as soon as they receive it – they need guidelines for how they will use it. The above passage is a good start, but it’s hardly the sum total of what the agency can do with the SBOMs it receives.

With the EO raising the bar for software verifications techniques and other software supply chain controls, additional scrutiny is being paid upon not just the software the vendors produce, but the business entities within a given software supply chain that may sell, distribute, store, or otherwise have access to the software code themselves. Departments and agencies looking to further enhance assessment of supplier software supply chain controls can perform additional scrutiny on vendor SDLC capabilities, security posture, and risks associated with foreign ownership, control, or influence (FOCI).

Wow! Where to begin on this one?

First, what does it mean to say “business entities within a given software supply chain that may sell, distribute, store, or otherwise have access to the software code themselves”? Once the code is compiled – which is done by the original supplier – no other entity has access to the code, unless they first decompile it. Decompilation is almost certainly going to yield a new code base that won’t perform as the original one, if it performs at all (which isn’t likely). Thus, an attack based on decompiling, altering and finally recompiling the code will have about zero likelihood of succeeding (I’ve certainly never heard of an attack like that, successful or not). In other words, this phrase makes no sense at all.

Of course, the SolarWinds attack was so devastating because the Russian attackers succeeded in inserting malware into the actual code of the product. But this was done before the code was compiled, not afterwards, and the attackers were Russian state actors, not business entities within the supply chain.

This isn’t to say that some organization within the supply chain couldn’t substitute a malicious product for a non-malicious one. However, since software is often (usually?) downloaded directly from the supplier or shipped by the supplier themselves, even this isn’t very likely. And in any case, this risk is nothing that SBOMs can address.

Regarding FOCI (foreign owned and corrupt influence), I understand there are certainly legitimate reasons for raising FOCI concerns, when the issue is hardware components of an intelligent device. But for software components, FOCI has almost no meaning. Consider the following:

1.      As I’ve already said, 90% of software components are open source. Open source communities don’t restrict their contributors to particular countries – in fact, they don’t even have a good way to verify in which country a participant resides. One small proprietary study of open source components in a major software product that I recently saw showed that every one of the small number of communities behind those open source products contained at least one member from Russia, China or Iran. Probably this is very typical for open source components. Should you remove all software from your network that has “tainted” open source components like these?  My guess is you may end up severely limiting your software procurement options, if you do that.

2.      And even commercial software is developed with teams from all over the world. As Matt Wyckhouse of Finite State pointed out last year, Siemens has 21 R&D hubs in China and employs about 5,000 Chinese staff there. Does this mean you should dump all your Siemens software, as well as the Siemens hardware that runs Siemens software?

3.      And since open source software usually lists its contributors by email address, how do you know that someone from say Cuba isn’t using a Gmail account?

This isn’t to say that, in cases where there might be some reason to be suspicious of the components in a particular product, it isn’t worthwhile for a federal agency to conduct a FOCI investigation of software. It also isn’t to say that, if there is a service available that examines FOCI of software components, it wouldn’t be worthwhile for an agency to utilize that service. But it would certainly be a bad idea to make a rigid rule like “We will not purchase any software that contains any open source component that includes a participant with a Chinese email address.”

Include flow-down requirements to sub-tier suppliers in agreements pertaining to the secure development, delivery, operational support, and maintenance of software. (page 245)

In my opinion, this requirement will cost suppliers a lot of money and effort, while producing very little benefit for their customers, the federal agencies. Let’s start with the fact that 90% of components are open source, and the software supplier has literally zero control over the terms under which they acquire open source components.

The product supplier does usually have agreements with commercial component suppliers, and they can certainly ask those suppliers to agree to meet certain criteria for development, delivery, etc. But what if a component supplier refuses? Their product is usually low cost and is used in thousands of products. If one of their customers, a final product supplier who is acting at the request of a federal agency, asks them to meet these criteria, they might well decide that the additional paperwork, etc. outweighs whatever they make from that customer.

The final product suppler will then have to tell the federal agency (their customer) that they’re unable to reach an agreement with the one component supplier. What should the agency do at this point? Drop this supplier altogether? Give them an ultimatum to replace the component made by the recalcitrant component supplier with one from a more amendable component supplier, despite the fact that components are seldom direct replacements for one another, and there will often not be any alternative component supplier?

The sentence listed above needs to be qualified with “If possible”, “If applicable”, etc. It shouldn’t be left as an absolute requirement.

Automatically verify hashes/signatures infrastructure for all vendor-supplied software installation and updates (page 245)

There can be many speed bumps in the road to automatic verification of hashes and signatures – in fact, the NTIA Initiative has spent endless hours discussing problems with creating and verifying component hashes, and is still nowhere near reaching a conclusion (there might be a daylong technical meeting next year on this subject, where the participants would be locked in a room – difficult to do for a virtual meeting, I’ll admit – until they come to some sort of agreement on this subject). The words “try to”, “where technically feasible”, “where possible”, etc. need to be included in this passage as well.

Ensure suppliers attest to and provide evidence of utilizing automated build deployments, including pre-production testing, automatic rollbacks, and staggered production deployments (page 245)

It is good if a supplier has all of these capabilities in place, but not all of them do. Is this an absolute requirement, meaning that no federal agency should ever procure software from a supplier who doesn’t meet each of these conditions? I for one believe that the incremental security benefit that would be gained from leaving this as an absolute requirement is more than offset by the fact that agencies would have to abandon suppliers they may be quite comfortable with, or whose product is unique in meeting a particular need of theirs. This requirement should be qualified with “Where possible”, as in the previous cases.

Lines 8461 to 8482 (page 246)

As unaccustomed as I am to saying nice things, I want to applaud NIST for these 32 lines about open source software controls. Note that these controls need to be applied by software suppliers, so federal agencies should request that their suppliers implement them, regarding the open source components in their software. Everything NIST recommends in this section is good, but here are my two favorites:

Apply procedural and technical controls to ensure that open source code is acquired via secure channels from well-known and trustworthy repositories

This is a real problem. There have been a number of attacks in which a software supplier has downloaded a component containing a backdoor in place of a legitimate component – sometimes through typosquatting, sometimes through substitution inside the repository, and sometimes through other means – and then included it in their software. The supplier needs to take care when downloading.

Supplement SCA source code-based reviews with binary software composition analyses to identify vulnerable components that could have been introduced during build and run activities

There are a number of problems having to do with software security and SBOMs that don’t have any neat solution; this is one example. This requirement points to the problem caused by the following facts:

1.      There’s general agreement that the best time to create an SBOM for a software product is as part of the final build.

2.      However, when you go to install that product, a number of other items often get downloaded with it: installers, separate libraries that are utilized by the software (so-called “runtime dependencies”), a container, etc. If any of these items contains vulnerabilities, those vulnerabilities can do as much damage as if the product itself contains those same vulnerabilities. The SBOM created during the final build won’t show any of these components (or their vulnerabilities).

3.      A binary software composition analysis tool will identify many of the components in these other items and alert the supplier to vulnerabilities in them. When they learn about these vulnerabilities (hopefully prior to actually providing the full download to customers), the supplier should patch them or take some other measure to remove the risk posed by the vulnerabilities. Otherwise, the supplier’s promise (if they make one) that their software is free of vulnerabilities when downloaded is really made with their fingers crossed, since the full package that the user runs may contain vulnerabilities anyway.

4.      You might well ask, “If the SBOM for the installation files is more complete, why doesn’t the supplier provide that to their customers, instead of the SBOM created from the final build?” Good question. The answer is that the SBOM for the installation files has to be developed through binary analysis, and that’s inherently more prone to errors and omissions than a build that uses source code – so in some cases, providing the full installation SBOM might cause more confusion than it would alleviate. The supplier should use a binary analysis tool to create an SBOM for all of the installation files and use that to identify and remediate vulnerable components in the installation files. However, the decision whether or not to share the latter SBOM with a particular customer should be a joint decision by the supplier and the customer (many customers will be happy to receive just the build time SBOM, since that one is essential).

5.      What should the agency do if they ask the supplier for an “installation SBOM”, and the supplier refuses? If they’re involved in a high-assurance use case, they might want to procure their own binary analysis tool and create the installation SBOM on their own; or they might hire a consultant to do that. In non-high-assurance cases, they made decide that they will accept the risk that comes with not knowing the components in the installation files, and therefore not being able to learn about their vulnerabilities.

My previous post pointed out a contradiction between two of the components of the draft IoT device labeling program that NIST is developing, which will be finalized in guidance due on February 6, 2022. Those components were NIST’s desire for a binary label (i.e. a seal of approval), and their desire to have outcomes-based criteria for that label, which IMO preclude a binary yes-or-no decision on which devices should receive the label.

But there’s another contradiction in what NIST is suggesting for the program, which is much more fundamental and has to do with how the program will be run. In the Discussion Draft they put out recently on this subject (and to some degree in the web workshop they had last week), they stated (page 1) their philosophy for the program very elegantly:

NIST will identify key elements of labeling program in terms of minimum requirements and desirable attributes – rather than establishing its own program; it will specify desired outcomes, allowing providers and customers to choose best solutions for their devices and environments. One size will not fit all, and multiple solutions might be offered by label providers.

What’s not to like about this? As a former student of Milton Friedman (a few years ago) at the University of Chicago, I’m all for letting markets determine for themselves how commerce should be conducted. But this assumes that no individual player in the market has the power to impose costs on other players, that can’t be fully compensated through the legal system.

Friedman had a great illustration for this principle: If a truck owned by Commonwealth Edison (the electric utility in Chicago, now part of Exelon, Inc.) hits my car, I have the right (or really my insurance company does) to sue them for the full damages; the damages can be clearly identified and quantified. However, if ComEd pollutes the air in Chicago (at the time they operated a couple big coal-fired plants just 4 or 5 miles from the U of C) and my daughter later develops asthma[i], it’s not at all clear that the asthma was ComEd’s fault.

Plus, ComEd can certainly argue that, while their coal plants might have increased my child’s chances of getting asthma, there are many other contributors to air pollution as well. Who could possibly sort all of these contributions out and determine the precise value of each source’s contribution to my daughter’s asthma? In cases like this, markets alone can’t assure fair outcomes for all participants. Government needs to step in.

In Friedman’s example, environmental regulations are required to prevent the pollution that could lead to my daughter getting asthma – and if ComEd doesn’t follow those regulations to a T, this fact allows me to sue ComEd for the cost of her asthma (although such suits are usually big class actions nowadays), without having to prove a specific connection between the pollution and the asthma. That is, regulations set boundaries for actors in free markets. To quote Robert Frost, “Good fences make good neighbors.”

This is all a roundabout way of saying that, while it’s great to set up a regulatory program - as NIST wants to do - in which the rules and their implementation are left up to the participants in the program, there needs to be some government entity that in the end ensures the rules are fair ones and they’re enforced fairly.

What bothered me about NIST’s idea for governance of the IoT device labeling program is that they seem to want to turn the entire program over to one or more “consumer labeling scheme owners”. How they’ll be chosen, how many there will be (one? Twenty?), as well as how the boundaries will be set between them, aren’t specified (at least they aren’t yet).

Even more importantly, the parameters of the program will be up to the scheme owners. To quote NIST (page 2), “The scheme owner would be responsible for tailoring the product criteria, defining conformity assessment requirements, developing the label and associated information, and conducting related consumer outreach and education.” In other words, there aren’t any real limits on the program a scheme owner can put together, other than that a) they have to use the criteria that NIST has identified (in the Discussion Draft), and b) the label needs to be binary. So what would happen if a scheme owner:

1.      Decides to assess devices against the criteria as leniently as possible (see my previous post, where I discussed the impossibility of performing a true binary assessment based on outcomes-based criteria), while at the same time charging high fees for assessments? The message will be clear to IoT device manufacturers: “The price this scheme owner charges me for the assessment is high, but I’m sure they’ll give me a label. It’s much better giving this scheme a shot, rather than going to a different scheme owner, who might really assess the device and not give me the label if they think I’m deficient.”

2.      Decides not to bother with the “consumer outreach and education” part? After all, the entities that are going to pay the fees to the scheme owner are the manufacturers, not the consumers. Given that there will probably be a lot of consumers who initially are willing to fork over an extra $10-$20 for a security camera that carries a cybersecurity label, why not charge a high assessment fee to manufacturers that see the importance of getting a label immediately, so they can sell to these early adopters? Then close up shop and let another scheme owner do the hard work of outreach and education, to grow the market further?”

3.      Decides to poach another scheme owner’s market? Let’s imagine that one scheme owner has had success providing labels for baby monitors; now they decide they ought to try their hand at a labeling scheme for smart appliances, despite not knowing anything about home appliances. But there’s already another scheme owner doing well in that market, and they’re not at all pleased at the idea of having an organization with no experience in appliances jump into that market and perhaps hand out meaningless labels, just so they can collect the assessment fees. NIST specifically points out that they don’t want to see consumers getting confused by multiple labels for similar products, but that’s exactly what would be the outcome in this case. Who is going to mediate this dispute?

During the web workshop, I asked in the chat how the scheme owners will make money. That didn’t get answered in the workshop, and it doesn’t seem to me that the NIST staff members who are developing the guidance for the IoT device labeling program have even asked themselves that question. But they really should. If you want to have private industry develop and run a regulatory program, you need to make sure you understand how they will make money doing this; if they can’t make money by what we’d consider “legit” means, they’ll find other means to make it. Those “other means” are likely to involve the scheme owners doing things that are unfair to other scheme owners, to manufacturers, and potentially to consumers as well, as shown in the three hypothetical examples above.

Yet preventing these abuses will require some power higher than the device owners to set and enforce rules. Let’s be clear: at a minimum, some government agency needs to police the device labeling program so that it operates fairly for all, even though the specific details of how the program will work are left up to the scheme owners.

In the web meeting, the presenters made clear – as I knew already – that NIST doesn’t run programs like this and isn’t a regulator. But how about another federal agency? In fact, the Executive Order, in the paragraph that orders the IoT device labeling program (which I quoted near the beginning of my last post), states that the Director of NIST should coordinate “with the Chair of the Federal Trade Commission” in developing the program and the criteria. Yet I haven’t seen or heard one word about the FTC, or any other agency, being involved with this program. Sure, NIST isn’t a regulator, but that doesn’t mean they can’t find a regulator within the government to set and enforce rules for this program.

So how about turning over implementation of the device labeling program to the FTC? They’re great at writing and enforcing rules. This doesn’t mean that NIST’s ideas need to go out the window. In general, I like the idea of having multiple scheme owners providing label schemes for different markets; and I also like the idea that the scheme owners should be free to construct their own programs.

But just like any other economic activity, there need to be government-enforced rules to make sure everyone is treated fairly. Anyone is free to build cars or trucks in whatever way they see fit. However, they all have to follow the safety standards set by the National Highway Traffic Safety Administration (NHTSA). When they advertise their car, they have to make true statements in their ads, or the FTC will come down on them. And when they communicate with shareholders, they need to follow the information disclosure guidelines set by the Securities and Exchange Commission (SEC). All this regulation obviously hasn’t prevented companies like Tesla and Rivian from producing innovative new cars and trucks that consumers and businesses want to buy.

So I like NIST’s ideas for allowing experimentation – and even competition - by the scheme providers. But make no mistake: This is a regulatory program. The EO ordered the government to develop a program to raise the level of cybersecurity in consumer IoT devices by creating a precious commodity – the label – that device manufacturers will desire because they know that a label will increase their sales. And even more importantly, not having a label might cause a manufacturer to go out of business.

NIST, please turn the implementation and operation of the IoT device labeling program over to a regulator to ensure it’s successful. I can guarantee it will fail if you don’t do that. Good fences make good neighbors.

P.S. My client Red Alert Labswill submit both this post and the previous one (slightly edited) to NIST as comments on the Discussion Draft paper.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the CISA’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

I’ve been working with an interesting client since last spring: Red Alert Labs, a French company that specializes in helping IoT device manufacturers understand and comply with cybersecurity regulations and standards, as well as helping organizations that utilize IoT devices (of course, that’s just about every organization in the developed world) understand how they can do so securely.

In the course of that work, I’ve written several posts about what’s going on with IoT regulation in the US, the most recent of which is this one. In a nutshell, there are two current programs to regulate IoT in the US. One is the IoT Cybersecurity Improvement Act of 2020, while the other is the consumer IoT device labeling program, described in the Executive Order of this past May. Here is how the EO described the labeling program:

Within 270 days of the date of this order, the Secretary of Commerce acting through the Director of NIST, in coordination with the Chair of the Federal Trade Commission (FTC) and representatives of other agencies as the Director of NIST deems appropriate, shall identify IoT cybersecurity criteria for a consumer labeling program, and shall consider whether such a consumer labeling program may be operated in conjunction with or modeled after any similar existing government programs consistent with applicable law. The criteria shall reflect increasingly comprehensive levels of testing and assessment that a product may have undergone and shall use or be compatible with existing labeling schemes that manufacturers use to inform consumers about the security of their products. The Director of NIST shall examine all relevant information, labeling, and incentive programs and employ best practices. This review shall focus on ease of use for consumers and a determination of what measures can be taken to maximize manufacturer participation.

Does this give you a clear idea of what the program should be about? If you say yes, you either didn’t read this carefully or you’re a liar. This paragraph was deliberately made as vague as possible, in order not to constrain NIST in what they develop. That’s all well and good, but February 6 is the 270^th day after the EO date, so by then, NIST has to have their final answers on both a) how the program will work, and b) what criteria will be used for the program.

NIST published draft criteriain August, and my guess is there wasn’t a lot of serious criticism of them (there was a comment period, and the comments – including the ones I helped prepare – seemed quite positive). They’re a subset of the criteria in NISTIRs 8259A and 8259B, which I think are excellent starting points for IoT device cybersecurity (NISTIR 8259A is about technical capabilities – or lack thereof – in the devices themselves, while 8259B is about the manufacturer’s practices and programs).

But besides publishing draft criteria, NIST was also required to draft how the consumer IoT device labeling program itself will work. That documentappeared in early December, and was then discussed in a four-hour web meeting last Thursday (the recording should be available hereby December 23). BTW, the webinar had 700 registrants from all over the world. I’ll bet the final number of viewers will be higher, after the recoding has been posted.

While the NIST presenters made a number of interesting individual points during the workshop, various issues were raised in the chat and as questions. Of course, the whole point of the workshop was to learn about what the issues were, and the staff seemed eager to hear about them. If you still want to comment, NIST will accept emailed comments – sent to labeling-eo@nist.gov - up until later this week (I believe the deadline is Thursday the 16th). Because NIST is under a very tight deadline for February 6 (the EO gave NIST a whole slew of deadlines for that date), NIST didn’t have a formal comment period this time.

While I have some minor issues with what NIST said, I have two major ones. These both have to do with what I see to be fundamental contradictions in what they’re proposing. If not addressed, I believe that either of these might sink the whole IoT device labeling program. I will address the first of those contradictions in this post, and the second in another post in the near future.

The first contradiction has to do with the type of label. This doesn’t mean whether it should be on white or yellow paper: NIST has already decided that there will be a small physical label attached to the IoT device (or perhaps included on a sheet of paper with the device), but that will have a URL or QR code that will take the consumer to a much more detailed discussion on the web.

NIST distinguishes between three different types of labels, in the answer to question 10 on pages 21 and 22 of the Discussion Draft: descriptive, graded, and binary. NIST defines them this way:

1.      “A descriptive (or informational) label provides facts about properties or features of a product without any grading or evaluation.”

2.      “A tiered (or graded) label indicates the degree to which a product has satisfied a specific standard, sometimes based on attaining increasing levels of performance against specified criteria.”

3.      “A binary label (sometimes called a “seal of approval”) is a single label indicating a product has met a baseline standard. Examples include Energy Star [EPA], USDA Organic [USDA], and the government of Finland’s cybersecurity label [FINLAND].”

NIST indicates they have decided that a binary label is best, meaning the product meets the baseline criteria listed in the document (more on those in a moment); they set out their reasoning in the answer to question 11 on pages 22 and 23. While I’m not crazy about binary, up-or-down judgments in cybersecurity matters, there might be some cases where they would work – so for the moment, let’s just say I accept NIST’s decision.

However, this directly conflicts with another decision they have made: that the criteria should be “outcomes-based”, meaning that each criterion should list an outcome to be achieved, rather than a particular means of achieving an outcome (those of you who are veterans of the NERC CIP Version 5 Wars of the early part of the last decade – has it really been that long? – will remember lots of discussion of prescriptive vs. objectives-based requirements. Of course, objectives-based means just about the same as outcomes-based).

NIST didn’t just express a preference for outcomes-based criteria; they actually rewrote all of the criteria from the August paper as outcomes-based. For example, the August paper proposed this wording for the Logical Access to Interfaces criterion:

1. The ability to logically or physically disable any local and network interfaces that are not necessary for the core functionality of the product component.

2. The ability to logically restrict access to each network interface to only authorized persons or devices.

3. The ability of the product component to validate that the input received through its interfaces matches specified definitions of format and content.

4. The ability to authenticate individuals and other IoT product components using appropriate mechanism to technology, risk and use case. Authenticators could be biometrics, passwords, etc.

5. The ability to support secure use of authenticators (e.g., passwords) including:

a. if necessary, ability to locally manage authenticators

b. ability to ensure a strong, non-default authenticator is used (e.g., not delivering the product with any single default password or enforcing a change to a default password before the product component is deployed for use)

Of course, these sub-criteria are all quite measurable, but they don’t allow for a device to utilize other means to achieve the same goal of interface access control. Contrast the above to the wording for the same criterion in the Discussion Draft (although the criterion is now called Interface Access Control):

1. Each IoT product component controls access (to and from) all interfaces (e.g., local interfaces, network interfaces, protocols and services) so as to limit access to only authorized entities. At a minimum, the IoT product and its components shall:

a. Use and have access only to interfaces necessary for the IoT product’s operation. All other channels and access to channels are removed or locked down

b. For all interfaces necessary for the IoT product’s use, access control measures are in place (e.g., unique password-based multifactor authentication)

c. For all interfaces, access and modification privileges are limited for the interfaces and users of the interfaces

2. The IoT product, via some, but not necessarily all components, executes means to protect and maintain interface access control. At a minimum, the IoT product shall:

a. Validate data sent to other product components matches specified definitions of format and content

b. Prevent unauthorized transmissions or access to other product components

c. Maintain appropriate access control during initial connection (i.e., on-boarding) and when reestablishing connectivity after disconnection/outages

NIST has rewritten all of the other criteria from the August paper in the same way. Of course, I don’t object to outcomes-based criteria, and in fact I think that, in domains like cybersecurity and safety, they’re the only good option. However, it’s when you put outcomes-based criteria together with an up-or-down assessment program (in this case, a binary label) that you get a problem: The binary label essentially says, “This product[1] has met all of the baseline criteria established by NIST.” Someone  is going to have to make the decision whether or not this is true for a particular device, and that decision has to be up or down; it can’t be “Well, they did a good job in this area, but not such a great one in this other area.” How will the decision be made?

In the case of prescriptive criteria (like the criteria in the August paper), it should be fairly easy to make such an up-or-down decision. As I look at the five sub-criteria in the first example above (and the two sub-sub-criteria under sub-criterion 5), it seems to me that it shouldn’t be hard to come to an objective decision on whether or not a particular IoT product meets each of those criteria. For example, “The ability to logically restrict access to each network interface to only authorized persons or devices” could be assessed by a) identifying each network interface on the device (this would include both interfaces for network cabling, as well as wireless “interfaces” like a wi-fi radio), and b) verifying that the device allows access to each interface to be restricted only to authorized persons or devices.

Now let’s look at the first outcomes-based criterion from the Discussion Draft in the example above (the one starting “Each IoT product component…”), and specifically the three sub-criteria under it. Before I can determine whether a device satisfies those three sub-criteria, I need to answer a number of questions, including:

1.      How can any party but the device manufacturer itself determine which interfaces the device “uses and has access to”? That can’t be determined by looking at any physical characteristics of the interfaces. An assessor would need to examine the code that runs on the device. But even being able to look at the code won’t necessarily let an outsider determine whether a particular interface is used or not, unless they have experience writing device software. And let’s face it: examining all of that code would be a very expensive, time-consuming process. Plus this assumes that the developers of the software – usually not the device manufacturer – would be willing to provide the assessor with access to that code, in exactly the version that happens to be on the device. Oh…and don’t forget the developer of the real-time operating system (RTOS) that runs the device, like Microsoft or Wind River; they’ll need to fork over their code as well. Lots of luck getting that to happen.

2.      How can any party but the device manufacturer determine which interfaces are “necessary for the IoT product’s operation”? Even an examination of the code probably can’t answer this question. It really has to be the product’s designer, who presumably is an employee of the manufacturer. The designer might well be willing to answer this question for the assessor, but then the assessment becomes a self-attestation. There’s nothing wrong with self-attestations per se, but when an up-or-down binary label is on the line (and not receiving the label might have a serious negative impact on sales of the device), is it likely that the product’s designer will reveal anything that is going to jeopardize their being awarded the label? In other words, a binary label that depends on the self-attestation of the manufacturer isn’t worth the paper (or electrons) it’s printed on.

3.      In the third sub-sub-criterion, how does an assessor determine whether “access and modification privileges are limited”? Does “limited” mean nobody is allowed to make any change to the device at all? Or should some people be allowed to make some changes, and one person (the super user) can make any change they want? Or maybe anybody can make any change they want, except for say disabling the device altogether? These – and myriad other scenarios – could all be interpreted as “limited modification privileges”.

A binary label is a very powerful thing. Receiving it or not might in some cases mean the difference between commercial success and failure of a product. A manufacturer that doesn’t get the label is going to be quite unhappy and challenge how the assessor arrived at their judgment on each criterion. And that’s what this assessment will really come down to – judgment. The assessor is going to have to answer a huge number of questions like the three above. They won’t have any objective way to answer most of them (except in some cases by just appealing to the manufacturer for an answer, which then creates its own problems). They will have to use their judgment to answer those questions, so they can award (or not) the label.

What’s the bottom line on all this? If you want to assess based on very prescriptive criteria (like those from the August NIST document), then a binary label is achievable. But if you want outcomes-based (or objectives-based, and there are other terms as well) criteria, you are going to open up a can of worms if you want to assess those in a binary fashion. With outcomes-based criteria, I think there is no “objective” way to make an up-or-down decision on whether the device meets a particular criterion.

Frankly, for the device labeling program, I think an informative label is the only way to go. Yes, the majority of consumers will have no idea what all that information means for them. But IMHO, it’s much better to give at least a few consumers some good information that will let them make an informed decision, than it is to give them a thumbs up that in the end rests on subjective judgments, and will be subject to endless challenges, with no real resolution of those challenges.

If we were talking about clear, objective criteria, like whether the device is properly electrically grounded, there’s no question that binary is best; you don’t want to just give someone information and hope they make the right decision, when the device could electrocute them if they make the wrong decision. But cybersecurity ain’t that way, which is why there are very few cyber regulations that even call for up-or-down decisions. NIST would make a big mistake were they to try to implement a binary IoT security label.

P.S. In general, I think the only workable cybersecurity regulations are those that are risk-based – that is, the organization is required to implement a risk management program to cover a certain domain, such as supply chain security (as in NERC CIP-013) or software vulnerability management. The organization is judged on how well they’ve developed and implemented their program.

However, this was clearly not what the EO had in mind when it ordered labeling programs for consumer IoT devices and consumer software, and I agree it’s hard to see Joe and Mary Smith conducting a full cyber risk assessment before buying a baby monitor – although that would definitely be in order before say a military purchase of an IoT device.

But given that the EO did order a consumer IoT labeling program, I don’t see any good alternative to an informative label. Let’s save binary and graded labels for the commercial, industrial and government use cases, where they make some sense.

P.P.S. I have another important disagreement with NIST on the IoT device labeling program, which I will hopefully write about in the very near future.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the CISA’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Both Kevin Perry and Dale Peterson made good comments on my last post, and both of them raised important issues. A lot of people viewed this post already, but you might want to check it again to see those comments (and my comments on Dale’s comments). 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the CISA’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

I was surprised to read this storyfrom ZDNet this morning, describing yet another devastating cyberattack on a critical infrastructure organization, this time an electric utility. Of course, even though the utility, Delta-Montrose Electric Association (DMEA) in Colorado, never used the word “ransomware” in their announcement of the attack, everyone interviewed for the article seemed to think it was a ransomware attack.

But even if it wasn't, what I'm most interested in is the fact that, by the utility's own reckoning, 90% of its internal systems (which I interpret as "IT network") were down. Yet the utility says their electric operations weren't affected at all. This simply shows that the utility followed a cardinal principle for critical infrastructure: Complete separation of the IT and OT networks, so there is no direct logical path by which an infected IT system might infect the OT network. 

I know of two other devastating ransomware attacks on critical infrastructure in the US. In both of these, the IT network was pretty much destroyed by the ransomware, while the OT network wasn’t touched by it. Yet in both of these cases, the OT network had to be completely shut down in the wake of the attack, along with the IT network. How could that happen, assuming the networks really were separated?

The more recent of these two attacks was, of course, Colonial Pipeline. In that attack, after most or all of the IT network was brought down, the OT network (and therefore the pipelines themselves) were also brought down. Colonial said at the time that they did this out of the usual "abundance of caution". However, a WaPo editorial pointed out that, with Colonial’s billing system (which was on the IT network, a normal practice even in critical infrastructure) being down, Colonial couldn't invoice for gas deliveries.

Even more importantly (and this was a fact I learned from somebody who commented on one of my postson Colonial – my longtime friend Unknown), since Colonial is a common carrier and doesn't own the gas it delivers, they would literally have been on the hook for the entire cost of all the gas they delivered but didn't invoice, if they'd continue to run their pipeline network. So the OT network had to come down as well, even though it wasn't directly impacted by the ransomware.

The previous attack was in 2018, when a very large US electric utility was hit with a devastating ransomware attack. As with Colonial and DMEA, the IT network was completely down but the OT network hadn't been directly affected by the ransomware. The IT department decided they had to wipe over 10,000 systems on their IT network and rebuild them from backups. According to two independent sources, the original plan was to leave the two grid control centers (part of the OT network) running during the approximately 24 hours it would take to do this.

However, the utility then decided that if they left the OT network running, they would run the risk that even a single system in the control center might have been infected, and then might re-infect the IT network as soon as the latter was brought back up – meaning they’d have to repeat the entire process of wiping and rebuilding. So they decided they had no choice but to wipe and rebuild the control center systems (about 2,000), including their VOIP phone system.

The result was that for 24 hours, the power grid in a multi-state area was run by operators using cell phones. It was pure luck that a serious incident didn't occur during this time, because power system events usually happen too quickly for humans to react properly; the event would probably have been over long before the control center staff would have been able to diagnose the problem and the solution through phone calls.[i]

In both of these previous attacks, the OT network was logically separated from the IT network, but from a larger point of view it wasn’t. In Colonial’s case, the problem was that there was a system on the IT network (the billing system), which had to be up in order for operations to continue. How could the OT shutdown have been prevented? Clearly, the billing systems should either have been on the OT network, or (since that might itself have caused problems), they should have been on a network segment of their own. The ransomware would never have reached them, and they would presumably have continued to operate after the attack. Thus, operations wouldn’t have had to be shut down.

And what could the utility in the second attack have done differently, to prevent having to shut down their control centers? It seems to me that the root cause of that shutdown was the fact that the utility didn’t trust its own controls, put in place to prevent exactly the sort of traffic between networks that they were worried might get through.

In case you’re wondering, these were high impact Control Centers under NERC CIP-002 R1. The CIP requirements that govern separation of networks, CIP-005 R1 and R2, should have been sufficient to prevent spread of malware between the networks. However, it’s also very easy to violate those requirements, and my guess is that somebody in IT didn’t want to take the chance that someone had slipped up in maintaining the required controls, no matter how unlikely it was that they had.

I guess the moral of this second story is that, if you’re ever in doubt about whether your IT and OT networks are really separated, you should take further steps to remove those doubts. That way, if this incident happens to your organization (utility, pipeline, oil refinery, etc.), you’ll be able to leave the OT network running without suffering a nervous breakdown.

Thus, the fact that DMEA was able to continue delivering electric power to their customers (although with delayed billing) shows they not only had the required separation between their IT and OT networks, but they also

a)      Didn't have any system dependencies linking their IT and OT networks, as in the case of Colonial, and

b)     Had enough controls in place that they didn’t doubt that the two networks were really logically separated.

Good for them!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by CISA’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Noted power industry cybersecurity consultant Patrick Miller recently put up a blog post consisting of a recorded interview and transcript of a discussion of software bills of materials. He did a good job of articulating the most important argument for having SBOMs: they will allow software users to find out about dangers lurking in components of the software they run, vs. being totally blind to those dangers.

However, Patrick gets off track when he discusses how the users will actually learn about those dangers, in this passage:

PATRICK:

Effectively what happens is (the users) take (the SBOM) from the manufacturer, they use a tool to compare the two, and they know what's in there. They can say, 'Yes, all the things we expect are in there and nothing else is in there. They didn't insert garlic into my soup and now I'm going to get sick,' for example.

GAIL:

As part of the critical infrastructure, I ask for the SBOM, the manufacturer produces an SBOM, I run my tool against what they've given me, and it comes back a little different. What happens then?

PATRICK:

Well, that's when you have a risk. You have to have the conversation with the vendor to find out: did you actually get what you expected?

In some cases, there are things like file integrity and certificates that can go a long way to helping this. In some cases, it may just be that there's a mismatch in the tools. But that's an interesting conversation for you to have with your vendor and should be the first thing you do if things don't match up. You need to talk to your vendor right away, because things should line up.

I think Patrick is describing a sequence like this:

1.      The user receives software (either new or an update to software they’re already using); the supplier provides an SBOM with it.

2.      As a check on the supplier, the user uses a binary analysis tool to create an SBOM from the delivered software binaries (this isn’t an easy thing to do for various reasons, but it’s possible for someone with the right software development experience).

3.      The user compares the SBOM they received from the supplier to the one they created and looks for discrepancies.

4.      If they find a discrepancy, the consumer talks with the supplier to find out whether this was due to an innocent mistake, or whether it was a deliberate attempt to hide something – perhaps there’s a vulnerable component in the software and they don’t want the customers to know about it?

However, this sequence simply couldn’t happen, for several reasons. First, SBOMs are almost all generated by automated tooling on the supplier side, usually as part of the software build process – and I’m talking about machine-readable SBOMs, which are the only ones that scale. There are single software products, including one that’s installed today on millions of corporate desktops across the US, that have upwards of 5,000 components; the average product contains over 100 components. It’s kind of hard to create those SBOMs by hand. While it would certainly be possible to edit the JSON or XML code that constitutes the SBOM after it’s been produced, it wouldn’t be easy.

More importantly, it’s just about 100% certain that there will be a mismatch between an SBOM produced from binaries (as in step 2 above) and one that’s created with the final software build, without the supplier having to obfuscate anything. There are various reasons for this, including that there are almost always other files or libraries that aren’t part of the software as built, but that are essential for it to run; the binary analysis will capture these files, as well as those for the “software itself” (and unfortunately, the fact that SBOMs created from the build process don’t include files installed with the software means that the SBOM your supplier sends you will almost always leave out these additional files – and therefore you won’t learn about vulnerabilities in those files. This is an ongoing problem with SBOMs, although it’s certainly not a fatal one).

A bigger reason is the one I alluded to in step 2 above: creating SBOMs using a binary analysis tool is always going to be much harder than creating one through the build process, since the output from the tool will inevitably miss a lot of components and misidentify others. That’s why there is a lot of judgment required to clean up the output from the tool; and even after applying that judgment, the resulting SBOM will virtually never match an SBOM produced from the final build.

Binary analysis is required in order to learn anything at all about the components of legacy software, for which no SBOM was produced at build time (i.e. just about all legacy software today). It’s like a colonoscopy: It has to be done, but it certainly isn’t a lot of fun.

More importantly, if the supplier decides to alter one SBOM (perhaps by renaming a few vulnerable components with the names of non-vulnerable components), they are going to have to replicate this work in future SBOMs for the same product. A new SBOM needs to be generated with every change to the software, which means with every new version and every patch application. The devious supplier will have to make the same change in all of these new SBOMs.

But the biggest reason why Patrick’s scenario is highly unlikely is this: Why should the supplier go to a lot of trouble to obfuscate vulnerable components, when virtually any SBOM you look at has vulnerable components (i.e. components for which one or more open CVEs are listed in the NVD) – and it’s certain that new vulnerabilities will be identified in those components in the future?

The problem isn’t so much that there are vulnerable components. The real question is how the supplier deals with component vulnerabilities, whenever they appear.

A 2017 studyby Veracode stated “A mere 52 percent of companies reported they provide security fixes to components when new security vulnerabilities are discovered.” I would hope that number would be higher nowadays, but one thing is certain: It will definitely be higher, once most software suppliers are distributing SBOMs to their customers. Components in the software will have fewer vulnerabilities, plus suppliers will work to patch those vulnerabilities more quickly than they would have otherwise (if they would have patched them at all).

Is this because the suppliers have all just experienced a Road to Damascus moment, and decided to completely change their former ways? No, it’s because it’s human nature to pay closer attention to doing your work correctly when somebody is looking over your shoulder. That’s why the SBOM effort by NTIA (and now CISA) is called the Software Component Transparency Initiative. It’s all about transparency.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the CISA’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Cybeats is going to unleash the fourth in their great series of webinars on software bills of materials on November 30 from 1-2 Eastern Time. Here’s their description of this one:

Software Bills of Materials (SBOMs) are beginning to arrive at critical infrastructure operators’ doors. The promises of much more rapid responses to cybersecurity vulnerabilities and other tangible benefits are waiting in the wings to be proven or disproven. What will be the impact of this new form of visibility and information sharing actually be?

Tune in on November 30th to learn more on how SBOMs may well prove critical to securing our critical infrastructures with our fourth episode of The State of Cybersecurity Industry: SBOMS impact on critical infrastructures.

The guests are:

·        Dr. Allan Friedman of CISA, leader of what was previously called the Software Component Transparency Initiative when he was at NTIA. The initiative continues, but its new contours will be discussed in December.

·        Ginger Wright of INL, my co-leader of the SBOM Energy Proof of Concept.

·        Tim Roxey, who has appearedin these posts a number of times in the past, and always has quite interesting things to say.

·        Chuck Brooks, who I only met recently but seems to have a good instinct for where the needle is moving in cybersecurity.

The event link is here. You don’t have to register, but they encourage you to; the button for that is on the same page.

See you then!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the CISA’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

When I’m looking for guidance on a decision, I often turn to the great 19^th century scholar Charles Dodgson, who wrote on mathematical logic. His two greatest treatises on that subject were written under the pen name Lewis Carroll: Alice in Wonderland and Through the Looking Glass.

Near the beginning of the first treatise, after Alice has fallen down the long rabbit hole and emerged in Wonderland, she has no idea where she is and has the following exchange with the Cheshire Cat:

Alice: ‘Would you tell me, please, which way I ought to go from here?’
The Cheshire Cat: ‘That depends a good deal on where you want to get to.’
Alice: ‘I don't much care where.’
The Cheshire Cat: ‘Then it doesn't much matter which way you go.’
Alice: ‘...So long as I get somewhere.’
The Cheshire Cat: ‘Oh, you're sure to do that, if only you walk long enough.’

What has been known until now as the Software Component Transparency Initiative of the National Technology and Information Administration (part of the US Department of Commerce) finds itself currently in somewhat the same position as Alice. The leader of the Initiative, Dr. Allan Friedman, moved a few months ago from the NTIA to CISA (which is of course part of the Department of Homeland Security).

The Initiative is a “multistakeholder process” – a special type of “organization” that the NTIA has deployed in many situations (there is currently a large multistakeholder process going on for 5G – much larger than the one for SBOMs). The idea is to have participants in an industry get together to agree on rules that apply to a new technology, without even mentioning the dreaded word “regulation”. However, CISA does things differently (although they aren’t interested in becoming a regulator any more than NTIA is, as their Director Jen Easterly made clear just last week), so this process can’t continue. And one can argue that the multistakeholder process has now outlived its usefulness, anyway.

There is agreement among the people who have been participating in the Initiative, that we would like to continue in some form. It is to discuss what that form will be, as well as to provide general instruction on what SBOMs are and how they can be used, that Allan has scheduled the first annual (hopefully) CISA “SBOM-a-rama” for December 15 and 16, at 12-3 PM ET on both days. This will be a two-day event:

1.      Allan describes the first day thusly, “The first session will focus on education, bringing the broader security and software community up to speed with the current understanding of technology and practices, and offer the opportunity for some questions and answers for those relatively new to the issue and technology.”

2.      Here’s his description of the second day: “The second day will focus on identifying the needs of the broader community around SBOM, and areas of further work deemed necessary for progress. This could include specific technical issues and solutions, operational considerations, or shared resources to support the easier and cheaper generation and consumption of SBOM and related data.” This is where I expect the two questions listed in the title of this blog to be asked. As long as there is agreement on at least the first question, I’ll be happy with that. Discussion beyond that will be exploratory, but will continue in future meetings, however they’re structured.

Who’s eligible to attend this. The requirements are quite rigorous, I’m afraid:

1.      You must have a working command of the English language.

2.      You must have an interest in SBOMs and how they can help you secure your organization, even if you know very little about them.

3.      You don’t have to have software development experience. If that’s a requirement, I can’t attend either.

I’ll publish the meeting information when it’s available.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by CISA’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

This week, my friend Maggy Powell of AWS put up a post on LinkedIn that provided a link to their most recent documentregarding NERC CIP, described by Maggy as the “AWS User Guide to Support Compliance with NERC CIP Standards”. She further states that “The User Guide describes key concepts for customers considering CIP regulated workloads in the cloud.”

Dale Peterson asked me for my comments on the document. Before I downloaded it, I pointed to this postfrom last year, where I tried to summarize the problem preventing NERC entities from deploying Medium or High impact BES Cyber Systems in the cloud (they’re free to deploy Low impact BCS in the cloud now). So I reviewed (skimmed, I’ll admit) the AWS document to see if it had anything to say that would change the situation enough to make it at least possible that Medium or High BCS could be put in the cloud.

It didn’t. Like the document and presentation that Microsoft Azure prepared for the NERC CIPC (remember the CIPC?) in around 2016, AWS seems to think that what needs to be done is just convince NERC and utilities that AWS has good security. That has nothing to do with the real problem, as my previous post explains. There’s literally nothing that AWS, Microsoft, or anyone else – other than NERC, the Regions, the NERC entities, and FERC – can do to change the situation, absent a wholesale revision of the CIP standards. I replied to Dale:

I skimmed through the AWS document, but it was unfortunately as I expected. It tells you everything you need to know about AWS security, except the one thing that matters for CIP: How AWS could possibly produce the evidence required for the utility to prove compliance with about 25 of the CIP requirements, if they put BCS in the cloud.

And the answer to that question remains what I wrote last fall: There's no way any cloud provider could do that, without breaking their business model.

NERC CIP won't permit BCS in the cloud until it's completely rewritten as a risk based compliance regime (which involves revising the NERC Rules of Procedure as well). What's also required is for the focus on devices to go away, and the new focus be on systems. This is exactly what the CIP Modifications SDT proposed in 2018 (a year or so after Maggy left as chairperson), and it got shot down by the big utilities, because they didn't want to have to make big changes to their procedures, etc.

That's the barrier. Until that's overcome, BCS will never be in the cloud, period. I don't see any movement toward this currently, but I'd be glad to help out the insurrectionists if they materialize.

I’ll close by paraphrasing the ending to my post linked above:

Of course, changing CIP will require a much more fundamental revision of the CIP standards than even CIP version 5 was. Doing what I’m suggesting will require widespread support among NERC entities, and I see no sign of that now. Does that mean BCS will never be allowed in the cloud?

I actually believe it will happen, although I won’t say when, because I don’t know (it definitely won’t be soon). I think the advantages the cloud can provide for NERC entities are so great that they will ultimately outweigh the general resistance to change. But the NERC entities themselves need to be able to change. Until that happens, there will be no BCS in the cloud, period.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the CISA’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Last week, Mariam Baksh of NextGovwrote quite an interesting article about how the Defense Department is rethinking its Cybersecurity Maturity Model Certification (CMMC) program for certifying the cybersecurity of defense contractors through third-party audits. Of course, rethinking a program isn’t too unusual in the government (or in private industry), but doing so a year after the program was launched is quite unusual.

However, I’m glad they did this, since I now realize – after not having paid much attention to the program before, to be honest – that CMMC would have been a disaster if it had really been implemented as written. I’m glad to see that DoD is going to actually – get this – consult with the contractors being regulated as they revise the program.

This is in contrast with the TSA, which gave the pipeline industry only three days to comment on the cybersecurity order they were developing, and enjoined anyone who had seen the order from revealing anything about what’s in it. Although I can’t particularly blame them for that last part, since what’s in the order is pretty  embarrassing.

The big problem: The TSA order requires a bunch of mitigations that are impossible to achieve (“Prevent users and devices from accessing malicious websites…”? Piece of cake! All you have to do is identify all of the “malicious websites” in the world, update the list minute-by-minute, and seamlessly block every URL. What could be simpler?). The only mitigation it doesn’t require is what would have actually preventedthe Colonial Pipeline attack. It seems that wasn’t even considered (aka “The light is better here”).

Fortunately, I’m certain the order will never be implemented, since some consultation with the pipeline industry would have shown TSA that full compliance with the order would probably be beyond the means of any pipeline company, period; and in the end, no regulation that literally can’t be complied with will be allowed to stand. Which brings me back to the CMMC. That’s based on NIST 800-171. This document has many more requirements than the TSA order, although they’re much more – how can I say it? – sensiblethan the TSA requirements. However, NIST 800-171 shares with the TSA order the fact that it lists mitigations, not risks.

It also shares with the TSA order the fact that it doesn’t address the most important risk in the domain being addressed. NIST 800-171, which is a supply chain cybersecurity risk management standard, omits any mention of software supply chain cyber risks, which are without doubt the most important supply chain risks today (my guess is 800-171 would be very different if it had been written after SolarWinds and EO 14028).

Cybersecurity is inherently a risk management process, requiring three steps of the organization:

1.      Identification of the high-level risks to be mitigated - i.e. the cybersecurity “domains” being addressed;

2.      Identification of low-level risks included in each domain, that are applicable to the organization and the environment in which it operates; and

3.      Identification of appropriate mitigations for those risks – meaning appropriate for the organization and the environment in which it operates.

Any cybersecurity standard needs either to require that the entities being regulated take these steps, or – if whoever drafts the regulations doesn’t trust those entities – take them on its own, and simply require the entities to implement mitigations for the risks that the regulator has identified (i.e. what I and others call the prescriptive approach). The latter is the approach that both the TSA pipeline order and CMMC/NIST 800-171 take. It’s also the approach that most of the NERC CIP requirements that were written as part of CIP v5 take (e.g. CIP-007 R2 and CIP-010 R1. Fortunately, literally all of the CIP requirements and standards written after CIP v5 are risk-based, since the industry seems to have finally learned its lesson about prescriptive cybersecurity requirements).

The prescriptive approach would work great if

A.     Whoever wrote the requirements had perfect knowledge of all current and future risks in the domain being regulated – e.g. pipeline or Bulk Electric System operations;

B.     Those persons also had perfect knowledge of all current and future mitigations for those risks, and could choose the best ones;

C.      The entities being regulated are similar enough that the risks and mitigations that are appropriate for one will be substantially the same as for another (of course, we know well that’s true in the power industry, where there’s very little difference between say ConEd and a coop in the middle of Nebraska); and finally

D.     The requirements are written so that, taken as a whole, they won’t pose an undue burden on an organization of any size, given that I know of no organization that has an unlimited budget for cybersecurity mitigation.

Needless to say, no person and no single organization meets the first or second criteria, and very few if any groups of regulated entities meet the third criterion. As for the fourth criterion, it would also take a group of people with godlike powers of perception to draft such requirements. The problem is that people who aren’t gods will inevitably err on the side of over-regulation. They’ll list a requirement for everything they can think of that could be important, with no consideration of whether having to meet every requirement might literally bankrupt most of the organizations that have to comply with the requirements.

Then how should a cybersecurity regulation be written? I’m glad you asked that. Looking at the three steps listed above, I believe the regulation itself should accomplish the first step – that is, the regulation should identify the high-level risks to be mitigated. This at least gives the organizations being regulated a place to start, as opposed to telling them to identify risks starting with a blank piece of paper.

Then the organization being regulated should take the second two steps on their own, although with oversight (and advice) from the regulator:

2.      Identify low-level risks included in each domain, applicable to the organization and the environment in which it operates; and

3.      Identify appropriate mitigations for those risks – meaning appropriate for the organization and the environment in which it operates.

Are there any cybersecurity requirements or standards that require these three steps – and nothing more? I’m sure there are a few, but the closest requirement that I know of is NERC CIP-010 R4, the requirement for cybersecurity of “Transient Cyber Assets” (e.g. laptops) and “Removable Media” (e.g. USB drives) used temporarily at power facilities like substations. This requirement doesn’t actually mention risk at all, but it requires a plan that includes ten sections describing mitigations for specific risks like “Introduction of malicious code”, “Software vulnerabilities” and “Unauthorized use”. These are high-level security domains, for each of which the utility has to develop appropriate mitigations.

The requirement even suggests high-level mitigations in each domain, that the utility might decide to implement. And these mitigations almost always provide the option of “Other methods…” of the utility’s own choosing. However, if the utility decides to implement another method, they need to convince an auditor that the method they chose does as good a job of mitigating the risk in question as the examples listed in the requirement.

How about NERC CIP-013? That’s definitely a risk management standard, but it doesn’t list high-level risks. It just tells the utilities to identify supply chain cybersecurity risks on their own, without stating that they should consider domains like software security, manufacturing security, software vulnerability management, etc. Therefore, in my opinion, it doesn’t make the cut. CIP-013 also doesn’t require mitigations at all – although that was clearly due to a simple oversightby the drafting team.  

So I’m glad to see that DoD is going to revise the CMMC program, since I simply don’t see how it can be fully implemented as written. And I’m especially glad to see that they’re going to get input from the contractors who will be regulated (some of them, anyway), rather than trying once again to require them to address every cybersecurity requirement that they could think of, with no regard for what’s the best way for contractors to mitigate the most cybersecurity risk possible on a non-infinite budget.

Of course, I can’t blame the folks at DoD for thinking that other organizations have infinite budgets. If DoD says something is needed and asks for it passionately, they’ll get it. And if they keep asking for more and more and that results in higher costs, they’ll get the funds needed to meet those higher costs, too. Plus, if a bunch of nosy reporters ask about the reasons for those higher costs, the answer will be – of course – classified. Just look at the F-35.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the CISA’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

I’ve said many times in this blog that I think software supply chain security attacks and ransomware attacks are the two biggest cyber threats in the world today. And in terms of growth rate, the former have it all over the latter. Ransomware is now a fairly mature threat, and while it’s definitely growing, it’s not growing at anywhere near the rate of software supply chain attacks.

In this post, I pointed out that Sonatype’s annual State of the Software Supply Chain report in September said that, in the 12 months starting May 2020, software supply chain attacks increased from below 2,000 to around 12,000, or 650%. By contrast, from February 2015 to June 2019, 216 software supply chain attacks were recorded. So we went from 216 attacks in more than four years to 12,000 attacks in the last 12 months. Wouldn’t just about any CEO be willing to sacrifice one or two limbs to achieve that rate of growth? And now ENISA, the EU Agency for Cybersecurity, predicts that there will be four times as many software supply chain attacks in 2022 as in 2021.

Fortress Information Security, as you may know, is in the business of supply chain cybersecurity, and they are quite aware of the importance of software supply chain attacks. That’s why they’re hosting a webinar on Thursday November 18 from 11 to 12 EST titled “Patch Poisoning: Critical Infrastructure’s Backdoor”. They’ve put out a white paper to go with it, titled “Patch Poisoning: Software Supply Chain Attack Detection and Prevention”, which does an excellent job of explaining a number of serious attacks that you (and I) have barely heard of. No developer credentials are required!

“Patch poisoning” – a wonderfully descriptive term that I’ve never heard before – is a great way of characterizing software supply chain attacks. They literally poison the trusting relationship that develops between suppliers and customers of any product, but especially software. Of course, Exhibit A is the 18,000 SolarWinds customers who allowed the seven “poisoned” updates of SolarWinds Orion to be automatically installed on their networks. And why shouldn’t they have done so? They’d trusted SolarWinds for years, and there was no reason to even consider not letting these updates install again. After all, the updates wouldn’t even have been allowed to install if they weren’t digitally signed by SolarWinds…And if you can’t trust an update signed by your supplier, what can you trust?

Good question. Sign up for the webinar (and read the white paper) to learn the answer. 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the CISA’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.