Sonatypecame out with their annual State of the Software Supply Chain report last week. Who’s Sonatype, you ask? I must confess that, until I fell in with a bad crowd at NTIA last year, I hadn’t heard of Sonatype, either – or for that matter, a lot of companies that play an important role in securing our software supply chain. I first learned of them when their 2020 report came out a year ago, and I wrote this postabout it.

What struck me most in last year’s report were

1.      The huge number of components found in the average software product (135 to be exact, although my guess is that number has gone up this year, since it’s been growing steadily. It was 73 in 2017. Of course many products have thousands of components);

2.      The large percentage of those that are open source (90%);  

3.      The fact that 11% of open source components have at least one vulnerability; and

4.      The fact that in 2020, a survey of 5,000 developers found that 21% had experienced an open source component-related breach in the past 12 months. This was down from 31% in 2018.

My big takeaway from last year’s report was that the biggest source of cyber risk in software is clearly the components that are included in it (as opposed to the small amount of code that the supplier of the software itself wrote), and the lion’s share of that risk is due to the open source components.

Does this mean you should tell your software supplier that they need to remove all open source components from their product, to reduce your risk? If they don’t laugh in your face (which would be dangerous at this time, given the Delta variant), they’ll probably say something like, “Sure we’d be glad to accommodate you. Let’s see…We already know that our product would be about 20-50 times more expensive if we had to replace the open source components with code we wrote ourselves – and if we even could write all of that code on our own.

“We invoiced you $100,000 when you bought our product recently. If you’d like our open source-free version when it’s ready in five years, that will cost you between $2 million and $5 million, which you’ll have to pay up front. Let’s say $5 million, to be safe. Will that be cash or credit card?”

You get the idea: we wouldn’t have anywhere near the volume of software we have today – and even then, we’d pay a much greater portion of our national income for software – if open source components weren’t available to effectively augment your supplier’s paid developers with an unseen worldwide army of unpaid developers. And software suppliers are becoming more and more dependent on open source components all the time.

However, the bad guys have also noticed that open source use is growing rapidly, so they’re continually finding new ways to take advantage of that growth. And this year’s report shows how successful they’ve been in that quest. In a section titled “Software Supply Chain Attacks Increase 650%” (page 10), the article points out that, in the 12 months starting May 2020, attacks increased from below 2,000 to around 12,000.

But this growth is even more amazing when you compare it to what came before: “From February 2015 to June 2019, 216 software supply chain attacks were recorded. Then, from July 2019 to May 2020, the number of attacks increased to 929 attacks.” So we went from 216 attacks in more than four years to 12,000 attacks in the last 12 months.

And what were these attacks? Little piddling attacks you never read about? Not at all. You can see the rogue’s gallery of attacks on page 12, but just in the last nine months, there were Solar Winds and Kaseya (which could count as 1500 attacks, since 1500 customers of MSPs that used Kaseya software ended up being compromised with malware. However, the article counts this as one attack).

How did these attacks increase so quickly? The article says (page 10):

Legacy software supply chain “exploits," such as the now infamous 2017 Struts incident at Equifax, prey on publicly disclosed open source vulnerabilities that are left unpatched in the wild. Next-generation software supply chain “attacks” are far more sinister, however, because bad actors are no longer waiting for public vulnerability disclosures to pursue an exploit. Instead, they are taking the initiative and injecting new vulnerabilities into open source projects that feed the global supply chain, and then exploiting those vulnerabilities before they are discovered. By shifting their attacks “upstream," bad actors can gain leverage and the crucial benefit of time that that enables malware to propagate throughout the supply chain, enabling far more scalable attacks on “downstream” users.

So it seems that the only thing growing faster than open source software use is open source software attacks. But there’s another really interesting trend. You might wonder if these attacks are happening because suppliers are getting sloppy about security. Au contraire. Later on in the paper (page 17), the authors discuss a metric called mean time to update (MTTU) – which is the mean time that a software supplier takes to update their open source components.

The reason this metric is so important, the paper shows, is that this metric is highly inversely predictive of the security level of the software product (i.e. relative absence of vulnerabilities). What’s really striking is how it’s changed:

⊲ 2011 average MTTU = 371 days

⊲ 2014 average MTTU = 302 days

⊲ 2018 average MTTU = 158 days

⊲ 2021 average MTTU (as of Aug 1) = 28 days

So at the same time that the number of attacks has gone through the roof, the developers have gotten security religion and improved this metric more than 10-fold since 2014 (302 days to 28 days). Why is this happening? Unfortunately, it seems that the bad guys are learning how to attack software even faster than the developers are learning how to protect it.

Now, do you doubt that software supply chain attacks are not only the most important source of supply chain cyber risk, but probably the most important source of cyber risk, period? I certainly think so.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the National Technology and Information Administration’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

The Energy SBOM Proof of Concept – sponsored by the Department of Energy and the National Technology and Information Administration – is entering a new education phase. Due to popular demand, we decided to provide an in-depth look at what goes into SBOMs – i.e. what are the ingredients in the software bill of materials cake, anyway?

In fact, we got so carried away with the cooking metaphor that we’re going to have a series of online “cooking classes”, at which noted SBOM “chefs” will demonstrate how they combine the elements of SBOMsand VEXes, as well as other ingredients, to produce software transparency. Truth be told, this is the ultimate goal of the NTIA software component transparency initiative, of which the energy PoC (and the contemporaneous healthcare and autos PoCs) is one part.

The first cooking class is next Wednesday, September 22 at noon ET. Julia Child being unavailable, we were quite happy to engage Steve Springett, who is co-leader of the OWASP group that develops and supports CycloneDX, one of the three major SBOM formats. Steve is also the initiator of the dependency-track “continuous component analysis platform” (which has been in operation since 2012 and has gained a wide following in its own right). In a few words, dependency track lets you upload SBOMs for the software your organization runs and track vulnerabilities found in components of that software. All for free, of course.

Here’s the webinar information (no registration is required):

Microsoft Teams meeting

Join on your computer or mobile app

Click here to join the meeting

Or call in (audio only)

+1 208-901-7635,,877158748#   United States, Boise

Phone Conference ID: 877 158 748#

Find a local number | Reset PIN

Learn More | Meeting options 

At the following biweekly meeting on October 6, we’re pleased to have Kate Stewart, VP of Dependable Embedded Systems (how’s that for a title?) of the Linux Foundation. Kate has been the leader of the team that developed – and continues to enhance and support - the SPDX format, which started 11 years ago. Two weeks ago, SPDX was “recognized as the international open standard for security, license compliance, and other software supply chain artifacts as ISO/IEC 5962:2021.” Quite an achievement!

Both Steve and Kate will do roughly the following, using a “basic” open source project from an OSS repository like GitHub:

1.      If possible, share the URL for the project with our mailing list in advance;

2.      Walk through how to build an SBOM (in CycloneDX or SPDX format, respectively) based on that project – with a lot of emphasis on explanation;

3.      Discuss basic use cases for the SBOM; and

4.      Show how the same method could be used to support a bigger and more complex project.

There will be time for Q&A. We also hope to be able to distribute some tasty samples, if we can overcome the (probably) small technical problem of decomposing them into digital bits and reassembling them into food at your computer. We have some people working on this problem as I write, and I fully expect they’ll have a solution by Wednesday. How hard can it be?

If you’d like to get a good preview of Steve and Kate, they did a great webinar on the “Roots of SBOM” recently, along with Chris Blask of Cybeats. The webinar was sponsored by Cybeats.

The PoC meetings are open to everybody, even if you’re not directly involved with the energy industry. You don’t have to sign up for the webinar, but if you’d like to be on our mailing list, drop an email to SBOMEnergyPOC@inl.gov.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the National Technology and Information Administration’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

My post yesterday discussed the concept of VEX, a document that can be described as a “negative security advisory”. The format for this document was recently developed by a working group in the NTIA software component transparency initiative, working in conjunction with the OASIS CSAF project. At the end of the post, I suggested that VEX could solve not just the problem it was designed to address - the fact that a large percentage of vulnerabilities found in components included in a software product aren’t in fact exploitable in the product itself - but a much wider problem.

The wider problem is that security advisories today (both positive and negative) are normally delivered in proprietary, non-machine-readable formats. If those advisories would be provided in the VEX format, there could be huge improvements in efficiency of vulnerability management, both in private industry and in government. In yesterday’s post, I suggested that this might happen someday, although I was careful not to say it was coming anytime soon.

After I put up the post, I decided to watch a video that Allan Friedman (leader of the US government effort to promote SBOMs, until recently at NTIA and now at CISA) had provided to the VEX working group that morning – and I realized afterwards that this should change the tone of what I wrote in that post. But since I don’t want my posts to be too long (God knows they’re long enough!), I decided I’d write this followup post today.

The video was of a presentation that Allan made with Jens Wiesner. Jens is with the German Federal Office for Information Security, which Jens explains is “more or less” the German CISA. Although Jens doesn’t say it, it seems he and the people who work for him are responsible for the government’s cyber vulnerability reporting in Germany. One of his staff members, Thomas Schmidt, worked with the NTIA VEX working group to create the VEX format as a profile in CSAF, the new open source vulnerability reporting standard that will be implemented soon (CSAF replaces the current CVRF standard, which was developed by the same OASIS group). A lot of large organizations worldwide, including Cisco and Oracle, are committed to using CSAF (and by implication VEX, since a VEX just uses a particular set of fields in CSAF) when it’s approved.

While Allan gave a good introduction to the concept of VEX in the first half of the video, I was most impressed by Jens’ discussion (in the second half) of a big problem that his team is experiencing:

1.      Since security advisories nowadays are in proprietary formats (i.e. Cisco’s advisories don’t look like Microsoft’s, which don’t look like Oracle’s, etc.) and they aren’t machine-readable, his staff members spend lots of time just reading advisories and summarizing them in a standard format.

2.      Once the advisories have been summarized, the data are published in a machine-readable format. Of course, if the advisories themselves were all machine-readable, the data could be transferred directly to the publishing formats – completely eliminating the huge amount of time spent having expert humans read and summarize the advisories.

3.      Jens discussed this at length, and you could hear the pain in his voice about the time wasted doing this (even worse, he’s aware that staff members are bored by this work, which of course means he’s probably always worried that some of them will quit). Ironically, there are probably lots of other governments and private organizations that have the same problem Jens has: they pay security professionals a lot of money to read and summarize security advisories.

4.      Another problem with non-automated advisories is that they have to be emailed out. If you’re looking for a particular advisory from a software company you don’t normally deal with, you’re going to have to do some digging around and perhaps calling to find it, since you’re probably not on the supplier’s mailing list to receive advisories. Machine-readable advisories can simply be made available at a URL and updated in real time. Wouldn’t that be nice?

5.      However, there is something holding up Germany’s officially adopting CSAF as the vulnerability reporting framework for government and industry. Jens said it’s the lack of “asset matching” tools on the user side. While he didn’t elaborate on what he meant, I can guess: There aren’t good tools now that will take the machine-readable vulnerability information reported through CSAF and figure out how this applies to individual devices, so they can be scanned and remediated. IMO, this is also the biggest issue holding up full adoption of SBOMs.

6.      Allan and Jens both pointed to this lack of tools as a business opportunity and called on existing or new companies to move to fill these gaps.

7.      However, Jens also pointed to another possible solution to this problem: Third party service providers could ingest VEX (and SBOM) data and perform the asset matching services needed to map vulnerabilities to individual devices on the user’s network. So the lack of asset matching tools doesn’t mean that VEXes and SBOMs can’t be used in an automated fashion; it just means the automation will usually be operated by a service provider, not by the individual user organization.

8.      Ultimately, I’m sure there will be inexpensive and effective tools that will allow at least larger and better-resourced organizations to directly utilize SBOMs and VEXes for vulnerability management; but this might not happen for a few years. In the meantime, I think third party service providers will provide a “bridge” to SBOMs and VEXes for these larger organizations. And medium- and smaller-sized organizations will perhaps always find it more effective and efficient to use these third parties.

9.      Allan and Jens both agreed that machine-readability is the only good solution for vulnerability management in both the near and longer terms, and that it seems inevitable that most security advisories will be machine-readable in the not-too-distant future.[i]

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the National Technology and Information Administration’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

More than a year ago, the NTIA Software Component Transparency Initiative came to the realization that there was a need for another new type of document, somewhat related to software bills of materials (SBOMs) but serving a different purpose. The initial name for the document was VEX, an acronym for Vulnerability Exploitability eXchange; this name has stuck. I’ve written two posts about this document, the more readable (and recent) of which is this one.  

I’ll let you read the previous post, but my purpose now is to describe why I’ve come to believe that VEXes might end up having as big an impact on software supply chain security as SBOMs, perhaps even more. The NTIA workgroup that’s been working on VEX has so far finished just one document – a one-pager – that describes VEX. It will be published this week (and will be available here). More documents will follow later, and I’m sure I’ll have more blog posts.

You may be wondering what the h___ the workgroup has been doing for a year, if all they’ve been able to accomplish is developing a one-page document. 90% of our work (I’ve been part of that workgroup) has been developing the format for VEX, and working with the OASIS Common Security Advisory Framework (CSAF) project to incorporate VEX as a “profile” within CSAF 2.0. CSAF 2.0 in an “official draft” status, meaning it’s completed the first of two or three steps required for it to be approved as an international standard. It will replace the existing CVRF format, which was developed by the same group. The German government is very involved with this project and intends to approve it as the official standard for vulnerability reporting in Germany (probably other European governments will do that as well).

That work is now done, so that a VEX will simply be a special case of a CSAF document. The purpose of “piggybacking” off an existing standard was to avoid (whenever possible) creating new formats. The NTIA initiative hasn’t created its own SBOM format, either. Instead, they have identified three existing SBOM formats – SPDX, CycloneDX and SWID – as equally worthy of consideration for a software supplier starting to produce their own SBOMs. In fact, last week SPDX was approved as an ISO standard; yet the three formats are different enough – and equally robust – that a supplier should consider all three  before they choose one (and a supplier can certainly produce SBOMs in multiple formats. There’s no need to stick to one).

Now that the format is settled, the VEX workgroup is starting to turn its attention to use cases and “rules of the road” for VEX. It turns out that, as I’ve delved further into these issues, I’ve come to realize that VEX isn’t just an adjunct document for SBOMs; it could come to be a key element of software supply chain security in its own right, as important as SBOMs themselves. Why do I say this? I’m glad you asked. Here’s some of what I’ve come to realize recently, although there’s much more that can be said.

Since the average software product has over 100 components (some have thousands), and since 90% of the average product consists of components, this means that vulnerabilities are far more likely to be identified in components of a software product than they are in the product itself. However, the NVD doesn’t list vulnerabilities in components of a product, just those found in the code actually written by the supplier you bought the product from.

That’s the bad news. The good news is that, for various reasons, the majority – and perhaps the great majority of these component vulnerabilities aren’t actually exploitable in the product itself. That is a good thing, but it does make it likely that a lot of time will be wasted by suppliers and software users, responding to false positive reports of component vulnerabilities.

The solution to this problem is for suppliers to issue notices to their customers that essentially say “Even though CVE Y is found in component X, and component X is included in our product, CVE Y isn’t in fact exploitable in our product.” This might be because the supplier has already patched that vulnerability, but it could also be for various technical reasons. However, these notices need to be machine-readable, just as SBOMs need to be. That way, they can be fed into automated tools for vulnerability and configuration management, rather than have a burdensome manual process in between the VEX (or the SBOM) and the tool.

Every security person is used to receiving notices from software suppliers about vulnerabilities that are found in their products (or “exploitable in their products”); I call these positive software vulnerability notifications. However, the VEX can be thought of as a negative vulnerability notification, since it says that a vulnerability isn’tfound in the supplier’s product.

Did VEX invent the idea of negative notifications? Not at all. Suppliers have been notifying users of non-exploitable vulnerabilities for years. The biggest example of this is patch notices: They say that if the user applies a patch to the software they’re running (or they download the current patched version), one or more vulnerabilities that were previously exploitable in the product will no longer be exploitable.

However, it is very likely that the number of VEXes will be overwhelming, far more than the number of patch notices currently put out. For example, since the average software product has 135 components, let’s say that each of these has a 1% chance of developing a vulnerability during any year; let’s also assume that the supplier’s own code in the product (which is actually called the “principal component” by the NTIA initiative) has a 1% chance of developing a vulnerability.

Since only the latter vulnerabilities are likely to appear in the NVD, this means that the NVD could be assumed to list only 1/136 of the total vulnerabilities that are found in the product – if you don’t take into account the fact that the majority of these vulnerabilities are unexploitable. If 90% of component vulnerabilities are unexploitable, this means that, once suppliers are regularly providing SBOMS (which means users will start looking up component vulnerabilities in the NVD), they will want to issue at least 120 VEXes every year, for every vulnerability identified for their product in the NVD. In fact, the number of VEXes per year is likely to be much larger than this, since my guess is more than one vulnerability is identified every year for the average software product.

However, a software product could easily have thousands of components (and some do, including products used every day by corporate and government users), at which point these numbers become huge. If a product has 5,000 components, this means there will be 5,000 component vulnerabilities identified every year, per my assumption. Since 90% of these won’t be exploitable and will need a VEX to state that fact, this means that the supplier of this product will have to issue at least 4,500 VEXes every year for the product.

This is of course a big number, but this explains why VEXes have to be machine readable. A user can receive SBOMs in a human-readable format (usually CSV or XLS files), but trying to keep track of VEXes will probably overwhelm – in a few years, not immediately – any manual effort to do that. However, it will be quite possible to track all VEXes, and to match them to a database of product components, through automated means.

And my feeling is that, once the tooling (or third party services) is developed to process VEXes, other types of negative software vulnerability notifications – especially patch notifications – will also move to the VEX format. Moreover, since VEXes can easily provide notification that a vulnerability is exploitable in a particular product, it seems to me that, in the not-too-distant future, those positive notifications will also move to VEX (right now, they’re mostly not machine-readable, and they’re provided in proprietary formats particular to each supplier. Note that suppliers will still be free to issue those advisories, even when they start providing positive VEX notices of exploitable vulnerabilities, since a lot of users will still want to have an advisory that they can read).

In other words, VEX is coming. And it might turn out to be an oncoming train. You should start thinking about how your organization might jump aboard.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the National Technology and Information Administration’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

This is my 843^rd post since I started this blog on the last day of January in 2013. I’ve completely forgotten about a lot of my old posts, but sometimes I see on my dashboard that there’s a cluster of people who have been looking at a post I’d completely forgotten. This happens more often, now that there’s a real search engine on my blog (you have to go to the blog’s main page to use it).

My most recent post, put up last Sunday, was about Joe Weiss. I’ve written about ten posts on him, all quite critical of him. Sunday’s post was no exception, and in it I referred to about four previous posts I’d written about him. As I expected, I saw afterwards that people were reading those posts.

But I was also surprised to see a post from January 2016,that I’d completely forgotten about, get a bunch of hits – in fact, even after reading it now, I have no real recollection of writing it (although it sounds like something I might write, so I have no reason to believe anybody broke into my Blogspot account and wrote it!).

You can read this post, but here’s my Cliff Notes summary of what’s important in it:

1.      In writing the post, I was reacting to a press release by the Foundation for Resilient Societies, which quoted Joe as complaining that “no current or proposed federal regulation requires encryption or other cyber-protection of grid communications with substations.”

2.      As a result, “foreign governments have been able to implant malware” in the grid, presumably by intercepting unencrypted communications between control centers (although he mistakenly calls these “control rooms”, which means something else in the power industry). And why have those governments been able to do this? Because the utilities are using the “public internet” to handle these sensitive communications.

3.      The allegation that communications between substations and control centers had been intercepted by foreign governments was of course a complete fabrication; nobody – and certainly not Joe – has ever introduced evidence that this has happened.

4.      And the allegation that utilities are using the public internet to communicate with substations? I said at the time (and still do), “ I know of no electric utility that uses the public internet to communicate with its substations, encrypted or otherwise. The communications channel is always private (whether carrier-owned or utility-owned), often serial or Frame Relay.” I should have added SONET to that list. Again, a 100% fabrication.

So what’s Joe’s solution? Very simple: The NERC CIP standards need to be revised to require encryption of communications between control centers and substations. For the moment, let’s put aside the fact that there’s no need for encryption on purely private channels. What would happen if we did it anyway?

It’s pretty clear what would happen. Substation communications require responses in fractions of a second. The latency that would be induced by encryption would cause a lot of needed commands (especially opening or closing a circuit breaker) to go unexecuted or to be executed too late to do any good, leading to a lot of grid reliability problems. And Joe knew this in 2016, since anybody involved with substation automation would have told him that.

And this is why FERC, when they ordered NERC to develop a standard for encryption of communications between control centers (which is much less sensitive to latency) specifically didn’t extend that requirement to substation communications. That order, Order 822 (which ordered development of CIP version 6, although the encryption requirement for control centers was incorporated into a new standard, CIP-012) came less than a week before I wrote the post.

The bottom line is that we’re lucky that nobody in the power industry took Joe’s statements seriously then (which they might well have done if they’d been supported by a single shred of evidence). And since Joe’s normal modus operandi of totally unproven allegations – nay, not just unproven, but fabricated out of thin air – continues, nobody in the industry takes what Joe says seriously today, either. Instead of substation communications, Joe now fulminates about the imminent danger from the Aurora vulnerability, level 0 attacks, and of course “hardware backdoors” (as in the Great WAPA Transformer Incident). He alleges – always without bothering to provide a shred of evidence - that all of these threats have been realized in successful attacks. But nobody in the industry believes him.

So why do I bother writing about Joe? It’s because, despite nobody in the industry believing what he says, he still has tremendous influence, due to the fact that so many people in DoE and the power industry are afraid of the trouble he – and his legions of woefully misled fans – can bring down on them. When someone brings up Joe’s latest lie (and there seem to be lots of people who are eager to do that. Joe has a bunch of devoted followers), these DoE people nod and scratch their heads and state very solemnly that yes, these are serious questions, and we need to look into them. Even worse, they do look into them, since they feel they have to – despite the fact that they know there’s no truth to them (for example, think of the huge amounts of time invested last year, in response to the EO, in searching for sources of cyber vulnerability in devices that don’t even have a microprocessor. About 20 of the 25 device types listed in the EO fell into this category).

Yet it’s only recently that government and industry have acknowledged that the most important source of cyber threat to the power grid, or almost any other industry, is software vulnerabilities, whether deliberately planted or (usually) due to poor development practices. Tracking down these vulnerabilities, and especially the poor practices that led to them, is much more difficult than going after the easy-to-understand movie plots about bad devices causing catastrophic grid failures that Joe traffics in. This is why the pressure Joe is exerting on people to investigate his fairy stories – and especially DoE employees who fear for their jobs if they stand up to him – is inevitably causing us to shortchange the real threats we’re facing. Hardware backdoors aren’t one of them.

In my last post, I talked about Senator Joe McCarthy’s lies, but I didn’t emphasize what a terribly destructive effect he had on the US government, and especially on the State Department. Because of him, the analysts who could size up a situation and make a rational decision on the best course of action were all pushed out (or worse) and replaced with hard-line anti-Communist ideologues. Those guys (almost all males, to be sure), led us into the tremendously destructive quagmire of the Vietnam War, as well as other foreign policy misadventures of the 1950’s and 1960’s.

By the same token, very minute a DoE or utility employee spends worrying that he or she will get canned if they don’t treat what Joe says with great respect is a minute they don’t spend acting on the really important threats faced by the grid. This is the real problem with Joe’s clown show. It’s time to call out Joe’s lies for what they are. Repeatedly.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the National Technology and Information Administration’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Someone forwarded me a blog post by Joe Weiss last week that made me think of an encounter between two other guys named Joe. This encounter occurred on June 9, 1954, the 30^thday of the Army-McCarthy Hearings. The hearings were called to investigate a series of conflicting charges between Sen. Joseph McCarthy and the US Army, arising from McCarthy’s year-long (mostly fruitless) investigation into possible Communist influence in the Army. The Army was represented by the eminent lawyer Joseph N. Welch. The hearings were broadcast live to a huge audience on the relatively new medium of television.

At one point in the hearings, McCarthy started attacking a young member of Welch’s law firm who had belonged to a left-leaning organization in college, but who was not involved in the hearings. Because this violated a previous agreement between Welch and McCarthy, Welch interrupted McCarthy with these memorable words: “Let us not assassinate this lad further, Senator; you've done enough. Have you no sense of decency, sir? At long last, have you left no sense of decency?” This 1-2 minute exchange is widely acknowledged to be what finally broke the back of McCarthy’s campaign of character assassination, which destroyed the careers of many people, especially government servants in the Department of State and other agencies.

I think it’s time to ask this same question of another person named Joe: Joe Weiss. Since last spring, he has been pushing an outright lie about a Chinese transformer that was ordered by the Western Area Power Authority (WAPA), which is part of DoE. The transformer was never installed at the substation for which it was intended, the Ault substation outside of Denver. Instead, when it arrived in the US last year, it was diverted to Sandia National Laboratories in Albuquerque, another part of DoE. There, it was presumably pulled apart in a search for…well, something. There has been no announcement of anything having been found, although Joe claims something was found – in fact, he says he knows all about it.

These are the facts:

1.      On May 11, 2020, Joe published a blog post that broke the story of the transformer, and said “When the Chinese transformer was delivered to a US utility, the site acceptance testing identified electronics that should NOT have been part of the transformer – hardware backdoors.” In other words, the transformer was shipped to the site (or at least put in the possession of WAPA), and there something called a “hardware backdoor” was identified in it; the transformer was then shipped to Sandia for analysis.

2.      On May 31, I put up a post pointing out that a Wall Street Journal article had said the transformer was shipped directly from the port of Houston (where it arrived in the US) to Sandia, and was never delivered to WAPA at all. So Joe’s statement that the backdoor was identified when the transformer was being installed was clearly wrong. I discussed a lot more that was wrong with his post, which I’ll let you read.

3.      Some time after that, Joe changed his story to say that the hardware backdoor was discovered at Sandia, not at the Ault substation. But then the question became why the transformer was shipped to Sandia in the first place. Did the government have some reason to believe that the transformer would arrive from the manufacturer already compromised? After all, the transformer had been on order for a year, and every single component installed in it was specified in advance by WAPA engineers – since they wanted to make sure there wouldn’t be any questions about something being planted in it. If there were concerns that a “hardware backdoor” would be installed in the transformer, why wasn’t the order cancelled early on (like in 2019?), saving WAPA a lot of money and their engineers a lot of time?

4.      But not to worry, Joe now has a new story that explains everything! It must have slipped his mind when he wrote the blog post last May, but in fact the backdoor was found when a similar transformer from the same manufacturer was installed at the Ault substation in 2019[i]! He made this assertion in a January articlein Forbes, which I wrote about in thispost.

5.      However, I regret to say that the January article didn’t convince me of the veracity of Joe’s new claim, since he provided no evidence for this, nor did he explain why he had never mentioned this previously when writing about the WAPA transformer.

6.      But I’m sure Joe wouldn’t have made this statement without evidence, so I was pleased to see that he did provide it during a joint press interview with Mark Weatherford in April. What’s the evidence? Here’s what Joe said (presumably with a straight face, which I certainly wouldn’t have been able to do): “I can read you—I won't even mention the country—an email I got from one of our closest allies. From someone very senior. And it's saying, ‘I am hoping you can help me with something. Regarding the transformer issue you discuss, can you please tell me to what level that information is confirmed?’”

7.      That’s Joe’s evidence! Somebody (high-ranking, of course. Joe never deals with anybody who isn’t high-ranking!) sent him an email asking for evidence on what he was saying about transformers (presumably the WAPA transformer, but maybe not). And Joe points to that email as evidence that a “hardware backdoor” was discovered in a Chinese transformer being installed at the Ault substation in 2019. Generally, when I look for evidence, I look for some sort of statement that indicates the specific event alleged actually happened – not a query from someone outside the country, asking for information on something Joe had told him. But I guess Joe and I don’t agree on what constitutes evidence.

Joe’s new blog post, as is typical for Joe, starts with the assumption that all readers know about this 2019 incident, so he doesn’t even bother to bring out his incontrovertible evidence again. However, this post goes on to implicitly accuse a huge number of federal employees, utility executives, and others (none by name, of course, since Joe has no evidence against anybody in this matter) of what can only be called treason: deliberately ignoring and covering up a massive Chinese threat to the US power grid – a cover-up that continues today, it seems.

·        While the post doesn’t specifically mention executives of WAPA, the clear implication of his story about the 2019 “incident” is that WAPA executives (and their superiors in DoE) were grossly negligent of their obligation to report such a serious event to the appropriate parties – because, as we all know, this never was reported in any way to the power industry. Obviously, the only way such a serious event (which if true should have been considered an act of war, IMO) could not have been reported is if there was a massive coverup. We’re really lucky that Joe is the only person on the planet who knows about this incident. It’s just too bad that Joe himself forgot to mention the event until this year. But I’m certainly not going to accuse Joe of treason!

·        And if Joe’s correct, the White House was seriously negligent in not issuing Executive Order 13920 until May 1, 2020. Joe has been insinuating since his original post last May that the WAPA “incident” led to the EO. So why did it take a year for the White House to act? Was it because the president at the time was a secret Chinese agent? I’m guessing that even Joe wouldn’t say that. But if what he says is true, there obviously was a serious cover-up going on at the White House for an entire year, perhaps not involving the president. Where’s Joe’s outrage about that?

·        Joe points out that the EO was designed to “reduce or eliminate” the use of Chinese made-equipment in the US electric grid, yet he says that 54 more Chinese-made transformers were installed in 2020, and more are on order now. This is of course shocking news! Why haven’t the top executives of every utility that placed one of those orders been hauled into court for violating the EO (at last up to the point that the EO was suspended this January)? In fact, if utilities had really followed the EO, they would have suspended all new investment in the US Bulk Power System (BPS) until they were sure that no product they were buying – and no single component of any product they were buying – wasn’t subject to foreign-owned and corrupt influence, pending the Secretary of Energy deciding exactly what that was (the EO gave the Secretary 150 days to do so). Essentially, all investment in the BPS, from May 1, 2020 through when the EO was suspended by President Biden this January, was illegal. Why hasn’t this been prosecuted?

·        I’ll tell you why the utility executives haven’t been hauled into court: DoE held two open virtual meetings for utility executives last May and June. In those meetings, high-level people went out of their way to tell the utilities not to change anything they were doing because of the EO, and especially not to stop any investment they planned on (although in the end, a lot of investment was deterred just because of the FUD surrounding the EO). Offhand, I’d say this doesn’t square with the idea that the EO was a reaction to finding a serious threat to the entire US power grid, do you? But if Joe’s right – and who am I to say he’s not? – then those DoE higher-ups who gave the briefings were engaged in something very close to treason, since they presumably knew all about the WAPA incident, yet they went so far as to tell utility executives to break the law by ignoring the EO. Why didn’t Joe call then for their immediate imprisonment and trial? And even though he’s more than a year late, why isn’t he calling for them to be tried now? After all, we’re still well within the statue of limitations for this alleged crime.

·        Joe also expresses outrage that, not only was the US utility industry not notified about what was found at Sandia[ii], but “this information has not been shared with our closest allies who also have these Chinese-made transformers.” This is terrible! Why isn’t Joe calling for top officials in the State and Energy departments (under the previous administration, of course) to be grilled by Congress on this, or better yet, immediately indicted? After all, those people knew all about a threat to our allies’ national security, yet they said nothing to them about it. Could it be that Joe is getting soft?

·        Joe doesn’t neglect already-installed Chinese equipment, either. He points out in the post that there were at least 150 utilities that installed Chinese equipment in the years 2018-2020, ranging from small to large (he names about 12 of them). Of course, this is much more serious, since it almost certainly means that there are lots of hardware backdoors installed up and down the US grid, just waiting for the command from their all-seeing Master in the Dark Tower in Beijing to send the entire US into darkness (as soon as the all-seeing Master learns what a hardware backdoor is, of course. And when he does, I hope he tells me).

·        Why isn’t Joe calling for all of this equipment to be pulled from service and thoroughly inspected for hardware backdoors (of course, his own company should do this work, since he’s the only person who knows what a hardware backdoor is)? And if the equipment needs to be replaced, it’s the utilities (or rather, their ratepayers) that should foot the bill for replacements, as well as for alternative power service while those replacements are being ordered and installed. This replacement alone might cause serious outages in many parts of the country, but I’m certain Joe will assure us that this is a small price to pay for ridding our grid of the Chinese hardware backdoors – whatever they are – that according to him are all over the place.

·        Joe specifically calls out two US corporations for being heavily involved in the dangerous activity of installing Chinese-made equipment on the US power grid. One is Alstom Grid, a huge equipment supplier in its own right and part of GE, who ordered a lot of Chinese equipment from 2018 to 2020, presumably for jobs on which they were performing an integration function. The other is Double Tree Systems, which Joe says “is associated with JSHP transformers and other Chinese equipment manufacturers connected to the Chinese government. Double Tree Systems continues to provide critical grid equipment and engineering services, including equipment explicitly addressed in EO 13920, to US utilities. Double Tree Systems not only imports and markets Chinese JSHP transformers in the U.S., but sells a variety of critical grid monitoring products and services.” (Joe often uses Big Bold Letters in his posts. This shows that he is Very Serious)

·        Don’t you think executives of both Alstom Grid (maybe even GE itself) and Double Tree (sorry, Double Tree) should be made to publicly apologize for putting the US grid at such obvious risk, and to pay (preferably out of their own deep pockets) for any and all costs their customers may incur in replacing the Chinese equipment with equipment made in the good ol’ US of A? It seems to me that’s the least they could do.

·        But Joe isn’t done yet. He moves on to EPRI (his former employer, a long time ago), pointing out that in 2014, EPRI announced at Distributech that they were organizing – as Joe says – “a demonstration program of the Chinese-made grid equipment”. Why does he say this? Because one of the companies listed in EPRI’s announcement was – you guessed it – Double Tree Systems. So EPRI should certainly be held to account for their evil deeds, as should such companies as Cisco and Schneider Electric, which had the temerity to be mentioned in EPRI’s announcement, alongside Double Tree Systems. They should have known in 2014 that there would be a serious Chinese-caused grid event in 2019, for which Joe now has incontrovertible email evidence. Or something like that. Who cares about the details?

So it seems there’s no real limit to the organizations that Joe feels should be held to account for the Chinese “penetration” of the US grid that exists mainly in his fevered imagination; and most importantly, there’s also no limit to the individuals in those organizations who will have their careers set back or ended because of this. But who cares about their careers! Let the investigations and purges begin!

In all seriousness, why am I writing yet another post debunking Joe’s claims, when my previous posts and other efforts (including a “Defense Use Case” produced by Rob Lee and Tim Conway of SANS last year) clearly haven’t stopped him from promulgating this set of lies (and let’s be clear, these are 100% fabrications, not based on any real incidents at all)?

I’m doing this because, for some reason I honestly don’t understand, Joe feels it’s important to double down on his lies by implicitly accusing what must have been thousands of people of participating in covering up serious national security threats, and most likely actively working with Chinese interests to further those threats. This includes large numbers of people who work or worked for major electric utilities and utility organizations like EEI and the ESCC (who of course should have sounded the alarm about the 2019 “incident”), the Departments of Energy and State, the White House, Cisco, Schneider, EPRI…and on and on.  

I must admit, I previously considered Joe’s stories to be a big joke – they make him look like an idiot, but I thought they were harmless to anyone else. But the events of January 6 show that fabricated allegations of near-treason can lead to real consequences for completely innocent people – who it seems, are usually government employees (fortunately for the people who attack them, government employees are almost never allowed to fight back against either physical or verbal attacks). On Jan. 6, those consequences included death and bodily injury.

The fact that Joe continues to hammer away at his made-up stories, and implicitly accuses these people of endangering the American way of life, makes it increasingly likely that Joe’s followers – and he does have a lot of devoted followers who are convinced he’s the only person who really understands the danger our country is in – will move to start destroying careers, like Joe McCarthy did. In fact, I heard this week of an incident that indicates this may already be starting. The person who reported it to me doesn’t want me to use his name, because – even though Joe’s allegations are easily proved to be false, he’s concerned his career will suffer anyway if it’s understood he’s standing up to Joe. Of course, this is exactly how McCarthy worked – he had every agency in Washington fearing they would be the next one in his sights if they so much as said a word against him.

That is, until Joe Welch said, “Have you no sense of decency, sir? At long last, have you left no sense of decency?”

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the National Technology and Information Administration’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

My last two posts have discussed the two mandatory cybersecurity “regulations” – the IoT Cybersecurity Improvement Act of 2020 and the IoT device “labeling” requirement in the May 12 Executive Order - that were promulgated within about six months of each other (under two very different presidents) recently. At first, it might seem that these couldn’t be more different. Here are some of the differences:

1.      The Act is a law, approved by both houses of Congress and by the president. The EO is simply an order that can’t in itself override a law and could be overridden by another law, if Congress were inclined to pass one.

2.      The Act focuses entirely on IoT devices, while the EO is a sprawling attempt to improve cybersecurity in the federal government, on many different fronts.

3.      The Act focuses entirely on federal agencies, requiring them to incorporate cybersecurity concerns into their procurement terms and conditions for IoT devices. Of course, there’s no doubt that the intention of the Act was to have the Feds set a standard for private industry, but there’s not a word in the Act itself about that. On the other hand, the device labeling provision in the EO, while also aiming directly at procurement by federal agencies, repeatedly speaks of “education” and “consumers”. For example, paragraph (s) of section 4 includes the phrase “educate the public on the security capabilities of Internet-of-Things (IoT) devices and software development practices”. And paragraph (t) says that the labeling program should be “compatible with existing labeling schemes that manufacturers use to inform consumers about the security of their products” (you can read both paragraphs in full in my previous post).

4.      The Act seems to be a variation on a fairly familiar theme: require federal contractors to be assessed against a new standard that would be developed by NIST (and has been. More on that in a moment). On the other hand, just the name “device labeling” was a signal that this is a very different type of cybersecurity regulation than has been seen previously in the US – although it has been used to a limited degree in Europe and Southeast Asia.

I had also considered these to be two very different regulations. However, when I recently sat down to look at them carefully, I began to see more similarities than differences. In fact, I’ve decided the two are so similar that I really doubt they’ll both be enforced. Since I see all of the momentum as being behind the EO, and since I haven’t seen any signs at all that anybody’s even preparing to enforce the Act (OMB is in charge of enforcing both regulations, since they’re the regulator/whip-wielder for federal agencies), I don’t think it will be long before the Act either officially or unofficially sleeps with the fishes.

So why do I think the Act and the device labeling requirement are so similar? Two things:

1.      The “meat” behind both the Act and the EO is requiring federal agencies to require contractors to be assessed based on a standard (or framework) to be developed by NIST. Even though the mechanism for this assessment is called a “device label” in the EO, if you dig into it and look into how the “requirements” for the “label” will be developed, they’ll be specified by NIST, just as the framework for the assessments under the Act will be specified by NIST. Since the subject of both the label and the framework is overall cybersecurity of IoT devices and they’re both developed by the same agency, does anyone doubt that they’ll be substantially the same?

2.      But if you do doubt this, you should put that aside now. Another thing I realized as I was studying the two “requirements” was that the framework behind both of them will be the same: NISTIR 8259D. Folks, when you’re talking about two different regulations, they can’t be any more similar than if they’re both based on exactly the same set of “requirements”.[i]

Maybe you now see why I came to believe that both the EO and the Act can’t stand. They require the same organizations (federal agencies) to take the same actions (require suppliers of IoT devices to be assessed for cybersecurity), with respect to the exact same subject matter (NISTIR 8259D). One often hears about redundancy in federal programs (I’ve sometimes mentioned the Department of Redundancy Department), but believe it or not, there are people who do watch for these things and flag them.

So why do I think the Act will fade away? Mainly because I haven’t heard anything about it being enforced, even though it gave OMB a six-month deadline to start the process (which would have probably passed in July, if not June). On the other hand, there’s been a lot of activity on the IoT device labeling requirement. NIST has scheduled a two-day conference(virtual, natch) for September on this and the other device labeling requirement, found in Section 4 paragraph (u) of the EO, on secure development practices for consumer software.

And just as in the case of SBOMs, NIST – this time in conjunction with the Chair of the FTC as well as representatives of other agencies – must make final decisions on the program by early February, 270 days after the date of the EO. Since a lot more decisions are due then, that will be a busy time around NIST. They might not be able to celebrate Groundhog Day next year.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the National Technology and Information Administration’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

In my most recent post, I described the first of two pending mandatory US cybersecurity regulations for IoT devices, the IoT Act of 2020. Now I’ll describe the second of these and the one that’s more important, the IoT “device labeling” requirement in the May 12 Executive Order on securing software sold to federal government agencies.

Paragraph (s) of section 4, on page 18 of the EO, reads: “The Secretary of Commerce, acting through the Director of NIST, in coordination with representatives of other agencies as the Director of NIST deems appropriate, shall initiate pilot programs informed by existing consumer product labeling programs to educate the public on the security capabilities of Internet-of-Things (IoT) devices and software development practices, and shall consider ways to incentivize manufacturers and developers to participate in these programs.”

Paragraph (t) on page 19 reads: “Within 270 days of the date of this order, the Secretary of Commerce acting through the Director of NIST, in coordination with the Chair of the Federal Trade Commission (FTC) and representatives of other agencies as the Director of NIST deems appropriate, shall identify IoT cybersecurity criteria for a consumer labeling program, and shall consider whether such a consumer labeling program may be operated in conjunction with or modeled after any similar existing government programs consistent with applicable law. The criteria shall reflect increasingly comprehensive levels of testing and assessment that a product may have undergone and shall use or be compatible with existing labeling schemes that manufacturers use to inform consumers about the security of their products. The Director of NIST shall examine all relevant information, labeling, and incentive programs and employ best practices. This review shall focus on ease of use for consumers and a determination of what measures can be taken to maximize manufacturer participation.”

Note that paragraph (u) mandates a labeling program for consumer software. This is similar to the IoT device labeling program and indeed, NIST is addressing these two programs together (e.g. they’ve scheduled a two-day conference in September that will address both programs).

What first struck me when I read these two paragraphs was that putting labels on devices seems to be an odd way to address cybersecurity. I normally think of a label as an indication that an organization has tested a product and finds it meets certain safety standards. In other words, the product is safe to use.

Cybersecurity (or security in general) can’t be reduced to a simple “safe/unsafe” decision. In fact, let’s be honest: Seldom (and perhaps never) in our personal or business lives do most of us decide to buy a software product or an intelligent device, based solely on a judgment of the product’s level of cybersecurity.

Instead, cybersecurity is at most a final gating factor: We make the decision to buy a particular product, and before we place the order (or put it in our shopping cart, whether physical or virtual), we may ask ourselves whether there are any cybersecurity skeletons in the closet of the supplier (a device manufacturer, a software developer, the online community that developed an open source software product, etc.) that are serious enough to warrant not going ahead with the purchase or procurement. But recent history tells us that even a serious cybersecurity breach that had a serious impact on customers (think Target or SolarWinds) in the end has very little impact on demand for what the breached company sells – and in fact the impact might be net positive if the company responded well to the attack (as did both Target and SolarWinds).

So what we’re really looking for regarding the security of an IoT device (or any other product containing or consisting of software or firmware) is an indication of the risks that we are going to undertake if we buy the product. If we know what those risks are, we can then take steps to mitigate at least some of them and accept the rest. And if we buy the product without even bothering to learn about the risks (which I’ll admit is my approach when purchasing products for personal use. My general reasoning is, “Given that well-known manufacturer XYZ develops this product and/or that well-known retailer ABC sells the product, it must be secure enough for my purposes.” Whether that’s good reasoning or not is left as an exercise for the reader), we’re accepting those risks (although we all presumably take some reasonable precautions with any product we buy, like at least set a somewhat secure password, when we can do that).

So the main purpose of supply chain cybersecurity in general (and not just for IoT devices) is to help us understand the risks we undertook when we made the decision to purchase the product, not to decide whether or not to purchase the product in the first place; we may or may not take steps to mitigate any of those risks (or even to learn about them), but if we want to do that, we can. And that’s why the idea of a device label for cybersecurity seemed strange when I read about it in the EO: Since a buy/no-buy decision very rarely depends on cybersecurity considerations, how can a label help you determine what the specific risks are that apply to the device, as well as what steps you could take to mitigate those risks? That’s too much information to fit on a label.

It turns out, although I didn’t know this until recently, that the idea of device labeling for IoT device cybersecurity has been around for a while in Europe. In fact, perhaps the best such program (albeit one that’s almost infinitesimally small compared with the program that will be required in the US) is one introduced by the Finnish government in 2019 (and no bad puns about whether or not the program is Finnished or whether or not it only applies to Finnished products. Bad puns are forbidden in this blog. I can assure you that I’ll never sink to using this low form of humor, such as asking whether the army fights to the (last) Finnish. After all, what kind of blogger do you think I am?).

There are three main components to the Finnish IoT device labeling program:

First, the manufacturer has to certify that the device complies with 18 of the 70-odd requirements, which are referred to as “Provisions”, in the ETSI 303 645 standard for IoT devices. This standard may become mandatory in Europe at some point (in the sense that the European Parliament will enact legislation requiring it). But more importantly, this will almost certainly become a de facto standard for commercial, industrial and home-based IoT devices in Europe.  

It may seem odd that the manufacturer has to certify compliance with just 18 of the requirements of ETSI 303 645. Why not all of them? For one thing, since Finland is part of the European Union, their IoT manufacturers will ultimately have to be in compliance with all of the standard anyway. But the real reason is that whoever drafted this law understood that the term “IoT device” covers a huge range of products, from baby monitors and doorbell cameras to protective relays and RTUs (remote terminal units).

This means that, for any particular IoT device, a large portion of the provisions in ETSI 303 645 won’t apply. The 18 requirements chosen by the Finnish were sufficiently general, that it can reasonably be expected that they’ll all apply to the great majority of IoT products. Examples of the 18 provisions are “a manufacturer should have a ‘public vulnerability disclosure policy’, ‘installation of updates should be secure’, ‘all unused interfaces must be disabled’, and ‘there should be no hard coded credentials in the system’.”

You might well ask, “How much good does it do for the user to know that the device they bought meets just 18 very general – and fairly easy to comply with – requirements?” My answer is “Probably not much.” However, the second step in the program directly addresses this gap. Instead of simply requiring that the organization comply with all of the ETIS 303 645 provisions or something like that, it requires that a third-party “inspecting body”, which is approved by a government agency called Traficom, should do a threat-modeling exercise for the device.

The term “threat modeling” is one of those terms that’s thrown around a lot and is used with a variety of meanings – so it’s almost meaningless (no pun intended here. Remember: This blog doesn’t do puns!) to discuss the meaning of threat modeling in general. However, in the context of the Finnish device labeling requirement, the term means that the inspecting body needs to consider both what the device will be used for (e.g. a baby monitor isn’t usually used for purposes that require high security, but a protective relay usually is. On the other hand, protecting personal information is a big concern for baby monitors, but it’s an almost nonexistent concern – except for login information, of course – for protective relays), as well as the setting in which it will be used (e.g. a room in a private home in the case of a baby monitor, vs. a Transmission substation where a cyberattack might affect thousands or even millions of people, in the case of the relay).

Using these two considerations, the inspecting body needs to identify all important cybersecurity threats that might apply to that device. Some of them will apply to many devices (like password strength), while others may be specific to the particular device. For example, one significant threat that applies to smart speaker devices like Amazon Echo™ is that, because they are continually listening to conversations for the magic word “Alexa”, if compromised they might be used to eavesdrop on conversations that occur in the home. In fact, DoD recently put out an advisory to employees working from home not to have smart speakers in the same room where they are working.

Third, when the inspecting body has identified the important cyber threats that apply to the product, they need to develop a test plan to determine whether each threat poses an actual risk. In my way of speaking, if a threat poses a risk, this means either the likelihood or impact of the threat being realized is high.

Since the device can have many uses, it’s very hard – nay, in my opinion it’s impossible – to determine the degree of impact of a threat being realized in all cases. Therefore, what’s important is likelihood, which can be high or low. If likelihood is high, the risk of the threat being realized is high, and mitigation measures should be taken to reduce the likelihood to low. And if likelihood is low, this means the risk is already mitigated (which might be by Mother Nature or other completely external forces, e.g. the risk that a forest fire will damage a region of the Sahara Desert is about as close to zero as it could be), and no further mitigation measures are warranted.

For example, in the case of the Amazon Echo threat discussed earlier, the inspecting body might decide that this threat is low, due to controls that Amazon has implemented to protect the audio stream the device “hears”. In this case, the body will deem this threat to pose minimal risk, meaning users need not take any further mitigation measures for it. So this threat will be deemed not to pose a risk in the Echo device, and it won’t be included in the test plan.

In the Finnish program, when the inspecting body has developed their test plan, they must submit it to Traficom (the government agency that operates the device labeling program). Traficom can disapprove the plan, in which case the inspecting body needs to revise it and resubmit it; otherwise, the plan is approved, and the inspecting body can go ahead and conduct the testing.

Finally, the testing body conducts the tests and submits the test plan and the results to Traficom. Traficom reviews these, then decides whether or not to grant the label. Traficom decides whether whatever residual risk remains (after the results of the device testing are considered) allows the device to be sold in Finland.

I think the Finnish model would be an excellent one for the US program, with two important modifications:

First, the US program has to operate with minimal government involvement, both for budgetary reasons and for general regulatory philosophical reasons (as long as human life isn’t directly at stake, the feds usually don’t want to be directly involved in auditing or testing. So the Consumer Product Safety Commission and the National Highway Traffic Safety Administration are directly involved in investigating cases that come up for them, since these could often have an impact on life safety. But it can’t be said that the cyber compromise of an IoT device would directly impact human life, except in rare cases (e.g. if a baby died because his or her parents couldn’t hear their distress cries, due to a cyberattack having disrupted communications with the baby monitor).

This means that the Office of Management and Budget, which is charged with implementing all of the EO’s provisions[i], won’t want to be in the business of certifying “inspecting bodies” – which may be called “laboratories”, as they are in Europe. Rather, they would certify the organizations that certify the labs, and might even go one step further: certify the organizations that certify the organizations that certify the labs.[ii]

The second difference between the Finnish IoT device labeling program and the likely US program is that the latter won’t be required for a device to be sold in the US, or even to the federal government. A manufacturer will be able to decide for themselves whether or not they want to participate in the program. If they decide not to, they won’t be shut out of the market, but they will have a harder time selling their product – especially to federal agencies, since the agencies will be required to ask their IoT device suppliers to participate in the labeling program, and they’ll have to justify to auditors any case in which they bought from a supplier that didn’t participate (i.e. the supplier doesn’t have a label at all, regardless of what it says).

So what will the “label” say, if it isn’t “This device is safe/unsafe to buy”? The label (which will probably be a document accessible by scanning a QR code or entering a URL found on the product or in its accompanying documentation) will list the threats that were identified through threat modeling and the degree of residual risk for each threat, after any mitigations that have been applied by the manufacturer. Additionally, the document will list mitigations that a user organization, which is concerned about one of the risks, can take to at least partially mitigate it. For example, if the organization (a federal agency, a private organization, etc.) wishes to partially mitigate the risk due to the fact that a device allows users to authenticate using weak passwords, they could locate the devices only in access-controlled rooms. 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the National Technology and Information Administration’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

The first time I ever wrote about IoT security was last December, when the IoT Cybersecurity Improvement Act of 2020 was signed into law by the former president. I realized it was something that would impact the energy industry, so I thought I’d let my readers know about it - not that I expected too many of them to get excited by it, of course. As it turns out, it was one of my most popular posts of recent years.

Then in May, the Executive Order (EO) included a very interesting provision on consumer IoT device security. This hasn’t received much attention, but it will next month, when NIST convenes a two-day virtual conference to discuss this and a related provision for consumer software.

Finally, in June I was engaged by a European company that deals with IoT security regulation to help them understand what’s going on in the US in that regard. This gave me the opportunity to learn about what’s going on in both the US and Europe (spoiler alert: Europe is ahead of the US in terms of having widely-followed guidelines for IoT, although the US is definitely now ahead in terms of mandatory requirements, at least ones that are on the books for future enforcement).

Since what’s going on in IoT cyber regulation will end up affecting all IoT users – home, commercial, industrial and government – very soon (including the energy industry), I’m going to summarize what’s most important (IMHO) to know about each set of regulations. This post is about the IoT Act, and my next post will be about the IoT provisions in the EO.

If you read my December post linked above, you’ll see I really like the way the IoT Act is structured; to be honest, I wouldn’t mind seeing all of the NERC CIP standards rewritten in this way (in fact, I think that if Medium or High impact BES Cyber Systems are ever going to be allowed to be deployed in the cloud – e.g. outsourced SCADA – rewriting the standards in this way will be required. Hopefully, I’ll have time to write a post on that topic in the near future, although I did touch on this topic in a recent post).

Here’s my summary of the Act, partly plagiarized from my own post:

1.      The Act starts with a great definition of IoT devices. They must “have at least one transducer (sensor or actuator) for interacting directly with the physical world, have at least one network interface, and are not conventional Information Technology devices, such as smartphones and laptops, for which the identification and implementation of cybersecurity features is already well understood.” Note that there’s no distinction here between “home” and “industrial” IoT. As long as there’s one transducer and one network interface and it’s not a smartphone or laptop, it’s an IoT device and it’s an IoT device. It’s safe to say that the great majority of organizations in the US, and a large percentage of households, are IoT users.

2.      But the Act doesn’t apply to device manufacturers, only consumers. And the only consumers it applies to are federal government agencies. Why is that the case? I’m sure this restriction made the bill much easier to write. Since there aren’t currently any mandatory federal cybersecurity regulations on any IT/OT/IoT suppliers, if the bill had been intended to apply to suppliers, it would have run into a firestorm of opposition and might have required years to get approved (as it is, the Act required two years for approval).

3.      But what about the fact that it’s restricted to federal agencies? Does it really do the general public a lot of good if nothing they use (unless they work for a federal agency) is directly covered by the Act? I think it definitely will, for the same reason that the Executive Order will do the general public a lot of good, even though it’s also restricted to federal agencies: Suppliers of any product, but especially an IT-type product, aren’t going to take the trouble to have two separate product lines – one for the Feds and one for the rest of us schmoes. It just doesn’t make for a great advertising tag line: “Sure, this IP camera is more likely to spy on you than the one the Feds use, but it costs $10 less!”

4.      So what are the federal agencies supposed to require of their IoT vendors? The Act wisely doesn’t prescribe anything (although see the next paragraph). It simply requires NIST to, within 90 days, develop “standards and guidelines for the Federal Government on the appropriate use and management by agencies of Internet of Things devices…including minimum information security requirements for managing cybersecurity risks associated with such devices.”

5.      But NIST doesn’t have a completely free hand in developing these guidelines. They have to address “secure development, identity management, patching and configuration management.” Of course, all four of these topics are ones that I would expect would be in any good cybersecurity framework.

6.      NIST, ever the eager beavers, met the 90-day deadline (give or take a month or two) with not one document but five: NISTIRs 8259B, 8259C and 8259D, along with NISTIR 8322 and draft SP 800-213. All of these documents are good. SP 800-213 is aimed at policies and practices of end user organizations (in this case federal agencies), while 8259D provides detailed guidelines for IoT device manufacturers who sell to the federal government. So I think these two documents will be the main two governing the IoT Act.

7.      The Act requires the Office of Management and Budget to develop implementation regulations within 180 days. That would probably have been sometime in July, but I haven’t seen anything yet.

While I think the IoT Act will have a positive impact on IoT security, I think the Executive Order will have a much greater one. In fact, it may work out that what the EO requires will end up superseding the IoT Act altogether. Is Tom right about this? Or is he full of it? You’ll be able to decide for yourself when the exciting conclusion to this series arrives in your inbox (or browser) in the very near future. Please try to contain your excitement.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the National Technology and Information Administration’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Two days ago, Politico published a really chilling articleabout how BlackBerry - which is now a software company, after earlier being a pioneer (in fact, the pioneer, in my opinion) in internet-connected personal devices - gravely mishandled reports of serious flaws in its QNX operating system. QNX is found in all sorts of devices like cars (200 million of them, according to the article) and the International Space Station. This was called to my attention by Eric Byres of aDolus, who wrote an excellent blog post that elaborated on Politico’s point about how having SBOMs could have made this a much easier problem to deal with, whether or not BlackBerry had cooperated.

Because the fact is that they didn’t cooperate. Microsoft first discovered this vulnerability in April in a whole host of device operating systems, not just QNX. The article continues, “In May, many of those companies worked with the Department of Homeland Security's Cybersecurity and Infrastructure Security Agency to publicly reveal the flaws and urge users to patch their devices. BlackBerry wasn’t among them.”

At first, BlackBerry denied that QNX even had the problem, even though CISA showed them it did (in some versions). Finally, they admitted the problem, but said they weren’t going to publicly announce the vulnerability. Rather, they were going to work with their direct customers privately to fix the problem.

Of course, on the surface, BlackBerry’s desire to limit disclosure to their direct customers was reasonable. Software companies do that all the time, especially if they haven’t yet distributed a patch for a vulnerability. After all, if they announce an important vulnerability to the whole world before their customers have been able to patch it, they’re inviting ruin on those customers.

But it turns out that BlackBerry doesn’t have any idea who are all the customers of QNX. BlackBerry sells it to large organizations that incorporate it into products that they sell to other organizations, who embed those products into their own products, etc. Once you get down below at least the second tier of this, users of devices that run QNX almost certainly have no idea that they have it – since it doesn’t appear on any invoice they’ve ever received.

Just as in the case of the Ripple 20 vulnerabilities, the right course of action in this case was to first prepare the patch, then let the whole world know about it. That way, in principleall users would be able to figure out if they have a product that contains QNX, and if they do,  get a patch from whoever sold them that product. The good news is BlackBerry finally took this course of action. The bad news is they took it two days ago, after Microsoft had revealed the vulnerability in April.

Of course, Microsoft deliberately didn’t name names in their April announcement, in order to give the O/S vendors a chance to patch their systems and then announce the vulnerability and the patch at the same time. It seems that BlackBerry missed that memo.

You may have noticed that I italicized “in principle” two paragraphs above. When you hear that X is true in principle, what do you normally think? I normally think, “There isn’t a snowball’s chance in hell that X is actually true in real life.” Indeed that’s the case here. In principle, every company could take an inventory of all the devices they operate, check the current software bill of materials (SBOM) for each device type, and instantly know which ones are running QNX.

And even if the device they own doesn’t run QNX, the O/S might be running inside a component within that device (take your car. Even if it’s one of the 200 million cars running QNX, in most cases QNX is probably running inside some device that’s a component of another device, etc). But in principle that’s not a problem either, since you’ll not only have an SBOM for your car, but you’ll have an SBOM for the entertainment system, the transmission system, the passive safety system, etc. So if one of those systems is running QNX, you’ll learn that.

Now, I won’t say that principles are worthless. But I will say that all these “in principles” aren’t going to get you very far, because without a doubt you don’t now have SBOMs for all of these devices – in fact, it’s very unlikely you have a single SBOM for anything in your car, let alone the car itself. Or just about any other software or device that you operate.

But this is changing, of course. Federal agencies are going to be required to get SBOMs from their software (and device) suppliers as of around August 12, 2021 – according to an OMB memothat came out last week, based of course on the May 12 Executive Order on software security. And it’s highly unlikely that this will end with the Feds – it will spread to the private sector, even though it’s not mandatory for them (and besides, the federal government buys lots of cars!).

If you’d like to learn more about SBOMs, I believe I’ve mentioned them once or twice in previous posts (he says while smirking). You can find some of them by searching on SBOM in the search bar that you see when you go to the main page of my blog, https://tomalrichblog.blogspot.com/. And you might want to join the Energy SBOM Proof of Concept meetings that I help coordinate. We’ll meet next Wednesday from noon to 1 ET. To get the URL, drop an email to SBOMenergyPOC@inl.gov.

I do wish to point out that we do have strict criteria for attending these meetings. You must be a user of electric energy. If you’re an off-the-grid kind of guy, and you’re reading this in a dark cabin in the mountains of East Nowhere, you probably won’t find the meeting appropriate. In fact, I don’t think you’ll be able to attend it, anyway (plus I don’t know how you’re reading this post in the first place).

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the National Technology and Information Administration’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

On August 1, I put up a postdescribing an email conversation I’d had with Tim Roxey, former VP and CISO of NERC, and someone whose words are always very interesting, but sometimes (sometimes!?) hard to understand. Fortunately, a number of people in the NERC security world agree that whatever words he does utter are worth pondering, since Tim doesn’t spout off just because he’s trying to fill up a page, as some unscrupulous bloggers have been known to do.

I’ll let you read that post, but one of its most interesting features was when Tim described a trip to Whole Foods:

I was in Whole Foods couple of weeks ago. Heavy storms moving in but I was in underground parking. 

I’m pushing about my cart when an announcement comes over the speakers. Please all shoppers stop shopping. We have lost our cash registers due to lightening in the areas. 

Me thinks. I have cash. I’m good. 

Me thinks wrongly. Somehow the Point Of Sale device can’t process the sales in cash cuz the credit side is down. 

Harumph. No, it was the people and a branch  point in their processing that broke. 

We are so dependent on our “usual processes” that we fail to see the alternatives. 

Colonial failed as well. 

If you are CIKR then this is Wrong. Be CIKR AND operate as such.

Last week, I received an email from Sam Chanoski of Idaho National Laboratories, who is someone I’ve known a long time and have a lot of respect for. He worked for NERC for eight years (including working with Tim, of course), the last 2 ½ of which were with the E-ISAC. During his period there, he was in the middle of almost everything the E-ISAC was doing at the time. After a stint at ABB, he joined INL last year.

Sam’s email to me focused on the above passage from my post (which was quoted pretty much verbatim from Tim’s email. Yes, Tim really writes that way!).

I may be able to help a little with some Tim-terpretation from Tim Roxey’s earlier response. He’s saying the same thing allegorically with his supermarket that can’t take his cash, that I’ve posited elsewhere: in any organization with a consequential mission, there are likely to be dependencies built into “normal” accomplishment of their critical functions – and while the people who implement these processes on technologies every day largely understand many of them, the organization as a whole is often blind to most of these critical dependencies.

Rooting out these dependencies and forcing the organization to appreciate them for the risks they present is the start of how we become more resilient to whatever life and the bad people throw our way, in whatever failure-of-our-imagination ways we experience it next, with the people, processes and technologies we have today. 

For the PPT we need for tomorrow, that’s where something like Cyber-Informed Engineering (CIE) might come in, to help us imagine, design, procure, build, operate, and maintain the energy systems of tomorrow with cybersecurity inextricably part of the DNA as much as safety is today. Even though CIE slightly predates the similarly named Consequence-driven Cyber-informed Engineering (CCE), it’s definitely less well known and less mature, but ultimately a lot more broadly applicable I think – https://inl.gov/cie/ is where we are now, with the next major parts coming out likely next summer when the national strategy on CIE is (hopefully publicly) published according to FY20 NDAA Section 5726. 

Noe of this is easy or fast or pleasant but it is necessary – as Gloria Steinem said, “the truth will set you free, but first it will p___ you off.” (Note: Since this is a family blog, I can’t quote Gloria exactly)

Sam makes a great point, and I must admit I didn’t see it when I wrote the previous post on what Tim said (now I do, though): People who work within a system day by day are probably the least able to tell you exactly how it works. They especially can’t tell you what’s needed in the way of “exogenous inputs” (as we used to say when I was working for an econometric modeling company, back in the days when people believed that computers were wonderful devices, rather than the instruments of the devil himself, as we all know to be the case nowadays). So the everyday workers need to have someone come in on occasion and tell them how their system really operates. That way, they can be prepared when one of those dependencies is lost.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the National Technology and Information Administration’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

In my last post, I made the case (although this wasn’t the main reason for the post) that the CIP standards, in order to solve the problems found in CIP today, must be rewritten as risk-based; this is a point I’ve made in a number of posts in the past year or so. One of the problems that I pointed out, that can be solved if CIP is rewritten, is the problem of BES Cyber Systems in the cloud, as in outsourced SCADA (note this is different from the problem of BCS Information in the cloud. A lot of NERC entities are storing BCSI in the cloud now, and the changes to the requirements that will make that completely “legal” have been approved).

Putting BCS themselves in the cloud is a very tricky question, mainly because doing so potentially puts the entity in violation of a whole host of CIP requirements (assuming the BCS is High or Medium impact. If it’s Low impact, there’s no restriction at all on putting BCS in the cloud today). The main reason why I say this is that there’s no feasible way to prove in a CIP audit that the cloud provider is actually performing all the acts required by the requirements in CIP-004, CIP-005, CIP-007, and CIP-010 (as well as some of the other CIP requirements).

Of course, this statement always seems strange to people who don’t understand how CIP is audited, since they will usually point out – with good reason – that if anything, the cloud providers have better security than any electric utility, as evidenced by the rigorous FedRAMP and SOC II audits that the major cloud providers have to pass. FedRAMP requires everything that CIP does and a lot more. You’ll get no argument from me on that point.

But what FedRAMP doesn’t require is that the cloud provider store evidence in a way that would enable them to pass a CIP audit; this is because NERC requires – for audits of all standards, not just the CIP standards – that the entity show evidence that they complied with a requirement in every instance. For example, CIP-004-6 R5.1 requires that the “individual’s ability for unescorted physical access and Interactive Remote Access upon a termination action” be removed within 24 hours of their termination (which includes termination for cause or just quitting). The cloud provider would have to retain evidence (e.g. screenshots) that this was done for every termination during the audit period, for any individual that had access to any system that contains “part” of the BCS (since the whole idea of the cloud is that particular functions are broken down into a lot of small functions, that can be executed almost simultaneously on many systems located in data centers that are probably all over the world - or at least all over the US).

But even this might not seem too hard – the cloud provider would just need to track everyone who is authorized to access the servers (physically or logically) that your BCS are stored on, and record the evidence that their access was removed within 24 hours when they left the provider. Is that so hard?

It is, when you consider that this applies to every data center where some part of the operations of your BCS  might have been stored during some period of time (even if that’s just a few minutes) during the audit period. Moreover, this applies to every employee who worked in one of those data centers during the audit period (which is usually three years) – since any employee who could have walked by a server containing some part of your BCS by definition has physical access to that server (unless it’s in a rack with a locked door, protected by a card reader that only provides access to a small number of data center employees).

In order to comply with this one requirement, the cloud provider would most likely have to provide evidence that they had removed access within 24 hours for any employee in the US who could have even walked by a system that housed some part of your BCS, for even the briefest amount of time during the three-year audit period. In other words, they would need to do this for a significant percentage of all of their US employees. And this is a walk in the park, compared to the evidence that would be required for compliance with CIP-007 R2 (patch management) and CIP-010 R1 (configuration management)!

So even though the cloud provider’s practices almost certainly far exceed what the NERC CIP standards require, it’s literally impossible that they could ever document that compliance. Could that problem be fixed by amending the CIP standards, so that cloud providers would just be able to point to their FedRAMP certification as the only evidence they need?

Absolutely it could (although it would very likely take at least 3-4 years before this change would actually come into effect, from the day that a Standards Authorization Request was written for it). And what would happen then? Literally every NERC entity with Medium and/or High impact BCS would immediately outsource everything they could to the cloud, since their compliance documentation from then on would just consist of one sentence: “See XYZ Cloud Provider’s FedRAMP certification.” Of course, I would hope those entities would find a way to employ the large number of people who were previously doing nothing but document CIP compliance – I’d hate to see so many people out on the street at once.

I agree that allowing cloud vendors to just point to FedRAMP as evidence of compliance with the various CIP requirements would be an extremely popular move, but I also believe some people might object that this could very well put the BES at much more serious risk than it’s ever been before. In fact, I’d have to withdraw what I’ve said previously, that there’s no way a cyberattack (or even a coordinated set of cyberattacks) could bring down the “whole” US power grid or even a substantial portion of it. If almost all electric utilities put most of their BCS in the cloud, it would just take attacks on the two or three major cloud providers to literally shut down a lot of the US power grid. But other than that, I don’t see anything wrong with this idea…

So what does this mean for the idea of having BCS in the cloud? Should you try it? Like a lot of things, it depends on your risk appetite. You might find that the auditors are very sympathetic to this idea – especially if you’re say a renewables producer who’s just starting out, and you have a heart attack when you compare the cost of running your Medium impact Control Center on outsourced SCADA vs. the cost of building everything yourself in a facility that you own. But it’s probably more likely that they’ll tell you to drop the outsourced SCADA and create your own Control Center in say one year – and if you don’t do that, they’ll throw the book at you.

But the choice is yours. Nothing in the CIP standards says you can’t outsource BCS to the cloud.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the National Technology and Information Administration’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Sometimes it seems that all questions regarding OT security ultimately come down to questions about NERC CIP. Of course, this is because NERC CIP is the oldest set of cyber regulations – outside of the nuclear and military domains – that directly addresses OT, and is still one of only a handful of cyber regulations that focuses solely on OT.

So far, I’ve tried to focus on the general OT security implications of the Colonial Pipeline ransomware incident, but the fact is that there are some very important implications for CIP in the incident. This week, I received an email from someone who’s been working in CIP for quite a while, but who wishes to remain unidentified. This person asked what CIP changes would be needed, based on the lessons learned from the Colonial attack. I think it’s time that we discussed this.

The most important part of the Colonial Pipeline attack, from a CIP perspective, is that the ransomware was – according to Colonial – confined to the IT network. But even then, the attack ended up shutting down Colonial’s entire pipeline system. And according to news reports, it seems that the vehicle for this happening was that Colonial’s customer billing system (which was on their IT network, of course) had to shut down; this somehow entailed the shutdown of the OT network.

Two questions inevitably arise in the mind of anyone (such as me) whose mind has been forever warped by spending too many hours pondering recondite NERC CIP issues:

1.      If the pipeline industry had been subject to compliance with the NERC CIP standards, would this incident even have happened?

2.      If something like this did happen in an electric utility – namely, that the utility had to shut down some or all of their transmission and/or distribution systems due to loss of a system on their IT network – would it have been because of a CIP violation, or could this have happened even in a utility that was completely CIP compliant? If the answer to the latter question is Yes, this implies that the CIP standards can’t prevent a successful cyberattack on the IT network from ever shutting down power operations. In turn, this would imply the CIP standards need to be expanded in some way to cover IT systems, as well as OT ones.

I brought this issue up (really for the second or third time, but for the first time as the subject of a whole post) in July, when I reported on a webinar I’d attended the day before. In that webinar, my longtime friend Jodi Jensen of WAPA had wondered how the loss of Colonial’s billing system could possibly have shut down pipeline operations. Why couldn’t operations have continued, on the understanding that Colonial would generate all outstanding bills once the billing system was back online?

Why was Jodi asking this question? I think it’s because NERC CIP (as well as good SCADA security practice, which Jodi knows a lot about) requires that every system whose loss or misuse could “affect” the Bulk Electric System in some way be included in the OT network, which is protected by an Electronic Security Perimeter – and at least in the power industry, the ESP protections (for assets that are classified as Medium or High impact) are quite strong; I have never heard of an OT compromise that began with a frontal assault on the ESP (which isn’t to say it’s never happened, of course) – although there have definitely been OT compromises that didn’t start with a breach of the “wall” between IT and OT. In fact, I cited two of those cases in my most recent post on the Colonial issue.

And how does an electric utility decide whether or not a system’s loss or misuse might affect the BES? It’s by running the systems through a bizarre methodology for classifying systems in operation at a NERC entity, which – if I didn’t know better, because I attended a number of the drafting team meetings when it was designed, and I engaged in not a few arguments with the then-chairman of the drafting team about this issue – I would guess was designed by a team consisting of Jack the Ripper, Suleiman the Magnificent and Rube Goldberg; this “system” is known commonly as CIP-002 R1 and Attachment 1.

The keystone of this classification system is the definition of BES Cyber System (well, really BES Cyber Asset, but don’t pay attention to details if you want to understand CIP-002 R1. You’ll go mad). I won’t bore you with the full definition of BCS, but its two main points are:

1.      If the system were “rendered unavailable, degraded, or misused”, it would “affect the reliable operation of the Bulk Electric System”.

2.      That effect needs to occur within 15 minutes. That is, a system isn’t a BCS if its loss or misuse won’t usually result in a BES impact within 15 minutes. Of course, when you’re dealing with electric power, if there’s any impact at all it’s usually going to occur within a second or two. The massive power outage in Florida in 2008 was detected literally within a couple seconds in Alberta.

Let’s say the Colonial Pipeline billing system were magically transformed to do electric power billing (although the two types of billing are very different) and installed at a large electric utility. Would it be classified as a BES Cyber System and thus be required to be installed within the ESP (i.e. on the OT network)? Or would it be just another IT system like payroll? Of course, if it were actually a BCS (again, at a Medium or High impact facility), the utility would be in a lot of trouble if a subsequent audit discovered it was on the IT network.

But how could a utility’s billing system (not the metering system needed to produce the data for bills. Metering is always part of the OT network, and of course metering is found everywhere power is distributed, including at your home) have a 15-minute on the BES? Ultimately, if an electric utility’s billing system is unavailable for an extended period of time, there might be some BES impact, but it certainly wouldn’t be within 15 minutes.

So the billing system wouldn’t be a BCS in a NERC CIP environment. This means it would be totally exempt from all CIP requirements. But does this mean the billing system doesn’t pose any risk at all to the BES? Does it really deserve to get off Scot free from CIP, even if it shouldn’t be subject to the same set of requirements as for example the utility’s Energy Management System, which has an undeniable 15-minute BES impact?

Rather than deal with a hypothetical system, let’s look at a real one that is found in literally every electric utility (in fact, it’s found in just about every industrial facility worldwide): the historian. This is a system that records what goes on inside the utility’s power network, so if there’s some sort of adverse event, the utility can go back through the data to trace what actually happened. Since the BES can run quite nicely, thank you, when the historian isn’t fulfilling this purpose, it’s normally not classified as a BCS.

But in some cases – as Kevin Perry pointed out in this post – the historian is used to provide a real-time view of what’s going on in the utility’s operations, meaning that the operators in the Control Center might make a split-second decision based on something the historian is telling them. Ever the auditor, Kevin correctly pointed out that, if the historian is used for that purpose, it is a BCS and needs to be installed inside the ESP, not on the IT network.

In the same post, Kevin described one case where he decided during an audit that a historian really was a BES Cyber System (actually a Critical Cyber Asset, the somewhat equivalent term used in CIP versions 1-4). He didn’t say whether that utility was fined or not (or whether they even received a violation), but at the least they had a very bad day when they found out they had to do all the re-engineering required to move their historian from the IT network into the ESP.

But what if the historian doesn’t perform any real-time monitoring function, so it’s correctly classified as not a BCS? Does that mean that its loss or compromise doesn’t affect the BES at all? Of course not. For example, if the historian malfunctioned (due to a cyberattack) at the same time a serious event occurred on the utility’s power network, the utility might not be able to determine the cause of the event, say a relay misconfiguration in a substation. Since the cause wasn’t detected, the misconfiguration might remain in place and cause another serious event. That’s a BES impact, although it’s not within 15 minutes. 

So at least with the historian, we have an example of a system whose loss or misuse will usually impact the BES, but which doesn’t have a 15-minute impact (Kevin mentions ICCP servers as another example). This means the historian isn’t a BCS and isn’t in scope for NERC CIP at all. But is that right? Shouldn’t systems whose loss or misuse can impact the BES – even though the impact won’t always or even usually be within 15 minutes – be in some way in scope for CIP?

In the same post referred to a couple times above, I quoted from an email Kevin sent me, in which he said, “I would argue that any ‘IT’ system or system component that is essential to (sustaining OT operations) needs to be considered OT and kept isolated from the rest of the IT world.” In other words, Kevin is suggesting that systems whose loss or misuse can affect the BES, but not within 15 minutes, should be included in the OT network, not the IT network.

Does this mean that such a system should be subject to the same requirements as an actual BCS? One might initially be inclined to argue “No, such systems should be required to be on the OT network – that is, within the ESP – but they shouldn’t have to comply with all of the requirements that BCS have to comply with.”

But this ignores an important security fact and an important CIP fact. The security fact is that, if a system is connected to an IP network and it gets compromised, it can be used as a launching point for attacks on other systems connected to the same network, which might have higher intrinsic value than the system that was first compromised. The CIP fact is that every system that is on the same network as a BCS needs to be declared to be a Protected Cyber Asset; and PCAs are subject to almost exactly the same set of requirements as BCS are.

So if we want to have a smaller set of CIP requirements that applies to these “intermediate systems” (i.e. systems whose loss will affect the BES, but not in 15 minutes. Of course, this is my name for them, and note that if Intermediate System is capitalized, it has a different meaning in CIP – so my name would never be usable in practice), we would probably need to put them on another network, separate from the IT and ESP networks and protected from both. At that point, we could apply a different set of CIP requirements just to these systems.

But which of the current CIP requirements should apply to the “intermediate systems”? I don’t know about your answer, but here’s mine: None of them. That is, I don’t want the current CIP standards extended any further than they already are. I totally agree that these intermediate systems do need to be included in a general OT cybersecurity compliance regime, and I’m fine if that regime is still called NERC CIP, but it needs to be a completely risk-based system.

I outlined the “new CIP” system I’d like to see in an article for a British publication in 2019 (this isn’t available online, but if you email me, I can send you a PDF of it). I also did a webinaron this topic for that same publication in 2018. Note my views have changed somewhat since then, but the general framework I laid out is still what I believe is needed.

And I’m not saying that what I’m proposing is the only way to do this. In fact, the current CIP Modifications drafting team proposed another idea in 2018, which I really liked. I wrote three posts on it: no. 1, no. 2, and no. 3. What happened to that idea? That’s a sad story: Briefly, the SDT was proposing some radical changes to CIP. These would have required NERC entities to revise a lot of their CIP compliance documents, as well as follow some new procedures. And it seems too many NERC entities simply ruled this out as impossible.

The result of this? The current CIP compliance regime (i.e. the standards and how they’re interpreted and enforced) is preventing CIP from being extended to intermediate systems, as well as to the cloud. The latter is especially unfortunate, since, even though I’m sure there are a few hardy souls who have actually outsourced Medium impact, and maybe even High impact, BCS to the cloud (e.g. in outsourced SCADA), they’re only doing this because they’re willing to live with the constant fear that an auditor will throw the book at them and make them shut down all BCS in the cloud, forcing these entities to reproduce their BCS in the nice, safe confines of a Control Center or substation that you can actually point to and walk into.

Folks, if you want to be able to put Medium or High impact BCS in the cloud and feel safe doing so (as well as other improvements, like much faster response to new threats and a much more efficient allocation of compliance resources, as well as of course incorporation of the intermediate systems we’ve just discussed), you’re going to have to accept that the CIP standards have to change. This means you’ll have to change your current procedures and documentation, period. Which do you want? BCS in the cloud or your current CIP procedures? You can’t have both.

Is it time to review your CIP-013 R1 plan? Remember, you can change it at any time, as long as you document why you did that. If you would like me to give you suggestions on how the plan could be improved, please email me.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the National Technology and Information Administration’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

We keep getting more reminders of why SBOMs are needed. One of the big reasons for having SBOMs is that, when a serious vulnerability is found in a software component that’s widely used, the first task of most organizations is figuring where this component is actually found in their environment – remember Ripple20? This can be a very time-consuming exercise, plus you’re not likely to have 100% - or even 25 or 50% - success. Having SBOMs for at least some of the software on your network could significantly reduce the amount of time you have to spend talking with your suppliers, scanning all your systems since you have no better way to find the vulnerability, etc.

Coincidentally, the Ripple20 vulnerabilities were announced a little more than a year ago. Then in December, Forescout Technologies announced the Amnesia:33vulnerabilities. One big similarity between Ripple and Amnesia is they they’re both vulnerabilities in IP stacks that were developed a long time ago (the Ripple20 vulnerabilities were in an IP stack developed by Treck, a company still operating in Cincinnati, in the 1990s). So the vulnerable components have had plenty of time to be embodied in software or embedded devices that were later incorporated into other products, which were later incorporated into still other products, etc. The result is that, in many cases, the maker of a device that contains the vulnerability has no idea that the component is even included in their device – this knowledge has passed out of the collective memory, just like the identity of the person who wrote the book of Genesis.

This week, Forescout (with JFrog) again announced serious vulnerabilitiesin an IP stack released in the mid-1990s, although this time the stack is used in a lot of OT systems (thanks to Kevin Perry for forwarding me the story); Forescout has named these vulnerabilities “Infra:Halt”. The original developer of the vulnerable software component, called NicheStack, was a company called InterNiche; this company is now part of HCC Embedded, although the product isn’t nowadays sold as a standalone component.

Of course, to truly learn about every instance of any software in your network, no matter how deeply embedded it is in other products, you would need a multi-level SBOM for every intelligent device or “standalone” software product on your network. When will you have this? Probably never, to be honest. In fact, if in three years you have just a one-level SBOM (i.e. a list of the components directly included in the software or device) for the great majority of the software you operate, I’ll consider that to be a big achievement (I also think it’s achievable).

However, here’s the thing about SBOMs: their benefits are entirely incremental. Having an SBOM for just one product is better than having none; having them for ten products is a lot better than having just one; etc. The way things are moving today (partly as a result of the May Executive Order), you’ll definitely be able to have a lot more SBOMs for products you use in one year, than you do now.

How can you take the first step into the world of SBOMs? You can join the Energy SBOM Proof of Concept, jointly sponsored by the National Technology and Information Administration (NTIA) and Idaho National Labs. Moreover, you can join our next biweekly meeting, on Wednesday August 11 at noon Eastern Time. To get an invitation and receive our mailings, drop an email to sbomenergypoc@inl.gov. See you then!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the National Technology and Information Administration’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Last Monday, I wrote a postabout the comments that Kevin Perry, former Chief CIP Auditor of SPP Regional Entity, made on this post, which discussed why it actually made sense for Colonial Pipeline to shut down their operations, due to a loss of their billing system in a ransomware attack.

The whole point of that post, as well as the previous posts I’d written on Colonial (which BTW you can find by searching on the main page of my blog. Until last summer, there was no way to search the blog, making it hard for me to find previous references to a subject, and close to impossible for readers to. I was quite glad when search was added) – starting with this one – was that a “purely IT” incident can affect the OT network, even if there’s no direct connection.

In the case of Colonial, the loss of their billing system meant that they couldn’t track who put how much gasoline into their pipeline and when, and who withdrew how much and when. For an electric utility, the loss of this capability wouldn’t require shutting down power transmission and distribution, since the utility can always bill for power used later (i.e. the meters will keep operating); and if the utility can’t bill later for some reason, they still need to provide power, because they’re…well, a utility.

But Colonial doesn’t own the gasoline in their pipeline; they’re transporting it, just as a mover transports your household goods to a new city. If  the mover loses your goods on the way, they’re on the hook for the entire value of those goods. By the same token, if Colonial keeps shipping gasoline while their billing system is down, they’ll literally lose track of what any one shipper has put into the pipeline, and will end up owing every shipper the entire value of their gasoline.

In last Monday’s post, I started by saying there were three questions that needed to be answered:

1.      How can we identify systems that don’t directly control operations, yet can have a huge impact on operations just the same (i.e., IT systems that perform functions required for operations)? And when we’ve identified them, what measures can we take to protect them better than other systems on the IT network that clearly have no direct operational impact, like say the systems that run the utility’s retirement plan?

2.      Should those systems be regulated by OT-focused cybersecurity compliance regimes, such as the dreaded…(and here I cross myself, despite not being Catholic)…NERC CIP?

3.      Or maybe we need to go beyond all this talk about regulation and protecting systems, and think about what the real problem might be?

To summarize what I think Kevin said in that post, he answered the first question by in effect saying, “Any system on the IT network whose loss or misuse can impact operations, like Colonial’s billing system, should be protected like OT systems are, including being isolated from other IT systems.”

Kevin answered the second question by in effect saying, “Any system whose loss or misuse can affect Bulk Electric System operations within 15 minutes (essentially, the BES Cyber Asset definition) should be classified as a BES Cyber System (BCS) and placed within the Electronic Security Perimeter (if the asset at which it’s installed is classified as Medium or High impact).”

An example he gave of this is a mistake he saw more than once in his ten-year NERC CIP auditing career: a NERC entity didn’t classify their historian as a BCS and installed it in the IT network, not the ESP. However, in the cases Kevin discusses, the historian was used for real-time monitoring purposes, and therefore should have been classified as a BCS. So it should have been installed in the ESP to begin with.

This is stretching what Kevin said a little, but one might draw the implication that, if a system’s loss or misuse doesn’t directly impact the process being controlled (which, in the case of an electric utility subject to the NERC CIP standards, is the smooth and uninterrupted operation of the BES. In the case of Colonial Pipeline, it’s the smooth and uninterrupted transport of natural gas in their pipeline system), then a) it’s OK to install it on the IT network, and b) it doesn’t need to be subject to special regulation, beyond a general obligation to follow good cybersecurity practices.

However, there are two cases I can identify in which the shutdown of the IT network directly required shutting down OT, even though there were no systems on the IT network that directly impacted the process being controlled by OT. One case is from 2018, when a serious ransomware attack on a very large electric utility’s IT network required shutting down the control centers as well – even though the ransomware never spread there.

The other case was cited by Tim Conway of SANS in a webinar earlier this year (which was quoted in Utility Dive). In 2017, the Not Petya malware (which was based on the Petya ransomware, except that not Petya didn’t even bother to save the encryption key after encrypting the victim’s systems – it simply threw the key away. The purpose of Not Petya was to cause havoc, pure and simple. And it did; about $10 billion worth of havoc – for which, naturally, Russia has never been held accountable. Do you notice a patternhere?) brought down the entire global operations of the Swedish shipping giant Maersk.

Tim pointed out in the webinar (reported in thispost) that no operational systems like cranes were affected by the attack on Maersk’s IT network. However, because of the loss of its IT systems, Maersk no longer knew what was in the containers it was shipping – meaning it really couldn’t guarantee that a container shipped to Company A was actually picked up by the correct recipient, rather than somebody else. This is very close to the situation that Colonial Pipeline faced when they lost their billing system. In both cases, the company shut down operations (although in the case of Maersk, operations were down for two weeks, vs less than a week for Colonial. On the other hand, given the devastation that Maersk suffered, the fact that it only took them two weeks to get up and running again isn’t much short of a miracle).

In other words, these two cases show us that the security of the IT network can be essential to the correct operation of the OT network, and – at least in the case of a complete loss of the IT network, as happened with Maersk and the utility in the 2018 incident – some IT incidents can require shutting OT down, even when there’s no particular system on the IT network whose loss requires the OT shutdown (as was the case with Colonial).

So we’re fooling ourselves if we think that our OT network is protected from all disturbances on the IT network, even though we may have made it impossible for an attacker to penetrate the OT network from IT – just like the French were fooling themselves when they built the Maginot Line after World War I, to prevent another German invasion – even though there was no way the Germans could have crossed the line to enter France. And this is just as true with Electronic Security Perimeters. True, CIP-005 R1 and R2 provide formidable protections against an “invasion” that comes through the IT network. But they don’t protect against all compromises, especially ones that magically bypass the ESP, like in the 2018 ransomware case.

So is the solution to apply the full NERC CIP requirements to IT systems, as well as OT systems? God forbid! I wouldn’t wish the current NERC CIP requirements – in all their prescriptive glory – on my worst enemy. However, if and when the NERC CIP standards are rewritten as risk-based, and when there are important changes made to NERC’s CIP compliance regime (as I discussed in this webinarin 2019), then it will be possible to regulate both IT and OT systems, but in different ways, commensurate with the risks posed by both types of systems.

To go back to my three original questions, Kevin and I answered the first two. But what about the third? That is, instead of just talking about regulating and protecting IT vs OT systems, maybe we need to think beyond that silo? What’s the real problem we need to address?

Fortunately, there’s someone who thinks about what the real problems are: Tim Roxey, who has appeared in this blog before. He replied to the same post that Kevin did, saying (in the inimitable English dialect known as Roxey-speak):

I was in Whole Foods couple of weeks ago. Heavy storms moving in but I was in underground parking. 

I’m pushing about my cart when an announcement comes over the speakers. Please all shoppers stop shopping. We have lost our cash registers due to lightening in the areas. 

Me thinks. I have cash. I’m good. 

Me thinks wrongly. Somehow the Point Of Sale device can’t process the sales in cash cuz the credit side is down. 

Harumph. No, it was the people and a branch  point in their processing that broke. 

We are so dependent on our “usual processes” that we fail to see the alternatives. 

Colonial failed as well. 

If you are CIKR then this is Wrong. Be CIKR AND operate as such. 

This was of course quite interesting, but it wasn’t…how can I say this?...definitive. So I wrote back to Tim and asked him two questions: “Do you think some sort of regulation of these systems is necessary? Or are you saying that changing the utility’s (or pipeline company’s) whole modus operandi is required to fix these problems?”

Tim replied:

Actually if we look at this differently, we see opportunity. 

Apply regulations that address People, Processes, and technology. Stop concerning ourselves with IT/OT as the technology of applicability.  If you can have the People pull the plug because their Processes (Recovery) or Technology (IT bleeding into OT) has led to a condition of uncertainty (The function of CEO is RISK) then the regulations were not so much fantastic. 

The regs in Colonial Pipeline simply do not exist. Their Issue was IT not OT and hence most NERC Regs would not apply even if they existed in TSA world. 

Requiring Baseline Regulations that hit all three factors;

• the People that operate inside
• Processes that control CI Functions that employ
• Technology to perform the Critical Infrastructure functions (National Security Functions)

Good Regulations address all three.  

Bottom line – Regulations tend towards baselines. Centers of excellence (Think INPO) tend towards Ceilings of excellent performance (best practices). Ceilings tend to include a better, more mature understanding of Risk. Not just the usual Vulnerabilities, Threats and Consequences stuff but also internal risks of how the People and Processes Parts and Technology parts interact. The People being unduly influenced by their knowledge of the processes (or lack thereof ) and the misunderstandings of the technology (IT really can touch OT) leads to enough uncertainty that conservative calls to pay Ransom are made.

As with all oracular statements (i.e. statements that a true oraclemakes. And no, that’s not Larry Ellison), these are subject to many interpretations. I’ve reproduced Tim’s exact words (with a couple minor grammar corrections), so that each of us can draw our own interpretation from them. Here’s mine:

·        You’re missing the boat if you focus all of your attention on the question of IT vs. OT. That’s not the issue.

·        The real issue – for both cyber regulations and best practices – is people, processes, and technologies. Get those right, and you won’t have to worry about IT vs. OT.

·        Don’t just pay attention to PPT in three silos, but look at how people, processes and technologies actually interact – as in the case of Whole Foods, where a needless dependence of cash payment systems on credit card payment systems made it impossible for this Whole Foods store to sell anything at all.

·        And just as important, make sure that people understand how the processes and technologies actually work, since for example a belief that OT exists safe behind its Maginot Line defenses can lead to a pretty rude awakening, just like in France in 1940.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the National Technology and Information Administration’s Software Component Transparency Initiative, for which I volunteer as co-leader of the Energy SBOM Proof of Concept. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Dr. Allan Friedman, who has been running the Software Component Transparency (SBOM) Initiative of the National Technology and Information Administration (NTIA) of the US Department of Commerce since its inception in 2018, announced recently that, as of later in August, he will move from NTIA to CISA (the Cybersecurity and Infrastructure Security Agency) in DHS.

When Allan announced this (and I’m sure he’s done it at least ten times in different meetings of the Initiative – including our Energy Proof of Concept meeting on Wednesday of this week), he has always immediately followed that by saying that he will still be completely involved in the work of the Initiative. But the Initiative will change, simply because CISA is a very different organization from NTIA.

Of course, it’s too early to know how it will change. Allan has promised (I believe him, too) that the whole group involved in the Initiative (there must be at least 200 people who attend at least one of the meetings in any given month, including from Europe and Japan) will meet with him in September to decide the way forward. This isn’t to say it will be a democratic process, but at least people will have their input.

Alan has pointed out many times over the past two weeks that the Initiative started from just about zero in 2018, and now has built up a substantial body of experience, knowledge and especially written guidance about SBOMs. This couldn’t have happened without the NTIA’s approach to launching a new technology (as they did with DNS in the 1980’s and 1990’s, and as they’re now doing with 5G).

To launch a new technology, NTIA doesn’t gather a bunch of wise people in a room (virtual or otherwise), who scratch their chins, offer profound thoughts, develop a very thoroughly-researched document describing in great detail all of the ins-and-outs and do’s-and-don’ts of the new technology, then go home and congratulate themselves on a job well done - whether or not anybody’s even looking at what they’ve written.

Rather, the NTIA gets the actual stakeholders together to figure out what’s needed for the new technology to succeed, and what’s the best way to get there; there are no preconditions, and all meetings and documents are completely public. In the case of SBOMs, a key tool is the industry-focused Proofs of Concept, of which there are currently three (healthcare, autos and energy). It’s possible the three PoCs will remain under NTIA’s auspices, simply because they’re working well and there’s no reason to mess with a good thing (the energy PoC is especially fortunate, since Idaho National Labs is providing support in many ways, including the web site and Ginger Wright, my very able co-leader in the effort). Of course, Allan will be able to participate in the meetings, no matter what agency they’re “under”.

So if everything was going so well, why is Allan making this switch? I believe (without having discussed it with him yet) that he looked at the number of cybersecurity professionals inside NTIA – a small number, certainly – vs. the number inside CISA (CISA had about 3400 employees last year, and I’m sure that number has already jumped a lot, especially as they keep getting more jobs added to their portfolio). And he saw that both he and the SBOM “movement” (cult?) can expand in all sorts of ways if they’re part of CISA, that they couldn’t even dream of under NTIA. There are some really huge possibilities, and Allan has just begun to explore them.

Good luck in the new gig, Allan! A new world is opening up for you and for SBOMs.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the National Technology and Information Administration’s Software Component Transparency Initiative, for which I volunteer. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

My post on the billing system at Colonial Pipeline brought out great comments from two wise men of the power industry cybersecurity world: Kevin Perry and Tim Roxey. As you’ll see, they didn’t the same thing at all, but they didn’t contradict each other, either. Rather, Tim’s comments built on Kevin’s.

Here’s a quick summary of my previous post, although I hope you’ll read it if you haven’t yet:

·        Even though the ransomware attack never reached Colonial’s OT network, it did bring down their billing system.

·        And even though it might seem odd that the loss of the billing system could bring down pipeline operations, there were actually good reasons for why that happened (which I’ll let you read).

·        I concluded by pointing out that “Tom’s First Law of OT Networks says that an ‘operations-focused’ company – as opposed to an information-focused company like an insurance company or a consulting firm – will be forced to bring their OT network down if their IT network falls victim to a ransomware attack.”

I stand by what I said, but Kevin’s and Tim’s email comments made me realize that I hadn’t asked the more interesting questions:

1.      How can we identify systems that don’t directly control operations, yet can have a huge impact on operations just the same (i.e, IT systems that perform functions required for operations)? And when we’ve identified them, what measures can we take to protect them better than other systems on the IT network that clearly have no direct operational impact, like say the systems that run the utility’s charitable operations?

2.      Should those systems be regulated by OT-focused cybersecurity compliance regimes, such as the dreaded…(and here I cross myself, despite not being Catholic)…NERC CIP?

3.      Or maybe we need to go beyond all this talk about regulation and protecting systems, and think about what the real problem might be?

Briefly, Kevin addressed questions 1 and 2; Tim took question 3 (not that I even thought of these questions until now, of course). I’ll start with what Kevin said, and cover what Tim said in my next post.

On Thursday, Kevin wrote this to me:

I would argue that any “IT” system, or system component that is essential to keeping to OT operational needs to be considered OT and kept isolated from the rest of the IT world.  As you noted, electric metering, whether at the customer point of delivery or in a tie substation, is OT.  The data from the meters are fed into the IT billing systems.  If the billing systems are down, bills will be delayed, but the meter data collection will continue until it can be transferred to the billing systems.  It is inexcusable that the OT must be shut down because an essential IT system is down.

Here are the points that I infer Kevin is making:

1.      This problem wouldn’t have happened in the electric power industry, since operations (including metering) can continue, even when the bills can’t be generated (no pun intended).

2.      The billing system is “essential to operations” in the pipeline industry (or at least in Colonial’s case), although not in the electric power industry (meaning it isn’t a BES Cyber System, or BCS).

3.      If there were a cyber regulatory regime like NERC CIP in place in the pipeline industry, the billing system would need to be considered the equivalent of a BCS.

4.      Regulation or no, the pipeline industry should protect their billing systems using at least some of the same measures (including isolation) used to protect OT systems.

I responded to Kevin’s email with the question, “If you think certain IT systems should be isolated, would you favor an expansion of the CIP standards to require network isolation, as well as perhaps some (although not necessarily all) of the other CIP requirements?”

I want to make one point here: CIP already covers a large group of systems that many electric utilities consider to be part of IT, not OT. Those are systems located in Control Centers. While these systems certainly perform an OT (and in many cases BES) function, they aren’t Industrial Control Systems, since they’re implemented on standard Intel-based hardware and run standard IT operating systems: Windows™ and Linux. A lot of the management that needs to be done on them is the same as what needs to be done for say financial systems.

And interestingly enough, Control Centers aren’t included in NERC’s 80-page “definition of the BES”. That definition requires an asset to be connected to the grid at 100kV or higher. The only reason systems in Control Centers are even included in CIP is because Control Centers are specifically called out in CIP-002 R1.1. So it wouldn’t be unprecedented if other “IT systems” were in scope for CIP, although CIP-002 would have to be amended for that to happen.

Kevin (a member of the NERC teams that drafted Urgent Action 1200, the CIP predecessor, as well as CIP versions 1 and 2, and who was then Chief CIP Auditor for the SPP Regional Entity for about ten years, until his retirement in 2018) replied to my email by saying:

A proper CIP-002 assessment of all Cyber Assets linked to the proper functioning of the readily identifiable OT should be sufficient.  In the early days, some entities tried to move systems out of scope simply by moving them out of the ESP (Electronic Security Perimeter).  My team always took a hard look at the historians that were outside the ESP and also their map board display systems.  Most entities simply used their historians for temporal data storage and non-real time engineering analysis, and keeping them out of scope was OK.  

But I am also aware of at least one entity that used their historian to drive their map board displays and also used the historian data for real-time decision making.  Their historians were Critical Cyber Assets (now BCS) because they were used for real-time operations.  At least one entity had map board displays that were not readily available on the dispatcher console, thus the map board also became a CCA/BCS.  And my team did not stop with systems used for the entity’s real-time operations.  An entity who declared their ICCP servers out of scope because they were not using the outbound data (destined for their RC or another BA or TOP) themselves found their decision frowned upon.  Even though they might not be receiving real-time data from a remote association, they were supplying real-time data essential to the recipient(s).  When they argued to the contrary, my team referred them to the TOP and IRO standards that compelled them to send what was initially known as “Appendix 4B” data.

So, apply the same logic to the billing system and you will see the meter data collection subsystem is absolutely a BCS if its failure causes you to shut down your OT (SCADA/EMS) systems.  The part of the billing system that sends the invoices and payments is not.  Processing invoices and payments can wait until you get that system back up.

Here is what I take away from what Kevin says that he doesn’t favor expanding the CIP requirements to include systems located on the IT network because, if a system on the IT network meets the definition of BES Cyber System (which the different examples he used all do, even though the entities that operate them hadn’t classified them as such), it must be treated as a BCS, including being located within the ESP (i.e. the OT network). Of course, this only applies at Medium and High impact BES assets. Low impact assets aren’t required to have ESPs.

So a system like the pipeline billing system – if it existed in the electric power world – would need to be treated as a BES Cyber System, subject to all the privileges (?) attendant on that august designation.

I then asked Kevin whether he thinks utilities should designate their meter data collection systems as BCS. His answer was nuanced, yet at the same time quite clear:

Inconsistent.  The meter data loss does not impact reliability within 15 minutes (Tom’s note: The definition of BES Cyber Asset/BES Cyber System requires that the loss or misuse of the system would have an impact on the Bulk Electric System within 15 minutes. If it has an impact but it will usually take longer than that to happen, it’s not a BCS).  But it also does not cause the utility to shut down the grid.  Loss of telemetry does not stop the revenue-quality meter from collecting data.  Loss of the meter itself does not stop the flow of electricity.  There are procedures for dealing with an occasional failure, including redundancy and inter-utility meter data reconciliation.

If the meter is only a revenue meter, then it does not need to be a BCS.  If the meter also reports real-time flows and/or voltage, then it is a BCS.  That is what I meant by inconsistent.

So Kevin is saying that, given the current NERC CIP requirements, there are only two choices: The meter data collection system is a BCS or it’s not. If it’s a BCS, it doesn’t get any break from any other BCS, in terms of the number or types of requirements that apply to it. If it’s not a BCS, it’s completely out of scope for CIP.

But there are certainly cases where a lack of good security on the IT network can result in an outage of the OT network. I described a dramatic example of that in thispost, where a ransomware attack that shut down the IT network but didn’t touch the OT network (as in the case of Colonial), in the end resulted in two large Control Centers being completely shut down for up to 24 hours, with the grid in a multistate area being run by cell phone.

It’s safe to say that none of the systems on the IT network of this utility met the definition of BCS, so there was no single system that led to the Control Centers being brought down – yet they were brought down anyway. This seems to me to point to the need for CIP to be extended in some way to cover IT assets – perhaps as some sort of “halfway house” asset. But there’s no way that the current CIP standards should be extended to cover anything else. They first need to be completely rewritten as risk-based. Then we can look at extending them to IT, based on the relative risk levels of OT vs. IT.

I’ll turn to Tim Roxey’s comments in my next post. 

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the National Technology and Information Administration’s Software Component Transparency Initiative, for which I volunteer. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Yesterday, I attended an excellent webinar on a topic I’ve been waiting to have someone explain to me, “Consequence Driven Cyber Informed Engineering (CCE) – Resilience Strategies”. It was sponsored by Midwest Reliability Organization (MRO), and featured two longtime friends of mine: Jodi Jensen of WAPA and Sam Chanoski of INL. Since a recording will be available on MRO’s website soon, I won’t try to reiterate what was said in the webinar, other than saying it’s worth your while to listen to the recording.

What inspired me to write this post was Jodi’s statement, regarding the Colonial Pipeline ransomware attack, that Colonial had said that they shut down the actual pipeline (i.e. their OT network) because of the loss of their billing system (which was on the IT network). Of course, the IT network was compromised, so it had to be shut down and the machines rebuilt.

Colonial insisted that their OT network wasn’t affected by the ransomware, but they had to shut it down anyway due to the loss of their billing system. Jodi wondered why the billing system was essential to operations. In other words, couldn’t they have continued shipping petroleum products through the pipeline and worried about billing later?

I wrote three posts after the Colonial incident: Here, here, and here(in that order). In all three of them, I discussed possible reasons why the OT network (and pipeline) had to be shut down, even though the ransomware didn’t penetrate it. I also linked to a postI wrote last October, describing an incident in 2018 in which a major utility – a BA for a multi-state area – had to shut down their Control Centers (i.e. an important part of their OT network) for up to 24 hours and run the grid from cell phones, when their IT network was hit by a ransomware attack that required rebuilding 12,000 computers from scratch.

Just like in the case of Colonial, the utility swore the ransomware never penetrated their OT network (and I have no reason not to believe them), but they couldn’t take the chance that just one machine in the Control Center had been compromised. If that had happened, that one machine might have then compromised all of the IT network when it was restarted, requiring another huge shutdown and rebuild (and I’m told that this becomes much less fun the second time around, to say nothing of the third or fourth time). Which is why they shut down and rebuilt all the systems in the Control Centers as well.

I brought up that incident because this might have been another reason why Colonial shut down their pipeline. And after I wrote the second post, one of the most prolific commenters on my posts, a person named Unknown, wrote in once again to say

Like you, I also believe that Colonial shut down because they could not accurately bill customers or track their customers' assets (i.e. refined petroleum products).

Pipelines are like banks and oil in the pipeline is like cash in the bank. If a bank loses its ability to track who gave them cash (or who they loaned it to), then there is no point opening the doors, even if they can safely store the money in the vault.

Unknown wrote this because I had pointed out in the post that the Washington Post had said in an editorial (which I paraphrase), “If they had kept their pipelines operating while the IT network was down, they wouldn’t have been able to invoice their customers.” I added, “And it’s safe to say that Colonial doesn’t feel that it should deliver gasoline through their pipeline solely as a charity.”

Unknown was pointing out that it was more than the wish to avoid operating as a charity that motivated Colonial to shut down. They don’t own the gasoline they ship in their pipeline, any more than a trucking company owns the furniture they ship or a bank owns the money in its vaults. If either one loses track of what’s been entrusted to them, the trucking company or bank has to repay the entire amount (and certainly with consequential damages) to whoever shipped the furniture or deposited the money.

In other words, this isn’t like an electric distribution utility, which – at least for a brief period of time – owns the electric power they’re distributing to their customers (I’ll omit discussion of Retail Choice here). That utility has to keep the lights on, no matter what it costs them, and if they can’t bill during an emergency, they can usually bill later (the meters needed for billing are all on the OT network, so presumably an IT network shutdown wouldn’t affect them anyway). Colonial isn’t obligated to keep the cars in Georgia full of gas (nor are they paid to do that, of course). They obviously can’t keep shipping gasoline if it’s likely they’ll end up having to pay the full cost of the gas to the shippers.

I concluded my third post on Colonial by articulating the first law of nature that I’ve ever identified. Tom’s First Law of OT Networks says that “an “operations-focused” company – as opposed to an information-focused company like an insurance company or a consulting firm – will be forced to bring their OT network down if their IT network falls victim to a ransomware attack.”

I’ve been told that this can’t be considered as a new law of nature because there are already enough of those. How about Newton’s Laws of Motion? They’ve been around since the 1600s, and Einstein showed they’re not applicable in extreme conditions. Why not drop one of them, and put my law in its place? Seems sensible to me…

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the National Technology and Information Administration’s Software Component Transparency Initiative, for which I volunteer. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

Next Thursday July 29, I’ll be speaking on a panel, with Ben Miller of Dragos, which will be asking (and at least trying to answer) the above question, at the “Transmission Infrastructure Investment, US” virtual conference. The panel will be led by Jim Cunningham of Protect Our Power.

Our panel addresses just one of ten live sessions in the conference, all of which look quite interesting. You can get an agenda and sign up here. Our session will run from 2:40 to 3:30 ET.

I’ll hope to see you there. If you run into me in the hallway or at lunch, please say hello.

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the National Technology and Information Administration’s Software Component Transparency Initiative, for which I volunteer. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

I anticipatedthat Josh Corman’s talk at this week’s Energy SBOM Proof of Concept meeting would be good, and I certainly wasn’t disappointed – in fact, it was great. I’m going to describe it a little here, but I’m pleased to announce that the video is available – so you don’t have to take my word on any of this. Josh’s talk starts a little after the 12-minute mark and goes on for 22 minutes (his connection went down at one point, but he came back very quickly).

The meeting was devoted to discussing use cases for SBOMs. It occurred to us in planning the meeting that one of the best ways to address this topic was to hear from the two people most responsible for the movement to make software bills of materials more than just a nice concept, but a regular practice with well-understood (but not mandated) guidelines for production and use. These two people were Dr. Allan Friedman, leader of the National Technology and Information Administration’s Software Component Transparency Initiative, and Josh, who coined the term SBOM. They both spoke at this week’s meeting on how they came to see SBOMs as an important need, and why.

Allan spoke first (and led the meeting, as he usually does). His talk was very good, and you should listen to it. However, Josh’s was exceptional. He covered two topics: The events that led him (and many others) to believe that SBOMs were needed, and SBOM use cases. The latter was based on the NTIA document whose development he led in 2019, Roles and Benefits for SBOMs across the Supply Chain, which is one of the three or four fundamental documents produced by the Initiative.

Below are some very interesting statements he made in the “history” part of his talk. They’re certainly nowhere near everything he said (he managed to get in lots of words in a short amount of time, without rushing his words. Fortunately, you can get everything he says if you’re not afraid to back up at a few points during his discussion), nor can I swear that I didn’t get a few things wrong.

1.      He remembers July 13, 2013 as the day that he woke up to the problem of software component vulnerabilities. On that day, servers running Apache Struts 2 – an open source component of many applications – were attackedthrough previously-unknown vulnerabilities.

2.      Josh’s reaction then was “It’s open season on open source. Who’s going to attack just one bank anymore, when they can attack lots of targets through one component?”

3.      At the time of that attack, Josh was in a high-level position at Akamai. However, he soon moved to Sonatype, an early leader in open source dependency (component) management – and now one of the leading software composition analysis tools.

4.      Probably the event that woke most of the rest of us out of our blissful ignorance of the problem of component vulnerabilities was the 2014 disclosure of the Heartbleed vulnerabilities in the OpenSSL cryptography library in 2014, which was estimated to be found in about half a million “secure” servers.

5.      Heartbleed – as far as I know – didn’t lead to any major breaches, but it required a huge effort by a huge number of organizations, just to find whether they had any vulnerable web servers - and if so, where. Why was that? OpenSSL is a component of other software, and often a component of other components, etc. Many organizations never even found all the instances of OpenSSL that they were running. For example, Josh says it took DHS six weeks to even answer the question of which federal agencies were affected by Heartbleed.

6.      Meanwhile, some financial companies knew in literally minutes or hours both whether and where they were affected. Why was this the case? Because they had kind of proto-SBOMs. Josh said the financial sector had woken up to this problem when he did – with the Apache Struts 2 attacks.

7.      After this, Josh decided to really dig into the idea of SBOMs and started reading Deming, who had stressed the importance of bills of materials for manufactured products. Having BOMs gave manufacturers the following advantages:

a.      They could have fewer, but petter parts.

b.      They could compare quality of different suppliers and buy more from the high-quality ones.

c.      They could track which parts went in which products, so that if there were a problem with a part, it could be tracked down and replaced in any product in which it had been used.

8.      Another seminal event for both Josh and awareness of component vulnerabilities was the 2015 SamSam ransomware attack on Hollywood Presbyterian Hospital. This attack exploited a vulnerability in the JBoss Java development platform (now called WildFly). The hospital had to shut down patient care for about one week.

9.      The hospital knew about SamSam, but didn’t have any idea whether it was affected by the vulnerability and if so, where. Ofcourse, this was because they had no SBOMs to provide them that information.

10.   It was this and the Wannacry attacks that caused the Food and Drug Administration, which regulates medical devices like pacemakers and infusion pumps, to put out a “Pre-market guidance” for those devices. While it didn’t require SBOMs immediately, it said they would be required in the future. This galvanized the medical community to start working on the problem of SBOMs and led to the creation of the NTIA Initiative.

But there’s a lot more. Watch the video!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. Nor are they shared by the National Technology and Information Administration’s Software Component Transparency Initiative, for which I volunteer. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.