In today’s data center, there are many ways to achieve system redundancy from a server connected to a fabric. Customers usually seek redundancy to increase service availability (such as achieving end-to-end AI workloads) and find system efficiency using different multihoming techniques.

In this post, we discuss the pros and cons of the well-known proprietary multi-chassis link aggregation group (MLAG) compared to standards-based EVPN multihoming (EVPN-MH). 

Introduction to MLAG  

Multihoming is necessary for all modern data centers, which enables a single host to connect to two or more nodes and serve in an all-active or single-active manner. All-active focuses on increasing capacity first and redundancy second. Single-active focuses primarily on redundancy.  

In the Internet Service Provider world, multihoming is a familiar concept, primarily for Point of Presence locations, where customer equipment interconnects with Provider Edge equipment locations. 

This connection is almost always a layer 3 routed connection and doesn’t introduce the challenges of the layer 2 world because it is intended to solve redundant site access or Internet access. However, in data centers, when we connect servers or end nodes into the network in a redundant way, we must get down to layer 2. 

MLAG came along in the early 2010s and many vendors implemented similar features that performed similar functionalities. One important thing to keep in mind is that MLAG is vendor-dependent proprietary technology. According to Wikipedia, MLAG’s “implementation varies by vendor; notably, the protocol existing between the chassis is proprietary.” This is a fundamental problem for MLAG that triggers many other issues. 

With MLAG (Figure 1), a client device can be a server or hypervisor, and a switch or router forms a classical link aggregation group (LAG) that typically bonds two physical links into a single logical link. On the other side of these links, you typically have two switches, which these links connect to. From an LACP point of view, these two switches act like a single switch with the same LACP system ID. This makes MLAG work from a server perspective. 

However, for the two MLAG participating switches, things are a bit more complex. Because they require state and MAC synchronization between them, a heartbeat is also needed to prevent split-brain situations and traffic flow over peer links in case one of the participating switches loses its uplink. This peer link makes the entire design non-standard, complex, and error-prone (not fitting in a CLOS leaf and spine architecture).  

There are efforts to make state and MAC synchronization standard. RFC7275 focuses on solving this problem and addressing it with a new protocol called Inter-Chassis Control Protocol (ICCP). However, different vendors still implement various flavors of RFC7275 and end up with the same issues. This MLAG solution solved the multihoming problem in a limited scope.  

While the future of MLAG is bleak, there’s a more flexible and technically superior multihoming solution: EVPN multihoming (also called EVPN-LAG or ESI-LAG). 

Benefits of EVPN multihoming  

Multihoming is no stranger to the ISP world and initially came along as a WAN technology. However, it became clear that modern data centers require their own way of implementing multihoming. 

Coincidentally, EVPN itself was first introduced as a WAN technology, then evolved into a data center technology. EVPN adopted multihoming functionality rather quickly. With RFC7432, EVPN-MH uses a new addressing field called the Ethernet Segment Identifier (ESI). This fundamental building block that makes EVPN-MH work is used everywhere across the fabric, as far as type-1 and type-4 routes are propagated. ESI is a 10-byte field that specifies a specific multihomed segment.

Let’s talk about what is under the hood of EVPN-MH, route types, and what makes it attractive compared to legacy and proprietary MLAG.  

EVPN-MH uses Border Gateway Protocol (BGP) as the control plane in contrast to ICCP, which MLAG uses. And, EVPN-MH uses several different types of EVPN route types as per RFC7432.

EVPN Route Type-1

EVPN Type-1 Route functions can be listed as mass withdrawal, aliasing, and load sharing (Figure 2).

Mass withdrawal 

Mass withdrawal makes sure that if a particular link goes down on an ES, you can withdraw all dependent MAC addresses connected to that particular link. This way, you achieve fast convergence by sending a mass withdrawal instead of one by one for each MAC. This assumes that a hypervisor is connected to that ES with many VMs, over the same VLAN, or over hundreds of VLANs. 

Aliasing and load balancing

Aliasing and load balancing makes sure that downstream traffic towards an ES is load-balanced across ES member switches, also known as EVI. This way, the ES member switches can receive traffic from other switches in the fabric in a load-shared manner, regardless of whether they are advertising that particular MAC behind their ES. 

EVPN Route Type-2

Type-2 (MAC/IP) routes are advertised by the same ES member leafs and they include the ESI value for each MAC attached to this Ethernet Segment (Figure 3).

Type-2 routes are not part of the EVPN-MH setup, however, they make use of ESI information when it’s present for a specific destination MAC.

EVPN Route Type-4

EVPN Type-4 routes are used for election of designated forwarder (DF) and autodiscovery of multihomed ES (Figure 4).

EVPN type-1 and type-4 routes make EVPN-MH work and provide standards-based interoperability. Type-4 routes are imported by only routers or leafs that participate in that particular ES. Other routers or leafs in the fabric that don’t participate in that ES don’t import type-4 routes. Type-4 routes are used for DF elections to select where to send local BUM traffic.As BUM traffic has to be flooded across the network, in multihomed scenarios, only the DF is responsible for sending BUM traffic to its clients (such as multihomed servers). 

Typical EVPN-MH topology can be seen in Figure 5.

Advantages of EVPN-MH: 

• Control-plane based MAC and state synchronization 
• Standards-based, BGP EVPN route types, and interoperability 
• Fabric-wide-route distribution of multihomed connections 
• Fast convergence, withdrawal 
• Capable of 2+ multihoming 
• No need for physical peer link connectivity 
• Future proof 
• Scalable with BGP 

Conclusion

EVPN-MH is a future-proof technology that uses BGP as its control plane. Its standards-based architecture, ability to provide multihoming to end hosts with more than two gateways, and active-active load balancing make it an attractive de facto solution in modern data center networks. Also, removing the need for a peer link between leafs fits EVPN-MH into the Clos architecture perfectly, reducing cost and complexity.  

I recommend using EVPN-MH for data centers with EVPN as the control plane, which will soon substitute all MLAG deployments in the field. Existing networks can stay with MLAG as they are already operational. However, new deployments and designs should certainly be based on EVPN-MH. 

For more resources, check out the NVIDIA Cumulus Linux Multi-Chassis Link Aggregation–MLAG configuration guide.

More than 40 million people had their health data leaked in 2021, and the trend is not optimistic.

The key goal of federated learning and analytics is to perform data analytics and machine learning without accessing the raw data of the remote sites. That’s the data you don’t own and are not supposed to access directly. But how can you make this happen with a higher degree of confidence?

Imagine a siloed federated learning scenario with a biotech company collaborating with a network of hospitals. They’re collaborating on an improved lung cancer detection model based on CT scans of images stored locally in local infrastructure.

Both the aggregator and the biotech data scientists are not permitted to access images directly or download them. They are only permitted to perform federated learning, training, and validation of remote models and to build better-aggregated models, which are then shared with all hospitals for improved generalization and better detection accuracy.

The goal of data protection seems to be obvious at first. It’s a question of permissions, roles, and maybe some encryption here and there. Unfortunately, it’s not as easy as it might seem.

Federated privacy defaults

Federated learning solutions tend to focus on the following:

• Transmission channel security
• Point-to-point trust (using certificates)
• Efficiency of the workflows
• Support for many existing algorithms

All these reduce the risk of inference of the raw data from the model itself.

However, most of the products and publications omit the important threat of a too-curious data scientist. Many products, by design or default, prioritize the near-absolute freedom of data processing by enabling external data scientists to send any queries or operations against remote data.

This is a perfectly acceptable setup for networks with virtually unlimited trust between the participants or whenever no sensitive data is being used. For instance, the network might be used for demonstration purposes or everything could happen within the boundaries of the data-owner organization.

This is how NVIDIA FLARE works by default when you want to use custom training and validation code.

No matter what encryption you use, how strong the transmission channel security guarantees are, or how up-to-date all the systems are on component vulnerabilities, when you permit data scientists to send any queries or code against remote data, you cannot guarantee data leakage protection.

On the contrary, you are fully dependent on trust levels for individuals or contracts. Or, you could worry later by analyzing logs of the operations and queries executed against the data that you don’t own in a federated network.

One or more data scientists may one day abandon honesty. Or, the damage could be accidental instead of malicious. Market data shows that the majority of data attacks are coming from the inside. Unfortunately, inside threats are also on the rise.

Proper employee education, contract clauses, trust, privacy awareness, and work ethics are important but you should provide much stronger guarantees using technical measures.

What to do about it

Data owners should be in full control: who does what when against what data.

Logging all data-related operations is often promised as a solution by multiple product vendors. Well, after the potential damage of the data leak is done, it’s too late. You must be proactive and prevent such things during the design phase.

What about existing permission systems? In the case of NVIDIA FLARE, it is either on or off, enabling remote data scientists to execute any jobs on remote sites (local policies). Also, the biotech administration can’t manage those permissions centrally, as they would be able to override local policies remotely.

Other solutions opt for binary Docker images pushed from a central repository to remote sites (hospitals), based on the trust that whatever is there can be trusted. This practically eliminates the data owners from the acceptance process, as they can only trust the closed box of the image. Technically, they could download the image, mount it, and review files but it’s impractical at scale.

There’s a more practical approach available.

Step up your data protection game using NVIDIA FLARE 2.3.2

Here’s how to lower the risk of data leakage significantly with all its consequences, both financial and reputational. You can deliver on the promise of data protection, which is a key element of federated learning and analytics using the latest features introduced in NVIDIA FLARE 2.3.2, as a result of our fruitful collaboration.

Job acceptance and rejection requirements

You need a solution that enables data owners to review the code to be executed against their data before it happens. Practically, in NVIDIA FLARE, it means Python code implementing the trainer, validation, and federated statistics plus configuration settings.

Data owners should be able to review and accept or reject the code themselves or use trusted third-party reviewers. Nothing should happen against data without the explicit acceptance of the data owner.

No job code should be changed from the previously accepted code to malicious code overnight. It should be rejected because its contents have changed and it must be re-reviewed and re-accepted.

Solution

NVIDIA FLARE 2.3.2 delivers enablement of custom event handlers that perform actions to enable the components to be created at the site and controlled by the site.

But why object creation? Can’t you just focus on the execution?

Because too-curious data scientists could easily inject the code into object initialization (constructor), object creation is essential. Act as early as possible to prevent the code that data owners don’t want from running against their data.

The following simplified flow is the default:

• A central server controlled by external data scientists submits jobs to clients.
• Clients schedule and execute those jobs.
• If the code contains uploading data to the cloud, virtually anything that Python permits is executed.
• Even worse, the code may change with each job submission and it won’t be detected or prevented from execution.

Jobs are sent from the orchestration node to local nodes for execution. They contain code and configuration.

After the federated network is built using the default NVIDIA FLARE configuration, there is unconditional trust in external data scientists by data owners. They are then permitted to submit jobs with local policies.

After the change, with a custom implementation:

• A central server controlled by external data scientists submits jobs to clients.
• A data owner reviews the job code and determines if it’s acceptable from the data leakage risk perspective.
• If the code is approved, the hash is added to the list of accepted hashes.
• The job code (hash, signature) is checked locally at the site against the list of accepted hashes of jobs.
• The job is executed on the client and results are returned to the server.

Here’s more detail about how the job workflow changes.

Figure 2 shows a federated learning network consisting of a BioTech orchestration node and three hospital nodes. The same federated training job is sent from the orchestration node for the local nodes. The job is not even created to prevent any malicious code as part of the initialization or constructor of the job object.

Figure 3 shows the flow of job acceptance and execution using a new type of event and event handler.

Job code acceptance strategies

Thanks to the open event–based model delivered in NVIDIA FLARE 2.3.2, it is possible to implement any suitable code verification strategy. Such strategies must always be designed, defined, and agreed upon in a federated network and then deployed on each node (a client, such as a hospital).

For demonstration purposes, you can compare the code content hash against the accepted code stored in a different directory.

For more real-world, enterprise-grade scenarios, you can also provide implementations based on digital signatures of the code and even co-signatures provided by the third party trusted by data owners. This is independent of biotech performing the federated training.

Code example

NVIDIA FLARE raises the BEFORE_BUILD_COMPONENT event before a new component (that is, a job) is instantiated. All you must do is write the event handler to analyze the code and determine if it was accepted. There’s no turnkey solution for that, as different federated networks may require different strategies. The following code example demonstrates such a handler. For demonstration purposes, the example only focuses on a subset of jobs.

focuses on a subset of jobs.
def handle_event(self, event_type: str, fl_ctx: FLContext):
    if event_type == EventType.BEFORE_BUILD_COMPONENT:
       
        # scanning only too curious data scientist jobs
        if self.playground_mode:
            meta = fl_ctx.get_prop(FLContextKey.JOB_META)
            log.info(f"meta: {meta}")
            if not "too-curious-data-scientist" in meta["name"]:
                return
           
        workspace: Workspace = fl_ctx.get_prop(key=ReservedKey.WORKSPACE_OBJECT) 
        job_id = fl_ctx.get_job_id()
       
        log.debug(fl_ctx.get_prop(FLContextKey.COMPONENT_CONFIG))
        log.debug(f"Run id in filter: " + job_id)
        log.debug(f"rootdir: {workspace.get_root_dir()}, app_config_dir: {workspace.get_app_config_dir(job_id)}, app_custom_dir: {workspace.get_app_custom_dir(job_id)}" )
       
        #making sure that approved_configs hash set is up to date (it's possible to update )
        self.populate_approved_hash_set(os.path.join(workspace.get_root_dir(), self.approved_config_directory_name))
        log.debug(f"Approved hash list contains: {len(self.approved_hash_set)} items")
       
        # check if client configuration json is approved (job configuration)
        current_hash = self.hash_file(os.path.join(workspace.get_app_config_dir(job_id), JobConstants.CLIENT_JOB_CONFIG))
        if current_hash in self.approved_hash_set:
            log.info(f"Client job configuration in approved list! with hash {current_hash}")
        else:
            log.error(f"Client job configuration not in approved list! with hash {current_hash}")
            log.error("Not approved job configuration! Throwing UnsafeComponentError!")
            raise UnsafeComponentError("Not approved job configuration! Killing workflow")
       
        # check if all classes added to custom directory are approved
        job_custom_classes = list(Path(os.path.join(workspace.get_app_custom_dir(job_id))).rglob("*.py"))
        for current_class_file in job_custom_classes:
            current_class_file_hash = self.hash_file(current_class_file)
            if current_class_file_hash in self.approved_hash_set:
                log.info(f"Custom class {current_class_file} in approved list!")
            else:
                log.error(f"Class {current_class_file} not in approved list!")
                log.error("Not approved job! Throwing UnsafeComponentError!")
                raise UnsafeComponentError(f"Class {current_class_file} not in approved list! with hash {current_class_file_hash}. Not approved job! Killing workflow")

As we demonstrated earlier, all the required contextual data is provided by NVIDIA FLARE to be able to perform required actions such as finding the custom code files, calculating their hashes, and so on.

There’s more

This does not solve all the problems related to data protection. However, in all the scenarios with data owners having limited trust for the remote data scientists training models on their data without seeing it, addressing this problem is imperative.

While this feature is not a definitive fail-safe measure against malicious users, it provides an additional layer of protection. It empowers nodes and fosters collaborative research through shared responsibility.

Next, consider focusing on other important areas, such as the following:

• Model inference attacks
• Differential privacy
• Transmission channel security
• Output filters

Summary

The principle of defense in depth in the case of federated learning and analytics makes it necessary to protect the data owner from possibly malicious remote code sent by external data scientists. In the case of truly federated scenarios, when there’s no full trust relationship between data owners and remote scientists, this is not optional.

The promise of remote data inaccessibility doesn’t deliver itself; you must empower data owners. It’s not guaranteed by default.

In this post, we demonstrated how to address this important threat using NVIDIA FLARE 2.3.2 to enable better data protection and build more secure federated learning networks today and in the future.

Generative AI is taking the world by storm, from large language models (LLMs) to generative pretrained transformer (GPT) models to diffusion models. NVIDIA is uniquely positioned to accelerate generative AI workloads, but also those for data processing, analytics, high-performance computing (HPC), quantitative financial applications, and more. NVIDIA offers a one-stop solution for diverse workload needs.

In quantitative applications for financial risk management, for example, NVIDIA GPUs offer incredible speed with great efficiency. NVIDIA H100 Tensor Core GPUs were featured in a stack that set several records in a recent STAC-A2 audit. 

STAC-A2 benchmark

STAC-A2 is a risk management benchmark created by the Strategic Technology Analysis Center (STAC) Benchmark Council for the assessment of technology stacks used for compute-intensive analytic workloads in finance. Designed by quantitative analysts and technologists from some of the world’s largest banks, STAC-A2 reports the performance, scaling, quality, and resource efficiency of any technology stack that can handle the workload: Monte Carlo estimation of Heston-based Greeks for path-dependent, multi-asset options with early exercise.

STAC-A2 is the technology benchmark standard based on financial market risk analysis. The workload can be a proxy extended to price discovery, market risk calculations such as sensitivity Greeks, profit and loss, and value at risk (VaR) in market risk. The benchmark as a proxy can also be extended to counterparty credit risk (CCR)  workloads such as credit valuation adjustment (CVA) and margin that financial institutions calculate for trading as well as risk management. 

Compared to all publicly reported solutions to date, this NVIDIA H100-based solution set numerous performance and efficiency records, including (but not limited to):

• The first sub-10ms warm time (8.9 ms) in the baseline Greeks benchmark.
• A cold time in the baseline Greeks benchmark of 38 ms, more than 3x faster than any previous benchmark.
• The fastest warm (0.51 s) and cold (1.85 s) times in the large Greeks benchmarks.
• The most correlated assets (400) and most paths (310,000,000) simulated in 10 minutes.
• The best energy efficiency (311,045 options/kWh).

Compared to a solution using 8x NVIDIA A100 SXM4 80GB GPUs, as well as previous generations of the NVIDIA STAC Pack and NVIDIA CUDA, this solution performed:

• 10x faster in the cold run of the baseline Greeks benchmark².
• 1.38x faster in the warm runs of the baseline Greeks benchmark¹.
• 1.38x faster in the cold run of the large Greeks benchmark⁴.
• 1.4x faster in the warm runs of the large Greeks benchmark³.
• With 10% more energy efficiency, benchmark 7.

Integrated hardware plus software solutions

One of the industry’s key concerns is risk calculation, which relies heavily on the latest technologies for real-time calculations and instant decision-making for trading and risk management. The HPE ProLiant XL675d Gen10 Plus server supports a dense node strategy for financial HPC, enabling up to 10 NVIDIA GPUs to efficiently populate a single HPC server. With this type of dense node, a compute farm for portfolio risk calculations can be realized with far fewer nodes achieving higher performance with lower operational costs for real-world calculations such as price discovery, market risk (such as VaR), and counterparty risk (such as CVA).

In areas such as CVA, such setups have been shown to reduce the number of nodes from 100 to 4 in simulation- and compute-intensive calculations (separately from STAC benchmarking).

This dense node solution enables an exciting strategy of scaling up with NVIDIA GPUs for a reduced number of nodes. It enables the highest performance at the lowest operating cost and the smallest data center footprint. The solutions can be extended to other workloads, such as AI language inference and backtesting as needed for a scaling strategy with such dense servers.

In addition to the hardware, NVIDIA provides all the key software component layers. This offers multiple options to developers, including the language of choice such as CUDA C++.

In calculations that are typically developed for fast run times, this implementation was developed on CUDA 12.0. The implementation uses the highly optimized libraries delivered with CUDA: cuBLAS, the GPU-enabled implementation of the linear algebra package BLAS, and cuRAND, a parallel and efficient GPU implementation of random number generators. 

The different components of the implementation are designed in a modular and maintainable framework using object-oriented programming. All floating-point operations were conducted in IEEE-754 double precision (64 bits). The implementation was developed using the various tools provided by NVIDIA to help debug and profile CUDA code. These tools include NVIDIA Nsight Systems for timeline profiling; NVIDIA Nsight Compute for kernel profiling; and NVIDIA Compute Sanitizer and CUDA-GDB for debugging.

Solution for risk HPC and AI convergence

NVIDIA H100 GPUs are an integral part of the NVIDIA data center platform. Built for AI, HPC, and data analytics, the platform accelerates over 4,000 applications. It is available everywhere, from data center to edge, delivering both dramatic performance gains and cost-saving opportunities with the aim of accelerating “every workload, everywhere.” 

The NVIDIA H100 PCIe GPU incorporates groundbreaking technology, such as NVIDIA Hopper architecture, with a theoretical peak performance of 51 TFLOPS for single precision and 26 TFLOPS for double-precision calculations. It uses 14,592 CUDA cores, plus 456 fourth-generation Tensor Core modules, which can deliver theoretical peak performance of 1,513 TFLOPS for BF16 and 51 TFLOPS for FP64 matrix calculations.

For HPC applications, the NVIDIA H100 almost triples the theoretical floating-point operations per second (FLOPS) of FP64 compared to the NVIDIA A100. It also adds dynamic programming instructions (DPX) to help achieve better performance. NVIDIA H100 GPUs feature fourth-generation Tensor Cores and the Transformer Engine with FP8 precision. 

With second-generation Multi-Instance GPU (MIG), built-in NVIDIA confidential computing, and NVIDIA NVLink, the NVIDIA H100 aims to securely accelerate workloads for every data center, from enterprise to exascale. The NVIDIA GPUs in SXM form share a switched NVLink 4.0 interconnect, providing high-speed GPU-to-GPU communication bandwidth. 

The H100 PCIe Gen 4 configuration used in this SUT provides most of the specified capabilities of H100 SXM5 GPUs in just 350 watts of thermal design power (TDP). This configuration can optionally use the NVLink bridge for connecting up to two GPUs for applications like deep learning in AI that are coded to take advantage of inter-GPU communications. (The STAC-A2 Pack does not use these fabrics.)

Summary

Whether at single-server scale or in larger scaling systems optimized for today’s most demanding HPC plus AI workloads, NVIDIA is uniquely positioned to accelerate workloads ranging from HPC quantitative financial applications and data processing to analytics and generative AI. In addition to risk calculations, organizations are converging NLP with generative AI, feeding inputs to quantitative calculations.

This is an active area of work, wherein the convergence of HPC and AI is happening as financial firms work on big-picture solutions combining various modeling techniques including HPC quantitative finance, machine learning (ML), and AI with neural net as well as NLP with generative AI. This enables firms to drive multiple complex business needs with converged HPC plus AI solutions that are increasingly a result of accelerated AI adoption, combining workloads for unique solution needs in the financial industry. 

To learn more, check out HPE and NVIDIA Financial Services Solution, Powered by NVIDIA, Sets New Records in Performance. 

Reach out to NVIDIA Financial Services with questions as you evaluate or apply accelerated compute to your critical business problems. Read the full report, CUDA 12.0 with 8x NVIDIA H100 PCIe 80GiB GPUs in an HPE ProLiant XL675d Gen10 Plus Server. 

References

¹1.38x faster in the warm runs of the baseline Greeks benchmark (STAC-A2.β2.GREEKS.TIME.WARM).
²STAC-A2.β2.GREEKS.TIME.COLD
³STAC-A2.β2.GREEKS.10-100k-1260TIME.WARM
⁴STAC-A2.β2.GREEKS.10-100k-1260TIME.COLD
⁵STAC-A2.β2.GREEKS.MAX_ASSETS
⁶STAC-A2.β2.GREEKS.MAX_PATHS

NVIDIA, working with Fujitsu and Wind River, has enabled NTT DOCOMO to launch the first GPU-accelerated commercial Open RAN 5G service in its network in Japan. This makes it the first-ever telco in the world to deploy a GPU-accelerated commercial 5G network.

The announcement is a major milestone as the telecom industry strives to address the multi-billion-dollar problem of driving improvements in performance, total cost of ownership (TCO), and energy efficiency. The solution unlocks the flexibility, scalability, and supply chain diversity promise of Open RAN.

DOCOMO and its partners confirmed that the solution is based on Fujitsu’s virtualized centralized unit (vCU) and virtualized distributed unit (vDU), the NVIDIA Aerial platform, and Wind River’s distributed cloud platform.

The 5G Open RAN solution is the first 5G vRAN for telecom commercial deployment using the NVIDIA Aerial platform. The platform brings together the NVIDIA Aerial vRAN stack for 5G, AI frameworks, accelerated compute infrastructure, and long-term software support and maintenance. It delivers innovative and transformational new capabilities for telco operators.

TCO and energy efficiency benefits

Working with vendors Fujitsu and Wind River, the new deployment uses the NVIDIA Aerial platform to deliver high performance, high cell density, and flexibility to DOCOMO’s 5G network, bringing better utilization of its network, lower TCO, and reduced power consumption.

DOCOMO notes that, when compared to its standard network based on a proprietary solution, this new solution reduces TCO by up to 30%, improves network design utilization by up to 50%, and reduces power consumption at base stations by up to 50%.

As of July 2023, DOCOMO serves over 22M 5G subscribers, from 20K+ base stations, with 5G coverage in over 815 cities. It uses 29 types of radio units (RUs) from four vendors and eight types of CUs from three vendors. The introduction of vRAN is expected to expand the capacity and coverage of the 5G network.

While the ability to mix and match products from different vendors promises improvements in flexibility and scalability from Open RAN networks, it poses two major concerns for operators:

• First, it makes it challenging to bring out the best performance from the different vendor products.
• Second, there are often technical issues that are only found at interoperability testing, which any operator deploying Open RAN must deal with.

NVIDIA, Fujitsu, Wind River 5G Open RAN partnership

DOCOMO launched OREX as an Open RAN service brand to address the challenges facing Open RAN. After the project was launched in February 2021, Fujitsu, NVIDIA, and Wind River worked together under OREX to develop the 5G vRAN solution, which is based on Fujitsu’s vDU and vCU.

The solution uses the following components:

• Commercial-off-the-shelf (COTS) servers
• Wind River distributed cloud platform
• Fujitsu’s 5G vRAN software
• NVIDIA Aerial vRAN stack
• NVIDIA Converged Accelerator

This is the first vendor consortium to deliver a commercial live 5G vRAN that meets NTT DOCOMO’s performance and interoperability requirements.

The NVIDIA Aerial platform includes NVIDIA Aerial vRAN stack software for the physical (PHY) layer 1 (L1), and NVIDIA Converged Accelerator with its combined data processing unit (DPU) and GPU for hardware acceleration of the computationally intense L1 workload. This makes it the world’s first DPU and GPU-accelerated (that is, NVIDIA-accelerated) 5G Open RAN to be deployed commercially to deliver a scalable, flexible, and cost-efficient network.

NVIDIA Aerial platform: Building blocks for wireless innovation

NVIDIA is driving innovation in the telecom industry with a portfolio of wireless frameworks, AI frameworks, and accelerated computing infrastructure. This enables the development of high-performance, fully software-defined, and AI-native networks with cloud economics (Figure 2).

 The accelerated computing infrastructure is made up of a combination of CPU, DPU, and GPU, together with a range of NVIDIA-certified COTS hardware servers.

NVIDIA Aerial is the platform with software, hardware, and support for delivering innovation in the wireless market segment. It brings together the NVIDIA Aerial vRAN stack for 5G, AI frameworks, other wireless frameworks, an accelerated compute infrastructure, and long-term software support.

Thanks to this combination of industry-shaping hardware, software, and the long-term support and maintenance typical for a commercial-grade software stack, this enables new performance thresholds for 5G networks. 

The key components of the platform are as follows:

• Software: NVIDIA Aerial vRAN stack
• Hardware: NVIDIA Accelerated Computing
• Carrier-grade support

Software: NVIDIA Aerial vRAN stack

This is an application framework for building high-performance, 100% software-defined, cloud-native, 5G vRAN, with O-RAN 7.2-x split. The NVIDIA Aerial vRAN stack is highly flexible and scalable and can deliver high performance for the L1.

NVIDIA Aerial has adopted a GPU-centric approach and relies on two notable subcomponents:

• NVIDIA cuBB SDK (CUDA baseband)
• NVIDIA DOCA GPUNetIO

cuBB provides GPU-accelerated 5G L1 processing. It delivers high throughput and efficiency by keeping all PHY layer processing within the high-performance GPU memory. The cuBB SDK also includes the 5G L1 high-PHY acceleration library cuPHY, which is optimized for NVIDIA GPUs. cuPHY offers unparalleled scalability by using the GPU’s massive computing capability and a high degree of parallelism.

NVIDIA DOCA GPUNetIO improves the performance of inline hardware acceleration. It provides optimized I/O and packet processing by exchanging packets directly between GPU memory and the DPU using direct memory access (DMA) technology.

Hardware: NVIDIA Accelerated Computing

This is the processing engine of the vRAN and is made up of CPUs, DPUs, and GPUs, together with COTS hardware servers.

The performance of the NVIDIA Aerial vRAN stack, especially the computationally intensive L1, is dependent on the choice of hardware it is deployed on. NVIDIA offers two hardware options for 5G network deployments. Their comparative performance is shown in Table 1 below.

• x86 and NVIDIA Converged Accelerator
• NVIDIA Grace Hopper and NVIDIA BlueField DPU

x86 and NVIDIA Converged Accelerator

This option combines the performance of NVIDIA DPUs and GPUs, together with an x86 CPU server, to deliver maximum performance for the 5G vRAN.

The integration of the GPU and DPU brings all front-haul enhanced common public radio interface (eCPRI) data traffic into the GPU without the CPU in the datapath. This is a full inline L1 offload, so the solution achieves high performance by avoiding the back-and-forth of eCPRI data between the CPU and the hardware accelerator in alternative systems across the host PCIe interface.

After the data is in the GPU, it benefits from the massive parallelism of the GPU architecture to optimize the processing capacity of the base station system. This brings improved RU capacity and processing power, provides a high-quality communications environment, and can handle high-load data processing along with future improvements in antenna technologies.

NVIDIA Grace Hopper and NVIDIA BlueField DPU

The NVIDIA Grace Hopper Superchip brings together the NVIDIA Grace CPU, which is based on the Arm architecture, and the high-performance NVIDIA Hopper GPU. The BlueField DPU helps to achieve the same performance from full inline L1 offload in a similar way to NVIDIA Converged Accelerator.

However, the biggest boost to performance comes from the integration of the CPU and GPU architectures using NVIDIA NVLink-C2C to deliver a CPU+GPU coherent memory model for accelerated workloads such as 5G vRAN.

NVIDIA NVLink-C2C is the NVIDIA memory-coherent, high-bandwidth, and low-latency interconnect. It delivers up to 900 GB/s total bandwidth: 7x higher bandwidth than the x16 PCIe Gen5 lanes commonly used in accelerated systems.

With the NVLink-C2C memory coherency, both CPU and GPU threads can concurrently and transparently access both CPU and GPU resident memory, enabling the RAN to optimize how it handles eCPRI data across CPU and GPU.

Carrier-grade support

The NVIDIA Aerial platform provides a full-stack, carrier-grade, hardened, and mature 5G solution with 10 years of long-life support and maintenance for telecommunications operators. This level of carrier-grade support provides assurances of reliability and resilience for telcos for field or commercial deployment using the platform.

OREX: Building out from Japan and beyond

The commercial deployment of a 5G Open RAN network by NTT DOCOMO, using the NVIDIA 5G platform, is a major milestone for the telecommunications industry. It showcases the capabilities of GPU-based acceleration for computationally intensive L1 PHY processing.

This new network comes with improved performance, flexibility, and scalability, plus higher cell density, significant improvements in energy efficiency, and a reduction in TCO. The delivery of this network paves the way for widespread adoption of GPU-based acceleration in cellular RANs.

DOCOMO and its partners in OREX are working together to promote a multi-vendor, Open RAN–compliant 5G vRAN to the global operator community. The commercial deployment in Japan is in alignment with the vision of OREX, enabling its members to validate their solutions commercially and then promote it to other operators globally.

NVIDIA continues to work with DOCOMO and other partners to support operators around the world to deploy high-performance, energy-efficient, software-defined, commercial 5G vRAN.

Sensor simulation is a critical tool to address the gaps in real-world data for autonomous vehicle (AV) development. However, it is only effective if sensor models accurately reflect the physical world. 

Sensors can be either passive, such as cameras—or active, sending out either an electromagnetic wave (lidar, radar) or an acoustic wave (ultrasonic) to generate the sensor output. When modeled in simulation, each modality must be validated against its real-world counterpart.

In previous posts, we detailed the validation process for camera and lidar models in NVIDIA DRIVE Sim. See Validating NVIDIA DRIVE Sim Camera Models and Validating NVIDIA DRIVE Sim Lidar Models. This post will cover radar, an essential sensor for obstacle detection and avoidance. 

There are multiple ways to approach radar validation. You can compare how an AV stack trained on real-world data behaves when encountering synthetic radar data, for example. Or, you can compare synthetic radar data to its physical counterpart in real-world experiments. 

Validating the model with an AV stack only evaluates its ability to the extent of triggering the AV function, which tests for a lower fidelity ceiling. For this reason, we will focus on the second approach.

Radar sensor pipeline

Radar sensors emit radio waves that reflect off objects in the scene and return to the sensor. The received signal then undergoes multiple processing stages that identify the returns from real objects and filter noises. These returns are then presented as a point cloud of the environment.

Such postprocessing methods are typically part of a sensor maker’s intellectual property, and thus NVIDIA DRIVE Sim sensor models aim to approximate them. Sensor suppliers in the DRIVE Sim ecosystem can include the exact implementations of their entire pipelines, including postprocessing.

DRIVE Sim uses ray tracing to model active sensors. Rays that embed the radar radiation pattern are fired into the scene. For each ray that hits an object in the 3D scene, secondary rays are created for reflections and transmissions based on the hit material’s wavelength-dependent properties. The materials in DRIVE Sim use bidirectional scattering distribution functions (BSDF). This enables simulating multipath effects.

DRIVE Sim ray tracing is time-aware. Each ray has its own timestamp and sees a different environment and sensor position to match that time. This enables simulating time-based effects, such as rolling shutter and Doppler.

For radar, after the stopping criteria for ray tracing are met, the sensor model consolidates the returns and processes them. Our radar model accounts for the sensor’s field of view (FOV), antenna directivity, resolutions, ambiguities, and the radar’s sensitivity pattern. A constant false alarm rate (CFAR) algorithm is used to extract valid detections over a simulated noise baseline. The detections are then encoded with the exact communication protocol as the real sensor to serve hardware-in-the-loop use cases.

Radar validation 

To validate the DRIVE Sim radar model, we designed three scenarios based on the technical product specification (TPS) of the real radar. The goal was to test various components of the radar sensor’s performance, including its detection capability over its FOV, separation capability, and accuracy in dynamic conditions. Then, we created a digital twin environment in simulation, collecting the equivalent data in DRIVE Sim for detailed analysis.

Data collection environment

For the data collection environment, we opted for an open spacious area—the Transportation Research Center in California. In this environment, we could minimize noise and unwanted reflections to simplify digital twin construction in DRIVE Sim.

Vehicle setup

We used radar sensors in the NVIDIA DRIVE Hyperion AV reference architecture, so developers building on NVIDIA DRIVE can easily transition between simulation and the real world. The sensors were mounted on a development vehicle (Figure 3). For this case, the front center radar (FCR) was the focus for evaluation.

Operating at a frequency of 77GHz, the radar under test included two scans: a near scan, with a wide FOV but limited range, and a far scan with extended range but a narrow FOV. Additionally, a 360° rotating lidar sensor (LD1) was mounted on top of the car to provide pseudo ground truth data.

Model validation process

Central to our three validation experiments were two trihedral corner reflectors. These are standard radar targets that reflect energy back in the incident direction. They are characterized by a radar cross-section (RCS) value, which is a measure of an object’s ability to reflect radar energy back to the receiver. 

We used one with a “high” RCS of 15.71 decibels relative to a square meter (dBsm), and another with a “low” RCS of 4.79 dBsm to characterize the model’s behavior across a wide RCS range.

The lidar’s pseudo-ground truth measurements were used to replicate the test setup virtually in DRIVE Sim with accurate material assignments. 

After collecting the virtual data, we compared the radar model outputs to the real radar. Results of the comparison are presented for the three scenarios below.

Scenario 1: FOV sampling with corner reflector

In the first scenario, we assessed the radar’s detection capabilities across its FOV and verified its range and azimuth accuracy. 

We placed a corner reflector at multiple grid positions within the radar’s FOV, as shown in Figure 5. We assumed the sensor’s behavior to be symmetric, and thus we only sampled half of the FOV to increase the sampling density. 

Altogether, we recorded a total of 579 positions for the high RCS corner reflector and 632 for the low RCS corner reflector.

Figure 6 depicts a top-down view of both real and simulated radar detections across all 1,211 high and low RCS corner reflector positions. We used this as a coherence check to start. Although we observed differences in FOV coverages above 80m, the overall coverage presented a noticeable similarity that is sufficient for the cross check.

The histograms in Figure 7 present the error distribution in range, azimuth, and RCS relative to the ground truth for both the high and low RCS corner reflectors, combined. Where applicable, we quantified the results by fitting a Gaussian distribution to the data. Results for the real radar are displayed on the left, while the DRIVE Sim data is shown on the right. 

We observed a high level of agreement between real and simulation data over various positions in the radar’s FOV, with both means and standard deviations sharing the same order of magnitude.

The discrepancies are primarily attributable to uncertainties in the ground truth. While the lidar sensor has millimeter-level accuracy, identifying the position and orientation of an object like a pole-mounted corner reflector can introduce errors in the centimeter range. Furthermore, while we calibrated the sensor positions prior to data collection, there might still be minor misalignments.

Overall, the agreement observed in RCS values, detection pattern, and the accuracies of the various detection properties validated the radar’s fidelity, wave propagation, and material modeling.

Scenario 2: Corner reflector separation capability test

In scenarios where road objects are near each other (stationary vehicles under a bridge, pedestrians or motorcyclists next to a vehicle or guard rail, or two closely parked cars, for example) radars can encounter difficulties in distinguishing individual objects. For this reason, it is crucial to accurately simulate this characteristic, known as separation capability.

We assessed this capability by placing the two corner reflectors in close proximity to each other. The data was sampled at four different distances from the sensor. For each distance, the corner reflectors were positioned as shown in Figure 8.

We selected different positions for the radar’s near and far scans, dependent on their corresponding FOV, to analyze their range and azimuth separation capabilities.

Tables 1 and 2 summarize the results for the near and far scans. The left column represents positions where CRs are close and we expect one detection per the TPS. The center and right columns represent positions where we expect two detections per the TPS. Each cell details the exact location of the CRs, and the number of detections observed for simulation and the real world. The percentage denotes the proportion of detections that followed our expectations in all considered scans. We define success when the simulated and real percentages are less than 20% apart.

Results for all positions at 0° and 50m, and -45° and 20m, demonstrated a high degree of similarity between real and simulated. We observed a minor discrepancy at 0° and 50m where CR2 (50.5, 0). In this scenario, the real radar returned two detections instead of one in 10% of the scans.

Comparisons made at -45° and 50m were mostly consistent, except for CR2 (36.36, -35.36), where the simulated radar returned two detections. 

As shown in Table 2, the results from the simulated and real-world sensors are largely in correlation. Significant deviations are noted at 0° and 50m where CR2 (50.5, 0). Furthermore, for 0° and 100m where CR1=CR2, the simulated radar returns two detections in 40% of scans, while real world never returns two detections. 

Upon further analysis of the deviations, we attribute them to the fact that the technical product specification only described the radar’s separation capability at a few angles. This made it difficult for us to estimate the exact layout of range and azimuth bins. 

In addition, our parameterization and implementation for the CFAR thresholding algorithm is an estimate, as it is a supplier’s intellectual property. The separation capability of the radar is expected to be quite sensitive to the CFAR behavior.

Overall, across both the near and far scans, we found the simulated separation capability to be close enough to the real sensor.

Scenario 3: Driving toward a corner reflector with constant speed

Doppler measurement enables radars to accurately detect the speed of moving targets. We evaluated the model’s performance in dynamic conditions, where the test vehicle drove directly toward high and low RCS corner reflectors, taking separate trips at constant speeds of 10kph, 40kph, and 80kph, as shown below.

The histograms in Figure 11 present the Doppler error results for both the high and low RCS corner reflectors.

We observed a remarkably high correlation in Doppler across all tested speeds. For 10kph, both real and simulated distributions exhibited similar peaks at approximately -3mps, -2mps, and 0mps. For 40kph, the peaks aligned around -10mps. For 80kph, peaks were observed at -20mps and 10mps. This high degree of accuracy was further demonstrated when plotting Doppler against range.

Figure 13 illustrates a top-down view of real and simulated radar detections across high and low RCS corner reflector positions and vehicle speeds.

Both the real and simulated Doppler measurements demonstrated substantial agreement in their mean and standard deviation values. However, we noticed deviations at higher speeds. 

We attribute these deviations to uncertainties during the creation of the digital twin. The position, speed, and orientation of the ego vehicle were estimated using lidar without the aid of differential GPS. The errors in these estimations are amplified at higher speeds as can be seen in 80kph.

In addition, we observed that DRIVE Sim is able to replicate the Radar Aliasing phenomenon, which occurs when an object’s radial velocity surpasses the radar’s maximum measurable unambiguous velocity, resulting in ambiguous velocity values. Real-world radars subtly shift the maximum measurable unambiguous velocity range with each cycle, enabling subsequent perception algorithms to disambiguate the velocities. 

Our simulation accurately replicated this behavior, as demonstrated by the alignment of the peaks in both the real and simulated data. Particularly, at a speed of 80kph, both the real and simulated radar exhibited similar velocity wrapping.

Conclusion

This study presents our first iteration for an in-depth validation of our simulated radar model using real-world data, including static and dynamic conditions. The analysis was designed to assess the model’s fidelity and accuracy across a variety of performance metrics.

Our results demonstrate a high degree of correlation between the simulated and real-world radar data with the model adeptly handling complex interactions such as multibounce effects. 

Upcoming experiments will focus on capturing radar data from more complex objects (vehicles, pedestrians, motorbikes) mimicking real-world scenarios. These objects not only have more complicated geometries, but are also composed of a variety of materials, thus introducing further complexities in radar wave interactions. Through these efforts, we aim to continually enhance the model’s fidelity, further bridging the gap between simulation and reality.

By validating accurate radar sensor behavior in simulated scenarios, we can improve system development efficiency, reduce dependence on costly and time-consuming real-world data collection, and enhance the safety and performance of AV systems. 

To learn more, see our previously published posts:

• Validating NVIDIA DRIVE Sim Camera Models
• Validating NVIDIA DRIVE Sim Lidar Models

NVIDIA Modulus 23.09 is now available, providing ease-of-use updates, fixes, and other enhancements.

Explore generative AI concepts and applications, along with challenges and opportunities in this self-paced course.

On Sept. 19, learn how NVIDIA TAO integrates with the ClearML platform to deploy and maintain machine learning models in production environments.

Organizations are integrating machine learning (ML) throughout their systems and products at an unprecedented rate. They are looking for solutions to help deal with the complexities of deploying models at production scale. 

NVIDIA Triton Management Service (TMS), exclusively available with NVIDIA AI Enterprise, is a new product that helps do just that. Specifically, it helps manage and orchestrate a fleet of NVIDIA Triton Inference Servers in a Kubernetes cluster. TMS enables users to scale their NVIDIA Triton deployments to handle large and varied workloads efficiently. It also improves the developer experience of coordinating the resources and tools required. 

This post explores some of the most common challenges developers and MLOps teams face when deploying models at scale, and how NVIDIA Triton Management Service addresses them. 

Challenges in scaling AI model deployment

Model deployments of any scale come with their own sets of challenges. Developers need to consider how to balance a variety of frameworks, model types, and hardware while maximizing performance and interfacing with the other components of the environment. 

NVIDIA Triton is a powerful solution built to handle these issues and extract the best throughput and performance from the machine it’s deployed on. But as organizations incorporate AI into more of their core workflows, the number and size of inference workloads can grow beyond what a single server can handle. The model deployments have to scale. A new scale of deployment brings with it a new set of challenges—challenges related to the cost and complexity of managing distributed inference workloads. 

Cost of deployment

As you deploy more models and find more use cases for them, it can quickly become necessary to scale out deployments to make use of a cluster of resources. A simple approach is to keep scaling your cluster linearly as you add more models, keeping all of your models live and ready for inference at all times. 

However, this is not an approach with infinite scale potential. Focusing on expanding the capacity of your serving cluster can result in unnecessary expenses when you have the option to improve utilization of currently available hardware. You will also have to deal with the logistical challenges of adding more resources on premises, or bumping up against quota limits in the cloud. 

Other approaches to scaling might appear less expensive, but can lead to steep performance trade-offs. For example, you could wait to load the models into memory until the inference requests come in, leading to long waits and an extended time-to-first-inference. Or you could overcommit your compute resources, leading to performance penalties from context switching during execution and errors from running out of memory on the device.

With careful preplanning and colocation of workloads, you can avoid some of the worst of these issues. Still, that only exacerbates the second major issue of large-scale deployments.

Operational complexity

At a small scale and early in the development of a process that requires model orchestration, it can be viable to manually configure and deploy your models. But as your ML deployments scale, it becomes increasingly challenging to coordinate all of the necessary resources. You need to manage when to launch or scale servers, where to load particular models, how to route requests to the right place, and how to handle the model lifecycle in your environment. 

Determining which models can be colocated adds another layer of complexity to these deployments. Large models might exceed the memory capacity of your GPU or CPU if loaded concurrently into the same device. Some frameworks (such as PyTorch and TensorFlow) hold on to any memory allocated to them even after the models are unloaded, leading to inefficient utilization when models from those frameworks are run alongside models from other frameworks. 

In general, different models will have different requirements regarding resource allocation and server configuration, making it difficult to standardize on a single type of deployment. 

Cost-efficient deployment and scaling of AI models

Triton Management Service addresses these challenges with three main strategies: simplifying Triton Inference Server deployment, maximizing resource usage, and monitoring/scaling Triton inference servers.

Simplifying deployment

TMS automates the deployment and management of Triton server instances on Kubernetes using a simplified gRPC API and command-line tool. With these interfaces, you don’t need to write out extensive code or config files for creating deployments, services, and Kubernetes resources. Instead, you can use the API or CLI to easily launch Triton servers and automatically load models onto these servers as needed. 

TMS also employs a method of grouping to optimize GPU or CPU memory utilization. This prevents issues that arise when different frameworks like PyTorch and TensorFlow models run on the same server and fail to release unused GPU or CPU? memory to each other.

Maximizing resources

TMS loads models on-demand and unloads them using a lease system when they are not in use, making sure that models are not kept active in the cluster unnecessarily. To bring up a model, you can submit an API request with a specified timeline or a checking mechanism. The system will keep the model available if it’s being used; otherwise, it will be taken down. 

TMS also automatically colocates models on the same device when sufficient capacity is available. To enable this, you need to prespecify the expected GPU memory use of your models during deployment. While there is no automated way to measure this yet, you can rely on Triton Model Analyzer and other benchmarking tools to determine ‌memory requirements beforehand. Together, these features enable you to run more workloads on your existing clusters, saving on costs, and reducing the need to acquire more computational resources. 

Monitoring and autoscaling

TMS keeps track of the health and capacity of various Triton servers for high availability reasons. Autoscaling is integrated into the system, enabling TMS to deploy Kubernetes Horizontal Pod Autoscalers automatically based on the model deployment configuration. You can specify metrics for autoscaling, indicating the conditions under which scaling should occur. Load balancing is also applied when autoscaling is implemented across multiple Triton instances.

How Triton Management Service works

To install TMS, deploy a Helm chart with configurable values into a Kubernetes cluster. This Helm chart delpoys the TMS Server control plane into the cluster, along with a config map that holds many of the configuration settings for TMS. You can operate TMS through gRPC API calls to the TMS Server, or by using the provided tmsctl command-line tool. 

The key concept in TMS is the lease. At its core, a lease is a grouping of models and some associated metadata that tells TMS how to treat those models, and what constraints exist for their deployment. Users can create, renew, and release leases. Creating a lease requires specifying a set of models from predefined repositories by a unique identifier, along with metadata including:

• Compute resources required by the lease
• Image/version of Triton to use for this lease
• Minimum duration of the lease
• Window size for detecting activity on the models in the lease
• Metrics and thresholds for scaling the lease
• Constraints on what models or leases with with the new lease can be collected
• A unique name for the lease that can be used to addressed it

When the TMS Server receives the lease request, it performs the actions listed below to create the lease:

• Check the model repositories to see if the models are present and accessible. 
• If models are present and accessible, check for existing Triton Inference Servers present in the cluster that meet the constraints of the new lease.
  • If none exist, create a new Kubernetes pod containing the Triton Inference Server container and a Triton Sidecar Container.
  • Otherwise, choose one of the existing Triton pods to add the lease to. 
• In either case, the Triton Sidecar in the Triton Pod will pull the models in your lease from the repository and load them into its paired Triton server. 

TMS will also create several other Kubernetes resources to help with management and routing for the lease:

• A deployment that will revive Triton pods if they crash.
• A Kubernetes service based on the lease name that can be used to address the models in the lease.
• A horizontal pod autoscaler to automatically create replicas of the Triton pods based on the metrics and thresholds defined in the lease.

Once the lease has been created, you can use the Triton Inference Server API or an existing Triton client to send inference requests to the server for execution. The Triton client does not need any modifications to work with Triton Inference Servers deployed by Triton Management Service. 

Get started with NVIDIA Triton Management Service

To get started with NVIDIA Triton Management Service and learn more about its features and functionality, check out the AI Model Orchestration with Triton Management Service lab on LaunchPad. This lab provides free access to a GPU-enabled Kubernetes cluster and a step-by-step guide on installing Triton Management Service and using it to deploy a variety of AI workloads. 

If you have existing compatible on-premises systems or cloud instances, request a 90-day NVIDIA AI Enterprise Evaluation License to try Triton Management Service. If you are an existing NVIDIA AI Enterprise user, simply log in to the NGC Enterprise Catalog.

Learn how to build and deploy production-quality conversational AI apps with real-time transcription and NLP.

Custom schemas in Universal Scene Description, known as OpenUSD or USD, are pivotal for developers seeking to represent and encode sophisticated virtual worlds. By formalizing data models, schemas enable the interpretation of raw data by USD-compliant runtimes.

Whether underpinning physics simulations or expanding digital twins, custom schemas provide the foundation for creativity, fidelity, and innovation in virtual environments.

In the third installment of this OpenUSD series, I share what developers must know about custom schemas. 

Specifically, we dive into:

• Data formalization: Custom schemas formalize data models, such as geometric meshes. These schemas populate the USD schema registry with canonical definitions.
• Data hierarchies: USD scenes, or stages, consist of prims—hierarchies of primitives—with each prim serving as a data container. Schemas formalize prim data, making them queryable and interpretable by USD-compliant runtimes.
• Data modeling compared to runtime behaviors: Schemas define data structure but not runtime functionality, highlighting the separation between the two aspects in USD development.
• Schema types: USD supports various schema types, including IsA (typed) schemas like UsdGeomMesh, and API schemas such as UsdPhysicsRigidBodyAPI to additionally annotate already-typed prims.
• Standardization process: Standardizing new schemas involves prototyping, formalization, internal reviews, whitepaper publication, and broad industry review, leading to adoption as a USD standard.

Custom schemas in USD open up numerous pathways for crafting more complex virtual worlds. OpenUSD includes core schemas like geometry and shading, with continuous development of custom schemas to broaden the digital landscape.

For the latest USD resources and tutorials, visit our OpenUSD resources page. Try the free RunUSD Validation Service to validate the compatibility of your OpenUSD assets and applications against a range of OpenUSD versions and configurations. 

If you’re a developer, start building OpenUSD-based apps and tools on the Omniverse platform. Stay up to date on the platform by subscribing to the newsletter, and following NVIDIA Omniverse on Instagram, Medium, and Twitter. Check out our forums, Discord server, Twitch, and YouTube channels.

AI is rapidly changing industrial visual inspection. In a factory setting, visual inspection is used for many issues, including detecting defects and missing or incorrect parts during assembly. Computer vision can help identify problems with products early on, reducing the chances of them being delivered to customers. 

However, developing accurate and versatile object detection models remains challenging for edge AI developers. Robust object detection models require access to comprehensive and representative datasets. In many manufacturing scenarios, real-world datasets fall short when capturing the complexity and diversity of actual scenarios. The constraints of narrow environments and limited variations pose challenges in training models to adapt to a range of situations effectively.

Teams can harness synthetic data for training models on diverse, randomized data that closely resemble real-world scenarios and address dataset gaps. The result is more accurate and adaptable AI models that can be deployed for a wide range of edge AI applications in industrial automation, healthcare, and manufacturing, to name a few.

From synthetic data generation to AI training 

Edge Impulse is an integrated development platform that empowers developers to create and deploy AI models for edge devices. It supports data collection, preprocessing, model training, and deployment, helping users integrate AI capabilities into their applications effectively.

With NVIDIA Omniverse Replicator, a core extension of NVIDIA Omniverse, users can produce physically accurate and photorealistic, synthetically generated annotated images in Universal Scene Description, known as OpenUSD. These images can then be used for training an object detection model on the Edge Impulse platform. 

NVIDIA Omniverse is a computing platform that enables individuals and teams to develop Universal Scene Description (OpenUSD)-based 3D workflows and applications.

OpenUSD is a highly versatile and interoperable 3D interchangeable and format that excels in synthetic data generation due to its scalability, performance, versioning, and asset management capabilities, making it an ideal choice for creating complex and realistic datasets. There’s a vast ecosystem of 3D content tools that connect to OpenUSD and USD-based SimReady assets that make it easy to integrate physically-based objects into scenes and accelerate our synthetic data generation workflows.

Omniverse Replicator enables randomization for USD data across several domains to represent scenarios that reflect real-world possibilities object detection models may encounter.

Using the synthetically generated images in USD to train models in Edge Impulse only takes a few clicks with the new Edge Impulse Omniverse extension.

Using the extension, which was developed using the Omniverse Kit Python extension template, users can connect to the Edge Impulse API and select the dataset to upload their synthetic data. The Kit Python extension template is a simple and self-explanatory resource for code snippet options and developing an extension quickly.

Generating synthetic data for an object detection model

To understand the workflow for generating synthetic data with Omniverse Replicator and using it to train a model in Edge Impulse, follow our example detecting soda cans model.

The first step in the process is building a virtual replica or a digital twin of the environment that represents the real scenario. The scene for generating synthetic images ‌consists of movable and immovable objects. The immovable set includes lights, a conveyor belt, and two cameras, while the movable objects consist of soda cans. Employing domain randomization, you can alter many properties, including location, lighting, colors, texture, background, and foreground of select immovable and movable objects. 

These assets are represented in Omniverse Replicator through OpenUSD. 3D model files can be converted into USD and imported into Omniverse Replicator using the Omniverse CAD Importer extension.

Lighting plays a pivotal role in realistic image generation. Rectangular lights can emulate light generated from a panel and a dome light brightens the entire scene. You can randomize various parameters for the lights like temperature, intensity, scale, position, and rotation of the lights. 

The following script shows temperature and intensity randomized through sampling from normal distributions, with scales randomized by a uniform distribution. The position and rotation of lights are fixed to remain constant.

python

# Lightning setup for Rectangular light and Dome light 

def rect_lights(num=1):

    lights = rep.create.light(

        light_type="rect",

        temperature=rep.distribution.normal(6500, 500),

        intensity=rep.distribution.normal(0, 5000),

        position=(45, 110, 0),

        rotation=(-90, 0, 0),

        scale=rep.distribution.uniform(50, 100),

        count=num

    )

    return lights.node

rep.randomizer.register(rect_lights)

def dome_lights(num=3):

    lights = rep.create.light(

        light_type="dome",

        temperature=rep.distribution.normal(6500, 500),

        intensity=rep.distribution.normal(0, 1000),

        position=(45, 120, 18),

        rotation=(225, 0, 0),

        count=num

    )

    return lights.node

rep.randomizer.register(dome_lights)

Most scenes have immovable objects important to the environment, like a table or in this case, a conveyor belt. The position of these objects can be fixed, while the material of the objects can be randomized to reflect real-world possibilities.

The following script generates a conveyor belt in USD that the cans will be placed upon. It also fixes its position and rotation. In this example, we don’t randomize the material of the conveyor belt.

python 

# Import and position the conveyor belt

conveyor = rep.create.from_usd(CONVEYOR_USD, semantics=[('class', 'conveyor')])

with conveyor:

    rep.modify.pose(

        position=(0, 0, 0),

        rotation=(0, -90, -90),

    )

To guarantee a high-quality dataset, it’s a good idea to use multiple cameras with different resolutions, and position them strategically in the scene. The position of the cameras can also be randomized. This script sets up two cameras of different resolutions strategically placed at various locations in the scene.

# Multiple setup cameras and attach to render products

camera = rep.create.camera(focus_distance=focus_distance, focal_length=focal_length,

                            position=cam_position, rotation=cam_rotation, f_stop=f_stop)

camera2 = rep.create.camera(focus_distance=focus_distance2, focal_length=focal_length2,

                            position=cam_position2, rotation=cam_rotation, f_stop=f_stop)

# Render images

render_product = rep.create.render_product(camera, (1024, 1024))

render_product2 = rep.create.render_product(camera2, (1024, 1024))

The last step is randomizing the position of the movable objects while also keeping them in the relevant area. In this script, we initialize five instances of 3D-can models, randomly selected from a collection of available can assets.

cans = list()

for i in range(TOTAL_CANS):

    random_can = random.choice(cans_list)

    random_can_name = random_can.split(".")[0].split("/")[-1]

    this_can = rep.create.from_usd(random_can, semantics=[('class', 'can')]) 

    with this_can:

        rep.modify.pose(

            position=(0, 0, 0),

            rotation=(0, -90, -90)

        )

    cans.append(this_can)

Then, the pose of the cans is randomized and scattered across two planes, keeping the cans on the conveyor belt while avoiding collisions.

with rep.trigger.on_frame(num_frames=50, rt_subframes=55):

    planesList=[('class','plane1'),('class','plane2')]

    with rep.create.group(cans):

        planes=rep.get.prims(semantics=planesList)

        rep.modify.pose(

            rotation=rep.distribution.uniform(

                (-90, -180, 0), (-90, 180, 0)

            )

        )

        rep.randomizer.scatter_2d(planes, check_for_collisions=True)

Annotating data, building the model, and testing with real objects

After being generated, the images can be uploaded to Edge Impulse Studio in a few clicks with the Edge Impulse Omniverse extension. In Edge Impulse Studio, datasets can be annotated and trained using models, such as the Yolov5 object detection model. The version control system enables model performance tracking across different dataset versions and hyperparameters, to optimize precision.

To test model accuracy with real-world objects, you can stream live video and run the model locally using the Edge Impulse CLI tool. 

If the model does not detect the objects accurately, the model must be trained on additional datasets. This iterative process is the norm when it comes to AI model training. An added benefit of synthetic data is that required variations in subsequent iterations can be done programmatically. 

In this example, an additional synthetic dataset was generated and used to train the model to improve performance. The additional dataset used a camera distance further from the conveyor. Other parameters like the angle of the camera and materials can be modified in additional datasets to improve performance. 

Taking a data-centric approach, where you create more data around the failure points of the model, is crucial to solving ML problems. Additional training and fine-tuning of parameters can enable a model to generalize well across different orientations, materials, and other relevant conditions.

Get started training and deploying edge AI with synthetic data

Generating physically accurate synthetic data is easy in Omniverse Replicator. Simply download Omniverse free and follow the instructions for getting started with Replicator in Omniverse Code.

With Edge Impulse, you can use synthetic data generated in Omniverse to train your ML models. Sign up and begin using embedded machine learning models today.

Join Amit Goel, director of product management at NVIDIA, at the Imagine 2023 keynote. Learn about industry insights on AI and machine learning, as well as use cases made possible by NVIDIA Omniverse and Omniverse Replicator.   

Stay up to date with NVIDIA Omniverse by subscribing to the newsletter, and following us on Instagram, Medium, and Twitter. For more resources, check out our forums, Discord server, Twitch, and YouTube channels.

Spear phishing is the largest and most costly form of cyber threat, with an estimated 300,000 reported victims in 2021 representing $44 million in reported losses in the United States alone. Business e-mail compromises led to $2.4 billion in costs in 2021, according to the FBI Internet Crime Report. In the period from June 2016 to December 2021, costs related to phishing and spear phishing totaled $43 billion for businesses, according to IBM Security Cost of a Data Breach. 

Spear phishing e-mails are indistinguishable from a benign e-mail that a victim would receive. This is also why traditional classification of spear phishing e-mails is so difficult. The content difference between a scam and a legitimate e-mail can be minuscule. Often, the only difference between the two is the intent of the sender: is the invoice legitimate, or is it a scam? 

This post details a two-fold approach to improve spear phishing detection by boosting the signals of intent using NVIDIA Morpheus to run data processing and inferencing.

Generating e-mails with new phishing intent

The first step involves using generative AI to create large, varied corpora of e-mails with various intents associated with spear phishing and scams. As new threats emerge, the NVIDIA Morpheus team uses the NVIDIA NeMo framework to generate a new corpus of e-mails with such threats. Following the generation of new e-mails with the new type of phishing intent, the team trains a new language model to recognize the intent. In traditional phishing detection mechanisms, such models would require a significant number of human-labeled e-mails.

Detecting sender intent

The first step targets the intent behind the e-mail. The next step targets the intent of the sender. To defend against spear phishing attacks that use spoofing, known senders, or longer cons that do not express their true intent immediately, we construct additional signals by building up behavioral sketches from senders or groups of senders. 

Building on the intent work described above, known senders’ past observed intents are recorded. For example, the first time a known sender asks for money can be a signal to alert the user. 

Syntax usage is also observed and recorded. The syntax of new e-mails is compared to the syntax history of the sender. A deviation from the observed syntax could indicate a possible spoofing attack. 

Finally, the temporal patterns of a sender’s e-mails are collected and cross-referenced when a new e-mail arrives to check for out-of-pattern behavior. Is the sender sending an e-mail for the first time at midnight on a Saturday? If so, that becomes a signal in the final prediction. These signals in aggregate are used to classify e-mails. They are also presented to the end user as an explanation for why an e-mail may be malicious.

Adapting to new attacks and improving protection

Existing machine learning (ML) methods rely nearly entirely on human-labeled data and cannot adapt to emerging threats quickly. The biggest benefit to detecting spear phishing e-mails using the approach presented here is how quickly the model can be adapted to new attacks. When a new attack emerges, generative AI is leveraged to create a training corpus for the attack. Intent models are trained to detect its presence in received e-mails. 

Using models built with NeMo generates thousands of high-quality, on-topic e-mails in just a few hours. The new intents are added to the existing spear phishing detector. The entire end-to-end workflow of creating new phishing attack e-mails and updating the existing models happens in less than 24 hours. Once the models are in place, ‌e-mail processing and inferencing become a Morpheus pipeline to provide near real-time protection against spear phishing threats.

Results

To illustrate the flexibility of this approach, a model was trained using only money, banking, and personal identifying information (PII) intents. Next, cryptocurrency-flavored phishing e-mails were generated using models built with NeMo. These e-mails were incorporated into the original training and validation subsets. 

The validation set, now containing the new crypto attacks, was then passed into the original model. Then a second model was trained incorporating the crypto attack intents. Figure 2 shows how the models compare in their detection. 

After training for the attack, the F1 score increased from 0.54 to 0.89 (Figure 3). This illustrates how quickly new attacks can be trained for and adapted to using NVIDIA Morpheus and NeMo.

Get started with NVIDIA Morpheus

Watch the video, Improve Spear Phishing Detection with Generative AI for more details. Learn more about how to use NVIDIA Morpheus to detect spear phishing e-mails faster and with greater accuracy using the NVIDIA AI workflow example. You can also apply to try NVIDIA Morpheus in LaunchPad and request a 90-day free trial to test drive NVIDIA Morpheus, part of the NVIDIA AI Enterprise software family.

The broadcast industry is undergoing a transformation in how content is created, managed, distributed, and consumed. This transformation includes a shift from traditional linear workflows bound by fixed-function devices to flexible and hybrid, software-defined systems that enable the future of live streaming.  

Developers can now apply to join the early access program for NVIDIA Holoscan for Media, a software-defined platform for developing and deploying media applications on-prem, in the cloud, and at the edge.

Using Holoscan for Media, broadcasters and solution providers can leverage the latest IT and provisioning technologies and a modern container-based approach to development, orchestration, and delivery.  

Holoscan for Media is an IP-based solution built on industry standards and APIs including SMPTE ST 2110, AMWA NMOS, RIST, SRT, and NDI.  

The platform integrates open-source and ubiquitous technologies, breaking from the proprietary and inflexible nature of SDI and FPGA-based systems. It also enables incorporation of the latest capabilities in production—such as generative AI—without additional infrastructure investments. With Holoscan for Media, countless NVIDIA application frameworks and SDKs are made accessible ​​to the industry for development.  

This framework provides several benefits to both broadcasters and solution providers, including: 

• Repurposability: Use a single platform for many applications. 
• Lower TCO: Benefit from the cyclical cost reductions. 
• Flexibility: The platform is cloud-native and independent of location. An application can be developed once and deployed everywhere. 
• Sustainability: Provisioning technologies that drive resource sharing means that overall less equipment is required. This means lower power and cooling costs and reduced impact from shipping to and from events. Ultimately, this leads to CO2 reductions.  

IP-based platform architecture 

NVIDIA Holoscan targets sensor data and media processing applications deployed at-scale across countless industries, in the cloud, on premises, and at the edge. Holoscan for Media tightens the focus on broadcast and live production workflows, with the first target being on-premises deployments. 

The hardware basis of the platform is therefore NVIDIA-certified systems from our partners, using NVIDIA Ampere architecture or later GPUs and NVIDIA BlueField-2 or later DPUs. The first systems are x86, but the entire software stack is multi-architecture to enable a wide range of systems and use cases with lower power consumption. In production, a minimal Holoscan for Media cluster consists of three nodes, and scales from there. 

The software stack begins with Kubernetes, the open-source container orchestration system for automating software deployment, scaling, and management. Partnering with the Red Hat OpenShift Container Platform brings enterprise-grade operation and support. 

The inclusion of Kubernetes plug-ins, known as operators, which provide and manage the hardware and underlay services, frees software developers to focus on their unique functionality. The open-source OpenShift Node Tuning Operator, NVIDIA GPU Operator, and NVIDIA Network Operator provide system, GPU, and high-speed secondary networking, tuned for performance and made available to every application that needs them. The GPU Operator can be used to assign one or more entire GPUs to an application. 

Support for MIG (Multi-Instance GPU) and vGPU (virtual GPU) enables GPUs to be securely shared between applications. The PTP Operator uses the PTP Hardware Clock on NVIDIA DPUs to provide precise timing from the secondary network to each application through a simple “get time” API. Other operators and plug-ins take care of IP address management (IPAM), DNS zone management, and more. 

Holoscan for Media also includes services such as an NMOS Registry and an easy-to-use graph-builder-based NMOS Controller user interface. These can be installed to support development and deployment of applications that act as media nodes and simplify integration with broadcast facility networks. 

Applications on the platform are packaged with Helm for simple, consistent deployment. A developer can indicate each container’s required capabilities and resources, including GPU, CPU, memory, and storage. This enables the platform to schedule and monitor applications to ensure each one is appropriately isolated, their requirements are met, and that best use is made of the available hardware. 

Developers can build applications using the growing list of NVIDIA SDKs supported on the Holoscan for Media platform. Traditional real-time video encoding and decoding with the Video Codec SDK, GPU-accelerated computer vision by CV-CUDA library, and any parallel compute algorithm using the  CUDA toolkit. On top of GPU-accelerated inference through TensorRT SDK or NVIDIA Triton Inference Server, new AI capabilities are offered by SDK and Cloud APIs like Maxine or NVIDIA Avatar Cloud Engine (ACE). Foundational SMPTE 2110 support and optimization of large media transfer is provided through NVIDIA Rivermax SDK. Developers can natively leverage Rivermax on the platform or through the DeepStream SDK, a complete streaming analytics toolkit based on GStreamer for AI-based media processing. Additionally, if developers have wider use cases beyond media, and want to consume and control other sensor types, NVIDIA provides the Holoscan SDK for creating real-time, AI-enabled sensor processing pipelines that meet latency requirements and scale from the data center to the edge.

Full source for a containerized reference application is available to Holoscan for Media developers. This uses NVIDIA DeepStream and can be configured as an NMOS-capable ST 2110 transmitter, receiver or transcoder gateway. 

Altogether, this open platform architecture provides the building blocks for the Dynamic Media Facility, using the latest scalable IT and provisioning technologies and open standards to benefit both broadcasters and software vendors. 

Get started with Holoscan for Media  

Holoscan for Media is now available for early access. Note that you must be registered in the NVIDIA Developer Program to apply for the early access release. You must also be logged in using your organization’s email address. We cannot accept applications from accounts using Gmail, Yahoo, QQ, or other personal email accounts.

To participate, fill out the short application form and provide details about your use case.

Streamline and accelerate deployment by integrating ETL and ML training into a single Apache Spark script on Amazon EMR.

Crossing the chasm and reaching its iPhone moment, generative AI must scale to fulfill exponentially increasing demands. Reliability and uptime are critical for building generative AI at the enterprise level, especially when AI is core to conducting business operations. NVIDIA is investing its expertise into building a solution for those enterprises ready to take the leap.

Introducing NVIDIA AI Enterprise 4.0

The latest version of NVIDIA AI Enterprise accelerates development through multiple facets with production-ready support, manageability, security, and reliability for enterprises innovating with generative AI.

Quickly train, customize, and deploy LLMs at scale with NVIDIA NeMo 

Generative AI models have billions of parameters and require an efficient data training pipeline. The complexity of training models, customization for domain-specific tasks, and deployment of models at scale require expertise and compute resources. 

NVIDIA AI Enterprise 4.0 now includes NVIDIA NeMo, an end-to-end, cloud-native framework for data curation at scale, accelerated training and customization of large language models (LLMs), and optimized inference on user-preferred platforms. From cloud to desktop workstations, NVIDIA NeMo provides easy-to-use recipes and optimized performance with accelerated infrastructure, greatly reducing time to solution and increasing ROI.  

Build generative AI applications faster with AI workflows

NVIDIA AI Enterprise 4.0 introduces two new AI workflows for building generative AI applications: AI chatbot with retrieval augmented generation and spear phishing detection. 

The generative AI knowledge base chatbot workflow, leveraging Retrieval Augmented Generation, accelerates the development and deployment of generative AI chatbots tuned on your data. These chatbots accurately answer domain-specific questions, retrieving information from a company’s knowledge base and generating real-time responses in natural language. It uses pretrained LLMs, NeMo, NVIDIA Triton Inference Server, along with third-party tools including Langchain and vector database, for training and deploying the knowledge base question-answering system.

The spear phishing detection AI workflow uses NVIDIA Morpheus and generative AI with NVIDIA NeMo to train a model that can detect up to 90% of spear phishing e-mails before they hit your inbox. 

Defending against spear-phishing e-mails is a challenge. Spear phishing e-mails are indistinguishable from benign e-mails, with the only difference between the scam and legitimate e-mail being the intent of the sender. This is why traditional mechanisms for detecting spear phishing fall short. 

Develop AI anywhere  

Enterprise adoption of AI can require additional skilled AI developers and data scientists. Organizations will need a flexible high-performance infrastructure consisting of optimized hardware and software to maximize productivity and accelerate AI development. Together with NVIDIA RTX 6000 Ada Generation GPUs for workstations, NVIDIA AI Enterprise 4.0 provides AI developers a single platform for developing AI applications and deploying them in production. 

Beyond the desktop, NVIDIA offers a complete infrastructure portfolio for AI workloads including NVIDIA H100, L40S, L4 GPUs, and accelerated networking with NVIDIA BlueField data processing units. With HPE Machine Learning Data Management, HPE Machine Learning Development Environment, Ubuntu KVM and Nutanix AHV virtualization support, organizations can use on-prem infrastructure to power AI workloads.  

Manage AI workloads and infrastructure

NVIDIA Triton Management Service, an exclusive addition to NVIDIA AI Enterprise 4.0, automates the deployment of multiple Triton Inference Servers in Kubernetes with GPU resource-efficient model orchestration. It simplifies deployment by loading models from multiple sources and allocating compute resources. Triton Management Service is available for lab experience on NVIDIA LaunchPad.

NVIDIA AI Enterprise 4.0 also includes cluster management software, NVIDIA Base Command Manager Essentials, for streamlining cluster provisioning, workload management, infrastructure monitoring, and usage reporting. It facilitates the deployment of AI workload management with dynamic scaling and policy-based resource allocation, providing cluster integrity.

New AI software, tools, and pretrained foundation models

NVIDIA AI Enterprise 4.0 brings more frameworks and tools to advance AI development. NVIDIA Modulus is a framework for building, training, and fine-tuning physics-machine learning models with a simple Python interface. 

Using Modulus, users can bolster engineering simulations with AI and build models for enterprise-scale digital twin applications across multiple physics domains, from CFD and Structural to Electromagnetics. The ​​Deep Graph Library container is designed to implement and train Graph Neural Networks that can help scientists research the graph structure of molecules or financial services to detect fraud. 

Lastly, three exclusive pretrained foundation models, part of NVIDIA TAO, speed time to production for industry applications such as vision AI, defect detection, and retail loss prevention. 

NVIDIA AI Enterprise 4.0 is the most comprehensive upgrade to the platform to date. With enterprise-grade security, stability, manageability, and support, enterprises can expect reliable AI uptime and uninterrupted AI excellence.

Get started with NVIDIA AI Enterprise

Three ways to get accelerated with NVIDIA AI Enterprise:

• Sign up for NVIDIA LaunchPad for ‌short-term access to sets of hands-on labs.
• Sign up for a free 90-day evaluation for existing on-prem or cloud infrastructure.

Purchase through NVIDIA Partner Network or major Cloud Service Providers including AWS, Microsoft Azure, and Google Cloud.

On Sept. 27, join us to learn recommender systems best practices for building, training, and deploying at any scale.

The camera module is the most integral part of an AI-based embedded system. With so many camera module choices on the market, the selection process may seem overwhelming. This post breaks down the process to help make the right selection for an embedded application, including the NVIDIA Jetson.

Camera selection considerations

Camera module selection involves consideration of three key aspects: sensor, interface (connector), and optics. 

Sensor 

The two main types of electronic image sensors are the charge-coupled device (CCD) and the active-pixel sensor (CMOS). For a CCD sensor, pixel values can only be read on a per-row basis. Each row of pixels is shifted, one by one, into a readout register. For a CMOS sensor, each pixel can be read individually and in parallel. 

CMOS is less expensive and consumes less energy without sacrificing image quality, in most cases. It can also achieve higher frame rates due to the parallel readout of pixel values. However, there are some specific scenarios in which CCD sensors still prevail—for example, when long exposure is necessary and very low-noise images are required, such as in astronomy. 

Electronic shutter 

There are two options for the electronic shutter: global or rolling. A global shutter exposes each pixel to incoming light at the same time. A rolling shutter exposes the pixel rows in a certain order (top to bottom, for example) and can cause distortion (Figure 1).

The global shutter is not impacted by motion blur and distortion due to object movement. It is much easier to sync multiple cameras with a global shutter because there is a single point in time when exposure starts. However, sensors with a global shutter are much more expensive than those with a rolling shutter. 

Color or monochrome 

In most cases, a monochrome image sensor is sufficient for typical machine vision tasks like fault detection, presence monitoring, and recording measurements.

With a monochrome sensor, each pixel is usually described by eight bits. With a color sensor, each pixel has eight bits for the red channel, eight bits for the green channel, and eight bits for the blue channel. The color sensor requires processing three times the amount of data, resulting in a higher processing time and, consequently, a slower frame rate.  

Dynamic range 

Dynamic range is the ratio between the maximum and minimum signal that is acquired by the sensor. At the upper limit, pixels appear white for higher values of intensity (saturation), while pixels appear black at the lower limit and below. An HDR of at least 80db is needed for indoor application and up to 140db is needed for outdoor application. 

Resolution 

Resolution is a sensor’s ability to reproduce object details. It can be influenced by factors such as the type of lighting used, the sensor pixel size, and the capabilities of the optics. The smaller the object detail, the higher the required resolution. 

Pixel resolution translates to how many millimeters each pixel is equal to on the image. The higher the resolution, the sharper your image will be. The camera or sensor’s resolution should enable coverage of a feature’s area of at least two pixels. 

CMOS sensors with high resolutions tend to have low frame rates. While a sensor may achieve the resolution you need, it will not capture the quality images you need without achieving enough frames per second. It is important to evaluate the speed of the sensor. 

A general rule of thumb to determine the resolution needed for the use case is shown below and in Figure 2.  The multiplier (2) represents the typical desire to have a minimum two pixels on an object in order to successfully detect it.

For example, suppose you have an image of an injury around the eye of a boxer. 

• 
• FOV, mm = 2000mm 
• Size of feature of interest (the eye), mm = 4mm

Based on the calculation, 1000 x 1000, a one-megapixel camera should be sufficient to detect the eye using a CV or AI algorithm. 

Note that a sensor is made up of multiple rows of pixels. These pixels are also called photosites. The number of photons collected by a pixel is directly proportional to the size of the pixel. Selecting a larger pixel may seem tempting but may not be the optimal choice in all the cases. 

Back-illuminated sensors maximize the amount of light being captured and converted by each photodiode. In front-illuminated sensors, metal wiring above the photodiodes blocks off some photons, hence reducing the amount of light captured.

Frame rate and shutter speed 

The frame rate refers to the number of frames (or images captured) per second (FPS). The frame rate should be determined based on the number of inspections required per second. This correlates with the shutter speed (or exposure time), which is the time that the camera sensor is exposed to capture the image. 

Theoretically, the maximum frame rate is equal to the inverse of the exposure time. But achievable FPS is lower because of latency introduced by frame readout, sensor resolution, and the data transfer rate of the interface including cabling. 

FPS can be increased by reducing the need for large exposure times by adding additional lighting, binning the pixels. 

CMOS sensors can achieve higher FPS, as the process of reading out each pixel can be done more quickly than with the charge transfer in a CCD sensor’s shift register. 

Interface

There are multiple ways to connect the camera module to an embedded system. Typically, for evaluation purposes, cameras with USB and Ethernet interfaces are used because custom driver development is not needed. 

Other important parameters for interface selection are transmission length, data rate, and operating conditions. Table 2 lists the most popular interfaces. Each option has its pros and cons. 

Optics 

The basic purpose of an optical lens is to collect the light scattered by an object and recreate an image of the object on a light-sensitive image sensor (CCD or CMOS). The following factors should be considered when selecting an optimized lens-focal length, sensor format, field of view, aperture, chief ray angle, resolving power, and distortion. 

Lenses are manufactured with a limited number of standard focal lengths. Common lens focal lengths include 6mm, 8mm, 12.5mm, 25mm, and 50mm. 

Once you choose a lens with a focal length closest to the focal length required by your imaging system, you need to adjust the working distance to get the object under inspection in focus. Lenses with short focal lengths (less than 12mm) produce images with a significant amount of distortion. 

If your application is sensitive to image distortion, try to increase the working distance and use a lens with a higher focal length. If you cannot change the working distance, you are somewhat limited in choosing an optimized lens. 

To attach a lens to a camera requires some type of mounting system. Both mechanical stability (a loose lens will deliver an out-of-focus image) and the distance to the sensor must be defined. 

To ensure compatibility between different lenses and cameras, the following standard lens mounts are defined. 

NVIDIA camera module partners 

NVIDIA maintains a rich ecosystem of partnerships with highly competent camera module makers all over the world. See Jetson Partner Supported Cameras for details. These partners can help you design imaging systems for your application from concept to production for the NVIDIA Jetson. 

Summary

This post has explained the most important camera characteristics to consider when selecting a camera for an embedded application. Although the selection process may seem daunting, the first step is to understand your key constraints based on design, performance, environment, and cost. 

Once you understand the constraints, then focus on the characteristics most relevant to your use case. For example, if the camera will be deployed away from the compute or in a rugged environment, consider using the GMSL interface. If the camera will be used in low-light conditions, consider a camera module with larger pixel and sensor sizes. If the camera will be used in a motion application, consider using a camera with a global shutter. 

To learn more, watch Optimize Your Edge Application: Unveiling the Right Combination of Jetson Processors and Cameras. For detailed specs on AI performance, GPU, CPU, and more for both Xavier and Orin-based Jetson modules, visit Jetson Modules. 

Explore how ray-traced caustics combined with NVIDIA RTX features can enhance the performance of your games.

Moment Factory is a global multimedia entertainment studio that combines specializations in video, lighting, architecture, sound, software, and interactivity to create immersive experiences for audiences around the world. 

From live performances and multimedia shows to interactive installations, Moment Factory is known for some of the most awe-inspiring and entertaining experiences that bring people together in the real world. These include dazzling visuals at Billie Eilish’s Happier Than Ever world tour, Lumina Night Walks at natural sites around the world, and digital placemaking at the AT&T Discovery District.

With a team of over 400 professionals and offices in Montreal, Tokyo, Paris, New York City, and Singapore, Moment Factory has become a global leader in the entertainment industry.

Streamlining immersive experience development with OpenUSD

Bringing these experiences to life requires large teams of highly skilled experts with diverse specialties, all using unique tools. To achieve optimal efficiency in their highly complex production processes, Moment Factory looked to implement an interoperable open data format and development platform that could seamlessly integrate all aspects, from concept to operation.

Moment Factory chose Universal Scene Description, also known as OpenUSD, as the solution. OpenUSD is an extensible framework and ecosystem for describing, composing, simulating, and collaborating within 3D worlds. NVIDIA Omniverse is a software platform that enables teams to develop OpenUSD-based 3D workflows and applications. It provides the unified environment to visualize and collaborate on digital twins in real time with live connections to Moment Factory’s tools.

Using OpenUSD with Omniverse enables Moment Factory to unify data from their diverse digital content creation (DCC) tools to form a digital twin of a real-world environment. Every member of the team can interact with this digital twin and iterate on their aspect of the project without affecting other elements

For example, a scenographer can work on a base set and unique scene pieces using Vectorworks, 3D design software. At the same time in the same scene, an AV (audio visual) and lighting designer can take care of lighting and projectors with Moment Factory’s proprietary live entertainment operating system and virtual projection mapping software, X-Agora.

Simultaneously, artists and designers can render and create eye-catching visuals in the scene using tools like Epic Games Unreal Engine, Blender, and Adobe Photoshop—without affecting layers of the project still in progress.

“USD is unique in that it can be fragmented into smaller pieces that enable people to work on their own unique parts of a project while staying connected,” said Arnaud Grosjean, solution architect and project lead for Moment Factory’s Innovation Team. “Its flexibility and interoperability allows us to create powerful, custom 3D pipelines.”

Digital twins simulate real-world experiences 

To simulate immersive events before deploying them in the real world, Moment Factory is developing digital twins of their installations in NVIDIA Omniverse. Omniverse, a computing platform that enables teams to develop OpenUSD-based 3D workflows and applications, provides the unified environment to visualize and collaborate on digital twins in real time with live connections to DCC tools.

The first digital twin they’ve created is that of Blackbox, which serves as an experimentation and prototyping space where they can preview fragments of immersive experiences before real-world deployment. It is a critical space for nearly every phase of the project lifecycle, from conception and design to integration and operation. 

To build the digital twin of the Blackbox, Moment Factory used USD Composer, a fully customizable foundation application built on NVIDIA Omniverse.

The virtual replica of the installation enables the team to run innumerable iterations on the project to test for various factors. They can also better sell concepts for immersive experiences to prospective customers, who can see the show before live production in a virtual environment.

One of the key challenges in the process for building large-scale immersive experiences is reaching a consensus among various stakeholders and managing changes. 

“Everyone has their own idea of how a scene should be structured, so we needed a way to align everyone contributing to the project in a unified, dynamic environment” explained Grosjean. “With the digital twin, potential ideas can be tested and simulated with stakeholders across every core expertise.”

As CAD drafters, AV designers, interactive designers, and others contribute to the digital twin of the Blackbox, artists and 2D/3D designers can render and experiment with beauty shots of the immersive experience in action.

To see the digital twin of the Blackbox in action, join the Omniverse Livestream with Moment Factory on Wednesday, September 13.

Developing Omniverse Connectors and extensions

Moment Factory is continuously building and testing extensions for Omniverse to bring new functionalities and possibilities into their digital twins.

They developed an Omniverse Connector for X-Agora, their proprietary multi-display software that allows you to design, plan and operate shows. The software now has a working implementation of a Nucleus connection, USD import/export, and an early live mode implementation.

Video projection is a key element of immersive events. The team will often experiment with mapping and projecting visual content onto architectural surfaces, scenic elements, and sometimes even moving objects, transforming static spaces into dynamic and captivating environments.

NDI, which stands for Network Design Interface, is a popular IP video protocol developed by NewTek that allows for efficient live video production and streaming across interconnected devices and systems. In their immersive experiences, Moment Factory typically connects a media system to physical projectors using video cables. With NDI, they can replicate this connection within a virtual venue, effectively simulating the entire experience digitally. 

To enable seamless connectivity between the Omniverse RTX Renderer and their creative content, Moment Factory developed an NDI extension for Omniverse. The extension supports more than just video projection and allows the team to simulate LED walls, screens, and pixel fields to mirror their real-world setup in the digital twin.

The extension, which was developed with Omniverse Kit, also enables users to use video feeds as dynamic textures. Developers at Moment Factory used the kit-cv-video-example and kit-dynamic texture-example to develop the extension.

Anyone can access and use Moment Factory’s Omniverse-NDI-extension on GitHub, and install it on the Omniverse Launcher or launch with:

$ ./link_app.bat --app create
$ ./app/omni.create.bat --/rtx/ecoMode/enabled=false --ext-folder exts --enable mf.ov.ndi

Extensions in Omniverse serve as reusable components or tools that developers can build to accelerate and add new functionalities for 3D workflows. They can be built for simple tasks like randomizing objects or used to enable more complex workflows like visual scripting.

The team also developed an extension for converting MPDCI, a VESA standard describing multiprojector rigs, to USD called the Omniverse-MPCDI-converter. They are currently testing extensions for MVR (My Virtual Rig) and GDTF (General Device Type Format) Converters to import lighting fixtures and rigs into their digital twins. 

Even more compelling is a lidar UDP simulator extension, which is being developed to enable sensor simulation in Omniverse and connect synthetic data to lidar-compatible software.

You can use Moment Factory’s NDI and MPDCI extensions today in your workflows. Stay tuned for new extensions coming soon.

To build extensions like Moment Factory, get started with all the Omniverse Developer Resources you’ll need, like  documentation, tutorials, USD resources, GitHub samples, and more.

Get started with NVIDIA Omniverse by downloading the standard license free, or learn how Omniverse Enterprise can connect your team. 

Developers can check out these Omniverse resources to begin building on the platform. 

Stay up to date on the platform by subscribing to the newsletter and following NVIDIA Omniverse on Instagram, LinkedIn, Medium, Threads, and Twitter.

For more, check out our forums, Discord server, Twitch, and YouTube channels.