The year 2022 has thus far been a momentous, thrilling, and an overwhelming year for AI aficionados. Get3D is pushing the boundaries of generative 3D modeling, an AI model can now diagnose breast cancer from MRIs as accurately as board-certified radiologists, and state-of-the-art speech AI models have widened their horizons to extended reality.

Pretrained models from NVIDIA have redefined performance this year, amused us on the stage of America’s Got Talent, won four global contests and a Best Inventions 2022 award from Time Magazine.

In addition to empowering researchers and data scientists, NVIDIA pretrained models are also empowering developers to create cutting-edge AI applications, by offering deep learning pretrained models and speedier convergence. To enable this, NVIDIA has spearheaded the research behind building and training these pretrained models for use cases like automatic speech recognition, pose estimation, object detection, 3D generation, semantic segmentation, and many more.

Model deployment can be streamlined, and users have already reaped the benefits over the last 3 months with 870 different NVIDIA pretrained models that support more than 50 use cases across several industries.

This post walks through a few of the top pretrained AI models that are behind groundbreaking AI applications.

Speech recognition for all

NVIDIA NeMo is serving a variety of industries with cutting-edge AI application development for speech AI and natural language processing. The use cases include the creation of virtual assistants in Arabic and the facilitation of state-of-the-art automatic speech recognition (ASR) for financial audio.

For language-specific ASR, the NVIDIA NeMo deep learning conformer transducer pretrained model and conformer-ctc (connectionist temporal classification) pretrained model are well-liked. These models have high accuracy, a low word error rate, and a low character error rate due to their pretraining on a range of datasets, such as Librispeech and Mozilla Common Voice Data. They also have a robust AI architecture.

These models are laying the groundwork for state-of-the-art Kinyarwanda ASR model, Kabyle, Catalan, and many low-resource language pretrained models, which are bringing the usage of enhanced speech AI to low-resource languages, regions, and sectors.

For more information, see NeMo automatic speech recognition models.

Verifying speakers for the greater good

To determine ‘who talked when,’ voice AI enthusiasts and application developers are fusing deep neural network speech recognition with speaker diarization architecture.

Beyond well-known uses like multi-speaker transcription in video conferencing, developers are gaining benefit from this AI architecture for special use cases:

• Clinical speech recordings and understanding medical conversations for effective healthcare
• Captioning and separating teacher-student speech in the education sector

Pretrained embeddings of the modified Emphasized Channel Attention, Propagation, and Aggregation in TDNN (ECAPA-TDNN) model are accessible with the NVIDIA NeMo toolkit. Fisher, Voxceleb, and real room-reaction data were used to train this deep neural network model for speaker identification and verification.

One of the best solutions for speaker diarization, ECAPA is based on the time-delay neural network (TDNN) and SE (squeeze and excite) structure with 22.3M parameters. It outperforms traditional TDNNs by emphasizing channel attention, propagation, and aggregation, as well as significantly reducing error rates.

For more information, see Speaker Diarization.

Visionary image control with SegFormer AI models

SegFormer is visionary research that uses AI to pioneer world-class image control. The original model and its variants are thriving in a variety of industries, including manufacturing, healthcare, automotive and retail. Its enormous potential is best demonstrated by applications like virtual changing rooms, robotic image control, medical imaging and diagnostics, and vision analytics in self-driving cars.

The semantic segmentation AI algorithm, a computer vision method for separating various objects in images, is the foundation of SegFormer. To increase performance to meet particular needs, the fine-tuned SegFormer is pretrained on datasets like ADE20k and CityScapes at several resolutions, such as 512×512, 640×640, 1024×1024, and so on. The AI design, which draws inspiration from the Transformer model architecture, produces cutting-edge outcomes in a variety of tasks.

For more information, see the NVlabs/SegFormer GitHub repo.

Purpose-built, pretrained model for automotive low-code developers

By detecting and identifying cars, people, road signs, and two-wheelers to comprehend traffic flow, TrafficCamNet has been driving smart city initiatives and detection technology for the automotive sector.

The model has been thoroughly trained using a vast amount of data that includes pictures of actual traffic crossings in US cities. The deep neural network model NVIDIA DetectNet_v2 detector is used with ResNet18 as a feature extractor.  The AI architecture, which is sometimes referred to as GridBox object detection, employs bounding-box regression on a regular grid in the input image. The NVIDIA TAO toolbox can be used to access and further fine-tune the purpose-built, pretrained model TrafficCamNet for best-in-class accuracy.

For more information, see Purpose-Built Models.

Award-winning models

NVIDIA pretrained models have won numerous awards for their cutting-edge performance, extraordinary research, and exemplary ability to solve real-world problems. Here are some notable wins.

World’s largest genomics language model wins Gordon Bell Special Award 2022

Researchers from Argonne National Labs, NVIDIA, the Technical University of Munich, the University of Chicago, CalTech, Harvard University, and others developed one of the world’s largest genomics language models that predicts new COVID variants. For their work, they won the 2022 Gordon Bell Special Award.

The model informs timely public health intervention strategies and downstream vaccine development for emerging viral variants. The research was published in October 2022 and presents GenSLMs (genome-scale language models), which can accurately and rapidly identify variants of concern in the SARS-CoV-2 virus.

The large genomics language models were pretrained on >110M gene sequences and then a SARS-CoV-2 specific model was fine-tuned on 1.5M genomes with 2.5B and 25B trainable parameters, respectively. This research enables programmers to further genetic language modeling by creating applications that can assist different public health initiatives.

For more information, see Speaking the Language of the Genome: Gordon Bell Winner Applies Large Language Models to Predict New COVID Variants.

State-of-the-art vision model wins Robust Vision Challenge 2022

The Fully Attential Network (FAN) Transformer model from NVIDIA Research won the Robust Vision Challenge 2022. The team adopted the SegFormer head on top of an ImageNet-22k pretrained FAN-B-Hybrid model, as described in the Understanding The Robustness in Vision Transformers paper. The model was then further fine-tuned on a composed, large-scale dataset, similar to MSeg.

NVIDIA Research developed all the models used. The model achieved a state-of-the-art 87.1% accuracy and 35.8% mCE on ImageNet-1k and ImageNet-C with 76.8M parameters. We also demonstrated state-of-the-art accuracy and robustness in two downstream tasks, semantic segmentation and object detection.

For more information, see the NVlabs/FAN GitHub repo.

Winning the Telugu automatic speech recognition competition

 NVIDIA recently won the Telugu-ASR challenge conducted by IIIT-Hyderabad, India. They trained a Conformer-RNNT (recurrent neural network transducer) model from scratch using 2K hours of Telugu-only data provided by organizers. Their efforts helped achieve the first position on the leaderboard for the closed track with WER 13.12%.

For an open competition track, they performed transfer-learning on a pretrained SSL Conformer-RNNT checkpoint trained on 36K hours from 40 Indic languages. With WER 12.64%, they won the competition. The fine-tuned winning model can be used by developers to create applications for automatic speech recognition that will benefit the 83M Telugu speakers globally.

NVIDIA pretrained models

NVIDIA pretrained models remove the need for constructing models from the start or experimenting with other open-source models that don’t converge, making high-performing AI development simple, rapid, and accessible.

For more information, see AI models.

The latest NVIDIA DRIVE OS release includes customization and safety updates for supercharging autonomous vehicle development.

The human hand is one of the most remarkable outcomes of millions of years of evolution. The ability to pick up all sorts of objects and use them as tools is a crucial differentiator enabling us to shape our world.

For robots to work in the everyday human world, the ability to deftly interact with our tools and the environment around them is critical. Without that capability, they will continue to be useful only in specialized domains such as factories or warehouses.

While it has been possible to teach robots with legs how to walk for some time, robots with hands have generally proven to be much more tricky to control. A hand with fingers has more joints that must move in specific coordinated ways to accomplish a given task. Traditional robotics control methods with precise grasps and motions are incapable of the kind of generalized fine motor control skills that humans take for granted.

One approach to these problems has been the application of deep reinforcement learning (deep RL) techniques that train a neural network to control the robot’s joints. With deep RL, a robot learns from trial and error and is rewarded for the successful completion of the assigned task. Unfortunately, this technique can require millions or even billions of samples to learn from, making it almost impossible to apply directly to real robots.

Applying simulation

Enter the NVIDIA Isaac robotics simulator, which enables robots to be trained inside a simulated universe that can run more than 10,000x faster than the real world and yet obeys the laws of physics.

Using NVIDIA Isaac Gym, an RL training robotics simulator, NVIDIA researchers on the DeXtreme project taught this robot hand how to manipulate a cube to match a provided target position and orientation or pose. The neural network brain learned to do this entirely in simulation before being transplanted to control a robot in the real world.

Similar work has only been shown one time before, by researchers at OpenAI. Their work required a far more sophisticated and expensive robot hand, a cube tricked out with precise motion control sensors, and a supercomputing cluster of hundreds of computers to train.

Democratizing dexterity

The hardware used by the DeXtreme project was chosen to be as simple and inexpensive as possible to enable researchers worldwide to replicate our experiments.

The robot itself is an Allegro Hand, which costs as little as 1/10th the cost of some alternatives, has four fingers instead of five, and has no moving wrist. We can use three off-the-shelf RGB cameras to track the 3D cube with vision, which can be repositioned easily as needed without requiring special hardware. The cube is 3D-printed with stickers affixed to each face.

DeXtreme is trained using Isaac Gym, which provides an end-to-end GPU-accelerated simulation environment for reinforcement learning. NVIDIA PhysX simulates the world on the GPU, and results stay in GPU memory during the training of the deep learning control policy network.

As a result, training can happen on a single Omniverse OVX server. Training a good policy takes about 32 hours on this system, equivalent to 42 years of a single robot’s experience in the real world.

Not needing a separate CPU cluster for simulation means a 10–200x reduction in computing costs for training at current cloud rental rates. Because we can use Isaac Gym to train the model, training time and cost can be dramatically reduced.

Perception and synthetic data

For the robot to know the current position and orientation of the cube that it’s holding, it needs a perception system. To keep costs low and leave open the potential for manipulation of other objects in the future, DeXtreme uses three off-the-shelf cameras and another neural network that can interpret the cube pose.

This network is trained using about 5 million frames of synthetic data generated using Omniverse Replicator and no real images whatsoever. The network learns how to perform the task under challenging circumstances in the real world. To make the training more robust, we use a technique called domain randomization to change lighting and camera positions, plus data augmentation to apply random crops, rotation, and backgrounds.

The DeXtreme pose estimation system is reliable and can perceive accurate poses even when the object in question is partly occluded from view, or when the image has significant motion blur.

Real robots are still challenging

One of the key reasons to use simulations is that training robots directly in the real world are riddled with various challenges. For example, robot hardware is prone to breaking after excessive usage. Experiment iteration cycles and turnaround time can also be slow.

During our experiments, we often found ourselves repairing the hand after prolonged usage, for example, tightening the loose screws, replacing the ribbon cables, and resting the hand to cool down after running 10-15 trials. Simulations enabled us to sidestep many of these issues by training on a robot that doesn’t wear out but also provides the large diversity of data needed to learn challenging tasks. At the same time, because simulations can run much faster than in real time, the iteration cycle is massively improved.

When training in simulation, the most significant challenge is bridging the gaps between the simulations and the real world. To address this, DeXtreme uses domain randomization of the physics properties set in the simulator: changing object masses, friction levels, and other attributes at scale across over a hundred thousand simulated environments at one time.

One interesting upshot of these randomizations is that we train the AI with all kinds of unusual combinations of scenarios, which translates to robustness when performing the task in the real world. For instance, most of our experiments on the real robot took place with a slightly malfunctioning thumb due to a loose connection on the circuit board. We were positively surprised that the policies transferred from simulation to the real world reliably, regardless.

Video 5. After over 32 hours of training, the DeXtreme robot was capable of repeated success at the task of rotating a cube to match a specific target

Sim-to-real

Future breakthroughs in robotic manipulation will enable a new wave of robotics applications beyond traditional industrial uses.

At the heart of the DeXtreme project is the message that simulation can be an incredibly effective tool for training complex robotic systems. This is true even for the systems that must handle environments with objects in continual contact with the robot. We hope that by demonstrating this using relatively low-cost hardware, we can inspire others to use our simulation tools and build on this work.

For more information, see DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Reality and visit DeXtreme.

For a further dive into simulators and how they can affect your projects, see How GPUs Can Democratize Deep Reinforcement Learning for Robotics Development. You can also download the latest version of NVIDIA Omniverse Isaac Sim and learn about training your own reinforcement learning policies.

X-ray-powered research is aiming to target sneaky hazardous materials making their way through airport security. The study, recently published in Scientific Reports, proposes a new design for a fast and powerful X-ray diffraction (XRD) technology able to identify potential threats. The work could be a notable step toward more accurate luggage scanning in airports. 

“The main goal of my project was to speed up this new X-ray imaging modality so it can be economically viable for airport security. Ultimately such a scanner could help find even the most creatively hidden explosives, drugs, and contraband, without excessive costs for the operator or delays for passengers,” said study author Airidas Korolkovas. He conducted part of the study while working as an X-ray physicist and imaging scientist at iTomography Corporation.

An awardee of the NVIDIA Academic Hardware Grant Program, Korolkovas was granted an NVIDIA TITAN V during his postdoctoral fellowship at Uppsala University in Sweden. Applicants must demonstrate how access to world-class computing resources could boost their research. 

In this case, Korolkovas designed and implemented a GPU-accelerated tomographic reconstruction algorithm for XRD to supplement existing computed tomography (CT) X-ray scans in airport security.  

Currently, airports rely on X-ray transmission alone to reveal luggage contents in 3D. X-ray beams can penetrate through, absorb, or scatter depending on the composition and spatial arrangement of atoms within each material. By measuring changes in the beam at various angles, sophisticated algorithms and computer vision technology can reconstruct 3D images of the bag contents. 

This gives airport security a peek into luggage without having to touch it. However, transmission-based CT scans have limitations. 

“Standard CT is sensitive to the average density and composition of materials. It is not sensitive to the internal arrangement of atoms, which makes all the difference when identifying a benign piece of plastic from a plastic explosive, or between sugar and cocaine,” Korolkovas said. 

According to the study, XRD could be a powerful new addition when scanning luggage, because it is sensitive to the internal arrangement of atoms.

This makes XRD especially well suited for identifying crystals as the repetitive arrangement of molecules in crystalline materials results in concentrated X-ray scattering along precise angles unique to each material. Access to this data could help airport security determine if objects in a bag contain threats like cocaine, crystal methamphetamine, or even explosives that are naturally crystalline, semi-crystalline, or crystalline powders.

Unfortunately, XRD scans of whole passenger luggage are quite slow, making them unusable in commercial aviation, which demands real-time results. 

To shorten the time, Korolkovas employed a novel scanner design aimed at high-intensity rather than high-resolution X-ray beams, which is traditionally a no-go as it degrades the XRD signal quality beyond recognition. 

Through a multistep approach, involving CT image segmentation and complex algebraic reconstructions, he was able to recover the XRD resolution, despite the limitations of beam intensity.

Korolkovas used the NVIDIA TITAN V to calculate the probabilities of all possible diffraction pathways.

“In this study, I was able to maintain acceptable resolution by combining data from transmission, diffraction, all the viewing angles, and the full spectrum of X-ray energies,” he said.

This can easily run into a quintillion mathematical operations for every slice of the bag.

According to Korolkovas, coding the reconstruction algorithm on a GPU was very helpful in keeping the computation time manageable. Changing from a CPU to a GPU, he was able to speed up the runtime from 10 hours to less than 1 hour. By further improving the algorithm and using multiple GPUs or cloud computing, he envisions eventually running the scan in real time. 

“X-rays can penetrate and scatter within the bag in every possible direction. CUDA texture mapping has turned out to be an efficient way to access the photon survival probabilities along any such pathway. The calculations of various pathways are partially independent of each other, and benefit from parallel computing afforded by CUDA,” he said.

Testing the approach on a simulated bag containing both benign and threat materials, he found that the XRD reconstruction adds material-specific information, improving what CT alone captures and threat detection. 

“An XRD imaging add-on to existing CT scanners is feasible and is well positioned to provide unique, material-specific information, at a low cost of installing an extra detector or two and developing suitable reconstruction software,” Korolkovas writes in the study. 

The next steps in the research include building an experimental prototype and testing the algorithm on real-world data. 

“The study has received encouraging feedback from Rapiscan Systems, a major manufacturer of X-ray scanners. Now that air travel is returning to pre-pandemic levels, there is renewed interest in advanced X-ray imaging and I hope to contribute to this endeavor,” Korolkovas said. 

He also plans on using machine learning to train neural networks that fingerprint the reconstructed diffraction patterns against a broad range of materials found in suitcases. This will improve the robustness of flagging threat materials, even with limited data that can be acquired in real time.

Read the study, Fast X-ray diffraction (XRD) tomography for enhanced identification of materials.
Access research code on the XRD_Tomography GitHub page.
Learn more about NVIDIA Higher Education and Research Developer Resources.

Funding for this research includes a grant from the U.S. Department of Homeland Security, Science, and Technology Directorate and a Titan V donated by NVIDIA.

The NVIDIA Optical Flow SDK 4.0 is now available, enabling you to fully harness the new NVIDIA Optical Flow Accelerator on the NVIDIA Ada architecture with NvOFFRUC.

Optical flow on the NVIDIA Ada Lovelace architecture

Starting from the NVIDIA Turing architecture, NVIDIA GPUs have dedicated hardware for optical flow computation between a pair of frames. NVIDIA has continued to invest in improving the optical flow hardware engine in the NVIDIA Ampere architecture and NVIDIA Ada Lovelace architecture generations, thanks to the continued feedback from application developers and researchers.

Significant performance improvements

The Optical Flow algorithm requires certain pre– and post-processing steps to improve the quality of the flow vectors.

In the NVIDIA Turing and NVIDIA Ampere architecture generation GPUs, most of these algorithms use a compute engine to perform the required tasks. As a result, when the compute engine workload is high, the performance of the NVIDIA Optical Flow Accelerator (NVOFA) could be affected.

On NVIDIA Ada-generation GPUs, most of these algorithms are moved to dedicated hardware within the NVOFA, reducing the dependency on the compute engine significantly.

In addition, NVIDIA Ada-generation GPUs bring several other optimizations related to reducing the overhead of interaction between driver and hardware. This increases the overall performance and context switches between various hardware engines on the GPU.

With these changes, the speed of the NVIDIA Ada Lovelace architecture NVOFA is improved ~2x compared to the NVIDIA Ampere architecture NVOFA.

Quality improvements

Based on the feedback from earlier generations of NVOFA, there are several quality improvements incorporated in the hardware. Using the same preset, you can see a 10-15% improvement in quality (tested on the KITTI2015 data set) compared to NVIDIA Ampere architecture GPUs.

For more information, see 1.4 NVOFA Quality and Performance.

Optical Flow SDK 4.0

The NVIDIA Optical Flow SDK enables you to access NVOFA functionality. The NVIDIA Optical Flow SDK is a set of Optical Flow C APIs, reusable C++ wrapper classes, and a set of sample applications. These APIs and C++ wrapper classes facilitate the programming of the NVOFA for the efficient computation of the optical flow between a pair of images.

Optical Flow SDK 4.0 comes with the following enhancements and features:

•  External hint support
•  NVIDIA Optical Flow-assisted Frame-Rate Up-Conversion (NvOFFRUC)

External hint support

When hints are generated with low evolution images or are available from other sources such as a game engine, NVOFA can refine the hints further to improve the quality of the flow vectors.

Though external hint support is already available through C-API, support was missing in earlier versions of SDK C++ wrapper classes.

Optical Flow SDK 4.0 adds necessary support in the C++ classes and the use of external hints is demonstrated in the sample application AppOFCuda. The hint format is the same as the output flow vector format: an array of NV_OF_FLOW_VECTOR structures. Each array element represents a motion vector for the corresponding block in raster scan order.

AppOFCuda accepts hints in Middlebury flo format but converts them into the required format (an array of NV_OF_FLOW_VECTOR structures) before passing it to the NVOF API. NVOFA prioritizes external hints when they are provided; you are expected to provide reasonable quality hints.

Frame-rate up-conversion (FRUC) is a technique that generates higher frame-rate video from lower frame-rate video by inserting interpolated frames into it. Such high frame-rate video shows smooth continuity of motion across frames, improving the perceived visual quality of the video.

The NvOFFRUC library exposes APIs that take two consecutive frames and generate an interpolated frame in between. The interpolation is instant and does not have to be exactly in the middle of the two frames: it can be specified arbitrarily. For more information, see the NVOFA FRUC Programming Guide.

These APIs can be used for up-conversion of any video content. Internally, the library uses the NVOFA hardware engine and CUDA compute cores. As a result, frame interpolation using the NvOFFRUC library is much faster compared to software-only methods.

For more information, see AV1 Encoding and FRUC: Video Performance Boosts and Higher Fidelity on the NVIDIA Ada Lovelace Architecture.

Optical Flow SDK 4.0 is available now.

The NVIDIA Base Command Platform enables an intuitive, fully featured development experience for AI applications. It was built to serve the needs of the internal NVIDIA research and product development teams. Now, it has become an essential method for accessing on-demand compute resources to train neural network models and execute other accelerated computing experiments. 

Base Command Platform simplifies AI experimentation workflows by providing a cohesive service that integrates users, jobs, and data. It provides easy access to a private registry to host custom containers as well as the breadth of software from the NGC Catalog. It offers all these features without sacrificing reliable NVIDIA performance, flexibility, and scalability. You can use Base Command Platform for experiments requiring a single GPU or a data center’s worth of them.

Base Command Platform interface and features

Base Command Platform supports a CLI, API, and web interface, all built into the NGC portal. The integrated web interface makes software discovery in the NGC Catalog and subsequent use in Base Command Platform smooth. You don’t have to transition between tools not designed to be used together.

In addition to providing access to the public NGC Catalog, you also gain access to a private registry dedicated to the Base Command Platform environment. The private registry is useful for keeping containers, models, and software private and secure, as dictated by developer requirements.

Base Command Platform provides a rich set of user management controls. When you are invited to use a Base Command Platform environment (called an organization), the administrator can restrict your ability to upload and interact with content on the platform through a set of role-based access controls. These controls can apply to the root organization and also to the concept of a team. 

A team can differ minimally or significantly from the root organization, depending on how an admin configures that team. For example, a team may only be provided access to a subset of private registry containers or resources. When onboarded to a team, you could be disallowed from uploading your own containers to the private registry. 

These capabilities can be mixed and matched to provide the right level of functionality for a given user or group by the org administrator. Administrators can also set hardware usage quotas in the organization for specific users, both GPU and storage capacity.

The Base Command Platform web interface places the key user interaction points front and (left of) center: 

• Jobs: A list of containers running on NVIDIA Base Command Platform compute resources.
• Datasets: Read-only data inputs that can be mounted into jobs.
• Workspaces: Read/write persistent storage that can also be mounted into jobs.

Simple yet powerful hardware abstraction

In Base Command Platform, the managed hardware resources are presented to the user through two concepts: accelerated computing environments (ACEs) and instances within an ACE. 

An ACE is a composition of a set of hardware resources: compute, network, and storage. An instance selects the CPU, RAM, and GPU resource quantities that a job requires from a system within an ACE. 

ACEs can support a variety of instance types depending on their underlying hardware composition. Administrators can restrict the use of these resources through a quota for GPU hours, as well as completely restricting instance type availability for specific users in the org. 

Base Command Platform resources are connected through industry-leading technology provided by the underlying infrastructure. NVIDIA NVLink, NVIDIA InfiniBand, and high-performance Ethernet connectivity are integrated as part of a Base Command Platform environment’s design to maximize the value of the managed hardware resources. 

The scheduler in Base Command Platform is designed to take advantage of topology awareness to provide optimal resource use for jobs as they are submitted.

Datasets and workspaces

Data management is core to Base Command Platform’s capabilities. Datasets, models, and source code must be made available to compute resources during experimentation. 

The dataset and workspace concepts are how Base Command Platform solves this problem. A dataset is a read-only storage construct after creation, with all the same sharing capabilities as private registry contents. They can be private to a specific user, shared with any number of teams in an org, or shared with the entire org. 

Workspaces are more flexible. They are readable and writable but can be marked read-only when used in a job if desired. Workspace-sharing capabilities are identical to what datasets support.

Datasets are created through either the web interface or the CLI at upload or conversion time. We cover conversion when jobs are discussed later in this post. A workspace is created first, then populated with data as part of a job, or through direct upload (similar to datasets).

So, why would you use one over the other? 

Datasets are a great fit for immutable data that must be widely shared as-is. Frequently, that is a dataset that no longer requires modification but could also include a license file or API key intended for shared use. They can be shared widely because there is no chance that they will be modified in place. 

Workspaces are a great fit as a landing place for data that is a work-in-progress: datasets, source code maintained outside a container, or even a collection of models under development. Workspaces can be shared widely but given that they are writable by default, wide sharing may require additional coordination and awareness between users.

The aggregate dataset and workspace capacity available for a given user in Base Command Platform is controlled by the user’s storage quota, set by the org administrator. You can see your storage quota along with your current storage usage on the Base Command Platform dashboard. 

There is an additional storage type, result, that factors into this capacity, which we discuss later in the context of jobs. When your quota is exceeded, you can request additional capacity if enabled by your environment administrator.

Bringing it all together in a job

Base Command Platform operationalizes data, compute resources, the NGC Catalog, and private registry contents with jobs. 

Job creation starts with resource selection. You are presented with available ACEs for use by your org and team. You may have access to more than one ACE, but a job must execute within a single ACE. 

After an ACE is selected, you can select from the available instance types in that ACE. For multi-node jobs, the only instances available are those that leverage the maximum CPU, RAM, and GPU resources available for a given system type within the selected ACE.

Choose an ACE, an instance type, and multi-node launch options, if necessary. Next, you can specify datasets and workspaces that are a part of the chosen ACE to be mounted into the target job, along with the desired mount point for each of them. Here, a workspace can be marked as read-only. 

The job’s result storage path must be specified as well. A result is a job-specific read/write storage pool intended to hold artifacts that you’d like to preserve from the job upon completion, along with stderr and stdout. As we mentioned previously, the capacity consumed by results counts against your quota.

Then, you must select a Container object and a valid container Tag. You can choose containers from the NGC Catalog as well as the private registry containers that you have permission to access. If you select a multi-node job, only containers marked in the NGC Catalog or private registry as supporting multi-node functionality are presented as options.

You now specify one or more commands, or even a service (such as JupyterLab, Visual Studio Code Server, or NVIDIA Triton Inference Server) to run inside the selected container when the job is active. If an external port is needed to expose a web interface or some other endpoint, one or more ports must be added with the Add a Container Port option.

Now that the job is specified, there are several more options available to configure: 

• Job priority level
• New job name
• How a job is capable of behaving if preempted
• Maximum runtime of the job
• Time slice interval for telemetry collection
• Custom labels

Interacting with running and finished jobs

After a job has been launched, you are redirected to a page specific to that job, where the launch details are on the Overview tab. In fact, an equivalent CLI version of the launch form is available under the Command section. 

Jobs are presented in a way that makes them easy to reproduce: either by copying a CLI representation or by cloning the job through the web interface. If ports were added to the job when launched, a URL endpoint is also available (Video 2).

Several additional tabs are present for a scheduled, running, or completed job:

• Telemetry: Key performance metrics for a job over time.
• Status History: A list of states that the job and associated replicas have been in.
• Results: A way to view the files that have been saved to a job’s results directory.
• Log: Searchable, live access to a job’s stdout output.

After a job completes, the job-specific page is still accessible and can be used as a reference for future jobs, or further debugging if something didn’t work out as intended. The results directory and log files from the job can be easily retrieved. Depending on how the job was written, the desired job artifacts could be in a workspace instead of the results directory. 

Base Command Platform provides CLI support for downloading data from a workspace. The resulting artifacts from a job can then be uploaded into Base Command Platform, either made public or kept in the private registry for the org. It provides the starting point for further experimentation in Base Command Platform or a critical component for a model deployed elsewhere.

For more information, see Managing Jobs.

MLOps integration using the NGC API

You can further augment and extend Base Command Platform capabilities with external software integration through the documented NGC API. 

The NGC API can be used for workflow integration or dashboards outside of Base Command Platform, such as third-party MLOps platforms and tools. MLOps software and service providers that have integrated their unique offerings with Base Command Platform include Weights & Biases and Rescale.

As Base Command Platform features evolve and expand, the NGC API enables new and existing software ecosystems to integrate its strengths into other purpose-built solutions.

Conclusion

Base Command Platform is one of the key NVIDIA tools for making AI infrastructure accessible to developers. To get a hands-on sense of how Base Command Platform works, NVIDIA offers a series of labs through NVIDIA LaunchPad. Some labs cover specific use cases around natural language processing and medical imaging, and others are tailored toward gaining experience with Base Command Platform capabilities. 

Learn how to use NVIDIA Base Command Platform to accelerate your containerized AI training workloads. 

The role of artificial intelligence (AI) in boosting performance and energy efficiency in cellular network operations is rapidly becoming clear. This is especially the case for radio access networks (RANs), which account for over 60% of industry costs. 

This post explains how AI is transforming the 5G RAN, improving energy and cost efficiency while supporting better use of RAN computing infrastructure.

5G is in full motion 

It is now over 4 years since the first 5G networks were launched. A GSMA study projects that there will be over 1.4 billion 5G connections by 2025. The deployment of standalone 5G networks globally is also beginning to increase. To learn more, see Operators Keep Pushing Forward on 5G Standalone Networks.

In the consumer market, 5G is the default upgrade for an ubiquitous cellular communications service. For the enterprise market, 5G has the optimal combination of high performance, mobility, flexibility, and security to provide the connectivity fabric for enterprise use cases (Figure 1). 

NVIDIA is driving innovation in cellular networks with a fully programmable NVIDIA Aerial SDK for building and deploying GPU-accelerated 5G virtual radio access networks (vRANs). This is providing the building blocks for a standard public 5G network for a telco or a private 5G network implementation with AI-on-5G. 

AI is shaping the current state—and evolution—of 5G 

The role of AI in cellular network operations is growing at a fast pace. AI delivers value through the terabytes of data collected every day—from network elements to customer interactions—and through the resulting insights. These insights are related to managing increased network demand, combating cyberthreats, optimizing services, and improving the customer experience.  

AI is currently applied across different domains in cellular networks such as RAN, core network, operations support systems (OSS), business support systems (BSS), and cloud infrastructure. These AI-enabled functionalities emerged in 4G, are becoming entrenched in 5G, and will become native in 6G. 

While AI will permeate the entire value chain of the industry, its impact on the RAN will be the most profound, particularly given the disproportionate share of industry capex and opex the RAN accounts for. Accordingly, both O-RAN and 3GPP have identified and are working on AI initiatives that improve the performance, flexibility, scalability, and efficiency of the RAN. To learn more, see Embracing AI in 5G-Advanced Towards 6G: A Joint 3GPP and O-RAN Perspective.

AI is transforming the RAN in four key ways: energy savings, mobility management and optimization, load balancing, and Cloud RAN. Read on for more details about each.

Energy savings

The rapid growth of 5G deployment has coincided with a rapid increase in energy costs globally, leading to concerns about high operational costs and carbon emissions. There is the additional concern that some 5G deployments may face hard limits on how much power can be supplied, even if the owners are willing to pay. These concerns create a powerful incentive to increase operational efficiency to achieve higher power efficiencies from current and future network deployments. See Take the Green Train: NVIDIA BlueField DPUs Drive Data Center Efficiency for more details. 

The industry has been using reactive and inflexible rule-based techniques to conserve energy, such as switching on/off cells based on different thresholds of cell load. However, AI offers a proactive and adaptive approach, enabling telcos to predict energy efficiency and load in future states. AI also provides better integration between the RAN and virtualized core network functions (with User Plane Function, for example) through offloading networking, security, and RAN tasks to NVIDIA GPUs and DPUs. 

Mobility management and optimization

Mobile communications systems have the distinct ability to support handovers of devices from one access point to another. This provides service continuity, supports mobility, and optimizes performance. Expectedly, disruptions, delays, and frequency of handovers add up to inefficiencies in network performance. 

Using AI to optimize paging and predict the next cell for handovers offers a significant opportunity to improve performance for advanced features. Such features include sophisticated dual connectivity options, conditional handover, and dual active protocol stack (DAPS) handover. 

AI prediction will rely on insights about the possible movement of the device. To achieve this, the Network Data Analytics Function (NWDAF) from 3GPP SA2 provides data from the core network, applications, and the OSS to improve handover performance, predict device location and performance, and steer traffic to achieve quality network performance.

NVIDIA continues to innovate around the NVIDIA Aerial SDK to support these new expectations for data collection and the use of AI for network management. 

Load balancing 

Handovers enable mobile networks to steer traffic to balance the load across different cell sites and to improve the use of spectrum, RAN, transport, and core infrastructure. This load-balancing decision is achieved by optimizing handover parameters and decisions using current or historical load information. 

However, this task is becoming more challenging due to the use of multiple frequency bands and interworking with different RANs. Current rules will increasingly struggle to cope with fast time-varying scenarios with high mobility, and dynamic traffic patterns with a large number of connections. 

AI models perform better and can predict load to optimize critical tasks using the collection of RAN data. This will improve network performance and user experience. This is a key driver in the development of proprietary AI tools and the current push for some industry standardization to unlock this opportunity at scale. 

Cloud RAN: Colocating RAN and AI in the cloud

While Cloud RAN is not a core application of AI in itself, it is the logical extension of the preceding load-balancing discussion. In this case, a softwarized and cloud-enabled RAN can be colocated on the same cloud infrastructure with AI workloads. Boosting RAN use beyond the typical average 25% of many sites, this will support telcos to extract efficiency gains from an asset that gulps at least 60% of industry capex. 

The suitability to share the same computational resources with other AI workloads, the availability of such orthogonal AI workloads, and the ability to use AI to switch dynamically between the different workloads are key to unlocking this opportunity. 

This Cloud RAN vision will begin in the 5G era and then mature in the 6G era, as the RAN as a workload in the cloud becomes the ultimate destination. By pooling baseband computing resources into a cloud-native environment, the Cloud RAN solution delivers significant improvements in asset use for both Cloud Service Providers (CSPs) and telcos. 

CSPs can run the RAN as a workload alongside their AI workloads within their existing data center architecture while telcos can increase RAN operational efficiency by more than 2x for an estimated >25% impact on EBITDA. 

NVIDIA is shaping this evolution with the Cloud RAN solution based on NVIDIA Spectrum switch, NVIDIA H100 CNX converged accelerator, and NVIDIA Aerial SDK. To learn more, see Unlocking New Opportunities with AI Cloud Infrastructure for 5G vRAN.

The role of AI in the RAN is only part of the overall role of AI in telecom network operations. In addition to the RAN, NVIDIA is working with partners on AI-powered operations to use data insights from telco data to create new revenues and improve operational efficiency. Visit the NVIDIA Telecommunications page to learn more.

Denoising is an advanced technique used to decrease grainy spots and discoloration in images while minimizing the loss of quality.

Learn about the NVIDIA Isaac ROS DNN Inference pipeline and how to use your own models with a YOLOv5 example in this December 1 webinar.

Our digital lives run on collections of computers tightly linked on high-speed networks, and the latest one is an AI supercomputer called NVIDIA DGX SuperPOD.

The University of Cambridge, in the UK, and an NVIDIA DGX SuperPOD point way to the next generation of secure, efficient HPC clouds.

Quantum computers, still in their infancy, are influencing a new generation of simulations already running on classical computers, and now accelerated with the NVIDIA cuQuantum SDK.

Fueled by data, ML models are the mathematical engines of AI, expressions of algorithms that find patterns and make predictions faster than a human can.

Software teams comprise a broad range of professionals, from software engineers and data scientists to project managers and technical writers. Sharing code with other team members is common when working on a project, and it is important to track all changes. This is where pull requests come in.

In software development, a pull request is used to push local changes into a shared repository (Figure 1). It is a way for you to request code review from other collaborators before pushing an approved update to the central server. This helps maintain version control.

This post discusses the benefits of pull requests and shares tips for creating and handling pull requests when working on software projects. Using this information, you will be better equipped to work with many collaborators on major projects.

The steps of a pull request

To create a pull request, follow the steps outlined below.

1. Create a new git branch to work locally using the following command:
   git -b BRANCH_NAME
2. Implement changes and push them frequently (so that they do not get lost) using the following command:
   git add NAME_OF_THE_FILE
git commit -m "DESCRIBE YOUR RECENT CHANGES"
3. After you have finished the implementation and committed your changes locally, you should get the latest changes from the shared repository to ensure there are no conflicting changes. You can get the latest changes using the following command:
   git pull origin BRANCH_NAME
4. Push your changes to the remote repository using the following command:
   git push --set-upstream-to origin REMOTE_BRANCH_NAME
5. Navigate to the user interface of the platform where your shared repository is located (GitLab, GitHub, BitBucket). There you are asked to write the name of the pull request and a short description. You also have the option to assign it to someone from your team to review it.

For a more detailed introduction to pull requests, see Making a Pull Request.

Key benefits of using pull requests 

Whether you are working on the frontend or backend of a project, pull requests can help with the code review process when working with a team. This section details the key benefits of using pull requests in your work.

Facilitate collaboration 

When it comes to collaboration, there are a few things that can make or break a team. One of those things is the ability to work together, even if members are responsible for different parts of the project.

Using pull requests, changes can be made without impacting the work of others. They are a great way to gather tips or code improvements from team members.

If you are unsure about a code change, submit a pull request for feedback. Other team members may have suggestions that you had not considered, and this can help you make better decisions about your code.

In any project, it is important to have experienced engineers review and accept or reject changes since you may miss some things that they can see from a fresh perspective. 

However, it is equally important to avoid bottlenecks when multiple team members are submitting changes to the project codebase. For those working on pull requests, it is critical to set expectations for expected review times. This ensures the project continues to move forward. 

Build features faster

Pull requests are a powerful tool that can help teams build features faster. Because pull requests can be reviewed with comments added, they provide an excellent way to communicate code changes.

First, they enable developers to submit changes to a project without having to wait for the project maintainer to merge the changes. This enables team members to work on code changes in parallel, which can speed up development.

Second, pull requests can be reviewed and comments can be added. Developers reviewing pull requests may need to ask questions or clarify potential errors. You can also use comments to share resources. 

Third, pull requests can be merged, so that changes can be integrated into the project quickly and easily when building new features.

Reduce risks associated with adding new code

There is no doubt that programming code comes with a degree of risk. After all, every time you add something new to your codebase, you are potentially introducing new bugs and vulnerabilities that affect the end user. 

Before a pull request is merged into the main codebase, other team members have the opportunity to review the changes to ensure compliance with the team’s coding standards. Bugs and errors can be addressed before they cause any problems in the live code.

With pull requests, you can always roll back to a previous version in case things go wrong. Pull requests become your safety net.

Improve code quality and performance

When you create a pull request, you are essentially asking for someone else to review your code and give feedback. By engaging a colleague, you can improve the quality of your code based on that feedback.

You can help reviewers understand your changes by writing descriptive commit messages and explanations in the description section of the pull request. 

You can also avoid potential problems if you make a change that someone else does not agree with. They can simply raise an issue with your pull request. This gives you the opportunity to fix the problem before it becomes a bigger issue. This is a powerful way to improve the quality of your code. 

Takeaways

Maintaining version control through pull requests is important for software teams. This approach enables team members to collaborate while tracking and managing changes to software systems. By using pull requests, teams can work on different parts of a system at the same time and then easily merge their changes together. This boosts team efficiency and prevents conflicts.

When used correctly, pull requests provide a clear and concise way to view changes that have been made to the code or file, facilitating discussion and feedback.

The importance of pull requests cannot be overstated. They are an essential part of the software development process, helping to ensure that relevant parties review code changes before they are merged into the main codebase. This helps to avoid bugs and other problems that could potentially cause serious issues. 

Learn to train an end-to-end TTS system and track experiments in this live workshop on December 8. Set up the environment, review code blocks, test the model, and more.

Machine learning (ML) security is a new discipline focused on the security of machine learning systems and the data they are built upon. It exists at the intersection of the information security and data science domains. 

While the state-of-the-art moves forward, there is no clear onboarding and learning path for securing and testing machine learning systems. How, then, should interested practitioners begin developing machine learning security skills? You could read related articles on arXiv, but how about practical steps?

Competitions offer one promising opportunity. NVIDIA recently helped run an innovative ML security competition at the DEF CON 30 hacking and security conference. Hosted by AI Village, the competition drew more than 3,000 participants. It aimed to introduce attendees to each other and to the field of ML security. The competition proved to be a valuable opportunity for participants to develop and improve their machine learning security skills.

NVIDIA AI Red Team and AI Village

To proactively test and assess the security of NVIDIA machine learning offerings, the NVIDIA AI Red Team has been expanding. While this team consists of experienced security and data professionals, they recognized a need to develop ML security talent across the industry. With more exposure and education, data and security practitioners are likely to improve the security of their deployed machine learning systems.

AI Village is a community of data scientists and hackers working to educate on the topic of artificial intelligence (AI) in security and privacy. The community holds events at DEF CON each year. 

The NVIDIA AI Red Team and AI Village joined together at DEF CON 30 to engage the information security community with a machine learning security competition. The topic was potentially new to many attendees. Members of the AI Village created challenges designed to teach and test elements of ML security knowledge. In addition to NVIDIA, these members represented AWS Security, Orang Labs, and NetSec Explained. 

AI Village Capture the Flag Competition

Capture the Flag (CTF) competitions include multiple challenges. Competitors play through the challenges and collect flags for those that are completed successfully. These flags are assigned various point values based on the level of challenge. Competitors win by collecting the most points. 

With this familiar format in mind, the AI Village and NVIDIA AI Red Team built The AI Village CTF @ DEFCON. Organizers partnered with Kaggle to use a platform familiar to the machine learning community. Similar to information security CTFs, Kaggle competitions provide a format for ML researchers to compete on discrete problems. 

Partnering with Kaggle provided the competition with a flexible and scalable platform that paired compute and data hosting with documentation and scoring. Although the challenge servers are no longer active, you can view the challenge descriptions.

Competitors reported onboarding and moving through the challenges with ease, with minimal additional infrastructure required from the AI Village. Furthermore, Kaggle has a large audience of skilled data scientists and machine learning engineers who were excited to explore the security domain. Kaggle also generously offered ongoing support and $25,000 in prizes. We could not have asked for a better partner for this event.

Over the month-long competition, over 3,000 competitors hacked their way through 22 challenges. This far exceeded expectations and included participants from over 70 countries, from first-time Kagglers to Grandmasters. The event succeeded in bringing the traditional information security and machine learning communities together to tackle a range of challenges from this new domain of ML security. 

Competitors used publicly available tools and innovative technique applications such as open source research, masking, and dimensionality reduction. In the process, they often reimplemented attacks from academic literature as well. 

Because one challenge remained unsolved, there was always a chance for someone to rise to the top of the leaderboard. For the final two weeks of the competition, the Kaggle Discussion Board and AI Village Discord were abuzz with theories and explorations of the remaining unsolved challenge. The organizers were checking hourly for a buzzer-beating leaderboard shift. Check out the challenge solutions.

Inference Challenge

In the Inference Challenge, participants had to execute a membership inference attack to identify training samples. They only had API access to an image classifier. When done successfully, the competitors would identify images that showed characters of the flag. 

Some competitors chose to randomly generate the images by permuting pixel values, effectively brute-forcing the problem. Other competitors assumed that the training data may have included a standard dataset and used EMNIST as their source data, leveraging open source data. Others made use of the Adversarial Robustness Toolbox, producing output similar to what is shown in Figure 2. 

Whatever method used, a successful challenger would be rewarded with the flag, spelling D3FC0N. This leetspeak encoding of the DEF CON conference name was used in several places on the conference website. 

Crop2 Challenge

Research and out-of-the-box thinking often help to solve CTF challenges. For instance, the one unsolved challenge in the competition was Crop2. In Crop2, participants were given a poisoned cropping model and had to create the poisoned sample (within some error bounds). They had one training data example to work with (Figure 3). 

This is a difficult problem without an efficient, standard algorithmic solution. When you think about all of the pixels in an image and all of the possible pixel values across three color channels, the search space explodes to over 800 billion options. Instead, competitors could combine reverse engineering, open source research, and assumptions to reduce the number of combinations.

After the competition ended, organizers gave hints to help competitors solve the Crop2 Challenge. Some of the key hints included using open source research to determine that pixel colors likely were generated by matplotlib default colormaps. This greatly reduces the search space into the hundreds of thousands. 

By making these informed assumptions, one competitor was eventually able to reach the Crop2 Challenge solution. One trait of great hackers is tenacity: still working tirelessly after the competition ended, this competitor diligently worked through the provided hints. The competitor reported that a hint “helped me realize that we only needed to use nine colors. Mate, I’d been fiddling around with 16 million. This made the search space manageable.” 

Competitor notebooks

Check out some of our favorite notebooks from competitors:

1. Chris Deotte – From a member of the Kaggle Grandmasters of NVIDIA (KGMoN), these solutions are very well organized and documented. We recommend Secret Sloth in particular.
2. Eric Bouteillon – Watch the flag appear character-by-character in Excuse Me. Also notice the different solve techniques for the MATH challenges. Have you heard of silhouette score?
3. John MacGillivray – John deduced that the Hotterdog model was based on MobileNet, enabling an offline attack. Great tradecraft.
4. Fournierp – A comprehensive writeup about model inversion for the Inference Challenge, written from scratch. You can also check out the MIFace in the Adversarial Robustness Toolbox.
5. Eoin O – Learn how you could have solved the Crop2 Challenge. More than 3,000 competitors tried to solve it for the greater part of a month. The day after the competition ended, organizers released several hints. Within a few hours, it was solved. It was great to see all of the competitors collaborating in Discord and the Kaggle Discussion Board after the competition ended.

Summary

The AI Village CTF @ DEF CON 30 competition showed that there is a significant appetite in both the security and data professions to improve machine learning security skills. As ML systems are deployed in increasingly security-critical contexts, it will become imperative to train professionals and develop tools and methods for security development, deployment, and testing. 

NVIDIA will continue driving innovation with a robust and secure ecosystem for AI, from embedded devices and laptops to supercomputers and the cloud. As part of this effort, our AI Red Team will empower ML security research and testing internally and establish security practices across the industry. We will host competitions, workshops, and release research and security tools in the future. If you’re interested in participating, contact us at threatops@nvidia.com.

Additional resources

• Learning to Defend AI Deployments Using an Exploit Simulation Environment
• Introduction to Adversarial Robustness
• Adversarial Machine Learning Reading List
• HackingNeuralNetworks GitHub repo 
• Coefficient GitHub repo 
• HackThisAI GitHub repo

Self-driving cars must be able to detect objects quickly and accurately to ensure the safety of their drivers and other drivers on the road. Due to this need for real-time processing in autonomous driving (AD) and visual inspection use cases, multiple AI models with preprocessing and postprocessing logic are combined in a pipeline and used for machine learning (ML) inference.

Speedup is required in every step of the pipeline to ensure a low latency workflow. Latency is the time it takes to get the inference response. Faster processing of AD data will enable more efficient analysis and use of the information, creating a safer driving environment. A delay with any single aspect can slow down the entire pipeline. 

To achieve a low latency inference workflow, electric vehicle manufacturer NIO integrated NVIDIA Triton Inference Server into their AD inference pipeline. NVIDIA Triton Inference Server is an open source multi-framework inference serving software.

This post explains how NIO orchestrated its pipeline of image preprocessing and postprocessing and AI models with NVIDIA Triton on GPUs. It also shows how NIO reduced network transmission to successfully speed up their AI inference workflow for AD use cases.

Join the NVIDIA Triton and NVIDIA TensorRT community and stay current on the latest product updates, bug fixes, content, best practices, and more.

Faster AI inference for real-time response

NIO designs, develops, jointly manufactures, and sells premium smart electric vehicles, driving innovations in next-generation technologies in autonomous driving, digital technologies, electric powertrains, and batteries. NIO Autonomous Driving Development Platform (NADP) is an R&D platform dedicated to the core autonomous driving service of NIO.

NIO chose NVIDIA Triton Inference Server because of several key technical and operational reasons, including:

• NVIDIA Triton supports DAG-based orchestration of numerous models, along with preprocessing or postprocessing modules
• Cloud-native deployment of NVIDIA Triton enabled multi-GPU, multi-node scaling in a lightweight way
• High-quality documentation and learning resources helped ease migration to NVIDIA Triton
• NVIDIA Triton’s stability and robust functionality are necessary for AD use cases 

NIO’s AI inference workflow for autonomous driving

Hundreds of AI models are used to mine data from autonomous vehicles. In a use case like autonomous driving, the inference workflow consists of multiple AI models with preprocessing and postprocessing logic stitched together in a pipeline. 

NIO moved the preprocessing and postprocessing of the pipeline from the client side, which runs on CPUs, to NVIDIA Triton running on GPUs. The NVIDIA Triton’s business logic scripting (BLS) functionality was used to orchestrate the pipeline to run optimally for AD use. 

By moving the preprocessing from CPU to GPU and leveraging efficient pipeline orchestration, NIO achieved 6x latency reduction in some core pipelines, improving the overall throughput by up to 5x. 

Before and after workflow pipelines are shown in Figure 1.

Model pipeline orchestration benefits of NVIDIA Triton

This section examines each of the benefits NIO realized by integrating NVIDIA Triton.

GPU-accelerated preprocessing

Preprocessing tasks such as decoding, resizing, and transposing were accelerated on the GPU by NVIDIA Triton using nvJPEG and NVIDIA DALI. This significantly offloaded the computing workload from the client CPU and reduced preprocessing latency.

Upgrading models without the need for client application modification

By moving the preprocessing and postprocessing of the model to NVIDIA Triton, each time the model is upgraded, the client side does not require any modification. This essentially speeds up the rollout of the model, helping it reach production faster.

Using a single GPU node to reduce network data transfer overhead

A unified preprocessing enables multiple copies of the input to be shared with multiple back end recognition models. The process uses GPU shared memory on the server side, without data transfer overhead costs. 

Figure 2 shows the pipeline can connect up to nine models using the NVIDIA Triton business logic scripting functionality.

For an input image of 2K resolution, the size of each frame is 1920 x 1080 x 3 x 8 = 47 Mb. Assuming a full frame rate of 60 fps, the amount of data input per second is 1920 x 1080 x 3 x 8 x 60 = 2847 Mb. In the previous workflow, each image is sent sequentially to the nine models over the network. Data transferred per second is 1920 x 1080 x 3 x 8 x 60 x 9 = 25 Gb = 3 GB.

In the new workflow, the nine models are orchestrated with the NVIDIA Triton business logic scripting. That means the models can access the image in the GPU shared memory and the images do not have to be sent over the network. Assuming a PCIe bandwidth of 160 Gb = 20 GB per second, theoretically the data generated per second can save 150 ms in data transfer if the data is transferred over PCIe.

Assuming an available bandwidth of 16 Gb = 2 GB per second, theoretically the data generated per second can save 1,500 ms in data transfer if the data is transferred over the network. All these result in speeding up the workflow.

Network transfer savings using image compression 

For accurate model prediction, the input image must be 1920 x 1080 x 3 x 8 bytes in the previous workflow and must be transmitted through the network. After introducing the server-side preprocessing, the original image can be altered to a compressed three-channel 720 pixel image (1280 x 720 x 3) within the allowed range of accuracy loss. 

As a result, it only takes a few hundred KB to transmit the bytes of the compressed image and resize with minimal accuracy loss to 1920 x 1080 x 3 x 8 bytes on the server. This leads to additional network transfer savings, speeding up the workflow. 

Ease of integration in NADP inference platform

NIO’s current inference platform based on NVIDIA Triton is a key component of their Autonomous Driving Development Platform (NADP), used in their autonomous driving solution.

As the NIO platform is built on Kubernetes (K8s), it was imperative for NVIDIA Triton to integrate well with Kubernetes. The components of the workflow are implemented as K8s CRD (native and custom) around NVIDIA Triton.

Continuous Integration/Continuous Delivery (CI/CD)

Argo is the engine used to orchestrate the workflow in Kubernetes. It helps with CI/CD for all the components involved in development, quantification, access, cloud deployment, pressure testing, and launch. NVIDIA Triton helps with CI/CD by triggering the next step in the workflow whenever the models are loaded.

In addition, use of the NVIDIA Triton Docker container helps with consistent functionality across development, test, and deployment environments.

Integrating the Jupyter environment into the NVIDIA Triton image was seamless. Jupyter provides a convenient development environment for debugging in case of a complex problem that requires online debugging or offline reproduction.

Ease of deployment with Istio

NVIDIA Triton natively supports gRPC protocol for communication with applications. However, as the Kubernetes native service cannot offer effective request-level load balancing for gRPC, NVIDIA Triton is integrated with the Istio service mesh. Istio is used to load balance traffic to NVIDIA Triton Inference Server and monitor the health of the service through liveness/readiness probes of NVIDIA Triton.

Ease of use with Apollo configuration management

Apollo Configuration Center is used for model name-based service discovery. Users can access the models without knowing the specific domain name where the model is deployed. Combined with the NVIDIA Triton model repository, users can directly trigger the deployment of models.

Metrics with Prometheus and Grafana

NVIDIA Triton provides a complete set of model service metrics based on model dimensions. For example, NVIDIA Triton can distinguish between inference request queueing time and GPU computation time, allowing for fine-grained diagnosis and analysis of online model service performance without entering the debug mode. 

Because NVIDIA Triton supports cloud-native mainstream Prometheus/Grafana, users can easily configure the dashboard and the alarms for each dimension to provide metrics support for high service availability.

Key takeaways

NIO’s optimized workflow that integrates NVIDIA Triton Inference Server resulted in a 6x latency reduction in some core pipelines. This improved overall throughput by up to 5x.

By moving the preprocessing logic to GPU using the NVIDIA Triton pipeline orchestration functionality, NIO achieved:

• Faster image processing
• Freed CPU capacity
• Reduced network transfer overhead
• Higher inference throughput

NIO achieved AI inference workflow acceleration using NVIDIA Triton Inference Server. NVIDIA Triton was also easy to integrate in a robust Kubernetes-based scalable solution.

Additional resources

• Get started with NVIDIA Triton and access a variety of beginner to advanced resources.
• Learn about the features needed in an inference platform when building a real-time or continuous data streaming application with Fast and Scalable AI Model Deployment with NVIDIA Triton Inference Server.
• Watch the NVIDIA On-Demand session, From Cloud to Car: How NIO Develops Intelligent Vehicles.

Learn how to decrease model training time by distributing data to multiple GPUs, while retaining the accuracy of training on a single GPU.

Timing is everything, especially when it impacts your customer experiences, bottom line, and production efficiency. Edge AI can help by delivering real-time intelligence and increased privacy in intermittent, low bandwidth, and low cost environments. 

By 2025, according to Gartner®, 75% of data will be created and processed at the edge, outside the traditional data center or cloud.¹ It’s no wonder that thousands of companies are turning to edge AI to drive transformation for their businesses.

As organizations undergo this shift, many IT and business leaders are still in the early stages of planning and executing their edge computing strategies. Because edge AI is a new concept, the process is difficult for many. 

NVIDIA, a leading AI infrastructure company with robust experience helping organizations, customers, and partners successfully deploy edge AI solutions, is no stranger to these new concepts. 

In an effort to help others, the learnings and recommendations from these experiences are presented in An IT Manager’s Guide: How to Successfully Deploy an Edge AI Solution. The whitepaper offers an in-depth look at building and executing a successful edge AI deployment.

This post features recommendations on some key considerations when configuring an edge system. 

Edge system configurations: Design recommendations 

There are many parameters to consider when sizing a system. The optimal PCIe server configuration will depend on the target workload for that server. 

Edge AI models incorporate various workloads into their applications, such as vision AI, natural language processing, recommendations based on industrial sensors, and predictive analytics. 

Edge computing sizing considerations

When it comes to designing full hardware and software solutions at the edge, it is important to look at the solution as a whole to understand how the parts work together. Below are some of the individual considerations that IT must evaluate for edge AI deployments.

Number of streams: Each camera feed is a stream requiring a certain amount of memory and compute for processing. Small configurations of 6-7 video processing streams require relatively small systems. Larger deployments may require high performance systems that are typically seen in the data center. 

Application examples: One of the first steps to a successful edge AI deployment is understanding what workload needs to be run to reach your goals. Vision AI applications like image recognition, people or vehicle detection, and segmentation are all common use cases. 

Once an application is determined, it is important to understand the intended scale. For example, are additional AI models needed? Typically, a proof of concept (POC) will consist of a single AI model and use case, but most production deployments ultimately incorporate multiple AI models. The next steps include quantifying the business value of the application, dictating any environmental constraints, and securing stakeholder alignment. 

Memory: Perhaps the most common way to under-resource an edge AI solution is to configure the edge systems with too little memory. Edge AI systems require significantly more memory than other applications to support the parallel execution of the inference engine across the CPU and GPUs. 

The data science team or application vendor who trained the AI will know the memory requirements of the latest model. IT teams should, at a minimum, double that number to accommodate the inevitable expansion of the model as it retrains. This will also provide some headroom for the additional AI models that will need to be deployed alongside the first one.

Another rule of thumb is to provision twice as much system memory as the total GPU memory, and never less than 1.5 times the total GPU memory. The memory should be evenly spread across all CPU sockets and memory channels for optimal performance.

Networking: As operations increasingly rely on digital technologies such as edge computing, resilience is key. There are two networks to consider when designing an edge solution: the network between the edge AI location and cloud, and the network between a sensor and the edge AI system. 

Understanding the type of network connectivity of your environment will help in determining the specific networking bandwidth requirements for your use case. For example, for a use case like robotics, where wireless connectivity may not be possible, 5G is the next best choice as it offers minimal congestion and guaranteed service and bandwidth. 

Accelerators: Most edge applications run adequately on single socket x86 or Arm CPUs. But when the edge applications incorporate AI capabilities, they are far more compute intensive. 

To run an inference engine at the edge, the edge hardware needs enough compute power to execute complex neural networks with massively parallel computations. CPUs execute all the independent cells of a neural network sequentially, while discreet accelerators can execute them in parallel. Hence, accelerators are architecturally suited for AI, providing meaningfully better performance. They have become an essential component of modern AI infrastructure.

Among the most effective discrete accelerators for edge AI are Graphics Processing Units (GPUs) and Data Processing Units (DPUs). 

Storage: Naturally, the edge server requires local storage, usually a solid state hard drive, for its operating system, network components, hardware drivers, and application software. Unlike other applications, edge AI solutions typically process a massive amount of unstructured input data such as images, voices, and sensor readings. Depending upon how much of this data needs to be stored, for how long, how securely, and the level of reliability, different storage options are called for.

The first step in determining what storage is necessary for an edge AI solution requires IT teams to think through a data strategy. The data strategy will dictate what and how much data will need to be stored locally or in the cloud, and in turn, guide what storage options are best for that particular solution. Without a proactive strategy, developers often make inconsistent and sub-optimal choices that create problems down the road. 

Security: Security is paramount for edge AI computing devices, as they are deployed in remote locations outside the data center firewalls and the physical protections that limit access to systems. For more details, see Edge Computing: Considerations for Security Architects.

When it comes to an edge AI solution, five areas should be understood and made part of the overall solutions architecture: end-to-end encryption, mutual authentication, physical security, zero trust networking, and real-time monitoring.

Management: A remote management plan is critical for edge environments because systems at the edge are distributed, always on, and often operate in remote settings. See Remotely Operating Systems and Applications at the Edge to learn more.

An edge management solution will have automatic deployment and provisioning capabilities, ongoing management, real-time alerting, auditing, and use modern, cloud-native tools. 

Organizations have the choice of whether to build or buy a management solution. The following are questions to consider: How quickly does a solution need to be set up? Is the appropriate team and expertise available? Does this provide secure management of my edge environment? 

Pillars of a successful edge deployment 

Deploying the infrastructure needed to support a scalable edge AI solution is a big challenge. The process is iterative and time consuming, yet critical to do correctly. Decisions that are made when building an edge AI solution have far-reaching implications that will impact an organization’s business outcomes. 

For more guidance on this topic, download An IT Manager’s Guide: How to Successfully Deploy an Edge AI Solution. 

References

1. Gartner, “Building an Edge Computing Strategy,” G00753920, September 2021.
GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved.

NVIDIA DRIVE OS host-side Docker containers on NVIDIA GPU Cloud (NGC) provide a quick and easy alternative to installing Debian packages to set up NVIDIA DRIVE OS on a host development machine. 

Developers using the NVIDIA DRIVE platform have requested the ability to run Docker containers directly on NVIDIA DRIVE AGX hardware for quick environment setup to run applications with many package dependencies. 

DRIVE OS 6.0.5 Linux now includes the NVIDIA Container Toolkit runtime as part of the target Root File System (RFS) that is flashed onto the board. It enables developers to run Docker containers directly on the target NVIDIA DRIVE AGX hardware. 

This post explains how to run a few sample applications inside a Docker container running on the target NVIDIA DRIVE AGX hardware.

Setup 

Install NVIDIA DRIVE OS on the host machine and flash the target board. See the DRIVE OS 6.0 installation guide for details.

After flashing DRIVE AGX Orin with DRIVE OS 6.0.5, you can pull target-side Docker images from NGC or Docker Hub, and run GPU-accelerated containers on the target right out of the box.

Running CUDA sample inside target-side Docker container 

The RFS flashed onto the target hardware using DRIVE OS 6.0.5 provides CUDA samples at /usr/local/cuda-11.4/samples. The matrix multiplication CUDA sample runs inside an Ubuntu 20.04 Docker container to demonstrate the accessibility of the GPU on DRIVE AGX Orin inside the container. 

$ cd /usr/local/cuda-11.4/samples/0_Simple/matrixMul && sudo make
$ sudo docker run --rm --runtime nvidia --gpus all -v $(pwd):$(pwd) -w $(pwd) ubuntu:20.04 ./matrixMul

The following output shows the successful running of the CUDA sample:

nvidia@tegra-ubuntu:/usr/local/cuda-11.4/samples/0_Simple/matrixMul$ sudo docker run --rm --runtime nvidia --gpus all -v $(pwd):$(pwd) -w $(pwd) ubuntu:20.04 ./matrixMul
[sudo] password for nvidia: 
[Matrix Multiply Using CUDA] - Starting...
GPU Device 0: "Ampere" with compute capability 8.7

MatrixA(320,320), MatrixB(640,320)
Computing result using CUDA Kernel...
done
Performance= 617.91 GFlop/s, Time= 0.212 msec, Size= 131072000 Ops, WorkgroupSize= 1024 threads/block
Checking computed result for correctness: Result = PASS

NOTE: The CUDA samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.

Running custom applications inside a target-side Docker container 

Two files—devices.csv and drivers.csv—are provided within the RFS flashed onto the board for applications inside Docker containers to access the devices, drivers, and shared libraries. These files each have a list of devices, drivers, and shared libraries needed for an application to run successfully inside the Docker container. 

These files must be edited to reflect all the devices, drivers, and shared libraries the application needs to run custom applications inside the Docker containers. To access these two files, see /etc/nvidia-container-runtime/host-files-for-container.d/.

For example, the CUDA sample cudaNvSci does not run out of the box, unlike the MatrixMul sample. An attempt to run the CUDA sample inside the Docker container without any changes to drivers.csv results in the following error:

nvidia@tegra-ubuntu:/usr/local/cuda-11.4/samples/0_Simple/cudaNvSci$ sudo docker run --rm --runtime nvidia --gpus all -v $(pwd):$(pwd) -w $(pwd) ubuntu:20.04 ./cudaNvSci
./cudaNvSci: error while loading shared libraries: libnvscibuf.so.1: cannot open shared object file: No such file or directory

The CUDA sample does work outside the Docker container. The output of running the sample directly on the target is shown below:

nvidia@tegra-ubuntu:/usr/local/cuda-11.4/samples/0_Simple/cudaNvSci$ ./cudaNvSci
1 GPUs found
Loaded './lenaRGB.ppm', 1024 x 1024 pixels
[cudaNvSciSignal] GPU Device 0: "Ampere" with compute capability 8.7
 
[cudaNvSciWait] GPU Device 0: "Ampere" with compute capability 8.7
 
created NvSciBufObj
created NvSciBufImageObj
created NvSciSyncObj
Wrote 'lenaRGB_out.ppm'

The output (above) should result if the sample is run inside the Docker container. Instead, an error results from loading the shared library libnvscibuf.so.1. To fix this error, look at the contents of drivers.csv:

dir, /usr/lib/firmware/tegra23x
 
lib, /usr/lib/libcuda.so.1
lib, /usr/lib/libnvrm_gpu.so
lib, /usr/lib/libnvrm_mem.so
lib, /usr/lib/libnvrm_sync.so
lib, /usr/lib/libnvrm_host1x.so
lib, /usr/lib/libnvos.so
lib, /usr/lib/libnvsocsys.so
lib, /usr/lib/libnvtegrahv.so
lib, /usr/lib/libnvsciipc.so
lib, /usr/lib/libnvrm_chip.so
lib, /usr/lib/libnvcucompat.so

This list does not include libnvscibuf.so.1 that the sample requires to run inside the Docker container. 

At this point, tools like ldd and objdump can help to determine the shared libraries required by the sample so the drivers.csv file can be edited to include them. 

Running ldd on cudaNvSci results in the following output: 

nvidia@tegra-ubuntu:/usr/local/cuda-11.4/samples/0_Simple/cudaNvSci$ ldd ./cudaNvSci
    	linux-vdso.so.1 (0x0000ffffa9f2b000)
    	libcuda.so.1 => /usr/lib/libcuda.so.1 (0x0000ffffa8933000)
    	libnvscibuf.so.1 => /usr/lib/libnvscibuf.so.1 (0x0000ffffa88c1000)
    	libnvscisync.so.1 => /usr/lib/libnvscisync.so.1 (0x0000ffffa888f000)
    	librt.so.1 => /lib/aarch64-linux-gnu/librt.so.1 (0x0000ffffa8877000)
    	libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000ffffa8846000)
    	libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000ffffa8832000)
    	libstdc++.so.6 => /lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000ffffa864d000)
    	libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000ffffa8629000)
    	libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000ffffa84b6000)
    	/lib/ld-linux-aarch64.so.1 (0x0000ffffa9efb000)
    	libnvrm_gpu.so => /usr/lib/libnvrm_gpu.so (0x0000ffffa844a000)
    	libnvrm_mem.so => /usr/lib/libnvrm_mem.so (0x0000ffffa8431000)
    	libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000ffffa8386000)
    	libnvos.so => /usr/lib/libnvos.so (0x0000ffffa8365000)
    	libnvscicommon.so.1 => /usr/lib/libnvscicommon.so.1 (0x0000ffffa8350000)
    	libnvsciipc.so => /usr/lib/libnvsciipc.so (0x0000ffffa8325000)
    	libnvrm_host1x.so => /usr/lib/libnvrm_host1x.so (0x0000ffffa8303000)
    	libnvsocsys.so => /usr/lib/libnvsocsys.so (0x0000ffffa82ef000)
    	libnvtegrahv.so => /usr/lib/libnvtegrahv.so (0x0000ffffa82dc000)
    	libnvrm_sync.so => /usr/lib/libnvrm_sync.so (0x0000ffffa82c4000)
    	libnvrm_chip.so => /usr/lib/libnvrm_chip.so (0x0000ffffa82b0000)

This list shows all the shared libraries the cudaNvSci sample requires, including libnvscibuf.so.1. 

Now, edit drivers.csv to include the shared libraries that are not already present in the list. After adding the required libraries, the content of drivers.csv is as follows (with bolded lines newly added):

dir, /usr/lib/firmware/tegra23x
 
lib, /usr/lib/libcuda.so.1
lib, /usr/lib/libnvrm_gpu.so
lib, /usr/lib/libnvrm_mem.so
lib, /usr/lib/libnvrm_sync.so
lib, /usr/lib/libnvrm_host1x.so
lib, /usr/lib/libnvos.so
lib, /usr/lib/libnvsocsys.so
lib, /usr/lib/libnvtegrahv.so
lib, /usr/lib/libnvsciipc.so
lib, /usr/lib/libnvrm_chip.so
lib, /usr/lib/libnvcucompat.so
lib, /usr/lib/libnvscibuf.so.1
lib, /usr/lib/libnvscisync.so.1
lib, /lib/aarch64-linux-gnu/librt.so.1,
lib, /lib/aarch64-linux-gnu/libpthread.so.0
lib, /lib/aarch64-linux-gnu/libdl.so.2
lib, /lib/aarch64-linux-gnu/libstdc++.so.6
lib, /lib/aarch64-linux-gnu/libgcc_s.so.1
lib, /lib/aarch64-linux-gnu/libc.so.6
lib, /lib/aarch64-linux-gnu/libm.so.6,
lib, /usr/lib/libnvscicommon.so.1

After editing the drivers.csv file, the cudaNvSci sample is now able to run inside the Docker container:

nvidia@tegra-ubuntu:/usr/local/cuda-11.4/samples/0_Simple/cudaNvSci$ sudo docker run --rm --runtime nvidia --gpus all -v $(pwd):$(pwd) -w $(pwd) ubuntu:20.04 ./cudaNvSci
1 GPUs found
Loaded './lenaRGB.ppm', 1024 x 1024 pixels
[cudaNvSciSignal] GPU Device 0: "Ampere" with compute capability 8.7
 
[cudaNvSciWait] GPU Device 0: "Ampere" with compute capability 8.7
 
created NvSciBufObj
created NvSciBufImageObj
created NvSciSyncObj
Wrote 'lenaRGB_out.ppm'

Similarly, you can edit devices.csv and drivers.csv to add the devices and drivers required by custom applications to run them inside the Docker container on the target device. 

Conclusion

DRIVE OS 6.0.5 Linux now supports running Docker containers directly on the NVIDIA DRIVE AGX Orin hardware. Many samples should run inside the Docker container right after flashing, but some applications might require access to other devices and drivers, which can be accomplished by editing the devices.csv and drivers.csv files.