A recently developed AI platform is giving medical professionals screening for breast cancer a new, transparent tool for evaluating mammography scans. The research, creates an AI model that evaluates the scans and highlights parts of an image the algorithm finds relevant. The work could help medical professionals determine whether a patient needs an invasive—and often nerve-wracking—biopsy.

“If a computer is going to help make important medical decisions, physicians need to trust that the AI is basing its conclusions on something that makes sense,” Joseph Lo, professor of radiology at Duke and study coauthor said in a press release. “We need algorithms that not only work, but explain themselves and show examples of what they’re basing their conclusions on. That way, whether a physician agrees with the outcome or not, the AI is helping to make better decisions.”

One in every eight women in the US will develop invasive breast cancer during their lifetime. When detected early, a woman has a 93 percent or higher survival rate in the first 5 years. 

Mammography, which uses low-energy X-rays to examine breast tissue for diagnosis and screening, is an effective tool for early detection, but requires a highly skilled radiologist to interpret the scans. However, false negatives and positives do occur, resulting in missed diagnosis and up to 40% of biopsied lesions being benign. 

Using AI for medical imaging analysis has grown significantly in recent years and offers advantages in interpreting data. Implementing AI models also carries risks, especially when an algorithm fails. 

“Our idea was to instead build a system to say that this specific part of a potential cancerous lesion looks a lot like this other one that I’ve seen before,” said study lead and Duke computer science Ph.D. candidate Alina Barnett. “Without these explicit details, medical practitioners will lose time and faith in the system if there’s no way to understand why it sometimes makes mistakes.”

Using 1,136 images from 484 patients within the Duke University Health System, researchers trained the algorithm to locate and evaluate potentially cancerous areas. This was accomplished by training the models to identify unhealthy tissue, or lesions, which often appear as bright or irregular shapes with fuzzy edges on a scan.

Radiologists then labeled these images, teaching the algorithm to focus on the fuzzy edges, also known as margins. Often associated with quick-growing cancerous breast tumor cells, margins are a strong indicator of cancerous lesions. With these carefully labeled images, the AI can compare cancerous and benign edges, and learn to distinguish between them.

The AI model uses the cuDNN-accelerated PyTorch deep learning framework and can be run on two NVIDIA P100 or V100 GPUs.

The researchers found the AI to be as effective as other machine learning-based mammography models, but it holds the advantage of having transparency in its decision-making. When the model is wrong, a radiologist can see how the mistake was made.

According to the study, the model could also be a useful tool when teaching medical students how to read mammogram scans and for resource-constrained areas of the world lacking cancer specialists.

The code from the study is available through GitHub.

Read the study in Nature Machine Intelligence. >>
Read more. >>

Deep learning models require vast amounts of data to produce accurate predictions, and this need becomes more acute every day as models grow in size and complexity. Even large datasets, such as the well-known ImageNet with more than a million images, are not sufficient to achieve state-of-the-art results in modern computer vision tasks.

For this purpose, data augmentation techniques are required to artificially increase the size of a dataset by introducing random disturbances to the data, such as geometric deformations, color transforms, noise addition, and so on. These disturbances help produce models that are more robust in their predictions, avoid overfitting, and deliver better accuracy.

In medical imaging tasks, data augmentation is critical because datasets contain mere hundreds or thousands of samples at best. Models, on the other hand, tend to produce large activations that require a lot of GPU memory, especially when dealing with volumetric data such as CT and MRI scans. This typically results in training with small batch sizes on a small dataset. To avoid overfitting, more elaborate data preprocessing and augmentation techniques are required.

Preprocessing, however, often has a significant impact on the overall performance of the system. This is especially true in applications dealing with large inputs, such as volumetric images. These preprocessing tasks are typically run on the CPU due to simplicity, flexibility, and availability of libraries such as NumPy.

In some applications, such as segmentation or detection in medical images, the GPU utilization during training is usually suboptimal as data preprocessing is usually performed in the CPU. One of the solutions is to attempt to overlap data processing and training fully, but it is not always that simple.

Such a performance bottleneck leads to a chicken and egg problem. Researchers avoid introducing more advanced augmentations into their models due to performance reasons, and libraries don’t put the effort into optimizing preprocessing primitives due to low adoption.

GPU acceleration solution

You can improve the performance of applications with heavy data preprocessing pipelines significantly by offloading data preprocessing to the GPU. The GPU is typically underutilized in such scenarios but can be used to do the work that the CPU cannot complete in time. The result is better hardware utilization, and ultimately faster training.

Just recently, NVIDIA took 3 out of 10 top places in the MICCAI 2021 Brain Tumor Segmentation Challenge, including the winning solution. The winning solution managed to reach a GPU utilization as high as 98% and reduced the total training time by around 5% (30 minutes), by accelerating the preprocessing pipeline of the system (Figure 1).

This difference becomes more significant when you look at the NVIDIA submission for the MLPerf UNet3D benchmark. It used the same network architecture as in the BraTS21 winning solution but with a more complex data loading pipeline and larger input volumes (KITS19 dataset). The performance boost is an impressive 2x end-to-end training speedup when compared with the native pipeline (Figure 2).

This was made possible by NVIDIA Data Loading Library (DALI). DALI provides a set of GPU-accelerated building blocks, enabling you to build a complete data processing pipeline that includes data loading, decoding, and augmentation, and to integrate it with a deep learning framework of choice (Figure 3).

Volumetric image operations

Originally, DALI was developed as a solution for images classification and detection workflows. Later, it was extended to cover other data domains, such as audio, video, or volumetric images. For more information about volumetric data processing, see 3D Transforms or Numpy Reader.

DALI supports a wide range of image-processing operators. Some can also be applied to volumetric images. Here are some examples worth mentioning:

• Resize
• Warp affine
• Rotate
• Random object bounding box

To showcase some of the mentioned operations, we use a sample from the BraTS19 dataset, consisting of MRI scans labeled for brain tumor segmentation. Figure 4 shows a two-dimensional slice extracted from a brain MRI scan volume, where the darker region represents a region labeled as an abnormality.

Resize operator

Resize upscales or downscales the image to a desired shape by interpolating the input pixels. The upscale or downscale is configurable for each dimension separately, including the selection of the interpolation method.

Warp affine operator

Warp affine applies a geometric transformation by mapping pixel coordinates from source to destination with a linear transformation.

Warp affine can be used to perform multiple transformations (rotation, flip, shear, scale) in one go.

Rotate operator

Rotate allows you to rotate a volume around an arbitrary axis, provided as a vector, and an angle. It can also optionally extend the canvas so that the entire rotated image is contained in it. Figure 7 shows an example of a rotated volume.

Random object bounding box operator

Random object bounding box is an operator suited for detection and segmentation tasks. As mentioned earlier, medical datasets tend to be rather small, with target classes (such as abnormalities) occupying a comparatively small area. Furthermore, in many cases the input volume is much larger than the volume expected by the network. If you were to use random cropping windows for training, then the majority would not contain the target. This could cause the training convergence to slow down or bias the network towards false-negative results.

This operator selects pseudo-random crops that can be biased towards sampling a particular label. Connected component analysis is performed on the label map as a pre-step. Then, a connected blob is selected at random, with equal probability. By doing that, the operator avoids overrepresenting larger blobs.

You can also select to restrict the selection to the largest K blobs or specify a minimum blob size. When a particular blob is selected, a random cropping window is generated, within the range containing the given blob. Figure 8 shows this cropping window selection process.

The gain in learning speed can be significant. On the KITS19 dataset, nnU-Net achieves the same accuracy in 2134 in the test run epochs with the Random object bounding box operator as in 3,222 epochs with random crop.

Typically, the process of finding connected components is slow, but the number of samples in the data set can be small. The operator can be configured to cache the connected component information, so that it’s only calculated during the first epoch of the training.

Accelerate on your own

You can download the latest version of the prebuilt and tested DALI pip packages. The NGC containers for TensorFlow, PyTorch, and MXNet have DALI integrated. You can review the many examples and read the latest release notes for a detailed list of new features and enhancements.

See how DALI can help you accelerate data preprocessing for your deep learning applications. The best place to access is the NVIDIA DALI Documentation, including numerous examples and tutorials. You can also watch our GTC 2021 talk about DALI. DALI is an open-source project, and our code is available on the /NVIDIA/DALI GitHub repo. We welcome your feedback and contributions.

Scientists at NVIDIA and Cornell University introduced a hybrid unsupervised neural rendering pipeline to represent large and complex scenes efficiently in voxel worlds. Essentially, a 3D artist only needs to build the bare minimum, and the algorithm will do the rest to build a photorealistic world. The researchers applied this hybrid neural rendering pipeline to Minecraft block worlds to generate a far more realistic version of the Minecraft scenery.

Previous works from NVIDIA and the broader research community (pix2pix, pix2pixHD, MUNIT, SPADE) have tackled the problem of image-to-image translation (im2im)—translating an image from one domain to another. At first glance, these methods might seem to offer a simple solution to the task of transforming one world to another—translating one image at a time. However, im2im methods do not preserve viewpoint consistency, as they have no knowledge of the 3D geometry, and each 2D frame is generated independently. As can be seen in the images that follow, the results from these methods produce jitter and abrupt color and texture changes.

MUNIT      SPADE       wc-vid2vid      NSVF-W       GANcraft

Enter GANcraft, a new method that directly operates on the 3D input world. 

“As the ground truth photorealistic renderings for a user-created block world simply doesn’t exist, we have to train models with indirect supervision,” the researchers explained in the study. 

The method works by randomly sampling camera views in the input block world and then imagining what a photorealistic version of that view would look like. This is done with the help of SPADE, prior work from NVIDIA on image-to-image translation, and was the key component in the popular GauGAN demo. GANcraft overcomes the view inconsistency of these generated “pseudo-groundtruths” through the use of a style-conditioning network that can disambiguate the world structure from the rendering style. This enables GANcraft to generate output videos that are view consistent, as well as with different styles as shown in this image!

While the results of the research are demonstrated in Minecraft, the method works with other 3D block worlds such as voxels. The potential to shorten the amount of time and expertise needed to build high-definition worlds increases the value of this research. It could help game developers, CGI artists, and the animation industry cut down on the time it takes to build these large and impressive worlds.

If you would like a further breakdown of the potential of this technology, Károly Zsolnai-Fehér highlights the research in his YouTube series: Two Minute Papers:

GANcraft was implemented in the Imaginaire library. This library is optimized for the training of generative models and generative adversarial networks, with support for multi-GPU, multi-node, and automatic mixed-precision training. Implementations of over 10 different research works produced by NVIDIA, as well as pretrained models have been released. This library will continue to be updated with newer works over time. 

If you’d like to dive deeper, grab the code on GitHub from the Imaginare repository, see the overview of the framework or read the detailed research paper.

Stay updated on more exciting research from NVIDIA at NVIDIA Research.

The study authors include Zekun Hao, Arun Mallya, Serge Belongie, and Ming-Yu Liu.

NVIDIA announces the newest release of the CUDA development environment, CUDA 11.6. This release is focused on enhancing the programming model and performance of your CUDA applications. CUDA continues to push the boundaries of GPU acceleration and lay the foundation for new applications in HPC, visualization, AI, ML and DL, and data science.

CUDA 11.6 has several important features. This post offers an overview of the key capabilities:

• GSP driver architecture now default on Turing and Ampere GPUs
• New API to allow disabling nodes in instantiated graph
• Full support of 128-bit integer type
• Cooperative groups namespace update
• CUDA compiler update
• Nsight Compute 2022.1 release

CUDA 11.6 ships with the R510 driver, an update branch. CUDA 11.6 Toolkit is available to download.

GSP driver architecture

The GSP driver architecture is now the default driver mode for all listed Turing and Ampere GPUs. The older driver architecture is supported as a fallback. For more information, see R510 Driver Readme.

Instantiated Graph Node API additions

We added a new API, cudaGraphNodeSetEnabled, to allow disabling nodes in an instantiated graph. Support is limited to kernel nodes in this release.  A corresponding API, cudaGraphNodeGetEnabled, allows querying the enabled state of a node. We’ve also added the ability to disable NULL kernel graph node launches.

128-bit integer support

CUDA 11.6 includes the full release of 128-bit integer (__int128) data type, including compiler and developer tools support. The host-side compiler must support the __int128 type to use this feature.

Cooperative groups namespace

The cooperative groups namespace has been updated with new functions to improve consistency in naming, function scope, and unit dimension and size.

CUDA compiler

• Added -arch=native compilation option to target installed GPUs during compilation. This extends the existing -gencode=arch=compute_xx,code=sm_xx architecture specification
• Add the ability to create PTX files from nvlink

Deprecated features

• The cudaDeviceSynchronize() used for on-device fork and join parallelism is deprecated in preparation for a replacement programming model with higher performance. These functions continue to work in this release, but the tools emit a warning about the upcoming change.
• CentOS Linux 8 has reached End-of-Life on Dec 31, 2021, and support for this OS is now deprecated in the CUDA Toolkit. CentOS Linux 8 support will be completely removed in a future release. 

Additional resources

• GTC sessions:
  • CUDA New Features and Beyond, by Stephen Jones
  • Nearly Effortless CUDA Graphs, by Rob Van der Wijngaart and Jiajie Yao
  • A Deep Dive Into the Latest HPC Software, by Tim Costa
  • Multi-GPU Programming Models, by Jiri Kraus
• Blog posts:
  • Revealing New Features in the CUDA 11.5 Toolkit
  • Reducing Application Build Times Using C++ Compilation Aids
  • Employing CUDA Graphs in a Dynamic Environment, by Rob Van der Wijngaart
  • A Complete Overview of Nsight Developer Tools, by Chaitrali Joshi

On January 13, NVIDIA hosted an online workshop to engage with the NVIDIA DOCA developer community in China. The core team at NVIDIA and leading partner representatives joined the workshop to discuss the application scenarios of NVIDIA BlueField DPUs and the NVIDIA DOCA software framework for cloud, data center, and edge. The workshop focused on the requirements for DOCA developers in key industries, such as consumer Internet, cybersecurity, and higher education, and designed a plan to expand the DOCA developer community in China.

Since the June 2021 launch of the DOCA community in China, nearly 1,000 developers have registered for the DOCA Early Access Program, accounting for almost half of global registrations. BlueField DPUs and DOCA show great potential for adoption in China, with developer numbers continuing to grow.

In the second half of 2021, NVIDIA has held three online bootcamps to introduce the application and future technology evolution of the BlueField DPU and DOCA in modern data centers. They also explored the software stack, development environment, developer resources, developer guides, and reference applications of the DOCA Software Development Kit (SDK). More than 3,500 developers participated in the bootcamps. At the latest bootcamp, the newest services and applications released with DOCA 1.2 attracted much attention. Technical information and success stories posted on social media and in knowledge communities have many developers and industry professionals excited about future advancements.

NVIDIA is working with leading global platform providers and partners, such as Juniper Networks, Excelero, VMware, and Palo Alto Networks, to integrate and extend solutions based on BlueField DPU and the DOCA software framework. Through the workshop, NVIDIA will help enable developers in China to develop applications in scenarios such as Zero Trust Security, Morpheus AI Security, edge network service platforms, and high-speed distributed storage. A rich developer program in China will be launched in 2022.

DOCA Developer Bootcamp and Virtual Hackathon

Following the hackathons in Europe and North America, NVIDIA intends to host the first spring DOCA Developer Hackathon in China in the second quarter of 2022. Before the hackathon, NVIDIA will host an online bootcamp to teach contestants about BlueField DPU and DOCA programming skills.

NVIDIA will invite teams of developers from partners, customers, and academia to learn, collaborate, and accelerate their software designs under the guidance of NVIDIA expert mentors. The aim is to foster innovative, breakthrough software projects based on BlueField DPU and DOCA 1.2 in high-performance networking, virtualization, cybersecurity, distributed storage, accelerated AI, edge computing, and video streaming processing. The program will empower the developer community in China to create revolutionary data center infrastructure applications and services. After evaluation, NVIDIA will reward outstanding innovation teams.

DPU and DOCA Excellence Center

Leadtek (Shanghai) Information Technology Co., Ltd. and Shanghai Zentek Intelligent Technology Co., Ltd. are highly familiar with NVIDIA products and solutions, including deep learning applications in cloud, data center, and edge scenarios. They have partnered with the NVIDIA Deep Learning Institute to train partners and customers. 

As the first group of members of the NVIDIA authorized DPU and DOCA Excellence Center, the two partners have set up their own Excellence Centers and begun their pilots. During the pilots, each partner will independently build and operate a virtual development platform based on the BlueField-2 DPU, establish a third-party DPU development environment, provide an online practice development environment for DOCA developers in China, and contribute to the DPU and DOCA ecosystem with NVIDIA.

The implementation of the DOCA developer program in China will help grow the global DOCA developer community, facilitate talent development, and enhance the capabilities of developers. It will also boost the performance advantages of solutions based on BlueField DPU and the DOCA SDK, and accelerate time-to-market, creating greater value for customers and partners.

Apply now to join the DOCA developer community and get early access to the DOCA software framework.

New weather-forecasting research using AI is fast-tracking global weather predictions. The study, recently published in the Journal of Advances in Modeling Earth Systems, could help identify potential extreme weather 2–6 weeks into the future. Accurate predictions of extreme weather with a longer lead time give communities and critical sectors such as public health, water management, energy, and agriculture more time to prepare for and mitigate potential disasters.

Climate change is amplifying the intensity and frequency of extreme weather events, with 2021 shattering storm, heatwave, flood, and drought records across the globe. According to a recent NOAA report, last year the US experienced 20 separate climate-induced weather disasters, each totaling over $1 billion in damage. 

Short-term and seasonal weather forecasting can play a large role in decreasing the socioeconomic and human costs of extreme weather. In 2019, meteorologists warned local and national leaders in the Philippines of a torrential rainstorm looming about 3 weeks out. The forecast gave communities time to weatherize structures and evacuate before the Category 4 Typhoon hit, saving lives, and reducing overall damage to the region. 

Current weather forecasting relies on supercomputers processing large amounts of global data such as temperature, pressure, humidity, and wind speed. These systems require massive computational resources and take time to process. 

Also, according to the authors, the ability to accurately predict forecasts further out, from several weeks to months, decreases significantly. 

Looking to improve current weather forecasting the researchers aimed to create a computationally efficient model, capable of accurately predicting upcoming weather called the Deep Learning Weather Prediction (DLWP). Originally introduced in a paper published in 2020, the DLWP relies on an AI algorithm that learns and recognizes patterns in historical weather data based on global grids.

The current work refines the DLWP by training a deep convolutional neural network on two additional data points—temperature at the atmospheric boundary layer and total column water vapor. They also improved the grid resolution at the equator to approximately 1.4°. 

Running on a single cuDNN-accelerated TensorFlow deep learning framework on an NVIDIA V100 GPU, the model runs 320 ensemble 6-week forecasts in just 3 minutes. The algorithm can process a 1-week forecast in 1/10th of a second. 

The DLWP is able to produce realistic forecasting of weather events such as Hurricane Irma, a Category 4 storm that hit Florida and the Caribbean in 2017. While the speedy DLWP model matches the performance of current state-of-the-art weather forecasters 4 to 6 weeks into the future, it has limitations predicting precipitation and is less accurate in shorter lead times of 2–3 weeks. 

According to the study, the DLWP may also prove a valuable tool for supplementing spring and summer forecasts in the tropics, a region that challenges current weather models.

The open-source code is available on GitHub.

Read the study in Journal of Advances in Modeling Earth Systems. >>

Read more. >>

The NVIDIA platform is the most mature and complete platform for accelerated computing. In this post, I address the simplest, most productive, and most portable approach to accelerated computing. There are three approaches that you can take for programming GPUs (Figure 1).

CUDA C++ and Fortran are the innovation ground where NVIDIA can expose new hardware and software innovations, and where you can tune your applications to achieve the best possible performance on NVIDIA GPUs. Many developers assume that this is how NVIDIA expects everyone to program for GPUs.

Instead, we expect that developers coming to the NVIDIA platform for the first time will use standard, parallel programming languages, such as ISO C++, ISO Fortran, and Python. In this post, I highlight some successes in using this approach to parallel programming to demonstrate the most productive path to entering the NVIDIA CUDA ecosystem.

The foundation of the NVIDIA strategy is providing a rich, mature set of SDKs and libraries on which applications can be built. NVIDIA already provides highly tuned math libraries, such as cuBLAS, cuSolver, and cuFFT; core libraries, such as Thrust and libcu++; and communication libraries, such as NCCL and NVSHMEM, as well as other packages and frameworks on which you can build your applications.

On top of this, NVIDIA layers the three different programming approaches:

• Standard language parallelism, which is the subject of this post
• Languages for platform specialization, such as CUDA C++ and CUDA Fortran for obtaining the best possible performance on the NVIDIA platform
• Compiler directives, bridging the gap between these two approaches by enabling incremental performance optimization

Each of these approaches makes tradeoffs in terms of performance, productivity, and code portability. As they can all interoperate, you don’t have to use a particular model but can mix any or all as desired.

If you start writing code using parallelism in standard programming languages, then you can come to the NVIDIA platform or any other platform with baseline code that is already capable of running in parallel. This is why we have invested more than a decade collaborating in the standard language committees on the adoption of features to enable parallel programming without the need for additional extensions or APIs. Standard language parallelism is a rising tide that raises all boats.

ISO C++

The C++ programming language is consistently among the top programming languages in recent studies of programming trends. It has seen a significant increase in usage in scientific computing. The richness of its Standard Template Library makes it a highly productive language for new code development and, since the release of C++17, it has supported several important features for parallel programming.

I’ve seen several applications get refactored away from traditional for loops in favor of these C++ parallel algorithms. Here are the results from a few of them.

Lulesh

Lulesh is a hydrodynamics mini-app from Lawrence Livermore National Laboratory (LLNL), written in C++. The mini-app has several versions for evaluating different programming approaches, both in terms of the quality of the code and performance. We worked with the developers to rewrite their existing OpenMP-based code to use C++ Parallel Algorithms. Figure 2 shows an example of just one of the application’s important functions.

The code on the left uses OpenMP to parallelize the loops in the code across CPU threads. To maintain both a serial and parallel version of the code, the developers used #ifdef macros and compiler pragmas. The result is repeated code and the introduction of an additional API, OpenMP, into the source.

The code on the right is the same routine, but rewritten using the C++ transform_reduce algorithm. The resulting code is much more compact, making it less error prone, easier to read, and more maintainable. It also removes the dependency on OpenMP, relying instead on the C++ standard template library, while maintaining a single source code for all platforms. This code is fully ISO C++ compliant, capable of being built by any C++ compiler that supports C++17. As it turns out, it is faster too!

As a performance baseline, we used the OpenMP code running on all cores of an AMD EPYC 7742 processor and built with GCC. Rebuilding this baseline code using NVIDIA nvc++ compiler achieves essentially the same performance on the CPU.

If you instead build the ISO C++ code using the same version of GCC and running on the same CPU, the performance improves by roughly 50%, due to various improved overheads and opportunities for the compiler to better optimize the code.

This turns into a 2X performance improvement when building this code using nvc++ and running on the same CPU. This is already an exciting achievement but to top that off, you can build this same code, changing only a compiler option to target an NVIDIA GPU instead of a multicore CPU. Now that same code runs more than 13X faster by running on an NVIDIA A100 GPU. There’s a 13.5X performance improvement from the original code, running in parallel both on the CPU and GPU, using strictly ISO C++ code.

STLBM

Another example of an application using C++ Standard Parallelism is STLBM, a Lattice-Boltzmann solver from the University of Geneva. Professor Jonas Latt discussed this application in several GTC sessions, showing how code written in ISO C++ without any external SDK dependencies can run with multiple compilers and on multiple hardware platforms, including NVIDIA GPUs. For more information, see Fluid Dynamics on GPUs with C++ Parallel Algorithms: State-of-the-Art Performance through a Hardware-Agnostic Approach and Porting a Scientific Application to GPU Using C++ Standard Parallelism

His application achieves more than a 12X performance improvement using GPUs. What is notable is that his baseline for comparison is a source code that is parallel by default, using the parallel algorithms in the C++17 standard template library to express the parallelism inherent in the application.

He categorized the experience of using ISO C++ to program for GPUs as “a paradigm shift in cross-platform CPU/GPU programming.” Rather than writing an application that is serial by default and then adding parallelism later, his team has written an application that is ready for any parallel platform on which they wish to run.

NVIDIA is heavily invested in the continued development of parallelism and concurrency in C++ and has coauthored a variety of proposals for the upcoming C++23 specification to further improve your ability to write code that is parallel-first.

ISO Fortran

Fortran remains a language whose primary focus is on scientific and high performance computing. Originally, the FORmula TRANslator, Fortran provides a variety of advantages both to developers and compilers, and also has a huge existing code base for modeling and simulation codes.

Fortran began adding features to support parallel programming in Fortran 2008, enhanced these capabilities in Fortran 2018, and continues to refine them in the upcoming version, currently referred to as Fortran 202X. Just as with ISO C++, NVIDIA has been working with application developers to use standard language parallelism in Fortran to modernize their applications and make them parallel-first.

Computational chemistry

My colleague Jeff Hammond, in his FortranCon2021: Standard Fortran on GPUs and its utility in quantum chemistry codes session, presented some promising results using Fortran do concurrent loops in kernels taken from the NWChem application and also GAMESS, another computational chemistry application.

For NWChem, he isolated several performance-critical loops that perform tensor contractions and has written them using several programming models. On multicore CPUs, these tensor contractions use OpenMP for threading across CPU cores. For GPUs, there are versions available using OpenACC, OpenMP target offloading, and now Fortran do concurrent loops.

Figure 5 shows that the do concurrent loops perform at the same level as both OpenACC and OpenMP target offloading on NVIDIA GPUs but without the need to include these additional APIs in the application. This is all standard Fortran.

High-performance flux transport

At the recent Workshop for Accelerator Programming Using Directives (WACCPD), collocated at the SC21 conference, a team of developers from Predictive Science Inc. showed their results in refactoring one of their production codes, which previously used OpenACC to run on NVIDIA GPUs, using do concurrent loops.

They compared the results of building this purely ISO Fortran application using NVIDIA nvfortran, gfortran, and ifort. They concluded that, for their application when using the nvfortran compiler, pure Fortran gave the performance that they required without the need for any directives. Furthermore, this code could run in parallel on GPUs and multicore CPUs without modification.

This paper received the award for best paper at the workshop, even though it required no directives at all for accelerator programming. When asked whether they would continue the standard language parallelism approach in their other applications, the presenter replied that they already have plans to adopt this approach in other important applications for their company.

Python with Legate and cuNumeric

The Python language has had a meteoric rise in popularity over the past decade. It is now commonly used in machine learning, data science, and even traditional modeling and simulation applications. Although Python is not an ISO programming language, like C++ and Fortran, we are implementing the spirit of standard language parallelism in the Python language as well.

In his keynote address at GTC’21 Fall, NVIDIA CEO Jensen Huang introduced the alpha release of cuNumeric, a library that is modeled after NumPy and which enables features similar to what I have discussed for ISO C++ and Fortran. The NumPy package is so prevalent in Python development that it is a near certainty that any HPC application written in Python uses it.

The cuNumeric package, written on top of a package called Legate, enables NumPy applications to automatically scale their work not only onto GPUs but across GPUs in a large cluster. I’ve seen for several example applications that simply replacing references to NumPy in the code to instead refer to cuNumeric, I could weakly scale that application to the full size of the NVIDIA internal cluster, Selene, which is among the 10 fastest supercomputers in the world.

For more information about cuNumeric, see NVIDIA Announces Availability for cuNumeric Public Alpha and watch the GTC On-Demand session, Legate: Scaling the Python Ecosystem.

Conclusion

I hope this post has inspired you to see that GPU programming is not as difficult as you may have heard. If you use standard language parallelism, it may even be possible without any code changes at all.

NVIDIA is encouraging you to write applications parallel-first such that there is never a need to “port” applications to new platforms and standard language parallelism is the best approach to doing this, as it requires nothing more than the ISO standard languages. This is why we continue to invest in the ISO programming languages and in bringing even more features for parallelism and concurrency to these languages.

In summary, using standard language parallelism has the following benefits:

• Full ISO language compliance, resulting in more portable code
• Code that is more compact, easier to read, less error prone
• Code that is parallel by default, so it can run without modification on more platforms

Here are several talks from GTC’21 that can provide you with even more detail about this approach to parallel programming:

• Shifting through the Gears of GPU Programming: Understanding Performance and Portability Trade-offs
• Accelerated Computing with Standard C++, Python, and Fortran
• Legate: Scaling the Python Ecosystem

For more information, see the following resources:

• Learn more about the compiler support and other posts at HPC SDK.
• Download the HPC SDK software for free.

A point cloud is a data set of points in a coordinate system. Points contain a wealth of information, including three-dimensional coordinates X, Y, Z; color; classification value; intensity value; and time. Point clouds mostly come from lidars that are commonly used in various NVIDIA Jetson use cases, such as autonomous machines, perception modules, and 3D modeling.

One of the key applications is to leverage long-range and high-precision data sets to achieve 3D object detection for perception, mapping, and localization algorithms.

PointPillars is one the most common models used for point cloud inference. This post discusses an NVIDIA CUDA-accelerated PointPillars model for Jetson developers. Download the CUDA-PointPillars model today.

What is CUDA-Pointpillars

In this post, we introduce CUDA-Pointpillars, which can detect objects in point clouds. The process is as follows:

• Base preprocessing: Generates pillars.
• Preprocessing: Generates BEV feature maps (10 channels).
• ONNX model for TensorRT: An ONNX mode that can be implemented by TensorRT.
• Post-processing: Generates bounding boxes by parsing the output of the TensorRT engine.

Base preprocessing

The base preprocessing step converts point clouds into base feature maps. It provides the following components:

• Base feature maps
• Pillar coordinates: Coordinates of each pillar.
• Parameters: Number of pillars.

Preprocessing

The preprocessing step converts the basic feature maps (four channels) into BEV feature maps (10 channels).

ONNX model for TensorRT

The native point pillars from OpenPCDet were modified for the following reasons:

• Too many small operations, with low memory bandwidth.
• Some operations, like NonZero, are not supported by TensorRT.
• Some operations, like ScatterND, have low performance.
• They use “dict” as input and output, which cannot export ONNX files.

To export ONNX from native OpenPCDet, we modified the model (Figure 4).

You can divide the whole ONNX file into the following parts:

• Inputs: BEV feature maps, pillar coordinates, parameters. These are all generated in preprocessing.
• Outputs: Class, Box, Dir_class. These are parsed by post-processing to generate a bounding box.
• ScatterBEV: Converts point pillars (1D) into a 2D image, which can work as a plug-in for TensorRT.
• Others: Supported by TensorRT. [OTHER WHAT?]

Post-processing

The post-processing parses the output of the TensorRT engine (class, box, and dir_class) and output-bounding boxes. Figure 6 shows example parameters.

Using CUDA-PointPillars

To use CUDA-PointPillars, provide the ONNX mode file and data buffer for the point clouds:

    std::vector nms_pred;
    PointPillar pointpillar(ONNXModel_File, cuda_stream);
    pointpillar.doinfer(points_data, points_count, nms_pred);

Converting a native model trained by OpenPCDet into an ONNX file for CUDA-Pointpillars

In our project, we provide a Python script that can convert a native model trained by OpenPCDet into am ONNX file for CUDA-Pointpillars. Find the exporter.py script in the /tool directory of CUDA-Pointpillars.

To get a pointpillar.onnx file in the current directory, run the following command:

$ python exporter.py --ckpt ./*.pth

Performance

The table shows the test environment and performance. Before the test, boost CPU and GPU.

Get started with CUDA-PointPillars

In this post, we showed you what CUDA-PointPillars is and how to use it to detect objects in point clouds.

Because native OpenPCDet cannot export ONNX and has too many small operations with low performance for TensorRT, we developed CUDA-PointPillars. This application can export native models trained by OpenPCDet to a special ONNX model and inference the ONNX model by TensorRT.

Download CUDA-PointPillars today.

Transportation monitoring systems, healthcare, and retail have all benefited greatly from intelligent video analytics (IVA). DeepStream is an IVA SDK. DeepStream enables you to attach and detach video streams in runtime without affecting the entire deployment.

This post discusses the details of stream addition and deletion work with DeepStream. I also provide an idea about how to manage large deployments centrally across multiple isolated datacenters, serving multiple use cases with streams coming from many cameras.

The NVIDIA DeepStream SDK is a streaming analytics toolkit for multisensor processing. Streaming data analytics use cases are transforming before your eyes. IVA is of immense help in smarter spaces. DeepStream runs on discrete GPUs such as NVIDIA T4, NVIDIA Ampere Architecture and on system on chip platforms such as the NVIDIA Jetson family of devices.

DeepStream has flexibility that enables you to build complex applications with any of the following:

• Multiple deep learning frameworks
• Multiple streams
• Multiple models combining in series or in parallel to form an ensemble
• Multiple models working in tandem
• Compute at different precisions
• Custom preprocessing and post-processing
• Orchestration with Kubernetes

DeepStream application can have multiple plug-ins, as shown in Figure 1. Each plug-in as per the capability may use GPU, DLA, or specialized hardware.

DeepStream is fundamentally built to allow deployment at scale, making sure throughput and accuracy at any given time. The scale of any IVA pipeline depends on two major factors:

• Stream management
• Compute capability

Stream management is a vital aspect of any large deployment with many cameras. Any large deployment cannot be brought down to add/remove streams. Such large deployment must be made failsafe to handle spurious streams in runtime. Also, the deployment is expected to handle runtime attachment/detachment of the use case to the pipeline running with specific models.

This post helps you in understanding the following aspects of stream management:

• Stream consumption with DeepStream Python API
• Adding and removing streams in runtime
• Attaching specific stream to pipeline with specific models in runtime
• Stream management on large-scale deployment involving multiple data centers

As the application grows in complexity, it becomes increasingly difficult to change. A well-thought-out development strategy from the beginning can help a long way. In the next section, I discuss different ways to develop a DeepStream application briefly. I also discuss how to manage streams/use-case allocation and deallocation and consider some of the best practices.

DeepStream application development

DeepStream enables you to create seamless streaming pipelines for AI-based video, audio, and image analytics. DeepStream gives you the choice of developing in C or Python, providing them more flexibility. DeepStream comes with several hardware-accelerated plug-ins. DeepStream is derived From Gstreamer and offers and unified API between Python and C language.

Python and C API for DeepStream are unified. This means any application developed in Python can be easily converted to C and the reverse. Python and C provide all the levels of freedom to the developer. With DeepStream Python and C API, it is possible to design dynamic applications that handle streams and use-cases in runtime. Some example Python applications are available at: NVIDIA-AI-IOT/deepstream_python_app.

The DeepStream SDK is based on the GStreamer multimedia framework and includes a GPU-accelerated plug-in pipeline. Plug-ins for video inputs, video decoding, image preprocessing, NVIDIA TensorRT-based inference, object tracking, and display are included in the SDK to make the application development process easier. These features can be used to create multistream video analytics solutions that are adaptable.

Plug-ins are the core building block with which to make pipelines. Each data buffer in-between the input (that is, the input of the pipeline, for example, camera and video files) and output (for example, the screen display) is passed through plug-ins. Video decoding and encoding, neural network inference, and displaying text on top of video streams are examples of plug-ins. The connected plug-in constitutes a pipeline.

Pads are the interfaces between plug-ins. When data flows from one plug-in to another plug-in in a pipeline, it flows from the Source pad of one plug-in to the Sink pad of another. Each plug-in might have zero, one, or many source/sink components.

The earlier example application consists of the following plug-ins:

• GstUriDecodebin: Decodes data from a URI into raw media. It selects a source plug-in that can handle the given scheme and connects it to decodebin.
• Nvstreammux: The Gst-nvstreammux plug-in forms a batch of frames from multiple input sources.
• Nvinfer: The Gst-nvinfer plug-in does inferencing on input data using TensorRT.
• Nvmultistream-tiler: The Gst-nvmultistreamtiler plug-in composites a 2D tile from batched buffers.
• Nvvideoconvert: Gst-nvvideoconvert performs scaling, cropping, and video color format conversion.
• NvDsosd: Gst-nvdsosd draws bounding boxes, text, and region of interest (ROI) polygons.
• GstEglGles: EglGlesSink renders video frames on an EGL surface (xOverlay interface and Native Display).

Each plug-in can have one or more source and sink pads. In this case, when the streams are added, the Gst-Uridecodebin plug-in gets added to the pipeline, one for each stream. The source component from each Gst-Uridecodebin plug-in is connected to each sink component on the single Nv-streammux plug-in. Nv-streammux creates batches from the frames coming from all previous plug-ins and pushes them to the next plug-in in the pipeline. Figure 3 shows how multiple camera streams are added to the pipeline.

Buffers carry the data through the pipeline. Buffers are timestamped, contain metadata attached by various DeepStream plug-ins. Buffer carries information such as how many plug-ins are using it, flags, and pointers to objects in memory.

DeepStream applications can be thought of as pipelines consisting of individual components plug-ins. Each plug-in represents a functional block like inference using TensorRT or multistream decode. Where applicable, plug-ins are accelerated using the underlying hardware to deliver maximum performance. DeepStream’s key value is in making deep learning for video easily accessible, to allow you to concentrate on quickly building and customizing efficient and scalable video analytics applications.

Runtime stream addition/deletion application

DeepStream provides sample implementation for runtime add/delete functionality in python and C language. The samples are located at the following locations:

• DeepStream source add remove (C): https://github.com/NVIDIA-AI-IOT/deepstream_reference_apps/tree/master/runtime_source_add_delete
• DeepStream source add remove (Python): https://github.com/NVIDIA-AI-IOT/deepstream_python_apps/tree/master/apps/runtime_source_add_delete

These applications are designed keeping simplicity in mind. These applications take one input stream, and the same stream is added multiple times to the running pipeline after a set interval of time. This is how a specified number of streams are added to the pipeline without restarting the application. Eventually, each stream is removed at every interval of time. After the last stream is removed, the application gracefully stops.

To start with the sample applications, follow these steps.

To create a Python-based application

1. Pull the DeepStream Docker image from ngc.nvidia.com.
2. Run git clone on the Python application repository within the Docker container.
3. Go to the following location within the Docker container:  deepstream_python_apps/apps/runtime_source_add_delete
4. Set up the Python prerequisites.
5. Go to apps/runtime_source_add_delete and execute the application as follows:

python3 deepstream-test-rt-src-add-del.py 
python3 deepstream_rt_src_add_del.py file:///opt/nvidia/deepstream/deepstream-/samples/streams/sample_720p.mp4

To create a C-based application

1. Pull the DeepStream Docker image from ngc.nvidia.com.  :
2. Run git clone on the C application repository at /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/ within the Docker container.
3. Go to deepstream_reference_apps/runtime_source_add_delete and compile and run the application as follows:

make
./deepstream-test-rt-src-add-del 
./deepstream-test-rt-src-add-del file:///opt/nvidia/deepstream/deepstream-/samples/streams/sample_720p.mp4

Application aspect: Runtime camera add-remove

DeepStream Python or C applications usually take input streams as a list of arguments while running the script. After code execution, a sequence of events takes place that eventually adds a stream to a running pipeline.

Here, you use the uridecodebin plug-in that decodes data from a URI into raw media. It selects a source plug-in that can handle the given scheme and connects it to a decode bin.

Here’s the list of sequence that takes place to register any stream:

1. The source bin is created from Curidecodebin plug-in by the function create_uridecode_bin. The function create_uridecode_bin takes the first argument source_id, which is an integer, and the second argument is rtsp_url. In this case, this integer is the order of the stream from 1…..N. This integer is used to create a uniquely identifiable source-bin name as source-bin-1, source-bin-2,… source-bin-N.
2. g_source_bin_list dictionary maps between the source-bin and id value.
3. After the source bin is created, the RTSP stream URLs from arguments to the program are attached to this source bin.
4. Later, the source-bin value of uridecodebin is linked to the sink-bin of the next plug-in, streammux.
5. Such multiple uridecodebin plug-ins are created, each for one stream, and attached to streammux plug-in.

The following code example shows the minimal code in Python to attach multiple streams to a DeepStream pipeline.

g_source_bin_list = []
for i in range(num_sources):
    print("Creating source_bin ",i," \n ")
    uri_name=argv[i]
    if uri_name.find("rtsp://") == 0 :
        is_live = True
    #Create source bin and add to pipeline
    source_bin=create_uridecode_bin(i, uri_name)
    g_source_bin_list[rtsp[i]] = source_bin
    pipeline.add(source_bin)

In a more organized application, these lines of code responsible for stream addition are shifted to a function that takes two arguments to attach a stream: stream_id and rtsp_url. You can call such a function anytime and append more streams to the running application.

Similarly, when the stream must be detached from the application, the following events take place:

1. source_id of the already attached stream is given to the function stop_release_source.
2. sink-pad of streammux attached to the source_id to be released is detached from the source bin of uridecodebin.
3. The source bin of uridecodebin is then removed from the pipeline.
4. The active source count is decreased by one.

The following code example shows a minimal code for Python and C to detach streams from a DeepStream pipeline.

def stop_release_source(source_id):
    pad_name = "sink_%u" % source_id
    print(pad_name)
    #Retrieve sink pad to be released
    sinkpad = streammux.get_static_pad(pad_name)
    #Send flush stop event to the sink pad, then release from the streammux
    sinkpad.send_event(Gst.Event.new_flush_stop(False))
    streammux.release_request_pad(sinkpad)
    #Remove the source bin from the pipeline
    pipeline.remove(g_source_bin_list[source_id])
    source_id -= 1
    g_num_sources -= 1

Deployment aspect: Runtime camera and use case management

Earlier, I discussed how to add and remove streams from the code. There are a few more factors, considering the deployment aspect.

Previously, you took all the input streams with command-line arguments. However, after the program is executed and while it is in deployment, you cannot provide any additional argument to it. How do you pass instructions to the running program on which stream to attach or detach?

Deployment required additional code that takes care of periodically checking whether there are new streams available that must be attached. The following streams should be deleted:

• Stream no longer requires monitoring.
• Camera issues lead to no streams.
• Previously attached stream must be used for another use case.

In the case of multiple data centers for stream processing, give priority to the stream source nearest to the data center.

The DeepStream pipeline runs in the main thread. A separate thread is required to check for the stream to be added or deleted. Thankfully, Glib has a function named g_timeout_add_seconds. Glib is the GNU C Library project that provides the core libraries for the GNU system and GNU/Linux systems, as well as many other systems that use Linux as the kernel.

g_timeout_add_seconds (set) is a function to be called at regular intervals when the pipeline is running. The function is called repeatedly until it returns FALSE, at which point the timeout is automatically destroyed, and the function is not called again.

guint g_timeout_add_seconds (guint interval, GSourceFunc function, gpointer data);

g_timeout_add_seconds takes three inputs:

• Interval: The time between calls to the function, in seconds.
• function: The function to call.
• data: The data and arguments to pass to the function.

For example, you call a function watchDog and it takes GSourceBinList. A dictionary map between streamURL and streamId. streamId is an internal ID (Integer) that gets generated after the stream is added to the pipeline. The final caller function looks like the following code example:

guint interval = 10;
guint g_timeout_add_seconds (interval, watchDog, GSourceBinList,  argv);

As per the current interval setting, the watchDog function is called every 10 seconds. A database must be maintained to manage and track many streams. Such an example database table is shown in Table 1. The function watchDog can be used to query a database where a list of all the available streams against their current state and use case is maintained.

Here’s an example of the bare minimum database structure (SQL/no-SQL) needed to manage many streams at the same time:

• Source ID: A unique ID, which is also the sink pad ID of nvstreammux where it is connected to. source_id would be useful to monitor nv-gst events, for example, pad added deleted EOS for each stream. Remember that in the earlier simple app, you considered making source bin as source-bin-1, source-bin-2,… source-bin-N in order of argument input. You use the same method with many cameras and track all active source bins in the application scope.
• RTSP URL: The URL that the source plug-in should use.
• Stream state: Helps in managing the states of the stream such as ON or OFF. The database client must also be able to change the camera, such as BAD STREAM, NO STREAMm CAMERA FAULT, and so on, according to what is perceived by the client. This can help in instant maintenance.
• Use case: Assigns a use case to the camera. This use case is checked and only those cameras whose model is currently active are attached.
• Camera Location:  Helps with the localization of the compute based on the cameras’ location. This check avoids unnecessary capture from a camera located at distant locations and could be better assigned to other nearby compute clusters.
• Taken: Assume that deployment is multiple GPUs with multiple nodes. When a DeepStream application running on any machine and any GPU adds any source, it sets the flag to True. This prevents another instance from repeatedly adding the same source again.

Maintaining a schema as described enables easy dashboard creation and monitoring from a central place.

Returning to the watchDog function, here’s the pseudo-code to check for the stream state and attach a new video stream according to the location and use cases:

FUNCTION watchDog (Dict: GSourceBinList)
INITIALIZE streamURL ⟵ List ▸ Dynamic list of stream URLs
INITIALIZE streamState ⟵ List ▸ Dynamic list of state corresponding to stream URLs
INITIALIZE streamId ⟵ Integer ▸ variable to store id for new stream
streamURL, streamState := getCurrentCameraState()

FOR X = 1 in length(streamState)
	IF ((streamURL[X] IN gSourceBinList.keys()) & (streamState[X] == "OFF"))
			stopReleaseSource(streamURL[X]) ▸ Detach stream

streamURL, streamState := getAllStreamByLocationAndUsecase()
FOR Y = 1 in length(streamState)
	IF ((streamURL[Y] NOT IN gSourceBinList.keys()) & (streamState [Y] == "ON")
			streamId := addSource(streamURL[Y]) ▸ Add new stream
			GSourceBinList(streamURL, streamId) ▸ update mappings
RETURN GSourceBinList

• An application enters the main function after module loading and global variable initialization.
• In the main function, the local modules and variables are initialized.
• As the application starts for the first time, it requests the list of streams from the database after location and use case filters are applied.
• After receiving the stream list ,all the plug-ins of the DeepStream pipeline are initialized, linked, and set to the PLAY state. At this point, the application is running with all the streams provided.
• After every set interval of time, a separate thread checks for the state of the current stream in the database. If the state of any already-added stream is changed to OFF in the database, the stream is released. The thread also checks if a new camera is listed in the database with ON state, the stream is added to the DeepStream pipeline after applying location and use case filter.
• After the stream is added, the flag in the Taken column of the database must be set to True so that no other process can add the same stream again.

Figure 4 shows the overall flow of the functional calls required to efficiently add remove camera stream and attach to the server running with an appropriate model.

Just changing the number of sources does not help, as downstream components to the source must be able to change their properties according to the number of streams. For this purpose, components of the DeepStream application are already optimized to change properties in runtime.

However, many of the plug-ins use batch size as a parameter during initialization to allocate compute/memory resources. In this case, we recommend specifying maximum batch size while executing the application. Table 2 shows a few such plug-in examples:

You can explicitly change the property when the number of streams is detected. To manually tweak the properties of the plug-in at runtime, use the set_property function in Python and C or the g_object_set function in C.

Best practices

• Always check your stream properties before adding to the pipeline. Stream properties can be checked with the gst-discoverer-1.0 command-line utility. It accepts a URI from the command line and prints all information regarding the stream. It is useful to find out what container and codecs have been used to produce the media, and therefore what plug-ins you must put in a pipeline to play it. Gst-Discover can be used with Python and C by using the respective APIs.
• Profile the DeepStream application when it is developed. This is the first step in optimizing and tuning your application. Profiling helps in the understanding of an application’s performance characteristics and can easily identify parts of the code that present opportunities for improvement. Find hotspots and bottlenecks in your application to help you decide where to focus your optimization efforts.
• Count the maximum number of streams that can be run on GPU by profiling the application. At runtime, make sure that you keep the maximum stream below the maximum supported number so that your application performance remains stable.

To increase performance, consult the DeepStream troubleshooting manuals.

For more information, see the following resources:

• NVIDIA DeepStream SDK Quickstart Guide
• GStreamer Tutorials
• DeepStream Basics: iPython notebooks
• DeepStream performance optimization cycle lab

Automation is the key to increasing operational efficiency and lowering OpEx, but it does not guarantee a successful data center deployment. While automation can confirm configuration integrity and prevent human errors in repetitive changes, it can’t validate intent and network requirements. Therefore, automation must be tested and validated before deployment, and the NVIDIA way of doing this is with a data center digital twin.

What is a data center digital twin network?

A data center digital twin network is a 1:1 simulation of a physical network environment, with logical instances of every switch, server, and cable. This enables it to be used for validating routing (BGP, EVPN), security policy compliance, automation, monitoring tools, and upgrade procedures.

This digital twin is hosted in the cloud, enabling teams to test their configuration at scale without the overhead of physical infrastructure. Data center digital twins offer a number of benefits:

• Decreases time to deployment
• Decreases network downtime
• Decreases lab costs
• Decreases need for network hardware and build while waiting on hardware to arrive
• Increases creativity and collaboration (design, monitoring, change management)
• Enhances the value of physical infrastructure–by giving more capabilities
• Simulates true-to-reality infrastructure
• Continuous integration–fixes and changes can be implemented and tested on an ongoing basis

How do I create a data center digital twin?

NVIDIA Air is a free platform for creating network digital twins. These digital twins can be clones of existing topologies, prebuilt topologies, or custom designed networks that can scale to 1000s of switches and servers. Each server and switch in the digital twin can be spun up in the NVIDIA Air cloud hosted environment for IT teams to extract the value of testing to its full potential.

Prebuilt network automation

Every developer values reusable sample code, and NVIDIA has Production Ready Network Automation. We publish working Ansible playbooks for complete leaf/spine topologies with BGP & EVPN all set up for you. These playbooks are built for the NetDevOps approach of Continuous Integration and are the same playbooks our professional services team uses. The playbooks are constantly updated based on learnings & best practices from actual customer deployments-and we have made our Production Ready Automation assets available free of charge.

Test your automation

Testing is a tradeoff between risk and cost. On one hand, to fully validate network functionality and reduce the risk associated with change management, the test network needs to be similar to the production network. On the other hand, creating a physical replica of the production environment is expensive both in CapEx and OpEx.

Using a virtual replica via a data center digital twin can significantly reduce the costs associated with such testing.

IT teams can integrate the data center digital twin into their CI/CD pipeline, deploy new changes, validate the configuration using NetQ and deploy to production confidently. This level of integration helps drive down the cost of validation even further.

Automate your testing

To shorten the time to deployment and decrease the risk of downtime, IT teams use NVIDIA Air to automate their testing process.

In addition to testing an ad hoc change, every change goes through a set of regression tests to eliminate degradation of the current functionality. Once both regression and ad hoc tests are passed, the ad hoc test is added to the regression tests suite and validated in future deployments.

Get started

Help your team learn best practices by testing changes in a risk-free environment by building your own data center digital twin. Easy to work with and free to use! Get started at NVIDIA Air.

For more information, see the following resources:

• NVIDIA Air
• Production Ready Automation
• SONiC in the Cloud (blog)
• Close Knowledge Gaps and Elevate Training with Digital Twin NVIDIA Air.
• Cumulus in the Cloud
• Air documentation

RAPIDS Accelerator for Apache Spark v21.10 is now available! As an open source project, we value our community, their voice, and requests. This release constitutes community requests for operations that are ideally suited for GPU acceleration. 

Important callouts for this release:

• Speed up – performance improvements and cost savings.
• New Functionality – new I/O and nested datatype Qualification and Profiling tool features. 
• Community Updates – updates to the spark-examples repository.

Speed up

RAPIDS Accelerator for Apache Spark is growing at a great pace in both functionality and performance. Standard industry benchmarks are a great way to measure performance over a period of time but another barometer to measure performance is to measure performance of common operators that are used in the data preprocessing stage or in data analytics.

We used four such queries shown in the chart below:

• Count Distinct: a function used to estimate the number of unique page views or unique customers visiting an e-commerce site.
• Window: a critical operator necessary for preprocessing components in analyzing timestamped event data in marketing or financial industry.
• Intersect: an operator used to remove duplicates in a dataframes.
• Cross-join: A common use for a cross join is to obtain all combinations of items.

These queries were run on a Google Cloud Platform (GCP) machine with 2xT4 GPUs each with 104GB RAM. The dataset used was of size 3TB with multiple different data types. More information about the setup and the queries can be found in the spark-rapids-examples repository on GitHub. These four queries show not only performance and cost benefits but also the range of speed-up (27x to 1.5x) varies depending on compute intensity. These queries vary in compute and network utilization similar to a practical use case in data preprocessing.

New functionality

Plug-in

Most Apache Spark users are aware that Spark 3.2 was released this October. The v21.10 release has support for Spark 3.2 and CUDA 11.4. In this release, we focused on expanding support for I/O, nested data processing and machine learning functionality. RAPIDS Accelerator for Apache Spark v21.10 released a new plug-in jar to support machine learning in Spark. 

Currently, this jar supports training for the Principal Component Analysis algorithm. The ETL jar extended the input type support for Parquet and ORC. It now also provides users with the functionality to use HashAggregate, Sort, Join SHJ and Join BHJ on nested data. In addition to support for nested datatypes a performance test was also run.

In the figure below, we show that the speed-up is observed for two queries using nested data type input. Some other interesting features that were added in v21.10 are, pos_explode, create_map and so on. Please refer to RAPIDS Accelerator for Apache Spark’s documentation for a detailed list of new features.

Profiling & qualification tool

In addition to the plug-in, multiple new features were also added to RAPIDS Accelerator for Apache Spark’s Qualification and Profiling tool. The Qualification tool can now report the different nested datatypes and write data formats present. It now also includes support for adding conjunction and disjunction filters, and filter based Regular Expressions and usernames.

The Qualifications tool is not the only one with new tricks: the Profiling tool now provides structured output format and support to scale and run a large number of event logs.

Community updates

We are excited to announce that we are in public preview on Azure and we welcome Azure users to try RAPIDS Accelerated for Apache Spark on Azure Synapse.

We invite you to view our talks presented at NVIDIA’s flagship event, GTC, held from Nov. 8-11, to learn how AI is transforming the world. The RAPIDS Accelerator team presented two talks; Accelerating Apache Spark gives an overview of new functionality and other upcoming features. Also, Discover Common Apache Spark Operations Turbocharged with RAPIDS and NVIDIA GPUs covers many microbenchmarks on Apache Spark.

Coming soon

The upcoming versions will introduce support for 128-bit decimal datatype, inference support for the Principle Component Analysis algorithm and additional nested data type support for multi-level struct and maps. 

In addition, lookout for MIG support for NVIDIA Ampere Architecture based GPUs (A100/A30) which can help improve throughput on running multiple spark jobs with A100. As always, we want to thank all of you for using RAPIDS Accelerator for Apache Spark and we look forward to hearing from you. Reach out to us on GitHub and let us know how we can continue to improve your experience using RAPIDS Accelerator on Apache Spark.

Deploying a Model for Inference at Production Scale

A lot of love goes into building a machine-learning model. Challenges range from identifying the variables to predict to experimentation finding the best model architecture to sampling the correct training data. But, what good is the model if you can’t access it?

Enter the NVIDIA Triton Inference Server. NVIDIA Triton helps data scientists and system administrators turn the same machines you use to train your models into a web server for model prediction. While a GPU is not required, an NVIDIA Triton Inference Server can take advantage of multiple installed GPUs to quickly process large batches of requests.

To get hands-on practice with a live server, the NVIDIA Deep Learning Institute (DLI) is offering a 4-hour, self-paced course titled Deploying a Model for Inference at Production Scale.

MLOps Overview

NVIDIA Triton was created with Machine Learning Operations, or MLOps, in mind. MLOps is a relatively new field evolved from Developer Operations, or DevOps, to focus on scaling and maintaining machine-learning models in a production environment. NVIDIA Triton is equipped with features such as model versioning for easy rollbacks. It is also compatible with Prometheus to track and manage server metrics such as latency and request count.

Course Information

This course covers an introduction to MLOps coupled with hands-on practice with a live NVIDIA Triton Inference Server. 

Learning objectives include:

• Deploying neural networks from a variety of frameworks onto a live NVIDIA Triton Server.
• Measuring GPU usage and other metrics with Prometheus.
• Sending asynchronous requests to maximize throughput.

Upon completion, developers will be able to deploy their own models on an NVIDIA Triton Server.

Start Learning >>

For additional hands-on training visit the NVIDIA Deep Learning Institute.

Deploying a Model for Inference at Production Scale

A lot of love goes into building a machine-learning model. Challenges range from identifying the variables to predict to experimentation finding the best model architecture to sampling the correct training data. But, what good is the model if you can’t access it?

Enter the NVIDIA Triton Inference Server. NVIDIA Triton helps data scientists and system administrators turn the same machines you use to train your models into a web server for model prediction. While a GPU is not required, an NVIDIA Triton Inference Server can take advantage of multiple installed GPUs to quickly process large batches of requests.

To get hands-on practice with a live server, the NVIDIA Deep Learning Institute (DLI) is offering a 4-hour, self-paced course titled Deploying a Model for Inference at Production Scale.

MLOps Overview

NVIDIA Triton was created with Machine Learning Operations, or MLOps, in mind. MLOps is a relatively new field evolved from Developer Operations, or DevOps, to focus on scaling and maintaining machine-learning models in a production environment. NVIDIA Triton is equipped with features such as model versioning for easy rollbacks. It is also compatible with Prometheus to track and manage server metrics such as latency and request count.

Course Information

This course covers an introduction to MLOps coupled with hands-on practice with a live NVIDIA Triton Inference Server. 

Learning objectives include:

• Deploying neural networks from a variety of frameworks onto a live NVIDIA Triton Server.
• Measuring GPU usage and other metrics with Prometheus.
• Sending asynchronous requests to maximize throughput.

Upon completion, developers will be able to deploy their own models on an NVIDIA Triton Server.

Start Learning >>

For additional hands-on training visit the NVIDIA Deep Learning Institute.

Consumer Electronics Show (CES), an annual trade show organized by the Consumer Technology Association, brings together thought leaders, products, and technologies working to transform traffic and roadways, an important cross-section of daily life.

With limited roadways and growing populations, cities increasingly look to automation, and simulation for managing traffic and constrained infrastructure. NVIDIA partners worldwide are deploying the NVIDIA Metropolis video analytics platform, leveraging real-time sensors and AI to design more efficient roadways and optimize traffic safety and operations. 

The following NVIDIA Metropolis partners are showcasing how they help manage traffic with AI at CES. 

Asilla: Asilla develops behavior recognition AI solutions that use posture estimation technology to enhance public safety. Asilla is helping cities and a wide range of industries improve safety and security by detecting abnormal behavior in real-time and enabling prompt response to events. Check out Asilla at booth #51127 in Sands Hall.

Bitsensing: Bitsensing uses GPU and radar technology to connect cities, roads, buildings, and individuals, building the complete autonomous, connected environment. With cutting-edge imaging radar technology, Bitsensing accelerates creating the ultimate smart city to bring a new level of reliability and convenience. Visit Bitsensing at booth #61059 in Eureka Park.

Ekin: Ekin develops the next generation of smart city solutions to optimize safety and security for cities. A forward-thinking provider of quantitative data to cities based on cutting-edge artificial intelligence technology, with a focus on traffic management, smart parking, smart city living, and public safety. Visit Ekin at LVCC – North Hall Booth #9136.

Nota: Nota produces an AI software optimization platform, which automates and optimizes customers’ AI applications. Nota creates lightweight AI models that are low latency, energy-efficient, and accurate. They work to optimize processor usage and proliferate lower-end edge devices. Check out Nota at booth #9646 in LVCC North Hall.

NoTraffic: NoTraffic’s real-time, plug-and-play autonomous traffic management platform uses AI and cloud computing to reinvent how cities run their transport networks. The NoTraffic platform is an end-to-end hardware and software solution installed at intersections, transforming roadways to optimize traffic flows and reduce accidents. Check out NoTraffic at booth #9130 in LVCC North Hall.

Ouster: Cities are using Ouster digital lidar solutions to capture the environment in minute detail and detect vehicles, vulnerable road users, and incidents in real time to improve safety and traffic efficiency. Ouster lidar’s 3D spatial awareness and 24/7 performance combine the high-resolution imagery of cameras with the all-weather reliability of radar. Check out Ouster and live demos at booth #3843 in LVCC West Hall.

Velodyne Lidar: Velodyne’s lidar-based Intelligent Infrastructure Solution (IIS) is a complete end-to-end Smart City solution. IIS creates a real-time 3D map of roads and intersections, providing precise traffic and pedestrian safety analytics, road user classification, and smart signal actuation. The solution is deployed in the US, Canada and across EMEA and APAC. Check out Velodyne Lidar at booth #6005 in LVCC West Hall.

Register for CES, happening Jan. 5-8 in Las Vegas.

Spotting painting forgeries just became a bit easier with a newly developed AI tool that picks up style differences with precision down to a single paintbrush bristle. The research, from a team at Case Western Reserve University (CWRU), trained convolutional neural networks to learn and identify a painter based on the 3D topography of a painting. This work could help historians and art experts distinguish between artists in collaborative pieces, and find fraudulent copies.

There are several methods to authenticating antique paintings. Experts often evaluate the style and condition of materials and use scientific methods such as microscopic analysis, infrared spectroscopy, and reflectography.

However, these exhaustive methods are time-consuming and can result in errors. They also cannot identify multiple painters of one piece of art. According to the study, painters such as El Greco and Rembrandt often employed workshops of artists to paint parts of a canvas in the same style as their own, making individual contributions unclear.

While analyzing artwork with machine learning is a relatively new field, recent studies have focused on combining AI methods with high-resolution images of paintings to learn a painter’s style and identify an artist. The researchers hypothesized that 3D analysis could hold even more data than an image, where features such as brushwork patterns along with paint deposition and drying methods could serve as an artist’s unique fingerprint. 

“3D topography is a new way for AI to ‘see’ a painting,” senior author Kenneth Singer, the Ambrose Swasey Professor of Physics at CWRU, said in a press release.

Extracting topographical data from a surface with an optical profiler, the researchers scanned 12 paintings of the same scene, painted with identical materials, but by four different artists. Sampling small square patches of the art, approximately 5 to 15 mm, the optical profiler detects and logs minute changes on a surface, which can be attributed to how someone holds and uses a paintbrush. 

They then trained an ensemble of convolutional neural networks to find patterns in the small patches, sampling between 160 to 1,440 patches for each of the artists. Using NVIDIA GPUs with cuDNN-accelerated deep learning frameworks, the algorithm matches the samples back to a single painter.

The team tested the algorithm against 180 patches of an artist’s painting, matching the samples back to a painter at about 95% accuracy. 

According to coauthor Michael Hinczewski, the Warren E. Rupp Associate Professor of Physics at CWRU, the ability to work with such small training sets is promising for later art historical applications with limited training datasets.

“Most of the other studies using AI for art attribution are focused on photos of entire paintings,” said Hinczewski. “We broke the painting down into virtual patches ranging from one-half millimeter to a few centimeters square. So we no longer even have information about the subject matter—but we can accurately predict who painted it from an individual patch. That’s amazing.”

Based on their findings the researchers view surface topography as an additional tool for attribution and forgery detection using an unbiased and quantitative analysis. In a collaboration with art conservation company Factum Arte based in Madrid, the team is working on artist workshop attribution and conservation studies on several works of the Spanish Renaissance painter El Greco.

The data and code associated with the research are available through GitHub. The work is a joint effort between researchers from the CWRU Department of Art History and Art, Cleveland Institute of Art, and the Cleveland Museum of Art.

Read the published research in Heritage Science. >>
Read the press release. >>