At GDC 2023, NVIDIA released new tools that make real-time path tracing more accessible to developers while accelerating the creation of ultra-realistic game worlds.

Generate frames with the latest breakthrough in AI rendering

Announced with the NVIDIA Ada Lovelace architecture, DLSS 3 raised the bar not just for visuals but also performance and responsiveness. Since its introduction in 2019 with the NVIDIA Turing architecture, DLSS has continuously set new standards for AI in game rendering.

Frame generation is the latest evolution.

DLSS Frame Generation is the new performance multiplier in DLSS 3 that uses AI to create entirely new frames. It is powered by NVIDIA GeForce 40 series and the Optical Flow Accelerator. This breakthrough has made real-time path tracing—the next frontier in video game graphics—possible.

Since that announcement, 28 top games and applications now use DLSS 3 to deliver realistic graphics with incredible performance, including A Plague Tale Requiem, Portal with RTX, and Cyberpunk 2077. In some cases, frames-per-second in games have almost tripled.

DLSS Frame Generation is coming to GDC as a plugin for developers through the Streamline 2.0 SDK. Streamline is the NVIDIA open-source cross-IHV framework that simplifies the integration of features like DLSS 3. Instead of manually integrating the DLSS Frame Generation SDK, you identify which resources (motion vectors, depth, and so on) are required for the desired plug-in and then execute the plug-ins to run in your rendering pipeline.

In addition, Epic Games announced the DLSS Frame Generation plugin coming to Unreal Engine in its next release. Coupled with the NVIDIA Reflex low latency technology available through UE5, you have all the tools to boost the performance of your games while providing a highly responsive experience for players.

Together, Epic and NVIDIA are pushing the bounds of AI in game development, bringing real-time path tracing a step closer to reality.

“NVIDIA DLSS 3 introduces truly impressive frame generation technology and the Unreal Engine 5.2 plugin will offer developers a great choice for increased quality and performance of their games,” said Nick Penwarden, vice president of engineering at Epic Games.

Integration of real-time path tracing technology simplified

Real-time ray tracing in games was made possible with the introduction of RT cores in the NVIDIA Turing architecture. Since then, NVIDIA has been hard at work on the next challenge.

Coming to GDC 2023, the NVIDIA RTX Path Tracing SDK will be available to developers everywhere.

The RTX Path Tracing SDK accurately re-creates the physics of all light sources in a scene to reproduce what the eye sees in real life. This new SDK gives you the flexibility and customizability to take advantage of proven NVIDIA technologies to suit the following use cases:

• Building a reference path tracer to ensure that your lighting during production is true to life, accelerating the iteration process.
• Building high-quality photo modes for RT-capable GPUs or real-time, ultra-quality modes that take advantage of the Ada Lovelace architecture.

The RTX Path Tracing SDK is the culmination of decades of NVIDIA research. This SDK demonstrates best practices for building a path tracer using the latest versions of the following tools and features:

• DLSS 3 for super-resolution and frame generation, to multiply performance.
• RTX Direct Illumination (RTXDI) for efficient sampling of a high number of shadow casting and dynamic lights.
• NVIDIA Real-Time Denoisers (NRD) for high-performance denoising of all light sources.
• Opacity Micro-Map (OMM) for improving RT performance in scenes with heavy alpha effects.
• Shader execution reordering (SER) for improving shader scheduling, thus increasing performance.

This SDK has all the necessary NVIDIA RTX components, documentation, and sample application for you to get started today.

Improving path tracing performance and increasing accessibility

NVIDIA continuously looks for more opportunities to improve the performance of real-time path tracing, which has led to the development of Opacity Micro-Map (OMM), announced at GTC 2022.

At GDC 2023, OMM SDK 1.0 will be available to all developers. OMM allows you to efficiently map complex geometries, such as dense vegetation and foliage, onto triangles and micro-meshes, providing high-level performance in detailed scenes.

To optimize all this new path tracing and AI rendering technology, an update to Nsight Systems is available now.  This release brings support for profiling OMM in Vulkan applications, enabling you to intercept malfunctioning OMM functions. An upcoming Nsight Graphics update will give you the ability to inspect and debug OMM’s performance gain.

NVIDIA Nsight Developer Tools has provided industry-leading performance insights and debugging guidance that has ensured optimal path-tracing integration for years now. Nsight Graphics dissects GPU activity, like exposing ray-gen shader metrics that root out stalls, and acceleration structures to optimize for ground truth lighting.

Recently, it was used to improve path tracing in the upcoming Cyberpunk 2077 Ray Tracing: Overdrive mode by resolving a shader inefficiency. Watch the demo video to learn more.    

Caustics

Lastly, NVIDIA is making it easier for all Unreal Engine developers to start their path-tracing journey with a set of new ray-tracing features in the NVIDIA Caustics branch of Unreal Engine 5.

Caustics is an optical phenomenon that exists all around us and is invisible to the naked eye until it hits a reflective material like glass and produces a curved region of light.

This technology is available now through the UE 5.1 Caustics branch and makes it easier to leverage caustic effects around metallic and translucent meshes and water surfaces.

Ray-traced depth of field is a standout feature of this branch. In traditional rasterization workflows, camera depth of field is a challenge to calculate translucent objects accurately. With the Caustics branch, such camera effects are achievable.

Figure 4 shows a traditional rasterization pipeline camera depth of field with translucency challenges.

Figure 5 shows a ray-traced camera depth of field bringing addressing translucency challenges.

In Figures 4 and 5, you can see the difference in detail between traditional caustics raster and ray-traced depth of field.

Next steps

For more information about other features introduced in this Caustics branch, see the Unreal Engine overview page.

For more information about free resources to re-create fully path-traced and AI-driven virtual worlds, see the NVIDIA Game Development resources page.

Check out the NVIDIA GTC 2023 game development sessions.

NVIDIA Base Command Platform provides the capabilities to confidently develop complex software that meets the performance standards required by scientific computing workflows. The platform enables both cloud-hosted and on-premises solutions for AI development by providing developers with the tools needed to efficiently configure and manage AI workflows. Integrated data and user management simplify both the user and administrator experience. 

Now, creating high-fidelity digital twins across teams and locations using NVIDIA Modulus with Base Command Platform is the newest tool available for high-performance computing (HPC) workflows. Creating and using digital twins is crucial to saving time and money for many use cases—from predicting optimal airplane maintenance schedules to simulating wind farms.

Getting started on these use cases can be daunting. However, a well-integrated solution makes all the difference and enables developers to focus on solving problems. Base Command Platform puts the full breadth of NGC catalog software a few clicks away and empowers the creation of powerful physics-informed machine learning (physics-ML) neural networks and climate models.

Climate modeling with FourCastNet

FourCastNet, part of the open-source Modulus platform, is focused on the creation of global weather forecasting at speeds that were previously impossible. It relies on Fourier neural operators and transformers to achieve this incredible leap in performance and resolution. FourCastNet is now compatible with Base Command Platform.

The ERA5 dataset, a complex weather dataset for the entire Earth across several decades, is used to train and validate such a complex model. FourCastNet is a key technology enabling the creation of the NVIDIA Earth-2 digital twin. For more information, see NVIDIA to Build Earth-2 Supercomputer to See Our Future.

The Modulus team is constantly looking to improve the performance of FourCastNet and recently updated it to use NVIDIA Data Loading Library (DALI) to ingest data to GPUs, further accelerating time to insight.

Boost scalability with Modulus on Base Command Platform

The full power of Modulus is unleashed when run in an environment that can scale across several GPU-based systems. There is no better way to run a highly scalable platform like Modulus to train large models like FourCastNet than Base Command Platform. 

To run these examples, we uploaded a slightly modified version of the Modulus NGC container to a Base Command Platform organization that has access to an accelerated computing environment composed of NVIDIA DGX A100 systems. We uploaded one terabyte of the ERA5 dataset to a workspace in that same environment.

To support coordinated multi-instance workloads, Base Command Platform integrates a tool called bcprun. bcprun simplifies multi-instance workload deployment by abstracting away complexity for machine learning (ML) practitioners and removing the need for additional software within workload containers (such as mpirun). It also provides an easier onboarding path for applications originally written for use with HPC schedulers, like Slurm.

The following code example shows a single-instance job launch of FourCastNet on Base Command Platform:

ngc batch run \
--name "bcp-dali.fcn.training.ml-model.modulus" \
--total-runtime 12H \
--org org-name \
--ace ace-name \
--instance dgxa100.80g.8.norm \
--workspace ERA5_test_21Vars:/era5/ngc_era5_data/:RO \
--result /results \
--image "nvcr.io/org-name/team-name/modulus:22.09-examples_0.4" \
--commandline "\
set -x && \
cd /examples/fourcastnet/ && \
ln -s /era5/stats . && \
python fcn_era5.py \
custom.train_dataset.kind=dali \
custom.num_workers.grid=1 \
training.max_steps=50000 \
training.print_stats_freq=500 \
network_dir=/results/network_checkpoint
"

To scale to two NVIDIA DGX A100 eight GPU instances (16 total), use the following command (with changes highlighted in bold):

ngc batch run \
--name "bcp-dali.fcn.training.ml-model.modulus" \
--total-runtime 12H \
--org org-name \
--ace ace-name \
--replicas "2" \
--array-type "PYTORCH" \
--instance dgxa100.80g.8.norm \
--workspace ERA5_test_21Vars:/era5/ngc_era5_data/:RO \
--result /results \
--image "nvcr.io/org-name/team-name/modulus:22.09-examples_0.4" \
--commandline "\
set -x && \
cd /examples/fourcastnet/ && \
mkdir -p /results/network_checkpoint && \
ln -s /era5/stats . && \
bcprun --nnodes \$NGC_ARRAY_SIZE \
--npernode \$NGC_GPUS_PER_NODE \
--cmd '\
python fcn_era5.py \
custom.train_dataset.kind=dali \
custom.num_workers.grid=1 \
training.max_steps=50000 \
training.print_stats_freq=500 \
network_dir=/results/network_checkpoint
'
"

The bcprun addition, along with the arguments added, ensures that the specified command (from the --cmd argument) runs on each replica created for the job (as specified by the --replicas and --nnodes arguments). The --npernode argument ensures that there is a process running on each instance for each GPU in that instance. This results in 16 total process launches for this job (eight in each replica, two total replicas). To scale to using four instances, set the --replicas argument to four instead of two.

Base Command Platform not only provides ease of use for ML practitioners and administrators but proves that peak performance has been achieved. The NVIDIA Selene supercomputer runs FourCastNet training with Modulus for comparison.

After testing the workload on Selene, we seamlessly reproduced the workload on a Base Command Platform deployment and achieved nearly identical results between the two environments. This outcome provides strong evidence that Base Command Platform can support the most demanding performance requirements for customers across both enterprise and scientific computing use cases.

An interview with developer Kaustubh Tangsali

To learn more about the experience of using NVIDIA Modulus on Base Command Platform, we spoke with Kaustubh Tangsali, a developer on the Modulus team. Kaustubh led the investigation of running FourCastNet and several other software examples on Base Command Platform. 

Briefly describe your industry background and experience.

I have primarily worked in the software industry with applications in simulation and computational fluid dynamics. I work on the development of the Modulus platform, which is a framework for domain experts and AI practitioners to develop physics-ML models. I have worked closely with internal partners like the NVIDIA Thermal team for using Modulus to design heat sinks and also several external partners to accelerate their workflows using Modulus.

How long have you been working with Modulus on Base Command Platform?

I have been using Modulus on Base Command Platform since mid-2020.

What does a routine day of use look like for you on Base Command Platform? What does your development cycle look like?

After I’ve done some local testing of my code or model, I typically mount my code in a Base Command Platform workspace and then launch jobs using either the NGC web interface or just using the command-line interface (CLI). The Jupyter interface is great for early debugging. When the model has run to completion, I download the checkpoints and results for further analysis. During the runtime, I also use the log feature and telemetry to monitor the job’s status.

How does a Base Command Platform environment compare to other environments you have used?

The web interface of Base Command Platform is something that I find useful. It is easy to monitor jobs, see the commands used to launch them, clone the jobs, and use different instance types, to name a few features. I think getting access to the latest and the best NVIDIA hardware is a big plus.

Do you have any advice for someone who is just getting started with Base Command Platform?

The NVIDIA Base Command Platform User Guide is well-documented and covers many common use cases that a data scientist might encounter, with command samples for single GPU, multi-GPU, and multi-instance jobs. As I mentioned earlier, I like to leverage the interactive nature of running jobs during the early stages of development before scaling the jobs, which the CLI streamlines.

Summary

Cutting-edge digital twin technology such as NVIDIA Modulus relies on powerful computing environments to make continued advancements. Base Command Platform harnesses the power of NVIDIA GPUs in an easy-to-use set of interfaces, continuing the NVIDIA mission of making advanced software capabilities broadly accessible to solve important problems. For more information, see Simplifying AI Development with NVIDIA Base Command Platform.

Kick-start your inference journey with short-term access in NVIDIA LaunchPad for Modulus. There’s no need for setting up your own environment.

For more information about NVIDIA Modulus, see the NVIDIA Deep Learning Institute course, Introduction to Physics-Informed Machine Learning with Modulus.

To get the latest release details, download and try NVIDIA Modulus.

Register for NVIDIA GTC 2023 for free and attend the Enterprises Share Their Experience with DGX Cloud session to learn about the wide array of use cases that are powered by Base Command Platform.

Polyphony Digital, a subsidiary of Sony Interactive Entertainment Inc and the creators of Gran Turismo, has exceeded 90M copies of cumulative sell-through sales of PlayStation software titles over three decades. Gran Turisimo 7, released in 2022, marked the 25th anniversary of the series’ beginning, and included implementation of Universal Scene Description (USD). USD is an open-scene description with APIs for describing, composing, simulating, and collaborating within 3D worlds.

Tools Pipeline Engineering Lead Megumi Ando sat down with NVIDIA to discuss the integration process in the company’s latest release as well as their adoption plans in Polyphony’s game development pipeline.

Why did you choose USD for your game development pipeline?

USD technology supports large-scale, multi-user, diverse asset pipelines for video production. Although it is a library designed for video production, USD’s mechanism for supporting large and complex pipelines could be used for in-game production. As USD already provides the functions needed for building pipelines, such as compositions, asset resolvers, file format plug-ins, and custom schemas, we adapted it to fit our pipeline.

What does switching to USD enable you to do that you could not do before?

USD is different from our previous format in that it can be edited in different tools, such as Houdini. Unlike previous formats, USD compositions can be used to edit data in a non-destructive manner. Even data containing unique specifications for games can be transferred back and forth between various tools without losing data.

In Gran Turismo, we have been creating assets such as cars and courses mainly in Autodesk Maya and the data is extremely complex. There are many tools for setting up various plug-ins and specifications and re-implementing those tools for Houdini was expensive.

In particular, shaders are complex and developing a new environment for previewing and editing shaders in Houdini is expensive and difficult to implement quickly. However, USD makes it possible to create material settings in Autodesk Maya during modeling, while material assignments can be done in Houdini.

Why did you choose USD over other file formats?

The ability to handle scene graphs non-destructively and procedurally across multiple tools. In the case of games, various exclusive specifications are set up in digital content creation (DCC) tools. In the conventional format, when editing with different tools, you couldn’t handle the parameters correctly unless they were handled by each tool.

In game production, as in video production, many people can produce many assets and shots simultaneously, while adding various game specifications. USD solves these problems as compositions can be used for nondestructive editing, so that you can smoothly move between multiple tools.

What challenges did you face in the process of integrating USD?

The current pipeline is huge, and it was impossible to convert it to USD all at one time.

We had to maintain the pipeline in operation and proceed with the migration in parts.

What are some recommended small steps to take first?

First, you need an independent project (preferably separate from the existing system) to understand how USD works before implementing it in a real project.

When using USD as an asset for a game, it is important to understand that, unlike video production, there is no established schema or editing tools specialized for game production. It is necessary to understand the USD specifications, data structure, API specifications, and so on. Before we introduced it into our asset production, we had to learn the file formats such as XML and JSON. It’s necessary to develop your own software according to the game’s specifications.

We introduced USD as a format to manage parameters for the conversion pipeline. Unlike asset data, everything is in small ASCII files. The structure of the data is easy to understand, and concepts such as layers, stages, and schemas are easy to grasp. The advantages of this system provided the foundation for introducing the system in asset creation.

Do you plan to continue using USD in your future game development?

Of course we do. Even with Gran Turismo 7, which used USD, implementation was still limited. We have been working on building a structure and publishing environment for assets that assume USD. We plan to take on challenges where we can, like improving the iteration of production.

What do you think is needed to make USD more widely adopted in the game industry?

In game production, there is no established schema, as there is for video production. There is no environment like KATANA or Houdini SOLARIS where you can lay out the USD, light it, render it, and so on. To be able to use the renderer in a game engine, you must either prepare a conversion pipeline instead of a renderer or use a game engine.

How have artists, technical artists, art directors, and studio executives reacted to the move to USD?

With the progress of USD support in DCC tools such as Maya and SOLARIS, and so on, artists and tech artists are happy with the increased flexibility, stability, and efficiency of data compatibility. The proactive use of open-source libraries has been a great benefit from a management perspective, as development resources can be focused on production-specific issues.

What do you think was the most compelling argument that convinced everyone to adopt USD?

With USD, you can test in a common environment that supports it immediately. As NVIDIA Omniverse, a real-time design collaboration and simulation platform for building and operating metaverse applications, and various other tools become more compatible, it will become even more of an advantage to adopt USD in the future.

Resources

For more information, see Polyphony Digital, Universal Scene Description, and resources for game developers.

Register for NVIDIA GTC 2023 for free and join us March 20–23 in our game development sessions. You may want to attend the following USD sessions:

• How USD Can Become the “HTML” of the Metaverse
• An Overview of USD for Building Virtual Worlds

For those attending GDC, see our sessions for game developers.

Generative AI has captured the attention and imagination of the public over the past couple of years. From a given natural language prompt, these generative models are able to generate human-quality results, from well-articulated children’s stories to product prototype visualizations. 

Large language models (LLMs) are at the center of this revolution. LLMs are universal language comprehenders that codify human knowledge and can be readily applied to numerous natural and programming language understanding tasks, out of the box. These include summarization, translation, question answering, and code annotation and completion. 

The ability of a single foundation language model to complete many tasks opens up a whole new AI software paradigm, where a single foundation model can be used to cater to multiple downstream language tasks within all departments of a company. This simplifies and reduces the cost of AI software development, deployment, and maintenance.

Introduction to creating a custom large language model 

While potent and promising, there is still a gap with LLM out-of-the-box performance through zero-shot or few-shot learning for specific use cases. In particular, zero-shot learning performance tends to be low and unreliable. Few-shot learning, on the other hand, relies on finding optimal discrete prompts, which is a nontrivial process. 

As explained in GPT Understands, Too, minor variations in the prompt template used to solve a downstream problem can have significant impacts on the final accuracy. In addition, few-shot inference also costs more due to the larger prompts.

Parameter-efficient fine-tuning techniques have been proposed to address this problem. Prompt learning is one such technique, which appends virtual prompt tokens to a request. These virtual tokens are learnable parameters that can be optimized using standard optimization methods, while the LLM parameters are frozen. 

This post walks through the process of customizing LLMs with NVIDIA NeMo, a universal framework for training, customizing, and deploying foundation models. 

What is NVIDIA NeMo?

NVIDIA NeMo is the universal framework for training, customizing, and deploying large-scale foundation models. NeMo takes advantage of various parallelism techniques to accelerate training and inference, and can be deployed on multi-node, multi-GPU systems on user-preferred cloud, on-premises, and edge systems. To learn more, see NVIDIA AI Platform Delivers Big Gains for Large Language Models and Accelerated Inference for Large Transformer Models Using NVIDIA Triton Inference Server.

The NeMo ecosystem consists of the following main components:

1. NVIDIA NeMo service: Provides a fast path to productionizing LLMs through NVIDIA-managed services. Developers can quickly and easily develop enterprise AI applications leveraging LLM capabilities without worrying about the underlying infrastructure. You can also experience Megatron 530B—one of the largest language models—through the cloud API or a web playground interface. Currently in early access.
2. NVIDIA NeMo framework: An end-to-end containerized framework that allows developers to efficiently train and deploy language models with billions and trillions of parameters, delivering high training efficiency across thousands of GPUs. NeMo framework container is currently in open beta and available through NGC.
3. NVIDIA/NeMo: An open-source conversational AI toolkit built for researchers working on speech AI and NLP, including LLM. Available through GitHub.
4. NeMo models: NVIDIA recently open-sourced pretrained NeMo framework models, ranging from small models such as the 1.3B GPT-3, 5B GPT-3, 3B mT5 model to large models, such as the 20B GPT-3.
5. NVIDIA/FasterTransformer: An open-source toolkit for high-performance inference of LLMs available through GitHub. To learn more about how to deploy the public NeMo framework models with FasterTransformer, see Deploying a 1.3B GPT-3 Model with NVIDIA NeMo Megatron.

This post explains how to use the NeMo framework container to customize a public NeMo model with prompt-learning techniques.

Prompt learning with NeMo

Prompt learning collectively refers to two parameter-efficient fine-tuning techniques, as detailed below. For more information, see Adapting P-Tuning to Solve Non-English Downstream Tasks.

• In prompt-tuning, soft prompt embeddings are initialized as a 2D matrix. Each task has its own 2D embedding matrix associated with it. Tasks do not share any parameters during training or inference. All LLM parameters are frozen and only the embedding parameters for each task are updated during training. NeMo prompt tuning implementation is based on The Power of Scale for Parameter-Efficient Prompt Tuning.
• In p-tuning, an LSTM model, or “prompt encoder,” is used to predict virtual token embeddings. LSTM parameters are randomly initialized at the start of p-tuning. All LLM parameters are frozen, and only the LSTM weights are updated at each training step. LSTM parameters are shared between all tasks that are p-tuned at the same time, but the LSTM model outputs unique virtual token embeddings for each task. NeMo p-tuning implementation is based on GPT Understands, Too.

Prompt learning for this example uses two open-source components of the NeMo ecosystem: the NeMo OSS toolkit and public NeMo models.

The process of prompt learning on a small GPT-3 345M parameter model is detailed in the NeMo Multitask Prompt and P-Tuning tutorial on GitHub. This tutorial demonstrates the end-to-end process of prompt learning: downloading and preprocessing data, downloading the model, training a prompt-learning model, and inferring on three different applications.

The sections below first walk through the notebook while summarizing the main concepts. Then this notebook will be extended to carry out prompt learning on larger NeMo models.

Prerequisite

You can experience NeMo through the NeMo Docker container. This provides a self-sufficient and reproducible environment for experimenting with NeMo. The NeMo Multitask Prompt and P-Tuning tutorial was tested with the NeMo 22.09 container, but you can try later releases of the same container. Download and run this container using the following script:

docker run  -u $(id -u ${USER}):$(id -g ${USER}) --rm -it --net=host nvcr.io/nvidia/nemo:22.09 bash

Then from within the container interactive bash environment, start Jupyter lab:

cd /workspace
jupyter lab --ip 0.0.0.0 --allow-root --port=8888

From Jupyter lab, you will find NeMo examples, including the above-mentioned notebook,  under /workspace/nemo/tutorial/nlp/Multitask_Prompt_and_PTuning.ipynb. 

In addition, you will need one GPU for working with the smaller 5B and 1.3B GPT-3 models, and four NVIDIA Ampere architecture or NVIDIA Hopper architecture GPUs to work with the 20B model, as it has four degrees of tensor parallelism (TP). 

Data preparation

The notebook will walk you through the data collection and preprocessing process for three different applications: Financial PhraseBank dataset for the sentiment analysis task, SQuAD dataset for the question/answer task, and Assistant Benchmarking dataset for the intent and slot classification task. 

The dataset should be in a .jsonl format containing a collection of JSON objects. Each JSON object must include the field task name, which is a string identifier for the task the data example corresponds to. Each should also include one or more fields corresponding to different sections of the discrete text prompt. See Figure 1 for an example.

Prompt template

You should determine and adhere to a pattern when forming the prompt. This pattern is called the prompt template and varies according to the use case. An example for sentiment analysis is shown below.

{
        "taskname": "sentiment",
        "prompt_template": "<|VIRTUAL_PROMPT_0|> {sentence} sentiment:{label}",
        "total_virtual_tokens": 10,
        "virtual_token_splits": [10],
        "truncate_field": None,
        "answer_only_loss": True,
        "answer_field": "label",
    }

The prompt contains all the 10 virtual tokens at the beginning, followed by the target sentence to classify. Next is a text marker (“sentiment:”), and finally the label of the sentence for training. The corresponding fields in the training data JSON object will be mapped to this prompt template to form complete training examples. NeMo supports pruning specific fields to meet the model token length limit (typically 2,048 tokens for NeMo public models using the HuggingFace GPT-2 tokenizer).

Training

The default NeMo prompt-tuning configuration is provided in a yaml file, available through NVIDIA/NeMo on GitHub. The notebook loads this yaml file, then overrides the training options to suit the 345M GPT model. The NeMo p-tuning enables multiple tasks to be learned concurrently. NeMo leverages the PyTorch Lightning interface, so training can be done as simply as invoking a trainer.fit(model) statement.

Inference

Finally, once trained, the model can be used for inference on new samples (omitting the “answer_field”) by invoking the model.generate(inputs=test_examples) statement.

Prompt learning on larger models

The 345M GPT-3 model process demonstrated in the notebook can be applied to larger public NeMo GPT-3 models, up to 1.3B GPT-3 and 5B GPT-3. Models of this size require only a single GPU of sufficient memory capacity, such as the NVIDIA V100, NVIDIA A100, and NVIDIA H100. After downloading the model, substitute the model name; in particular, in the following cell:

# Download the model from NGC
gpt_file_name = "megatron_gpt_345m.nemo"
!wget --content-disposition https://api.ngc.nvidia.com/v2/models/nvidia/nemo/megatron_gpt_345m/versions/1/files/megatron_gpt_345m.nemo -

Instead of downloading the 345M GPT model from NGC, download either the 1.3B GPT-3 or 5B GPT-3 models following the instructions on HuggingFace, then point the gpt_file_name variable to the .nemo model file. 

Note that for the 5B model there are two variants, one with a TP degree of 1 (nemo_gpt5B_fp16_tp1.nemo) and other with TP=2 (nemo_gpt5B_fp16_tp2.nemo, nemo_gpt5B_bf16_tp2.nemo). The notebook can only support the TP=1 variant. With everything else left unchanged, you can execute the same notebook end-to-end. 

Multi-GPU prompt learning

Due to the limitations of the Jupyter notebook environment, the prompt learning notebook only supports single-GPU training. Leveraging multi-GPU training for larger models, with a higher degree of TP (such as 4 for the 20B GPT-3, and 2 for other variants for the 5B GPT-3) requires use of a different NeMo prompt learning script. This script is supported by a config file where you can find the default values for many parameters.

Models

This section demonstrates the process of prompt learning of a large model using multiple GPUs on the assistant dataset that was downloaded and preprocessed as part of the prompt learning notebook.

You can download either the 5B GPT model with TP=2 (nemo_gpt5B_fp16_tp2.nemo) or the 20B GPT-3 model with TP=4. Note that these models are stored in a .nemo zip archive. To speed up model loading substantially, unzip the model beforehand, and use this unzipped folder in the NeMo configuration. Use the following script:

tar -xvf nemo_gpt5B_fp16_tp2.nemo -C nemo_gpt5B_fp16_tp2.nemo.extracted

Then use the extracted directory nemo_gpt5B_fp16_tp2.nemo.extracted in NeMo config.

Configuration

A configuration file suitable for the assistant dataset (intent and slot detection application) is shown below:

name: megatron_virtual_prompt_gpt

trainer:
  devices: 2
  accelerator: gpu
  num_nodes: 1
  precision: 16
  logger: False # logger provided by exp_manager
  enable_checkpointing: False
  replace_sampler_ddp: False
  max_epochs: 25 # min 25 recommended
  max_steps: -1 # consumed_samples = global_step * micro_batch_size * data_parallel_size * accumulate_grad_batches
  log_every_n_steps: 10 # frequency with which training steps are logged 
  val_check_interval: 1.0 # If is an int n > 1, will run val every n training steps, if a float 0.0 - 1.0 will run val every epoch fraction, e.g. 0.25 will run val every quarter epoch
  gradient_clip_val: 1.0
  resume_from_checkpoint: null # The path to a checkpoint file to continue the training, restores the whole state including the epoch, step, LR schedulers, apex, etc.
  benchmark: False


exp_manager:
  explicit_log_dir: null
  exp_dir: null
  name: ${name}
  create_wandb_logger: False
  wandb_logger_kwargs:
    project: null
    name: null
  resume_if_exists: True
  resume_ignore_no_checkpoint: True
  create_checkpoint_callback: True
  checkpoint_callback_params:
    monitor: val_loss
    save_top_k: 2
    mode: min
    save_nemo_on_train_end: False # Should be false, correct prompt learning model file is saved at model.nemo_path set below, 
    filename: 'megatron_gpt_prompt_tune--{val_loss:.3f}-{step}'
    model_parallel_size: ${model.tensor_model_parallel_size}
    save_best_model: True

model:
  seed: 1234
  nemo_path: ${name}.nemo # .nemo filename/absolute path to where the virtual prompt model parameters will be saved
  virtual_prompt_style: 'p-tuning' # one of 'prompt-tuning', 'p-tuning', or 'inference'
  tensor_model_parallel_size: 1 # intra-layer model parallelism
  pipeline_model_parallel_size: 1 # inter-layer model parallelism
  global_batch_size: 8
  micro_batch_size: 4

  restore_path: null # Path to an existing p-tuned/prompt tuned .nemo model you wish to add new tasks to or run inference with
  language_model_path: ??? # Path to the GPT language model .nemo file, always required
  save_nemo_on_validation_end: True # Saves an inference ready .nemo file every time a checkpoint is saved during training. 
  existing_tasks: [] # List of tasks the model has already been p-tuned/prompt-tuned for, needed when a restore path is given
  new_tasks: ['intent_and_slot'] # List of new tasknames to be prompt-tuned
  


  ## Sequence Parallelism
  # Makes tensor parallelism more memory efficient for LLMs (20B+) by parallelizing layer norms and dropout sequentially
  # See Reducing Activation Recomputation in Large Transformer Models: https://arxiv.org/abs/2205.05198 for more details.
  sequence_parallel: False

  ## Activation Checkpoint 
  activations_checkpoint_granularity: null # 'selective' or 'full' 
  activations_checkpoint_method: null # 'uniform', 'block', not used with 'selective'
  # 'uniform' divides the total number of transformer layers and checkpoints the input activation
  # of each chunk at the specified granularity
  # 'block' checkpoints the specified number of layers per pipeline stage at the specified granularity
  activations_checkpoint_num_layers: null # not used with 'selective'

  task_templates: # Add more/replace tasks as needed, these are just examples
  - taskname: "intent_and_slot"    
    prompt_template: "<|VIRTUAL_PROMPT_0|>Predict intent and slot: {utterance} \nLabel:{label}"
    total_virtual_tokens: 10
    virtual_token_splits: [10]
    truncate_field: null
    answer_only_loss: False
    "answer_field": "label"


  prompt_tuning: # Prompt tunin specific params
    new_prompt_init_methods: ['text'] # List of 'text' or 'random', should correspond to tasks listed in new tasks
    new_prompt_init_text: ['some init text goes here'] # some init text if init method is text, or None if init method is random

  p_tuning: # P-tuning specific params
    encoder_type: "tpmlp" # ['tpmlp', 'lstm', 'biglstm', 'mlp'] 
    dropout: 0.0
    num_layers: 2  # number of layers for MLP or LSTM layers. Note, it has no effect for tpmlp currently as it always assumes it is two layers.
    encoder_hidden: 2048 # encoder hidden for biglstm and tpmlp
    init_std: 0.023  # init std for tpmlp layers

  data:
    train_ds: ???
    validation_ds: ???
    add_eos: True
    shuffle: True
    num_workers: 8
    pin_memory: True
    train_cache_data_path: null  # the path to the train cache data 
    validation_cache_data_path: null  # the path to the validation cache data 
    test_cache_data_path: null  # the path to the test cache data 
    load_cache: False  # whether to load from the cache data


  optim:
    name: fused_adam
    lr: 1e-4
    weight_decay: 0.01 
    betas: 
    - 0.9
    - 0.98
    sched:
      name: CosineAnnealing
      warmup_steps: 50
      min_lr: 0.0 # min_lr must be 0.0 for prompt learning when pipeline parallel > 1
      constant_steps: 0 # Constant steps should also be 0 when min_lr=0
      monitor: val_loss
      reduce_on_plateau: false

Thanks to the yaml text format and comments, most of the hyperparameters are self-explanatory. Using the Jupyter lab interface, create a file with this content and save it under /workspace/nemo/examples/nlp/language_modeling/conf/megatron_gpt_prompt_learning_intent_n_slot.yaml.

Most important in the config file is the prompt template shown below:

 prompt_template: "<|VIRTUAL_PROMPT_0|>Predict intent and slot: {utterance} \nLabel:{label}"
    total_virtual_tokens: 10
    virtual_token_splits: [10]
    truncate_field: null

Here, 10 virtual prompt tokens are used together with some permanent text markers. 

Training

To begin training, open a terminal window from within the Jupyter lab interface (File → New  → Terminal). Then issue the bash command:

python /workspace/nemo/examples/nlp/language_modeling/megatron_gpt_prompt_learning.py \
    	--config-name=megatron_gpt_prompt_learning_intent_n_slot.yaml \
    	trainer.devices=2 \
    	trainer.num_nodes=1 \
    	trainer.max_epochs=25 \
    	trainer.precision=bf16 \
    	model.language_model_path=/workspace/nemo/tutorials/nlp/nemo-megatron-gpt-5B/nemo_gpt5B_fp16_tp2.nemo.extracted \
    	model.nemo_path=/workspace/nemo/examples/nlp/language_modeling/intent_n_slot.nemo \
    	model.tensor_model_parallel_size=2 \
    	model.pipeline_model_parallel_size=1 \
    	model.global_batch_size=16 \
    	model.micro_batch_size=1 \
    	model.optim.lr=1e-4 \
    	model.data.train_ds=[/workspace/nemo/tutorials/nlp/data/assistant/assistant_train.jsonl] \
    	model.data.validation_ds=[/workspace/nemo/tutorials/nlp/data/assistant/assistant_val.jsonl]

Note the following:

• model.tensor_model_parallel_size should be set to 2 for the 5B GPT model (nemo_gpt5B_fp16_tp2.nemo) or 4 for the 20B GPT-3 model
• trainer.devices should be set to a multiple of the TP value. If 4 for the 5B models, there will be two data-parallel workers, each with two GPUs
• model.language_model_path should be set to the absolute path of the model extracted directory
• model.data.train_ds, model.data.validation_ds should be set to the location of the train and validation data

Inference

Finally, once trained, carry out inference in NeMo using the following script:

python /workspace/nemo/examples/nlp/language_modeling/megatron_gpt_prompt_learning_eval.py \
            virtual_prompt_model_file=/workspace/nemo/examples/nlp/language_modeling/intent_n_slot.nemo \
            gpt_model_file=/workspace/nemo/tutorials/nlp/nemo-megatron-gpt-5B/nemo_gpt5B_fp16_tp2.nemo.extracted  \
            inference.greedy=True \
            inference.add_BOS=False \
            inference.tokens_to_generate=128 \
            trainer.devices=2 \
            trainer.num_nodes=1 \
            tensor_model_parallel_size=2 \
            pipeline_model_parallel_size=1 \
            data_paths=["/workspace/nemo/tutorials/nlp/data/assistant/assistant_test.jsonl"] \
            pred_file_path="test-results.txt"

Note the following:

• model.tensor_model_parallel_size should be set to 2 for the 5B GPT model (nemo_gpt5B_fp16_tp2.nemo) or 4 for the 20B GPT-3 model 
• trainer.devices should be set to equal the TP value (above)
• pred_file_path is the file where test results will be recorded, one line per test sample

Get started customizing your language model

This post walked through the language model customization process using NVIDIA NeMo. From a single public checkpoint, these models can be adapted to numerous NLP applications through a parameter-efficient, compute-efficient process. 

Visit NVIDIA/NeMo on GitHub to get started with LLM customization. You can also sign up for the NVIDIA NeMo service early access program.

Register for GTC 2023 for free and join us March 20–23 for How to Build Generative AI for Enterprise Use Cases and targeted session tracks for generative AI models for science and art, autonomous vehicles, content creation, and much more. 

A retailer’s supply chain includes the sourcing of raw materials or finished goods from suppliers; storing them in warehouses or distribution centers; and transporting them to stores or customers; managing sales. They also collect, store, and analyze data to optimize supply chain performance.

Retailers have teams responsible for managing each stage of the supply chain, including supplier management, logistics, inventory management, sales, and data analytics. All these teams and processes work together to ensure that the right products are available to customers at the right time and at the right price.

It is important to make informed decisions about retail sales and operations by collecting, analyzing, and interpreting data from various sources, such as point-of-sale (POS) systems, customer databases, and market research surveys.

Big data processing is a key component of retail analytics, as it enables retailers to handle and analyze large volumes of data from various sources with low latency. Retailers can gain valuable insights into customer behavior, market trends, and operational efficiency.

This post provides an overview of retail applications that can benefit from GPU-accelerated Apache Spark workloads. We provide detailed step-by-step instructions with a sample retail use case on how to get started with using GPU acceleration on Spark workloads on Dataproc. The example shows you how to speed up data processing pipelines for retailers. We highlight new RAPIDS Accelerator user tools for Dataproc that help set you up for application tuning and also provide insights into GPU run details.

To follow along with this post, access the notebook on the NVIDIA/spark-rapids-examples GitHub repository.

Types of data analysis for retail applications

Depending on your business questions and goals, several types of complex analysis can be performed on retail data:

• Inventory forecasting: Analyzing the sales trend, demand, and quantity in stock to predict future inventory needs.
• Demand forecasting: Predicting future customer demand for products to optimize inventory levels, pricing, and promotions to meet that demand.
• Price optimization: Analyzing the competitor’s price, cost of goods, and transportation cost to determine the optimal price for each product.
• Sales performance analysis: Analyzing the sales status and purchase history of individual customers to identify patterns and target marketing efforts.
• Supply chain analysis: Analyzing the supplier ID, shipping cost, and warehouse cost to identify opportunities for cost savings and efficiency improvements.
• Customer segmentation: Analyzing the customer demographics, purchase history, and contact information to segment customers and target marketing efforts.
• Product analysis: Analyzing the sales of different products and identifying top-selling products and potential new products to introduce.
• Location analysis: Analyzing the location of the sales and identifying potential new locations to expand to.
• Retail markdown optimization: Analyzing large amounts of data to replace the manual process of lowering product prices.

For instance, you can optimize in-store or online inventory levels by processing recent product sales data from different sources.

Retail data sources

Retail data can derive from a variety of sources:

• Sales data: Data on the products that are being sold, such as the product name, price, quantity sold, and date of sale. This data can come from POS systems, online sales platforms, or other systems where the sales transactions are recorded.
• Stock data: Data on the products that are currently in stock, such as the product name, quantity in stock, location of the stock, and the date it was received. This data can come from inventory management systems, warehouse management systems, or other systems where the stock is recorded.
• Supplier data: Data on the products that are being ordered from suppliers, such as the product name, quantity ordered, price, and the date the order was placed. This data can come from purchase order systems, supply chain management systems, or other systems where supplier orders are recorded.
• Customer data: Data on customers such as demographics, purchase history, and contact information. This data can come from customer relationship management (CRM) systems, e-commerce platforms, or other systems where customer information is recorded.
• Market data: Data on the market conditions, competitor’s prices, sales trends, and demand forecasts. This data can come from external sources such as market research firms, government agencies, or other data providers.
• Logistic data: Data on logistics such as shipping, transportation, and warehouse management. This data can come from logistics management systems, shipping carriers, or other systems where logistic information is recorded.

Data from these sources may be in different formats (CSV, JSON, Parquet, and so on).

In this post, you use a synthetic data generator code to mock up the data sources and formats explained earlier. The following Python code example can generate huge volumes of synthetic sales data, stock data, supplier data, customer data, market data, and logistic data in different formats. There are some data quality issues that can be used as an example for simulation purposes.

# generate sales data
# Define the generate_data function which takes an integer i as input and generates sales data using random numbers. The generated data includes sales ID, product name, price, quantity sold, date of sale, and customer ID. The function returns a tuple of the generated data. The multiprocessing library is used to generate the data in parallel

def generate_data(i):
    sales_id = "s_{}".format(i)
    product_name = "Product_{}".format(i)
    price = random.uniform(1,100)
    quantity_sold = random.randint(1,100)
    date_of_sale = "2022-{}-{}".format(random.randint(1,12), random.randint(1,28))
    customer_id = "c_{}".format(random.randint(1,1000000))
    return (sales_id, product_name, price, quantity_sold, date_of_sale, customer_id)

with mp.Pool(mp.cpu_count()) as p:
    sales_data = p.map(generate_data, range(100000000))
    sales_data = list(sales_data)
    
print("write to gcs started")
sales_df = pd.DataFrame(sales_data, columns=["sales_id", "product_name", "price", "quantity_sold", "date_of_sale", "customer_id"])
sales_df.to_csv(dataRoot+"sales/data.csv", index=False, header=True)
print("Write to gcs completed")

Access the full notebook on the spark-rapids-examples GitHub repository.

Data cleaning, transformation, and integration

The raw data may have to be cleaned, transformed, and integrated before it can be used for analysis. This is where Apache Spark SQL and DataFrame APIs come in, as they provide a powerful set of tools for working with structured data. They can handle large amounts of data from different sources to process and extract useful insights and information.

First, extract the data, which can be in different formats, such as CSV, JSON, or Parquet. Then, load data into Spark using the DataFrame API. Spark SQL is used to perform data cleaning and preprocessing tasks:

• Remove missing values.
• Handle outliers.
• Transform the data into a suitable format for analysis.

After the data is cleaned and preprocessed, use Spark SQL and DataFrame API to perform various types of analysis for inventory optimization.

For example, you can analyze sales data with SQL queries to identify top-selling and underperforming products. To perform advanced analysis, such as time series analysis, use the DataFrame API to forecast future demand and optimization algorithms or analyze purchase patterns in datasets.

Finally, apply analysis results to make decisions, such as how much stock to hold in inventory, how much to order from suppliers, and when to do so. Use Spark’s Dataframe API or integrate with other systems to update inventory levels automatically.

Also, using Spark SQL and DataFrame API to process large amounts of data from different sources such as sales, stock, and supplier data enables a more efficient and accurate inventory management system.

You can accelerate the process and save compute costs by running the data pipeline on a GPU-powered Google Dataproc cluster. The rest of this post is a step-by-step guide on the process of creating GPU-powered clusters and running the accelerated data processing pipeline using RAPIDS Accelerator.

Create a RAPIDS Accelerator GPU-enabled Dataproc cluster

Retailers’ various data sources push their raw data to Google Cloud Storage, which serves as a source for processing it on the GPU-enabled Dataproc cluster. In the Dataproc cluster, you can enable the Jupyter lab component gateway to run a notebook that performs data cleaning, merging, and analysis on top of the merged data.

In Figure 1, I kept the generated retail source datasets on cloud storage and used the Dataproc 2.x cluster to process the data. GPU enablement with RAPIDS Accelerator can be done based on that process.

To create a GPU-enabled Dataproc cluster, run shell commands using Cloud Shell. To do this, first enable the Compute and Dataproc APIs to gain access to Dataproc. Also, enable the Storage API as you need a Google Cloud Storage bucket to store your data. This process may take a few minutes to complete.

gcloud services enable compute.googleapis.com
gcloud services enable dataproc.googleapis.com
gcloud services enable storage-api.googleapis.com

The following example configurations help you run GPU-enabled workloads on GCP. Adjust the sizes and number of GPU based on your needs.

To launch a GPU-enabled cluster with RAPIDS Accelerator, run the following command in the CLI:

gcloud dataproc clusters create sparkrapidsnew \ 
--region us-central1 \ 
--subnet default \  
--zone us-central1-c \ 
--master-machine-type n1-standard-8 \ 
--master-boot-disk-size 500 \ 
--num-workers 4 \ 
--worker-machine-type n1-standard-8 \ 
--worker-boot-disk-size 1000 \ 
--worker-accelerator type=nvidia-tesla-t4,count=2 \ 
--image-version 2.0-debian10 \
--properties spark:spark.eventLog.enabled=true,spark:spark.ui.enabled=true \
--optional-components HIVE_WEBHCAT,JUPYTER,ZEPPELIN,ZOOKEEPER \ 
--project PROJECT_NAME \
--initialization-actions=gs://goog-dataproc-initialization-actions-us-central1/spark-rapids/spark-rapids.sh --metadata gpu-driver-provider="NVIDIA"  \
--enable-component-gateway

Figure 2 shows creating a GPU cluster with one T4 GPU each on the worker nodes. The script in the initialization actions installs the latest version of RAPIDS Accelerator libraries in the cluster.

You can also build a custom dataproc image to accelerate cluster init time. For more information, see the Getting started with RAPIDS Accelerator on GCP Dataproc quick start page.

Run PySpark on Jupyter Lab

To use notebooks with a Dataproc cluster, select the cluster under the Dataproc cluster and choose Web Interfaces, Jupyter Lab.

Data cleansing

Run the following command to complete these tasks:

1. Read all the data in different formats like JSON and CSV.
2. Perform data cleaning tasks, such as removing missing values and handling outliers.
3. Perform data transformation tasks, such as converting date columns to date type and changing case on strings.

# remove missing values
sales_df = sales_df.dropDuplicates()
# remove duplicate data
sales_df = salesdf.dropna()
# convert date columns to date type
sales_df = sales_df.withColumn("date_of_sale", to_date(col("date_of_sale")))
# standardize case of string columns
sales_df = sales_df.withColumn("product_name", upper(col("product_name")))
# remove leading and trailing whitespaces
sales_df = sales_df.withColumn("product_name", trim(col("product_name")))
# check for invalid values
sales_df = sales_df.filter(col("product_name").isNotNull())

After you’ve cleaned the data, join all the data on a common column (product_name or customer_name) and write the cleaned and transformed data to the Parquet file format.

This is an example of how PySpark can be used to perform data cleaning and preprocessing tasks on large datasets. However, keep in mind that the specific cleaning and preprocessing steps vary depending on the nature of the data and the requirements of the analysis.

Retail data analysis

You can perform various retail data analytics using PySpark. In the demo notebook, PySpark is reading cleaned data in Apache Parquet format, creating new columns based on certain conditions, calculating the rolling average of sales for each product, and using a window function for forecasting.

Then, it performs various aggregations and group-by statements to get the following:

• Total sales
• Quantity sold by product and location
• Total quantity in stock and total sales by the supplier
• Number of perishable products compared to non-perishable products per location
• Number of good compared to bad sales status per location
• Count of the number of sales that contain a 10% off promotion

The results of the aggregations are then saved in Apache Parquet format to disk. The execution time of the code is also being measured and printed.

Bootstrap GPU cluster with optimized settings

The bootstrap tool applies optimized settings for the RAPIDS Accelerator on Apache Spark on a GPU cluster for Dataproc. The tool fetches the characteristics of the cluster, including number of workers, worker cores, worker memory, and GPU accelerator type and count. Then, t uses the cluster properties to determine optimal settings for running GPU-accelerated Spark applications.

Usage: spark_rapids_dataproc bootstrap --cluster  --region 

The tool produces the following example output:

##### BEGIN : RAPIDS bootstrap settings for sparkrapidsnew
spark.executor.cores=4
spark.executor.memory=8192m
spark.executor.memoryOverhead=4915m
spark.rapids.sql.concurrentGpuTasks=2
spark.rapids.memory.pinnedPool.size=4096m
spark.sql.files.maxPartitionBytes=512m
spark.task.resource.gpu.amount=0.25
##### END : RAPIDS bootstrap settings for sparkrapidsnew

Tune applications on the GPU cluster

After Spark applications have been run on the GPU cluster, run the profiling tool to analyze event logs of the applications to determine if more optimal settings should be configured. The tool outputs a per-application set of config settings to be adjusted for enhanced performance.

Usage: spark_rapids_dataproc profiling --cluster  --region 

The tool produces the following example output:

With optimized settings, you can run the data cleansing and processing code and compare it with the CPU counterpart. You can also analyze the profiling tool output results and further tune the job as per the respective insights.

Table 1. Speed-up and cost savings calculation on GPU Dataproc for a retail pipeline

First, the pipeline was run on a Dataproc cluster with CPU only. Then, it was run on different configurations of a Dataproc cluster enabled with T4 GPUs.

Table 1 shows a speed-up of 2.45x of running the pipeline on a GPU cluster over an equivalent CPU cluster, with a cost savings of 20% when moving to a GPU cluster.

Next steps

There can be different motivations to move a job from CPU clusters to GPU clusters, such as accelerating the performance, saving costs, meeting your SLA, or solving any resource contention issues for long-running jobs.

This example scenario explores how to save data processing costs by configuring the cluster size accordingly. You can try different combinations of GPUs and virtual machines to meet your goal.

If you’re looking to speed up data processing, machine learning model training, and inference, join us at GTC 2023 for our upcoming Accelerate Spark with RAPIDS For Cost Savings session, where we discuss benchmarks showing the performance and cost benefits of leveraging GPUs for Spark ETL processing.

This post is the fifth in a series about optimizing end-to-end AI.

NVIDIA TensorRT is a solution for speed-of-light inference deployment on NVIDIA hardware. Provided with an AI model architecture, TensorRT can be used pre-deployment to run an excessive search for the most efficient execution strategy.

TensorRT optimizations include reordering operations in a graph, optimizing the memory layout of weights, and fusing ops to a single kernel to reduce memory traffic to VRAM. To apply these optimizations, TensorRT must have the full network definition and its weights.

The evaluated strategy is serialized within a TensorRT engine, which is shipped with the application to achieve the best inference performance in production. Nothing else apart from this engine is needed during deployment to execute the network.

The inclusion of compiled kernels and the serialization to a file make this engine compatible only with a GPU of the same compute capability. The file is also specific to a TensorRT version but will be compatible with future versions after 8.6.

Generating a performant and correct engine

A network graph definition and its weights can be either parsed from an ONNX file or by building a network manually using the NetworkDefinition API. Building the graph manually provides the most granular access to TensorRT but also requires the most effort.

A quick way to evaluate TensorRT on a given ONNX file is to use trtexec. This tool can also be used to generate an engine file that can be deployed with the Python or C++ API later through the --saveEngine option. Run the following command to create an engine:

trtexec –onnx="model.onnx" –saveEngine="engine.trt"

When you are using TensorRT through ONNX Runtime, only some of the TensorRT native build features provided by trtexec or the TensorRT API are exposed. For more information about TensorRT through ONNX Runtime, see TensorRT Execution Provider.

In this post, I briefly discuss what is needed for acquiring a performant engine from an ONNX file using trtexec. Most of the command-line options can be easily translated to TensorRT C++/Python API functions.

For investigating TensorRT performance, precision, or compatibility, useful tools include polygraph or trex. In case the operations in the graph are not supported by TensorRT, you can write plugins to support custom operations in an engine.

Before going through the effort of implementing a custom operator, evaluate whether the operation can be realized through a combination of other ONNX operators.

Precision

TensorRT performance is heavily correlated to the respective operation precision INT8 or FP16 and FP32.

When using trtexec with an ONNX file, there is currently no option to use the precision specified inside the ONNX file. However, you can enable TensorRT to cast weights to the respective precision and evaluate the inference cost. Use --fp16 or --int8. Lower precision usually results in faster execution, so those weights are more likely to be selected in the resulting engine.

Model quality can be affected by reducing the numeric precision, so you can manually label specific layers to be executed at higher precision using the --layerPrecisions option. To get the most performance by using Tensor Cores it is important to have tensor alignment of, for example, eight elements for FP16. This is not only true for TensorRT but also for ONNX optimization in general.

When the model requires FP32 precision, TensorRT still enables TF32 on NVIDIA Ampere architecture and later by default, but it can be disabled using --noTF32. When TF32 is used, the multiplicands of a convolution are rounded to their nearest FP16 equivalent. The exponent and therefore the dynamic range of FP32 are preserved. After these multiplications, the result of the convolution multiplications is summed using FP32. For more information, see TensorFloat-32 in the A100 GPU Accelerates AI Training, HPC up to 20x.

Input format

In combination with the used precision, it can also be useful to change the I/O buffer type using --inputIOFormats and --outputIOFormats.

As Tensor Cores require an HWC layout, it can be beneficial to ingest data in that layout if possible. That way, TensorRT does not have to do an internal conversion, especially if the data is already present in FP16. If you have data in HWC and FP16, run the following command for compilation:

--inputIOFormats=fp16:hwc --outputIOFormats=fp16:hwc

All other precisions are also supported in CHW and HWC.

Workspace size

As I mentioned earlier, TensorRT has different tactics for executing operations in the graph. A 2D convolution, for example, can be executed without an intermediate buffer by loading the full kernel.

For large kernels, it can make sense to execute the convolution in two 1D convolution passes, requiring intermediate buffers. Use the –memPoolSize argument in trtexec to specify the maximum amount of VRAM that TensorRT can allocate in excess. For example, –-memPoolSize=workspace:100 limits the size of intermediate buffers to 100 MB. Experimenting with this value can help you find a balance between inference latency and VRAM usage.

For more information about optimizing TensorRT inference costs, see Optimizing TensorRT Performance.

Deployment target

After finding the ideal way of serving a network using TensorRT, it comes down to how this model can be deployed.

For this, it is important to pinpoint the requirements that must be fulfilled and compare the strategies. Those change heavily with the deployment target: server, embedded, or workstation.

For the server and embedded deployment scenarios, the hardware variety is usually limited, which makes shipping a dedicated engine together with the binary an attractive solution.

This post series focuses on workstation deployment for which a wide variety of systems must be considered. During development, end-user workstations are completely unknown, making it the most difficult deployment scenario.

The challenges that TensorRT has on workstations are two-fold.

First, depending on the AI model complexity, compiling engines on the user’s device can be a lengthy process (from seconds but also up to multiple minutes), which could be hidden in an installer for example. The GPU should ideally be idle during engine generation as inferences are timed on the GPU and the best performing is selected. Running any other tasks on the GPU simultaneously distorts timings and results in suboptimal execution strategies.

Second, a TensorRT engine is specific to a certain compute capability (for example, for the NVIDIA Ampere and NVIDIA Ada Lovelace architectures). For a single model, you’d ship multiple precompiled engines to support multiple GPUs. When you’re trying to squeeze the best inference performance out of the GPU, you’d have to ship an engine for each GPU SKU (for example, 4080, 4070, and 4060) instead of one for the whole compute capability. Shipping many engines of the same model is counterintuitive. It’s often not feasible as each engine holds all the needed weights already and it just bloats the binary shipment size.

Based on those challenges, two solutions are best suited for workstations:

• Shipping prebuilt engines
• Shipping a prepopulated timing cache

For both approaches, you must compile engines on all supported GPU architectures before deploying.

Be aware that these solutions are not just a consideration for the first deployment, as workstation hardware is subject to change. If you decide to upgrade your GPU to the latest generation, you must generate a new engine.

Shipping prebuilt engines

Shipping prebuilt engines is especially useful for server or embedded appliances and is usually the best solution. In those scenarios, the hardware is already known and an engine fitting for this hardware can be included into the deployment binaries.

For workstations, it is not usually feasible to ship all required engines. A download must be executed during installation or upon requesting a specific feature.

For this download, you must query the GPU’s compute capability and download the respective engine. In that case, TensorRT AI deployment does not impact an application’s shipping size at all, apart from the shared library. The original weights and network definition in the form of an ONNX file, for example, do not have to be shipped.

Usually, a TensorRT engine is much smaller than its corresponding ONNX file. The obvious downside of this is that an offline installer is not possible. This is not necessarily a big problem nowadays.

Shipping a prepopulated timing cache

First, I want to explain what a timing cache is and how you can acquire one.

While building an engine, you can evaluate the inference time of the evaluated execution plans. The results of this timing can be saved to a timing cache for later reuse through the --timingCacheFile option of trtexec.

A timing cache does not have to be model-specific. In fact, I recommend using a system-wide timing cache for all TensorRT build processes. Whenever a kernel timing is needed, the process checks the timing cache and either skips timing or extends the timing cache. This drastically reduces the engine build time!

The timing cache is usually only a few MB big or even under 1 MB. This makes it possible to include multiple caches in the initially shipped binaries of an application and drastically reduce build times.

You are still required to precompile all engines on multiple compute capabilities to populate the cache. However, for shipment, only a small additional file is needed.

As with the engines, it would be ideal to have one cache for every single GPU, but it is possible to ignore those mismatches in deployment across a compute capability. With this file present, building an engine on the user side is a CPU-only operation that can be done during installation without worrying about a busy GPU during timing. Using this approach of shipping a timing cache and the ONNX model enables a controllable engine-building process on the user device without downloading files on demand.

Further acceleration is possible using a new heuristic algorithm selection (--heuristic flag). It currently only prunes convolution tactics that are unlikely to be the fastest but it will be extended to other operations in future releases.

Conclusion

This post shows the options for TensorRT engine deployment to workstations without shipping vast amounts of binaries. The options also avoid putting you through a time-consuming and possibly incorrect building process in case of a busy GPU.

All the trtexec options mentioned in this post can be found in the open-source code of trtexec or by searching the C++ or Python API documentation for the respective features.

After reading, you should be able to decide which method is the optimal deployment choice for your application and find a balance between binary size and compute time on a user device. Be aware that mixing the strategy on multiple models is also an option.

For commonly used models, you might want to compile them in the installer with a timing cache. For other models that not all users need, download the respective ONNX or TensorRT engine on demand.

Learn more about TensorRT deployment in our forum, AI & Data Science.

Attend GTC 2023 for free from March 21-23 and see our AI for Creative Applications sessions.

This post is part of a series on accelerated data analytics:

• Accelerated Data Analytics: Speed Up Data Exploration with RAPIDS cuDF discusses how the pandas library provides efficient, expressive functions in Python.
• This post walks you through the common steps of time series data processing with RAPIDS cuDF. 

Because it is generally constrained to a single core, a standard exploratory data analysis (EDA) workflow benefits from accelerated computing with RAPIDS cuDF, an accelerated data analytics library with a pandas-like interface. Time series data notoriously requires additional data processing that adds time and complexity to your workflow, making it another great use case for leveraging RAPIDS. 

With RAPIDS cuDF, you can speed up your time series processing for “Goldilocks” datasets that are not too big and not too small. These datasets are onerous on pandas but don’t need full distributed computing tools like Apache Spark or Dask.

What is time series data?

This section covers machine learning (ML) use cases that rely on time series data and when to consider accelerated data processing.

Time series data is ubiquitous. Timestamps are a variable in many types of data sources, from weather measurements and asset pricing to product purchase information and more. 

Timestamps come in all levels of granularity, such as millisecond readings or monthly readings. When timestamp data is eventually leveraged in complex modeling, it becomes time series data, indexing the other variables to make patterns observable. 

The following prevalent ML use cases rely heavily on time series data, as do many more: 

• Anomaly detection for fraud in the financial services industry
• Predictive analytics in the retail industry
• Sensor readings for weather forecasting
• Recommender systems for content suggestions 

Complex modeling use cases often require the processing of large datasets with high-resolution historical data that can span years to decades, as well as processing real-time streaming data. Time series data goes through transformations, such as resampling data up and down to make the period of time consistent among datasets and being smoothed into rolling windows that denoise patterns.

pandas provides simple, expressive functions to manage these operations, but you may have observed in your own work that the single-threaded design can quickly get overwhelmed by the amount of processing required. This is especially applicable for larger datasets or use cases that need fast data processing turnaround, which time series–analysis use cases often do. The subsequent wait time for pandas to process the data can be frustrating and can lead to delayed insights. 

As a result, these scenarios make cuDF a uniquely good fit for time series data analysis. Using a pandas-like API, you can process tens of gigabytes of data with up to 40x speedups, saving the most valuable asset in any data project: your time. 

Time series with RAPIDS cuDF

To showcase the benefit of acceleration for exploring data with RAPIDS cuDF and how it can be readily adopted, this post walks through a subset of time series–processing operations from Time Series Data Analysis. This is a robust notebook analysis of a publicly available dataset of real weather readings, available in the RAPIDS GitHub repository.

In the complete analysis, RAPIDS cuDF executed with a 13x speedup (see the benchmarking section later in this post for exact numbers). Speedups typically increase as the full workflow becomes more complex. 

Extrapolating to the real world, this gain has real impact. When an hour-long workload can be completed in under 5 minutes, you can meaningfully add time back to your day.

Dataset

Meteonet is a realistic weather dataset that aggregates readings from weather stations located throughout Paris from 2016-2018, with missing and invalid data. It is about 12.5 GB in size. 

Analysis approach

For this post, imagine you are a data scientist who has received this aggregate data for the first time and must prepare this for a meteorological use case. The specific use case is open-ended: it could be a forecast, report, or input in a climate model. 

As you review this post, most of the functions should be familiar as they are designed to resemble operations in pandas. This analysis is structured to perform the following tasks: 

• Format the DataFrame.
• Resample the time series.
• Run a rolling-window analysis.

This post disregards several data inconsistencies that are addressed in the end-to-end workflow demonstrated in the notebook, Time Series Data Analysis Using cuDF.

Step 1. Format the DataFrame 

First, import the packages used in this analysis with the following command:

# Import the necessary packages
import cudf
import cupy as cp
import pandas as pd

Next, read in the CSV data.

## Read in data
gdf = cudf.read_csv('./SE_data.csv')

Begin by focusing on the meteorological parameters of interest: wind speed, temperature, and humidity. 

gdf = gdf.drop(columns=['dd','precip','td','psl'])

After the parameters of interest are isolated, perform a series of quick checks. Start the first transformation by converting the date column to the datetime data type. Then, print out the first five rows to visualize what you are working with and assess the size of the tabular dataset.

# Change the date column to the datetime data type. Look at the DataFrame info
gdf['date'] = cudf.to_datetime(gdf['date'])
gdf.head()
Gdf.shape

Output

The DataFrame shape (127515796, 8) shows 127,515,796 rows by eight columns. Now that the size and shape of the dataset are known, you can start investigating a bit deeper to see how frequently the data is sampled.

## Investigate the sampling frequency with the diff() function to calculate the time diff
## dt.seconds, which is used to find the seconds value in the datetime frame. Then apply the 
## max() function to calculate the maximum date value of the series.
delta_mins = gdf['date'].diff().dt.seconds.max()/60
print(f"The dataset collection covers from {gdf['date'].min()} to {gdf['date'].max()} with {delta_mins} minute sampling interval")

The dataset covers sensor readings from 2016-01-01T00:00:00.000000000 to 2018-12-31T23:54:00.000000000, at a 6-minute sampling interval. Confirm that the dates and times expected are represented in the dataset.

After a basic review of the dataset is complete, get started with time series-specific formatting. Begin by separating the time increments into separate columns.

gdf['year'] = gdf['date'].dt.year
gdf['month'] = gdf['date'].dt.month
gdf['day'] = gdf['date'].dt.day
gdf['hour'] = gdf['date'].dt.hour
gdf['mins'] = gdf['date'].dt.minute
gdf.tail

The data is now separated into columns at the end for year, month, and day. This makes slicing the data by different increments much simpler.

Experiment with the updated DataFrame by selecting a specific time range and station to analyze.

# Use the cupy.logical_and(...) function to select the data from a specific time range.
import pandas as pd
start_time = pd.Timestamp('2017-02-01T00')
end_time = pd.Timestamp('2018-11-01T00')
station_id = 84086001
gdf_period = gdf.loc[cp.logical_and(cp.logical_and(gdf['date']>start_time,gdf['date']

The DataFrame has been successfully prepared, with 13 variables and 146,039 rows. 

Step 2. Resample the time series

Now that the DataFrame has been set up, run a simple resampling operation. Although the data updates every 6 minutes, in this case, the data must be reshaped to fall into a daily cadence.

First set the date as the index, so that the rest of the variables adjust as time does. Downsample the data from samples every 6 minutes to one record per day, to yield one record for each variable on each day. Retain the maximum value of each variable for each day as the record for that day.

## Set "date" as the index. See what that does?
gdf_period.set_index("date", inplace=True)
## Now, resample by daylong intervals and check the max data during the resampled period. 
## Use .reset_index() to reset the index instead of date.
gdf_day_max = gdf_period.resample('D').max().bfill().reset_index()
gdf_day_max.head()

The data is now available in daily increments. Refer to the table as a check that the operation yielded the desired result. 

	date	number_sta	lat	lon	height_sta	ff	hu	t	year	month	day	hour	mins
0	2017-02-01	84086001	43.81	5.15	672.0	8.1	98.0	283.05	2017	2	1	23	54
1	2017-02-02	84086001	43.81	5.15	672.0	14.1	98.0	283.85	2017	2	2	23	54
2	2017-02-03	84086001	43.81	5.15	672.0	10.1	99.0	281.45	2017	2	3	23	54
3	2017-02-04	84086001	43.81	5.15	672.0	12.5	99.0	284.35	2017	2	4	23	54
4	2017-02-05	84086001	43.81	5.15	672.0	7.3	99.0	280.75	2017	2	5	23	54

Step 3. Run a rolling-window analysis

In the previous resampling example, points are sampled based on time. However, you can also smooth data based on frequency, using a rolling window. 

In the following example, take a rolling window of length three over the data. Again, retain the maximum value for each variable.

# Specify the rolling window.
gdf_3d_max = gdf_day_max.rolling('3d',min_periods=1).max()
gdf_3d_max.reset_index(inplace=True)
gdf_3d_max.head()

Rolling windows can be used to denoise data and assess data stability over time. 

	date	number_sta	lat	lon	height_sta	ff	hu	t	year	month	day	hour	mins
0	2017-02-01	84086001	43.81	5.15	672.0	8.1	98.0	283.05	2017	2	1	23	54
1	2017-02-02	84086001	43.81	5.15	672.0	14.1	98.0	283.85	2017	2	2	23	54
2	2017-02-03	84086001	43.81	5.15	672.0	14.1	99.0	283.85	2017	2	3	23	54
3	2017-02-04	84086001	43.81	5.15	672.0	14.1	99.0	283.35	2017	2	4	23	54
4	2017-02-05	84086001	43.81	5.15	672.0	12.5	99.0	283.35	2017	2	5	23	54

This post walked you through the common steps of time series data processing. While a weather dataset was used for the example, the steps are applicable to all forms of time series data. The process is similar to your time series analysis code today. 

Performance speedup

When running through the complete notebook with the Meteonet weather dataset, we achieved a 13x speedup on an NVIDIA RTX A6000 GPU using RAPIDS 23.02 (Figure 1).

Pandas on CPU (Intel Core i7-7800X CPU)	User: 2 min 32 sec Sys: 27.3 sec Total: 3 min
RAPIDS cuDF on NVIDIA A6000 GPUs	User: 5.33 sec Sys: 8.67 sec Total: 14 sec

Key takeaways

Time series analysis is a core part of analytics that requires additional processing compared to other variables. With RAPIDs cuDF, you can manage the processing step faster and reduce time to insight with the Pandas functions you are accustomed to.

To further investigate cuDF in time series analysis, see rapidsai-community/notebooks-contrib on GitHub. To revisit cuDF in EDA applications, see Accelerated Data Analytics: Speed Up Data Exploration with RAPIDS cuDF.

Register for NVIDIA GTC 2023 for free and join us March 20–23 for related data science sessions.

Acknowledgments 

Meiran Peng, David Taubenheim, Sheng Luo, and Jay Rodge contributed to this post.

]]>