NVIDIA Omniverse Replicator accelerates AV development


The gap between reality and simulation is getting smaller and smaller.

In the GTC keynote speech, NVIDIA founder and CEO Jensen Huang announced the launch of NVIDIA Omniverse Replicator, an engine used to generate synthetic data with ground truth to train AI networks. In the demonstration, Huang demonstrated the powerful functions of Omniverse Replicator when using DRIVE Sim to develop autonomous vehicles.

DRIVE Sim is a simulation tool built on Omniverse, which utilizes many functions of the platform. The data generated by DRIVE Sim is used to train the deep neural network that constitutes the perception system of autonomous vehicles. For the NVIDIA DRIVE team, synthetic data has always been an effective and critical part of its AV development workflow.

The deep neural network supporting the perception of autonomous vehicles consists of two parts: the algorithm model and the data used to train the model. Engineers invested a lot of time to improve the algorithm. However, due to the limitations of real-world data, the data aspect of the equation is still underdeveloped. These data are incomplete, time-consuming, and costly to collect.

This imbalance usually leads to the stagnation of DNN development, which hinders the progress of data that cannot meet the needs of the model. Through synthetic data generation, developers can better control data development and customize it according to the specific needs of the model.

Although real-world data is a key component of AV training, testing, and verification, it presents major challenges. The data used to train these networks is collected by sensors on the fleet during actual driving. Once captured, the data must be marked with real conditions. The labeling is done manually by thousands of labelers—a process that is time-consuming, costly, and may be inaccurate.

Using synthetic data to augment real-world data collection eliminates these bottlenecks, while allowing engineers to adopt a data-driven approach to DNN development, significantly accelerating AV development and improving real-world results.

Sim-Real domain gap problem

Synthetic data generation is a well-known tool for artificial intelligence training-researchers have been experimenting with video games, such as Grand Theft Auto The data was created as early as 2016.

However, unlike video games, the quality of perceptual DNNs is largely affected by the fidelity of the data to the real world-training on data sets that are not converted to the physical world will actually degrade the performance of the network.

This gap between simulation and reality is mainly manifested in two ways.The appearance gap corresponds to the pixel-level difference between the simulated image and the real image, which is caused by how The simulator generates data. The fidelity and material properties of renderers, sensor models, 3D assets all contribute to this.

The content gap may be caused by the lack of real-world content diversity and the difference between the simulated and real-world environments. These inconsistencies occur when the context of the scene does not match reality. For example, the real world contains dirty roads, sunken cars, and emergency vehicles on the side of the road, all of which must be reproduced in simulation. Another important factor is the behavior of participants, such as traffic and pedestrians-real interaction is the key to real data output.

Use Omniverse Replicator to narrow the gap between Sim-Real domains

Omniverse Replicator aims to bridge the gap between appearance and content.

In order to close the appearance gap, DRIVE Sim uses Omniverse’s RTX path tracking renderer to generate physically-based sensor data for cameras, radar, lidar, and ultrasonic sensors. Real-world effects are captured in the sensor data, including phenomena such as LED flickering, motion blur, rolling shutter, lidar beam divergence, and Doppler effect. These details even include high-fidelity vehicle dynamics, which is important because, for example, the movement of the vehicle during a lidar scan can affect the point cloud generated.

The other half of the sensor equation is the material. The materials in DRIVE Sim are physically simulated to achieve accurate beam reflection. DRIVE Sim includes a built-in lidar material library and an upcoming radar and ultrasonic material library.

The sensor functions of DRIVE Sim include path tracking cameras, radar and lidar models, which can capture real-world effects such as motion blur, LED flickering, rolling shutter and Doppler effect.
Early morning scene rendered with NVIDIA DRIVE Sim RTX path tracer.
DRIVE Sim uses the RTX path tracker to render these morning and night scenes with incredible fidelity.

One of the main ways to solve the content gap is to use more diverse assets at the highest fidelity. DRIVE Sim leverages the capabilities of Omniverse to connect to various content creation tools. However, to generate a suitable scene requires the context to be correct.

Behind the scenes, Omniverse Replicator uses a feature called domain randomization to organize data for quick scene manipulation. DRIVE Sim includes tools for this and scenario building to create a large amount of different data while maintaining the real-world context. Because Omniverse Replicator also has time accuracy and certainty, data sets can be created in a repeatable manner.

DRIVE Sim provides tools to generate random scenes in a controllable and repeatable way, adding variety to the generated data.

Real result

DRIVE Sim has achieved significant results in NVIDIA’s use of synthetic data to accelerate perception development.

An example is the migration to the latest NVIDIA DRIVE Hyperion sensor set. The NVIDIA DRIVE Hyperion 8 platform includes sensors for complete production AV development. However, before these sensors came out, the NVIDIA DRIVE team was able to use synthetic data to provide DNNs for the platform. DRIVE Sim generated millions of images and real data for training. Therefore, once the sensors are installed, the network can be deployed, saving precious months of development time.

In another example, when the vehicle is not in the center of the lane, the PathNet DNN that detects the drivable lane space is difficult to determine the path. Collecting this type of data is difficult because partly driving out of the lane is dangerous (and violates NVIDIA’s data collection policy). By training the network on millions of off-center driving path synthetic images, DRIVE Sim significantly improves the accuracy of PathNet.

The same is true for LightNet, which detects traffic lights, and SignNet, which detects and classifies road signs. Due to lack of data, these networks have difficulty identifying extreme angle lights and misclassified signs under certain conditions. Engineers can design data to augment real-world data sets and improve performance.

By training two DNNs on synthetic data covering these problem areas, the performance is rapidly improved and the bottleneck in the development process is eliminated.

See what humans can’t see

Synthetic data changes the nature of DNN development. It saves time and economy, and provides engineers with the freedom to generate customized data sets on demand.

Developers can specify elements such as weather, lighting, pedestrians, road debris, etc. They can also control the distribution of elements, such as specifying specific combinations of trucks, buses, cars, and motorcycles in a given data set.

Synthetic data provides basic facts that humans cannot label. Such as depth information, speed and multi-sensor tracking. This kind of ground truth information can significantly enhance perception.

It also helps to label components that are difficult (and sometimes impossible) to mark. For example, pedestrians walking behind a car cannot be correctly marked by humans when they are blocked. However, through simulation, even if the information is not visible to humans, accurate ground truth can be automatically obtained at the pixel level.

A key feature of synthetic data is to provide accurate ground truth labels for difficult or impossible scenes in the real world, such as scenes where pedestrians are blocked when a car passes by.

Clear the way forward

As a modular, open and extensible synthetic data generation platform, Omniverse Replicator brings powerful new functions to deep learning engineers. DRIVE Sim uses these new features to provide AV developers with the ultimate flexibility and efficiency of simulation testing.

It allows engineers to create data sets needed to accelerate work.

The result is that DNN is more accurate, has shorter development time, and can bring safer and more efficient autonomous driving technology to the road faster.

Learn more about DRIVE Sim and accelerate the development of today’s safer and more efficient vehicles.

Catch up with the full GTC keynote:

Source link

Related Articles

Leave a Reply

Your email address will not be published.

Back to top button