Self-driving cars often grab headlines because of their on-road activity, such as when Alphabet’s Waymo recently launched its Waymo One taxi service in the Phoenix area, or when a Volvo serving as an Uber test vehicle struck and killed a pedestrian in Tempe, Arizona, last year. But much of the effort to turn driverless cars into reality takes place out of public view, in the virtual world. 
                                                                                                
Autonomous vehicle companies use highly detailed computer simulations to safely tweak their vehicles before putting them on the road. This software-based technology allows engineers to drive many more miles — potentially millions of miles a day — and try out many more scenarios than they could ever hope to go through in the real world. 
 
According to ABI Research, “accruing the tens of billions of miles necessary to have confidence in autonomous systems would require the deployment of at least three million unproven autonomous vehicles over the course of 10 years.” But Waymo, which leads the market in autonomously driven miles, covers as many autonomous miles in one day with simulations as it did from 2009 to 2018 with its physical prototypes, ABI wrote in a recent report.
 
“As people have grown in their maturity about what needs to be done for autonomous vehicles, it’s become clear to them that the number of cases for which they need to validate the vehicle is just physically impossible to do” with on-road testing, said Sandeep Sovani, global director for the automotive industry at Pennsylvania-based software firm ANSYS. “And, therefore, unless they do simulations, they will not be able to validate their vehicles for safety and performance across all the situations they are likely to encounter.”
 
Simulation allows automakers to try out scenarios they cannot or do not want to recreate in the real world. Cruise, a San Francisco-based firm that is working with parent company General Motors to develop driverless cars, said its simulation team can safely “simulate the most horrific traffic accidents and unexpected road experiences to train how our cars can respond.”
 
Driverless car simulations are sophisticated. For example, Cognata, an Israeli simulation firm that has partnered with Audi’s self-driving unit, Autonomous Intelligent Driving, uses “artificial intelligence, deep learning and computer vision to create a realistic automotive simulation environment in its entirety,” says Heikki Laine, Cognata’s vice president of product and marketing. “Virtual cars travel virtual roads in virtual cities — all in remarkably true-to-real-world conditions, including weather and geo-specific AI-based traffic.” 
 
Some car companies, such as Waymo, have developed their own simulators in-house. But such know-how is difficult and costly to acquire. Sensing a major business opportunity, many software firms are now offering simulation products to car firms.
 
When simulation provider Metamoto was founded in 2016, “we saw that massively scalable simulation was going to be needed” by the driverless car industry, said Chad Partridge, CEO of the Redwood City, California-based firm. “Building a simulator is not necessarily a core competency of any autonomous system business.” 
 
In-house simulation
                                                                                       
Waymo, formerly the Google self-driving project, says it uses a suite of custom-built sensors, including cameras, laser radar (lidar) and radar, to build virtual replicas of particularly “tricky” hazards, such as a busy intersection with a flashing yellow arrow for left turns. It then tests the ability of its software to navigate those simulated hazards.
 
“Every time we update the [self-driving] software, we can test the change at the same intersection in a variety of driving conditions,” the Mountain View, California-based company wrote in its 2018 safety report. “That’s how we were able to teach our vehicles to naturally inch forward at that flashing yellow light, and slot in after oncoming traffic.”
 
To account for as many scenarios as possible, Waymo varies road conditions in its simulator, such as by changing the speed of oncoming vehicles, altering the timing of traffic lights, and adding simulated pedestrians, motorcycles riding between lanes, or joggers darting across the street.
 
“Each day, as many as 25,000 virtual Waymo self-driving vehicles drive up to 8 million miles in simulation, refining old skills and testing out new maneuvers that help them navigate the real world safely,” Waymo said.
                                                                                       
Pittsburgh-based Argo AI, which is working with Ford to develop self-driving cars, also devised its simulation capability in-house. Argo’s simulation uses both “software in the loop,” in which self-driving software is run on servers, and “hardware in the loop,” in which the software is tested on the computer hardware that will be used in vehicles.
 
Zoox, which was founded in 2014 and is developing all-electric, self-driving rideshare vehicles, is constantly improving its simulator and is testing it against a range of environmental, road and traffic conditions, according to Bert Kaufman, the head of corporate and regulatory affairs at the Foster City, California-based firm.
 
“We've been developing our simulation capabilities since day one and have a team of simulation developers who come from a range of disciplines, including many from the video game industry,” Kaufman said.

Metamoto says its service allows customers to quickly create unique scenarios. Image: Metamoto.
 
Simulation providers
 
Among simulation providers, Metamoto is one of the newest. In August, the startup launched Simulation as a Service, a cloud-based, subscription-based, video-like product that autonomous vehicle makers and their suppliers can use to try out their wares in virtual scenes created by high-definition maps. 
 
Simulation as a Service consists of three main parts: a designer, which builds scenarios; a director, which runs simulations; and an analyzer, which assesses and debugs software. To put cars or components through their paces, Metamoto customers can vary environmental factors, including weather, traffic, road conditions and pedestrians, and hardware, including vehicle properties, sensor placement and settings, and latency in communication between components. 
 
“The breadth of this is very, very big,” Partridge said. “We’ve got the whole end-to-end covered.”
 
Metamoto continues to refine its product. In January, it added a “rapid custom scene generation” tool provided by another new software developer, VectorZero of Carlsbad, California. In addition, it recently arranged to begin offering its product through AutonomouStuff, a global marketplace for the robotics and autonomous systems industry.
 
Partridge said he is pleased with the reception that Simulation as a Service has received. As of mid-January, Metamoto had lined up a dozen customers, including car manufacturers, top-level car suppliers, providers of autonomy-enabling sensors and software, and ride-hailing businesses. 
                                 
The feedback has “been extremely positive so far,” he said. “We haven’t lost a customer.”
 
Partridge, a former AUVSI board member, says he expects Simulation as a Service to have applications to many areas besides driverless cars, such as unmanned aircraft and ground robotics.
 
“We’re already getting contacted by mining groups and by groups that are doing warehouse robotics and farming,” he says.  
 
MathWorks, a Boston-area software developer founded in 1984, introduced the Automated Driving System Toolbox in 2017. 
 
The Toolbox, which is typically installed on desktop and laptop computers, allows engineers to build and test the three types of software that run a self-driving car: controls, which affect the vehicle’s movement; perception, which extracts and fuses information from camera, lidar and radar sensors; and planning, which lays out the vehicle’s path.
 
The Toolbox permits all three software types to be tested together to simulate driving a car. It also allows individual components of each software type to be tested so that design errors can be detected before the components have been integrated.
 
“Full system integration testing is critical, but finding bugs during system integration testing is expensive, so we believe engineers should test individual components first, then incrementally integrate to achieve system-level integration,” MathWorks says.
 
ANSYS, which was founded in 1970, has a mix of new and enhanced software products that can simulate a driverless car or individual components. Its products can test both hardware and software.
 
“We are working with many key driverless car manufacturers around the world, as well as most of the major component manufacturers,” Sovani says. 
 
NVIDIA, which introduced Drive Constellation at a technology conference last year, says its cloud-based product provides “photo-realistic” simulation. During Drive Constellation’s “hardware-in-the-loop” testing, one server simulates the driving environment, while another contains the computer that will eventually drive the real car.

Danny Shapiro, NVIDIA’s senior director of automotive, said the Santa Clara, California-based firm, has designed its product to be “open,” which allows third-party developers to “provide countless cityscapes, vehicle dynamics models and traffic scenarios. The same situation can be run over and over again, with a nearly infinite number of changes each time — from time of day to weather to an animal crossing the road.”

Shapiro also touted his product’s “human-in-the-loop testing,” in which engineers test a driverless car by taking over other vehicles in the virtual environment and seeing how their vehicle responds to sudden cut-offs or tailgaters. This feature, he said, allows NVIDIA “to add another layer of unpredictability.”

Above: Argo AI, which is working with Ford to develop self-driving cars, devised its own simulation capability. Photo: Ford. Below: One of Waymo's fleet of self-driving vans. Photo: Waymo