The Waymo Driver’s training regimen: How structured testing prepares their self-driving technology for the real world
Imagine a tiny city where you control everything that happens on the streets. You manage how many cars zip down the roads and how fast they are going. You dictate how many cyclists are on a roundabout or whether they follow the road rules. The “weather” around the vehicle can change multiple times a day from blue skies and sunshine one minute to heavy rain showers the next, but only if you want it that way. One may say such a city doesn’t exist, but if you drive out to the middle of Merced County in California, you’ll find it at Castle, a former Air Force Base their team uses to help build the World’s Most Experienced Driver™.
Spanning 113 acres, Castle is a state-of-the-art, closed-course testing facility — which has been specially designed and redesigned over the years for Waymo’s unique testing needs to prepare the Waymo Driver to handle challenges on the road, evaluate new software before it’s released to their self-driving fleet, and validate their system’s performance. While their closed-course testing at Castle has always been a critical component of their development process, over these past few months of shelter-in-place, Castle has produced even more benefits. It has helped test and scale the new policies they’ve put in place to keep their team and entire community safe, while unlocking some key technology advancements for their teams working at home. Now that their vehicles are back on the road, they can accelerate the Waymo Driver’s deployment by enabling the entire development cycle — from gathering real-world data and analyzing it for interesting encounters, to rigorously testing updates they make to the system and evaluating its performance.
Mastering the fundamentals to handle the one in a million
Waymo’s testing site within the former base is set up like an adaptable city, including everything from wide avenues and suburban driveways to a railroad crossing and roundabouts, similar to where their vehicles operate in the real world. At Castle, they stage complex and rare scenarios — such as a person walking out of a porta-potty onto a street or a pile of garbage falling out of the truck in front of their— in a safe, controlled environment and test them over and over again, changing different variables, to ensure that their self-driving technology can handle the wide range of situations it might come across.
Over the years, they’ve been able to amass a library of over 40,000 structured testing scenarios — not counting all the variations they create with each situation. These scenarios include things that they have never seen on public roads but could imagine happening and others that occur once every hundreds of millions miles. Additionally, they create some relatively mundane tests that can also be quite challenging for self-driving vehicles and humans alike. For example, driving behind a large garbage truck on a narrow street — that stops every hundred feet or so and leaves empty trash cans strewn about — may create a lot of complexity on the road.
Their teams develop these scenarios in a variety of ways. They use national databases to identify the most common traffic accident scenarios, recreate situations their trained drivers see while testing on public roads, create scenes specifically to test new driving functions developed by their engineers, and field suggestions from team members based on their driving experience. By exposing the Waymo Driver to a wide variety of scenarios and teaching their self-driving technology fundamental skills rather than only to handle individual situations, their vehicles become more equipped to handle any situation they encounter on the road, even if it hasn’t seen that specific scenario before.
How structured testing and simulation complement each other
Testing is not a one and done task, but rather a never-ending feedback loop that includes structured testing, simulation, and public road operations. When they develop a scenario to test new software, they can use either one of these tools or a combination of them. For example, after executing a test on their private track, they can then create and run hundreds of variations of that scenario in simulation. Their simulation technology allows them to do this in a matter of seconds right at their developers’ desks as they work on new features for the Waymo Driver.
As much as simulation can help scale the value of structured testing, structured testing complements their simulation. While they maintain a high level of realism in simulation, structured testing allows them to evaluate both their software and hardware stack. By recreating scenarios their vehicles have already successfully completed in simulation, they can continuously verify their simulator and assess small nuances that may affect vehicle behavior. Adding this extra layer of testing and redundancy into their development cycles helps them to ensure they have done their due diligence before their vehicles drive on public roads.
Testing Waymo’s fifth-generation hardware suite on the electric Jaguar I-PACE at Castle
Not only does structured testing help verify software updates, it also allows them to test and validate their hardware. With closed-course testing, they can evaluate their integrated system’s performance across their vehicle and sensing systems.
Over the past decade, they’ve changed the number of vehicle platforms and sensor suites they test and operate on five times — from Lexus 450 SUVs to their custom-built Firefly, from Chrysler Pacifica minivans to 18-wheeler trucks, and now to all-electric Jaguar I-PACES. All these platforms have different body builds and require different modeling for sensors, perception, motion control, and planning. Structured testing allows them to test various system capabilities, such as how their sensing and computer vision systems work together to detect and identify a speeding motorcycle at long distances. With structured testing, they can also measure how well their motion control system follows the intended path, and validate their system’s performance and reliability across more diverse operating domains through weather testing in rain tunnels and thermal chambers, across speed bumps and potholes.
Like the platforms that came before it, their custom-built 5th generation sensor suite has been rigorously tested at Castle to help ensure its safety and readiness for public roads. With each generation, they have matured their models and their methods, so that each new bring up is smoother and quicker, enabling them to get on the roads to serve their riders and Waymo Via partners sooner. And, if you are in the Bay Area, Phoenix, or Detroit area, you should see more of their next-generation Waymo Driver driving around.
As they continue to build the hardware and software to power the Waymo Driver, they’re looking for people to join their growing team. At Waymo, their teams motivate and inspire one another, see their research implemented in tangible ways, and together make real steps toward a positive impact on the world of mobility. Whether you’re an engineer, researcher, or a curious and critical thinker driven to make the roads safer for everyone, they’re looking for people to help them tackle real-world problems.
