On October 8th, Waymo opened its fully autonomous ride-hailing service to the general public in Phoenix. Right now members of the public are hailing vehicles with no human driver controlling the car – either in the vehicle or remotely – to help them get to where they’re going as part of their everyday lives.
This milestone is not just the output of many years’ experience in developing the Waymo Driver – it’s also the result of over a decade of developing safety practices. Since it started in 2009 as the Google Self-Driving Car Project, it has spent years refining a comprehensive set of robust methodologies that assess safety across its technology, operations, and team culture, and ultimately guide the deployment and safe operations of the Waymo Driver.
Waymo take responsibilities to the communities it serve seriously. Now that it has opened up fully autonomous ride-hailing service to the public, also sharing in detail the safety framework that guides its progress.
Waymo wants to give everyone who rides with the Waymo Driver a deeper understanding of the incredibly high standards to which it holds itself in the safety practices that govern its deployments. Waymo is publishing two papers that explain the processes it use to drive fully autonomously on public roads and validate the safety of its operations.
The first, Waymo’s Safety Methodologies and Safety Readiness Determinations include details of its Safety Framework – the careful and multilayered approach to safety that has helped make it possible to deploy fully autonomous vehicles on public roads.
The second, Waymo’s Public Road Safety Performance Data, analyzes the miles it has driven on public roads in Arizona – one of 10 states it has driven in the U.S. since its founding – to provide data about its safe operations in practice.
This is the first time an autonomous technology company has released a framework describing the safety of its fully autonomous commercial operations. It believes this transparency and accountability are important for demonstrating the trustworthiness of its operations, and critical to deepening the dialogue around autonomous driving safety. Waymo looks forward to ongoing engagement with experts and academia to help ensure its work can continue to evolve and grow.
Waymo’s Safety Framework: Safety Methodologies and Safety Readiness Determinations
There is currently no universally accepted approach for evaluating the safety of autonomous vehicles – despite the efforts of policymakers, researchers, and companies building fully autonomous technologies. Since it began its work in 2009, it has worked to develop a robust Safety Framework – with multiple complementary methodologies – that embeds safety in all aspects of technology and product development. This framework draws on insights from the research and automotive communities and is based on its extensive experience in building autonomous systems.
By publishing the framework, Waymo aims to transparently share details about its approach; show how it evaluates its safety performance; demonstrate how it understands and measure its readiness for safe autonomous operations at scale, and invite perspectives from experts as it continues to expand its operations.
As industry progresses, it encourage other autonomous driving companies to develop and publish guiding frameworks and principles that can demonstrate how autonomous systems are safely and responsibly deployed.
Waymo’s Safety Framework, available here, maps out its approach to safety in three fundamental layers:
1. The Hardware Layer
Waymo call its automated driving system (ADS) the Waymo Driver. A vehicle equipped with the Waymo Driver has four main subsystems, which form the ‘hardware layer’. This includes the vehicle itself; the systems used for steering and driving; the sensor suite built into the vehicle; and the computational platform used to run its software.
Safety Framework explains:
- How Waymo verify and validate the performance of these components.
- How Waymo select its base vehicles with safety performance in mind.
- Its use and testing of backup steering and control actuators.
- How Waymo specify and measure the performance of vehicle sensors.
- The backup computers and power sources within computational platform.
In combination, these steps have led to the development of a robust hardware stack that has been designed around the goal of ensuring safety in any scenario. You can read about the latest generation of hardware here.
2. The ADS Behavioral Layer
There are three primary capabilities upon which it evaluate the performance of the Waymo Driver’s behavioral layer:
- Avoiding incidents with other vehicles or users of the roads.
- Successfully completing trips in fully autonomous mode.
- Adhering to applicable driving rules where the vehicle is operating.
Waymo use a series of methodologies to evaluate these capabilities, including:
- Hazard analysis techniques to design for the robustness of the Waymo Driver from the beginning of the development process.
- Scenario-based testing, both in simulation and closed-course test environments, to verify the behavior and performance of the Waymo Driver.
- Simulated deployments to evaluate the aggregate performance of the Waymo Driver over a large number of miles and a wide variety of situations.
3. The Operations Layer
The first two parts of the Safety Framework lead to the development of safe, capable autonomous vehicles. The final part – operations – reflects the importance of the systems, approaches and culture it has been building around its technology to help ensure it is safe. Safety Framework sets out the inner workings of several important operations processes at Waymo:
- Fleet Operations includes a broad range of monitoring and support functions to help ensure the safe operation of the Waymo Driver and vehicles. For example, while the Waymo Driver handles the entire driving task, fleet response team can confirm what its ADS is seeing and provide additional information.
- Risk Management Program enables to proactively identify and resolve potential safety issues that are triggered by ongoing technological change, categorizing and prioritizing potential risks so they can be mitigated.
- Finally, Field Safety Program helps collect, assess and resolve potential safety concerns that occur during real-world operations. Waymo draw from many sources, including employees, riders, and the public.
Waymo’s governance structures are important tools that help monitor and review safety-related decisions within company. Safety Framework details this work, which includes the role of the Waymo Safety Board. This group works together to help resolve safety issues, approve new safety activities, and ensure the overall approach to safety continues to evolve and improve. Waymo uses these methodologies to assess and continuously improve the safety of the Waymo Driver. Waymo also use them collectively to make an assessment about whether the Waymo Driver is ready to make the next step – as it did before opening fully autonomous operations for ride-hailing to the public in Phoenix earlier this month.
Waymo establishes rigorous performance criteria for its Driver in a particular operating environment and drawing on all safety methodologies to determine whether the Driver is ready to be tested or deployed in new conditions without creating any unreasonable risks to safety.
Waymo’s Public Road Safety Performance Data
As broaden the availability of Waymo services and vehicle platforms enabled by the Waymo Driver, Waymo wants to be sure that the communities in which it operates understand the safety approach it takes so they can be confident in the safety of its products.
In addition to the Safety Framework, today it is sharing a white paper on Waymo’s Public Road Safety Performance Data. There is not yet a globally or nationally recognized framework to assess the relative safety of autonomous vehicles – so it is taking this opportunity to share safety-relevant data with peers, stakeholders, and communities.
The paper includes results from 6.1 million miles of automated driving with a trained vehicle operator in Arizona in 2019. It also includes an additional 65,000 miles of fully autonomous operations – operating at Level 4 autonomy, with no human driver behind the wheel and no remote operators – from 2019 and the first nine months of 2020.
This data represents over 500 years of driving for the average licensed U.S. driver – a valuable amount of driving on public roads that provides a unique glimpse into the real-world performance of Waymo’s autonomous vehicles. The data covers two types of events:
- Every event in which a Waymo vehicle experienced any form of collison or contact while operating on public roads
- Every instance in which a Waymo vehicle operator disengaged automated driving and took control of the vehicle, where it was determined in simulation that contact would have occurred had they not done this
The paper also includes analyses of the events by type, and provides details of the most potentially significant events, including diagrams. This data is unlike almost all driving data available to the world, as it includes every collision or contact, even those with no damage that would not be reported to insurance companies or local police departments. The majority of incidents in its report fall into this category of typically unreported minor collision or contact.
In addition to sharing data, the paper describes the simulation processes Waymo uses to understand how events would likely have played out in situations with on-board vehicle operators where they chose to disengage the Waymo Driver. These simulations use models of human behavior, and help us better understand how vehicles and other road users interact.
The paper is available at www.waymo.com/safety.
