Alphabet Inc’s Waymo and General Motors’ Cruise are leading a backlash in opposition to a California reporting requirement on self-driving vehicle test data that the companies claim is not relevant or appropriate in measuring performance or progress.
Data released Wednesday by California revealed Waymo and Cruise once again had the greatest number of test miles between “disengagements,” when a human driver must take over control from a self-driving system during testing on public roads.
The disengagement data is largely used as a yardstick to compare companies testing self-driving automobiles on California roads, and is often cited as proof that Waymo leads the sector.
Waymo tweeted that the disengagement metric does not give “relevant insights” nor does it distinguish the company’s “performance from others in the self-driving space”.
At an investor conference on Wednesday, Dan Ammann, CEO of GM’s Cruise subsidiary, sidestepped a query regarding the relevance of the California disengagement data, stating “there is no really great way to track” the company’s progress, other than their own internal data.
Data provided to California revealed Waymo had 13,219 miles between disengagements, compared with 11,017 miles in 2018. Cruise reported an even higher improvement, with 12,221 miles between disengagements in 2019, compared with 5,205 miles the earlier year.
Waymo stated its self-driving test vehicles logged 1.45 million miles on California roads last year, where as Cruise vehicles tallied 831,000 miles, majority of them on the streets of San Francisco.
Mark Rosekind, a former head of the National Highway Traffic Safety Administration (NHTSA) and current chief safety innovation officer for self-driving startup Zoox, addressed the inconsistency between testing on congested city streets and on freeways, and said “we have found that not all miles are created equally”.
“Our focus has been on the quality of these miles, not the quantity,” Rosekind said, noting that Zoox more than doubled the amount of test miles in 2019.
Aurora co-founder Chris Urmson, who earlier headed Waymo’s self-driving program, wrote last month that the numbers mean little when there’s no obvious definition of “what constitutes a disengagement”.
The larger self-driving firms want to incorporate more and different data when calculating performance and progress, including miles logged in computer simulations.
Waymo also noted that most of its real-world system validation data comes from Phoenix, where the company has been testing self-driving vehicles for years and runs a small commercial ride sharing fleet.
The company’s self-driving performance in Phoenix is “largely unrelated” to their California testing, which Waymo said is mostly for engineering development.
“We don’t think (California) disengagement data should be used to compare performance, or judge readiness or competency,” Waymo stated.
