Google Duplex AI Calls: We Tested It

Unveiled at Google I/O 2018, Google’s AI system for accomplishing real-world tasks over the phone called Google Duplex, will make its live debut this summer. We talked to it while in Mountain View, California, and it works just as demonstrated during Google’s developer event reveal.

When it was first presented on stage, Google Duplex unquestionably wowed the crowd, creating awe and surprise in the developer community. When word reached outside the event, some were wondering how people would react to an AI call, and whether the AI could even get the job done in a real situation. Was the demo extremely scripted?

If you are unfamiliar with Google Duplex, watch the official video below and/or the Google IO reveal recording:

Real-world test

When Google asked if we wanted to try the AI, I drove to Mountain View, CA. After a short briefing on the technology, journalists took on the role of the restaurant staff at Oren’s Hummus Shop, answering reservation calls from Google Duplex. We could talk to it however we wanted, and we could even try to challenge the AI. It was a revealing experience.

For my part, I opted to be as natural as possible and even verbally confirmed the reservation parameters at the end. Google Duplex understood, checked and confirmed. Others challenged the system, adding some errors in the reservation to see if the AI would catch them.

"THE AI WOULD CATCH THE ERROR AND POLITELY CORRECT THE PERSON"

For example, if the AI tried to book a table for three people, one of us would say at the end : “Okay, so it’s a reservation for four,”. The AI would catch the error and politely correct the person “hum… actually, t’s for three”. Overall, it worked very well, and Google estimates that 4/5 calls would end up with the task being performed. While not perfect, it is an excellent start and should improve over time.

At the beginning of the call, Google Duplex introduces itself as being an automated system: “Hi, I’m calling to make a reservation. I’m Google’s automated booking service, so I’ll record the call”.

This is something people asked about because the AI voice can sound freakishly human. During the initial demo at Google I/O, many were laughing because the human on the line didn’t realize he/she was speaking to a machine. It’s funny but potentially disturbing enough that disclosure is needed. Google Duplex will follow the company’s AI principles:

• Be socially beneficial
• Avoid creating or reinforcing unfair bias
• Be built and tested for safety
• Be accountable to people
• Incorporate privacy design principles
• Uphold high standards of scientific excellence
• Be made available for uses that accord with these principles

Designed for particular tasks

It is essential to understand that Google Duplex is not designed to be a general AI, even if it sounds very savvy. Like many other types of perceptual computing, the system is designed to perform a specific task. Here, it is “booking a restaurant reservation”. If during a Duplex call, the human starts asking for the weather report, Google Duplex will not be able to answer that, and will try to get back to the main topic (in a polite way).

" I’M GOOGLE’S AUTOMATED BOOKING SERVICE"

During development, engineers have created a certain scope of events that the AI is training for. These are possible scenarios that are commonly encountered when performing the task, and they serve as waypoints or sub-tasks that indicate progress. For a restaurant booking, the AI knows that it needs to confirm the time, date, number of people and if the booking was made.

There are various things that have not been worked out. For example, if the restaurant asks about possible allergies, this is not something that Duplex can answer today, although it might be possible at some point. Making reservations is one skill that has been developed so far, and for additional tasks, the AI will need specific training.

60% of small businesses who rely on customer bookings do not have an online booking system set up (Google Consumer Survey, April 2018)

Not perfect, but already extraordinary

During our tests, it was possible to push the system to the limit by asking for things it didn’t know about (gluten allergies, and other things indirectly related). If the AI can’t make progress with the reservation, it can fall back to a human operator who will take over. There are corner cases that Google might not have bumped into yet and the human fallback will not only make sure that the businesses don’t waste time. Also, it should help Google identify new friction points and correct them.

Sometimes our attempts to challenge the system seemed slightly unnatural, but it was a good experiment to see how far one could push it. Everyone’s assessment of “how good” it is will vary. From a user perspective, this is certainly good enough for a real-world deployment on a small scale. This is exactly what Google intends to do.

"THE START OF SOMETHING MUCH LARGER"

This feels like the start of something much larger. This technology will surely improve drastically with more field experience. It could eventually be extended to other languages, more situations and could open the door to more applications. It is exciting.

Our metric for success is: Google Duplex should save time. It is not very difficult to envision situations in which spending a few seconds asking Google Assistant to make a call via Google Duplex could save you minutes, if not much more, depending on the wait time (Imagine calls to the DMV or many administrations…).

Other companies like Microsoft are also working on similar technologies, but Google is firmly ahead at the moment. Google does not yet know if and when the service can be extended to more businesses and people. This summer’s test should provide more data for the next step.

Hubert Nguyen
Categories: General
Tags: AI (Artificial Intelligence), Editorspick, Google, Google Assistant, Voice Assistant