Google Duplex: Understanding the Core Technology Behind Assistant's Phone Calls

Top Google executives, led by CEO Sundar Pichai, took the stage at the company's annual developer conference to reveal new features for its products and services. Some that we found pretty interesting (not to mention useful) were the new Smart Compose feature in Gmail and the integration of Assistant in Maps.

But the feature that stole the show has got to be that demo powered by Google Duplex, the company's new experimental technology which lets an AI-powered bot make phone calls on your behalf. Sounds crazy, right?

Well, Google just showed the world that such a technology exists and it will be available via Google Assistant in the near future. The demo - which showed a voice assistant call a hair saloon and book an appointment on behalf of a customer - was nothing short of amazing, and it just goes to show how far ahead Google is in terms of voice-based AI assistants and conversational, natural language processing.

At the core of it, Duplex is a recurrent neural network that can be trained for highly specialised tasks and it uses Google's automatic speech recognition technology, so it can interface with the user. Thanks to WaveNet, an AI-based generative program that's part of Google's DeepMind division, Duplex can sound a lot closer to a human than ever before, complete with “umms" and “aahs”, as seen in the video above.

The cool thing about Duplex is that it enables the Assistant to relieve you of mundane tasks, like having to make a restaurant reservation or a hair salon appointment. The creepy part is the other party has no idea they are talking to a machine, something that can be abused in any number of different ways.

This feature is said to be rolling out in the coming weeks in the US as an experiment to “normal users”. Google will be focusing on a few specific scenarios initially, before expanding it further.

“It is something that we are really thoughtful about and thinking hard, about how we bring this to market,” said Lillian Rincon, Product Management Director for the Google Assistant during a media briefing at Google I/O in Mountain View, California.

So, how does this actually work? From the looks of it, it would be like giving any other command to the Assistant. For example, when booking an appointment at a salon, you will have to give the Assistant some key parameters to work with, like the name of the salon and the timings you wish to book. If the company or business has an online booking service, then it will book it through that, else it would make a phone call.

Rincon explained that the Assistant will also check the working hours, so if it's too early in the day, it will let you know, and suggest booking later in the day. If the booking is successfully done, you then get a notification and a calendar entry, where you can check if the appointment was made correctly. Also, Duplex won't place a traditional call to the business as it will use a different number for this, instead of using your phone number.

Image credit: Google

Getting a machine to mimic a human is no easy feat, which is why Duplex will mostly be used for very specific tasks that have the lowest chance of things going wrong. Google makes it very clear in a blog post that it cannot carry out general conversations.

Duplex is designed to respond in a natural manner and adapt to responses in real time, while understanding the context of the conversation. However, if it finds the conversation too complex, Rincon says, “the algorithm is smart enough to understand it's failing or can't do something and in that case, it will get a human to come and take over.” This means there would be some factor of human intervention in the process, should things not go as planned.

This convenience also raises a couple of pertinent questions, the biggest one being, would the person on the other end of the line be notified that he or she was speaking to a machine? While Rincon didn't address this question directly, a quick look at a Google's blog post on Duplex suggests that businesses would in some way have to be “supported by Duplex,” which means the person who answers the call would be aware that calls coming in might not always be from humans.

Furthermore, Rincon points out, that the goal of the experiment is not to trick the other person into thinking its a human. “We are still deciding how we are going to introduce this technology to the world,” she stated. Other areas of concern would be, can someone misuse it to make random reservations or appointments? And how much of your personal data and information would Duplex access during its experimental phase?

Google Duplex is still very much an early experiment so its deployment depends entirely on how successful its trial runs are and how people respond to it. We are quite excited to see how this technology evolves as based on what we've seen, it could prove to be a real game changer when applied correctly.

Disclosure: Google sponsored the correspondent's flights and hotel for the trip to Google I/O.

We discussed Android P, Google Assistant, Google Photos, and also the most important things that Google did not mention during its I/O 2018 keynote, on Orbital, our weekly technology podcast, which you can subscribe to via Apple Podcasts or RSS, download the episode, or just hit the play button below.