New Lip-Sync Technology Uses Audio Clips to Generate Realistic Video

Advertisement
By Press Trust of India | Updated: 12 July 2017 18:39 IST

Scientists have developed new computer algorithms that can turn audio clips into a realistic, lip-synced video of the person speaking those words.

The researchers successfully generated highly-realistic video of former US President Barack Obama talking about terrorism, fatherhood, job creation and other topics, using audio clips of those speeches and existing weekly video addresses that were originally on a different topic.

Advertisement

"These type of results have never been shown before," said Ira Kemelmacher-Shlizerman, an assistant professor at the University of Washington (UW) in the US.

"Realistic audio-to-video conversion has practical applications like improving video conferencing for meetings, as well as futuristic ones such as being able to hold a conversation with a historical figure in virtual reality by creating visuals just from audio," said Kemelmacher-Shlizerman.

Advertisement

In a visual form of lip-syncing, the system converts audio files of an individual's speech into realistic mouth shapes, which are then grafted onto and blended with the head of that person from another existing video.

Advertisement

The team chose Obama because the machine learning technique needs available video of the person to learn from, and there were hours of presidential videos in the public domain.

"In the future video, chat tools like Skype or Messenger will enable anyone to collect videos that could be used to train computer models," Kemelmacher-Shlizerman said.

Advertisement

Because streaming audio over the Internet takes up far less bandwidth than video, the new system has the potential to end video chats that are constantly timing out from poor connections.

"When you watch Skype or Google Hangouts, often the connection is stuttery and low-resolution and really unpleasant, but often the audio is pretty good," said Steve Seitz, professor at UW.

"So if you could use the audio to produce much higher-quality video, that would be terrific," he said.

By reversing the process - feeding video into the network instead of just audio - the team could also potentially develop algorithms that could detect whether a video is real or manufactured, researchers said.

The new machine learning tool makes significant progress in overcoming what is known as the "uncanny valley" problem, which has dogged efforts to create realistic video from audio.

When synthesised human likenesses appear to be almost real - but still manage to somehow miss the mark - people find them creepy or off-putting.

"People are particularly sensitive to any areas of your mouth that don't look realistic," said Supasorn Suwajanakorn, a doctoral graduate at UW's Allen School of Computer Science & Engineering.

"If you do not render teeth right or the chin moves at the wrong time, people can spot it right away and it is going to look fake. So you have to render the mouth region perfectly to get beyond the uncanny valley," Suwajanakorn said.

 

Get your daily dose of tech news, reviews, and insights, in under 80 characters on Gadgets 360 Turbo. Connect with fellow tech lovers on our Forum. Follow us on X, Facebook, WhatsApp, Threads and Google News for instant updates. Catch all the action on our YouTube channel.

Further reading: Science, Barack Obama, Apps
Advertisement

Related Stories

Popular Mobile Brands
  1. OnePlus Nord 6, Nord CE 6 Get ColorOS 17 Stable Update
  2. Here's How Much the Vivo X500 Series, Watch GT 2 Could Cost
  3. Google Gemini Tasks Rolls Out to Free Users in India, US and Other Markets
  4. Google Unveils Gemini 4 Neon With Four Thinking Levels, Cybersecurity Features
  1. GTA 6-Inspired Meme Coin DacityCoin Lists on Bitrue Futures With 10x Leverage
  2. OnePlus Nord 6, Nord CE 6 Get ColorOS 17 Stable Update in India
  3. Vivo X500 Series, Watch GT 2 India Prices and Storage Variants Leak Ahead of Launch
  4. Apple Reveals 45 Best Apps and Games of 2026: Crimson Desert, Adobe Premiere Make the Cut
  5. Google Unveils Gemini 4 Neon With Four Thinking Levels, Cybersecurity Features
  6. Nothing's First AI-Native Device Leaked With Physical Talk Button, Red and Green LED Lights
  7. Sony WH-CH730N, WH-CH530 Wireless Headphones Launched in India: Price, Specifications
  8. Pay for Siri? iOS 27.3 Could Cap AI Usage Unless You Upgrade to iCloud+
  9. GTA 6 Download Size on PS5 and Xbox Series X/S: How Much Space Do You Need?
  10. Vivo X Fold 7 Said to Be in Development With 2nm Snapdragon, MediaTek Chip Options
Download Our Apps
Available in Hindi
© Copyright Red Pixels Ventures Limited 2026. All rights reserved.