YouTube's New Automatic Caption Tool Can Now Describe Sound Effects

YouTube introduced captions in its videos back in 2007, and made automated captions for speech available a few years later. The company will soon also start describing sound effects in videos through machine learning. YouTube has developed a sound effect captioning system for its video platform collaborating with Sound Understanding and Accessibility teams. The automatic sound effect captioning system will identify and label sounds in the video without manual input.

With machine learning, YouTube will be able to automatically detect the existence of sound effects in a video and transcribe it to appropriate classes or sound labels. YouTube will soon start showing sound effects like [APPLAUSE], [MUSIC], and [LAUGHTER]. The company explains that "these were among the most frequent manually captioned sounds, and they can add meaningful context for viewers who are deaf and hard of hearing."

YouTube stresses that the new changes will help the 360 million people around the world who have problems in hearing. The company has so far made several changes to cater to these users, and claims that the number of videos with automatic captions now exceeds 1 billion while adding that people watch videos with automatic captions more than 15 million times per day.

"We started this project by taking on a wide variety of challenges, such as how to best design the sound effect recognition system and what sounds to prioritise. At the heart of the work was utilising thousands of hours of videos to train a deep neural network model to achieve high quality recognition results," said Noah Wang, Software Engineer in a blog post.

The company adds that its new captioning tech is still in the early stages of recognising sound effects automatically. YouTube lists some more challenges that will make video watching experience even better for the targeted users. "Future challenges might include adding other common sound classes like ringing, barking and knocking, which present particular problems -- for example, with ringing we need to be able to decipher if this is an alarm clock, a door or a phone."