MIT Develops Smartwatch App That Detects Emotions Using a Variety of Signals

MIT researchers have developed new smartwatch software that understands the tone of a conversation, and can tell you if the person is happy, sad, or neutral based on their speech patterns and vitals. The researchers demoed this feature on a Samsung Simband research-friendly wearable, hinting at how future devices could serve as 'social coaches' for people with anxiety or Asperger's syndrome.

Researchers Tuka AlHanai and Mohammad Mahdi Ghassemi have built an algorithm that can analyse speech and tone in real-time, as the person speaks. The video demo shows how the graph fluctuates as a person relays his story fluently. The experiment was conducted by asking people to wear the Samsung Simband research device, and tell a sad or happy story. The device captures high-resolution physiological waveforms to measure features such as movement, heart rate, blood pressure, blood flow, and skin temperature. The system also captured audio data and text transcripts to analyse the speaker's tone, pitch, energy, and vocabulary.

It then showed real-time data of the person's emotions as they spoke. MIT claims that the system can analyse audio, text transcriptions, and physiological signals to determine the overall tone of the story with 83 percent accuracy.

"As far as we know, this is the first experiment that collects both physical data and speech data in a passive but robust way, even while subjects are having natural, unstructured interactions. Our results show that it's possible to classify the emotional tone of conversations in real-time," Ghassemi says in the report. Furthermore, the system can also provide a 'sentiment score' for specific five-second intervals within a conversation.

The researchers plan to improve the algorithm by enabling it to decode boring, tense, and excited tones as well, over and above it labelling them as positive or negative. It looks to use this technology to leverage social coaching, but first it needs to do various testing and collect data from other commercial wearables like the Apple Watch.