Google's Latest Mixed Reality Tech Reveals Person Behind the VR Headset

At I/O 2016, Google announced its Daydream VR platform cementing its focus on virtual and mixed reality moving forward. It even released Daydream View headsets alongside the Pixel phones last year. The company has been introducing several VR-centric features on its video platform YouTube as well, to promote immersive content. While few videos shot at YouTube's VR Labs do show how spectators can view the inside world of virtual reality, the content creators are often seen wearing VR headsets - making them look emotionless. Daydream Labs and Google Research have tried to address that problem and have found a way to eliminate those big VR headsets from mixed reality videos.

The folks at Daydream Labs and Google Research used a combination of machine learning, 3D computer vision, and advanced rendering techniques, to remove the headset and show the full face of the creator. For this experiment, Google used a HTC Vive headset, a green room, and a video camera. The user's face was scanned; an identical dynamic 3D model was created complete with blinking eyes and looking in different directions. Separately, the Vive headset was fitted with eye-tracking for understanding where the creator is looking, and then this 3D model was superimposed on the real person's face. This superimposition didn't completely remove the headset, but it just replaces it with a transparent version that shows the eyes and the other portions of the face that is otherwise hidden behind the headset.

This 3D designed face moves, looks, and blinks in real-time with the user, adding more credibility to the mixed reality video. It removes the valley effect that the obstruction of the headset brings, and looks to give content creators and viewers a more qualitative experience.

"The end result is a complete view of both the virtual world and the person in it, including their entire face and where they're looking. Our initial work focused on mixed reality is just one potential application of this technology. Seeing beyond VR headsets could help enhance communication and social interaction in VR. Imagine being able to VR video conference and see the expressions and nonverbal cues of the people you are talking to, or seeing your friend's reactions as you play your favourite game together," Google posts on its blog.

While Apple steers clear of VR, Google, Facebook's Oculus, HTC (with its Vive headset), Samsung, Acer Starbreeze, and Sony are betting big on the virtual reality industry. All of them have joined hands to create the Global Virtual Reality Association (GVRA) - a new non-profit organisation with a vision to build a regulatory body for the ever-rising virtual reality industry.