Google Research shows expressions VR glasses wearer in mixed reality

Daydream Labs and Google Research are experimenting with technology to show outsiders what a person with VR glasses sees in virtual reality, including that person’s facial expressions. They ‘remove’ the VR glasses in the mixed reality display.

According to Google, it is a problem that you cannot see what a VR glasses wearer sees and it is also an obstacle that several people in a VR environment cannot see each other’s facial expressions. The latter is possible with avatars, but Google is testing with mixed reality to arrive at a representation in which virtual reality and the real person are combined.

The researchers use a camera to generate a 3D model of a person’s face, which takes less than a minute. They then build a database of expressions such as gaze directions and eye blinks by having users look at a moving point on the screen.

For the mixed reality images, a camera simply records the user in front of a green screen, and then projects the representation of the virtual reality world into the background. A marker on the headset helps with calibration between camera and VR glasses. This calibration allows the 3D face model to be placed over the VR glasses, even if the user moves his head.

Eye tracking techniques from SMI have been incorporated into the HTC Vive that the researchers are using. The information from those sensors is linked to the eye movement database, allowing a real-time representation of the eyes to be superimposed on the VR glasses. Google uses 3D vision and machine learning technology to make lighting and other adjustments that make the display more natural.

Daydream Labs and Google Research expect social interaction in VR environments to increase. Their experiments to make the headset less present can help more natural rendering of video conferencing and gaming in VR, they expect.