Facebook’s Creepy VR Avatars Look and Move Exactly Like You
Facebook has been secretly working on a sophisticated system that creates ultra-realistic virtual avatars that strikingly resemble people in both their facial features and mannerisms. While the system is not yet fully developed and might take years to see the light of the day, it’s already fairly advanced.
The system goes by the moniker Codec Avatars and Facebook is feverishly working on the cutting-edge technology behind the innovation at its Facebook Reality Labs (FRL) based in Pittsburgh. Facebook believes this virtual reality-based social capabilities engine will be a game-changer in social interactions thanks to its ability to generate very accurate virtual representations of ourselves. The system is quite sophisticated and offers exciting prospects even though seeing the full-face representations of people’s faces is also a tad creepy.
What is the Goal of Facebook’s VR Avatars?
Facebook hopes its virtual avatars will eventually help overcome the challenge of long distance communications between people and systems. By visualizing someone’s face as if they were right in front of you, the tech giant is hoping its groundbreaking innovation will help foster relationships between people in a virtual environment in the future. Facebook Reality Labs is utilizing an innovative 3D capture technology and artificial intelligence systems to develop the technology behind its Codec Avatars at its Pittsburgh facility.
Avatar Creation Process
To pull off the VR avatars, Facebook Reality Labs is deploying a contraption called Mugsy. This is a dome which has been fitted with 132 camera lenses and 350 lights and all of these are aimed at the central position where a subject must be seated and make several facial impressions. This process will take a few hours after which the system will generate a unique photorealistic face. These impressions are subsequently used to train machine learning systems.
There is also a second area that is fitted with an even larger number of cameras and the role of this section is to map out how the subject’s body moves. The information captured here will help animate the subject’s virtual avatars to create a highly realistic impression.
It is not yet known when a consumer-facing product based on this technology will be released into the market from the FRL. Facebook Reality Labs has cautioned that the technology is still “years away” from realization in consumer virtual reality headsets. The development is however a key step forward in the computer modeling of human behavior in virtual environments.
Creating lifelike avatars is a development at the frontiers of innovation which might get less cumbersome down the road once the system is fully developed. Currently, large quantities of high quality audio and video information of the research participants must be captured using a complex and highly-customized system of microphones and cameras in the FRL to create these realistic avatars. The data captured from the subjects is subsequently used in modeling artificial intelligence systems which may one day make it possible for users to use the Codex Avatar system in rapidly building virtual avatars from just a few photo snaps or video shots.
The current cumbersome avatar creation process is not yet consumer-friendly. As a result, the main focus of research at the Facebook Reality Labs is the use of machine learning to rapidly generate highly detailed avatars based on large amounts of input data captured from subjects over several hours.
Facebook hopes that some time in the future, the artificial intelligence system will be so sophisticated that it would be able to generate the lifelike virtual avatars simply from your existing Facebook or Instagram photos.
The Capture Research
FRL uses an extensive system of cameras to capture user data for the research including one that is placed in the nose gap, inside the head-mounted device used in the capture research. This setup allows the labs to capture extremely detailed data from its subjects.
The virtual reality headsets used in the capture research are relatively darker on the inside to avoid blinding the subjects who will sit for hours exposed to bright light. The features to be captured are illuminated with infrared lights which are not visible to the human eye. In normal headset applications, these lights are used in the “Constellation” positional tracking system in the Oculus Rift for eye tracking purposes. In the Facebook capture research, these lights are now being used to illuminate bodily features for capture.
Codex Avatars Still a Distant Dream
In spite of the huge amount of resources put into the research, the technology is still a distant dream. Oculus has been carrying out extensive research on face tracking and realistic avatars for many years now and we are still far from a practical usable consumer-facing product.
In 2015, Oculus carried out a research in partnership with USC for a hardware prototype designed with one unwieldy armed connected to a development kit. The bodily expressions would be read by extra sensors that were positioned in the foam lining of the headset where the camera was shielded from view. However, FRL had not been part of this earlier effort.
In the current research, FRL says it’s working through several possible solutions and will need certain technological breakthroughs to come to fruition for the product to go mainstream. The FRL team in Pittsburgh hopes to develop a successful system and eventually transfer it to product teams that will develop it for the mainstream market. The main challenge that needs overcoming is how to generate realistic avatars from minimal amounts of data and that will take years of machine learning.
Facebook’s efforts to create realistic avatars are not just focused on its Codec Avatars at its Facebook Reality Labs in Pittsburgh. The company also has another team based in Sausalito, CA that is exploring different approaches to generating realistic avatars.
The Sausalito team is exploring physics-based avatars capable of interacting with any virtual environment. This effort combines fundamental research in multiple disciplines including neuroscience, biomechanics, motion analysis as well as physically-driven simulations.
According to Facebook, this alternative approach is still dependent on live data capture just like Codec Avatars. However, rather than relying on live sensor data that drives a neural network, it drives a physics-based model based on human anatomy. Facebook might divulge more information about this alternative approach later in the year.
Revolutionizing the Nature of Communication
Should this see the light day, realistic avatars will revolutionize the nature of virtual communication. Instead of cartoon approximations that have limited expressions, users will be communicating with real humans over long distances who will have very realistic and lifelike expressions and movements.
In virtual reality research, the reproduction of nuanced human expressions is currently regarded as one of the hardest solutions to implement in VR experiences. Possibly, with the FRL research, the realistic avatars may one day constitute the main form of remote communications, thereby eliminating the limitations that we currently face with video calls.
In spite of the opportunities for realistic avatar communications that it portends, this technology also carries the risk of impersonation. If an appearance, mannerisms and expressions can simply be generated from computer images, then there is the risk of these being used by fraudsters to wear these like costumes and impersonate people. The possibilities for beneficial uses, however, far outweigh the risks.