Meta’s photorealistic avatars can be generated with just an iPhone
Scanning a photorealistic avatar for virtual reality (VR) once required more than 100 cameras, but now researchers at Meta have revealed it can be done with just an iPhone.
Facebook first announced “Codec Avatars” in 2019, which are a class of learned photorealistic face models that accurately represent a person’s geometry and texture in 3D and therefore virtual reality (VR). Avatars are nearly indistinguishable from video, and the company hopes realistic avatars will be important in the future, especially given its metaverse ambitions.
Meta used a specialized platform to generate the codec avatars called MUGSY, which contained 171 high-resolution cameras, as shown below. But as reported To downloadhe recently made a breakthrough and was able to bypass MUGSY by using analysis from a smartphone with a forward-facing depth sensor, such as an iPhone with FaceID.
To capture the Avatar Codec, the phone must be panned across a neutral face and then back up again while copying a series of 65 facial expressions. Researchers say the scanning process takes an average of three and a half minutes, although users will have to wait six hours for a machine with four high-end GPUs to render the avatar in detail. It is likely that if this product were made available to the general public, it would render on a cloud GPU and not on the user’s device.
The iPhone’s scanning system uses the Universal Prior Model (UPM) “hypernet”, a neural network that helps generate the person-specific codec avatar. This UPM hypernet is trained by scanning the faces of 255 different individuals using an advanced capture platform, similar to MUGSY but with 90 cameras.
Other researchers have already demonstrated the generation of avatars from a smartphone scan, but Meta says the quality of its results is state-of-the-art. However, the system is limited to scanning only a person’s head and is unable to interpret glasses or long hair.
Avatars codec still far from its release
Meta avatars currently have a basic cartoon art style, so that kind of high fidelity still seems a long way off. On that note, the Meta avatars featured on products like the Oculus Quest 2 have actually lost realism over time to better adapt to complex environments in apps like Horizon Worlds.
Last February, Mark Zuckerberg said in an interview with Lex Fridman that in the future, virtual reality users may use an “expressionistic” avatar in casual games and a “realistic” avatar in business meetings.
Facebook declined to say when the codec avatars might be available to general users, but the company is confident it’s getting closer.
Picture credits: Photos via Authentic Volumetric Avatars from Photo Scan, Meta Reality Labs