Mark Zuckerberg's impressive Codec Avatar was rendered on a powerful workstation
Meta's newer Codec Avatars are rendered on workstations, but an older version is already running on standalone headsets.
Yaser Sheikh, head of Meta's Codec Avatar research, gave a presentation at the Nvidia GTC conference last week, and MIXED was there.
Last September, Meta demonstrated its Codec Avatars tech in a Lex Fridman podcast. Fridman and Mark Zuckerberg spoke for an hour over a long distance with photorealistic Codec Avatars. Both had been meticulously scanned beforehand. During the interview, they wore a wired Meta Quest Pro that uses five sensors for eye and face tracking. Until now, it was unknown what other technology was needed.
A picture from Sheik's presentation shows the actual recording situation and the workstation used. | Image: Meta
In the presentation, Sheikh said that a workstation with 4 GeForce RTX 4090 graphics cards was used to render the avatars on both sides. This means the technology is still a long way from running at this quality level on standalone headsets like Meta Quest 3, which only have a fraction of the processing power.
According to Sheikh, Codec Avatars 2.0 were used for the podcast. However, Sheikh later said in the Q&A session that the lab has gotten the older Codec Avatars 1.0 to work for multiple people on standalone headsets. These avatars can be created using a smartphone face scan. The video below demonstrates Codec Avatars 1.0.
Sheikh also showed a new generation of the technology, Codec Avatars 3.0, which uses Gaussian splatting to render people even more realistically, down to individual hairs. Like Codec Avatars 2.0, these run on workstations.
Meta has not announced when its Codec Avatars research will be turned into a product.
The Codec Avatars 3.0 are more detailed than those in the podcast. | Image: Meta
Codec Avatars starting in 2025?
Meta has been working on Codec Avatars for many years. This research was first shown to the public in 2019. The long-term goal is to enable photorealistic telepresence on affordable headsets.
Meanwhile, the pressure is on for Meta to finally deliver more realistic avatars: Apple Vision Pro already offers a well-functioning telepresence solution with Personas. The tech has not yet crossed the uncanny valley and the avatars are only displayed two-dimensionally within a window, but we can expect Apple to greatly improve the technology in the coming months and years. Personas also require a facial scan, which is created using Vision Pro, as seen in the video below.
According to a leaked hardware roadmap, Meta's Codec Avatars tech will find its way into a new premium headset. Could this be the headset Meta is working on with LG, which will reportedly come to market in 2025?
Note: Links to online stores in articles can be so-called affiliate links. If you buy through this link, MIXED receives a commission from the provider. For you the price does not change.
