Meta on eye tracking: "It's super hard"
Meta's XR chief Andrew Bosworth talks about two big technological hurdles in VR: eye tracking and full-body tracking.
Meta is launching its first VR headset with eye-tracking this year, Project Cambria. Eye tracking should enable more realistic avatars and eye contact in social VR apps, and could bring new, gaze-driven input methods and interactions.
Foveated rendering, on the other hand, is still a thing of the future. In this rendering technique, the VR headset determines which area of the image the eye is focusing on and computes only that area in full detail. This could save energy or computing power. At least in theory.
Eye tracking is not yet cracked
In his recent Q&A on Instagram, Andrew Bosworth makes it clear that eye-tracking is far from mature and also explains why.
By coverage, he means that eye tracking must work for 100 percent of people. For foveated rendering, the requirements are particularly high. Here, you not only have to track eye movements fast enough, but you also have to be able to predict them somewhat, Bosworth says. This is "really hard," he adds. Meta is still far from the vision associated with foveated rendering, he says.
Body tracking: Meta is (also) researching external solutions
Another difficult hurdle is full-body tracking, at least if you forgo external tracking accessories. Meta Quest 2 only captures head and hand movements, it does not recognize movements of the lower body. The integrated cameras can't solve this task alone because the legs and feet are not always visible.
Capturing all body movements and transferring them to virtual reality would have many advantages: Like eye tracking, it enabled more realistic avatars and new forms of interaction, and could also help evaluate physical activity, which would be particularly useful in the context of VR fitness.
"So outside in body tracking is probably necessary for some of the use cases people have in mind. That's one of the things we're looking at," Bosworth says.
The big drawback here is that you need more components, which translates into additional cost and extra setup effort, Bosworth says. He calls a report last week that Meta is working on a body tracking solution for Quest 2 premature.
The lack of full-body tracking is why Meta's avatars are floating torsos with no legs - a fact that has been met with much criticism and ridicule.
Meta's stock market crash and more
The head of Reality Labs also addressed the following point:
- Meta's stock crash. Meta had experienced similar things in the past and he is not worried about it because he thinks in the long term. Meta is still making a lot of money, he said. Because investments in VR and AR would only pay off in the longer term, that doesn't mean they're bad.
- That it's so cumbersome to share VR content (images, videos) you create yourself on Instagram, Messenger and Whatsapp is a valid criticism, he said. Meta is working on it.
- The Quest Store is open to streaming platforms like Disney+, Bosworth said. It's up to the providers, not Meta, to make such content available on Quest.
- Bosworth doesn't see a Game Pass-like offering for the Quest platform coming anytime soon. For that model to be viable, the app catalog would have to grow first and VR content would have to be offered on multiple platforms.
- On the topic of children and virtual reality, Bosworth points to Meta Quest 2's age restriction, existing control systems (casting options, age ratings for VR apps), and parents' supervisory responsibilities. "I don't know of any system that is going to verify ages that can't be circumvented," Bosworth says.
- Bosworth did not say when there would be updates on Project Cambria. However, he said the team continues to make good progress.
Read more about Meta:
- Meta Quest 2: insurer records increase in claims
- I love Virtual Reality - but I couldn't care less about the Metaverse
- The best Meta Quest 2 Fitness games 2022
Note: Links to online stores in articles can be so-called affiliate links. If you buy through this link, MIXED receives a commission from the provider. For you the price does not change.
