Meta Quest 2: Researchers show body tracking using modded VR controllers
A research team demonstrates an experimental method for body tracking in virtual reality. How does it work?
VR systems capture the movements of the head, hands and arms in virtual reality by internal or external tracking. This is done with the help of cameras and sensors that are either directly attached to the VR headset (internal tracking) or use an infrared or laser beam network generated via external stations for tracking (external tracking).
Both systems have advantages and disadvantages, but they have one thing in common: They mainly focus on the movements of the VR headset and the controllers. Full-body tracking either barely works or can only be achieved with accessories.
A New York research team is now presenting an approach that enables body tracking via cameras on the VR controllers.
VR body tracking via controller cameras
The VR controllers each have two fisheye cameras with wide-angle lenses, one on the tracking ring and one at the bottom of the grip. The goal is to capture as much of the body as possible during movement.
A software assembles the four controller images into a digital skeleton in real time.| Image: ControllerPose
The ControllerPose software merges the images of the left and right controllers in real time. From the total of four images, it thus creates a 185-degree view of the user. The system locates the poses with an accuracy of up to 3.38 inches (8.59 cm), according to the researchers.
For processing, the researchers tried two variants. The camera images were either streamed to a PC, which would be cheaper but cause higher latencies, or processed directly on the controllers via a Raspberry Pi Compute Module and Google Coral.
As example applications, the researchers show VR games they programmed themselves, such as a movement game reminiscent of OhShape. The player strikes certain poses with his arms and legs to match approaching shapes. The team also demonstrates a field hockey app that involves deflecting the puck with arms and legs.
Does it work?
Especially for VR fitness, full-body tracking would be useful as you could train specific body parts. However, the current prototype is not yet running smoothly. It has one major shortcoming in particular: the VR controllers only have a good enough view of the body about 32 percent of the time during a typical VR application.
According to the researchers, they are either too close to the body or too far away from it. One approach to solving this problem could be a system that uses tracking cameras on the controllers and the VR headset. Both images together could then enable reliable body tracking.
Will Meta bring full-body tracking to the Meta Quest 2?
Meta also repeatedly researched a solution for body tracking in the past. Hints of full-body tracking for the Meta Quest 2 appeared in the official Quest developer documentation earlier this year.
Meta's VR head Andrew Bosworth dismissed the rumors shortly after in a Q&A session on Instagram, noting that the positions of the cameras on current VR headsets are not suitable for properly capturing legs or feet.
However, Meta's upcoming Cambria VR headset will likely have tracking cameras built into the controllers. Recently, Meta also brought camera-based body tracking to Instagram AR effects.
Note: Links to online stores in articles can be so-called affiliate links. If you buy through this link, MIXED receives a commission from the provider. For you the price does not change.
