Meta's new AI research could improve Quest 3's capabilities

Meta's new AI research could improve Quest 3's capabilities

Meta wants to give AR glasses and VR headsets a better understanding of how humans interact with objects and hopes that this will lead to new applications.


Machine understanding of hand-object interactions is still a challenge due to hand occlusions and the complicated way hands move.

Researchers at Meta Reality Labs have released a dataset called HOT3D (an acronym for"Hand and Object Tracking"). It is intended to help in the development of a vision-based system that better understands hand-object interactions and thus enables new applications.

"For example, such a system
will enable transferring manual skills between users by first
capturing expert users performing a sequence of hand-object
interactions (when assembling a piece of furniture, doing a tennis
serve, etc.), and by using the captured information to guide
less experienced users, e.g., via AR glasses," the researchers write in their paper. "The skills could be
similarly transferred from humans to robots, enabling autonomous
robots that can learn on the fly. The system could also help an
AI assistant to better understand the context of a user’s actions
or enable new input capabilities for AR/VR users, e.g., by turning
any physical surface to a virtual keyboard, or any pencil to a
multi-functional magic wand."

The dataset is available on Meta's HOT3D project page. There is also a link to the research paper.



Recorded with and for Quest and other (upcoming) Meta devices

The dataset contains over 800 minutes of egocentric video recordings showing interactions with 33 everyday objects. In addition to simple scenarios in which objects are picked up, looked at, and put down, the dataset also includes typical actions in kitchen, office, and living room environments.

Two Meta devices were used to capture the video data: The Project Aria research glasses and the Quest 3 VR headset. Because Meta uses these devices, the dataset will likely be used primarily to train AI-based systems for the company's existing and future AI/AR/VR glasses and headsets.