Meta's new AI research could improve Quest 3's capabilities
Meta wants to give AR glasses and VR headsets a better understanding of how humans interact with objects and hopes that this will lead to new applications.
Machine understanding of hand-object interactions is still a challenge due to hand occlusions and the complicated way hands move.
Researchers at Meta Reality Labs have released a dataset called HOT3D (an acronym for"Hand and Object Tracking"). It is intended to help in the development of a vision-based system that better understands hand-object interactions and thus enables new applications.
"For example, such a system will enable transferring manual skills between users by first capturing expert users performing a sequence of hand-object interactions (when assembling a piece of furniture, doing a tennis serve, etc.), and by using the captured information to guide less experienced users, e.g., via AR glasses," the researchers write in their paper. "The skills could be similarly transferred from humans to robots, enabling autonomous robots that can learn on the fly. The system could also help an AI assistant to better understand the context of a user’s actions or enable new input capabilities for AR/VR users, e.g., by turning any physical surface to a virtual keyboard, or any pencil to a multi-functional magic wand."
The dataset is available on Meta's HOT3D project page. There is also a link to the research paper.
Recorded with and for Quest and other (upcoming) Meta devices
The dataset contains over 800 minutes of egocentric video recordings showing interactions with 33 everyday objects. In addition to simple scenarios in which objects are picked up, looked at, and put down, the dataset also includes typical actions in kitchen, office, and living room environments.
Two Meta devices were used to capture the video data: The Project Aria research glasses and the Quest 3 VR headset. Because Meta uses these devices, the dataset will likely be used primarily to train AI-based systems for the company's existing and future AI/AR/VR glasses and headsets.
Note: Links to online stores in articles can be so-called affiliate links. If you buy through this link, MIXED receives a commission from the provider. For you the price does not change.
