A Quest 3 experiment: Is hand tracking finally ready for prime time?
Hand tracking is considered the VR input of the future. But how close to mainstream is it really? I tested it with someone who has no prior experience with VR or hand tracking.
Most experts agree that hand tracking will change the way we interact in and with VR and MR. More controversial is how well hand tracking works today and how long it will take for it to catch on and replace controllers as the primary input - or if it ever will.
Some in the industry believe that hand tracking will completely change the industry in as little as two years, while others are more skeptical. The debate has been further fueled by the launch of the Apple Vision Pro, which is known to rely solely on eye and hand tracking.
Meanwhile, the hype surrounding the device has died down since its launch in February. Apple's first spatial computer will not change the VR industry overnight, and some might argue that it has raised rather than lowered awareness of the benefits of controllers, its precision controls and haptics.
For hand tracking to become the primary input method, it must be flawless and intuitive to use for most consumers. To put it to the test, I looked for a suitable test subject: My friend Esther (name changed for this article), who is in her early 70s and has no prior experience with VR, hand tracking, or video games.
Hand tracking has been praised for its accessibility. If Esther could use the technology, then nothing should stand in the way of hand tracking. Correct?
First steps
I let Esther experiment with Meta Quest 3's hand tracking for about an hour.
First, I walked her through the Meta Quest hand tracking tutorial. Esther immediately understood the concept of touch input, which she was familiar with from mobile devices, but it didn't always work the way she thought it would. For example, she had to select and tap certain items several times before the system interpreted the input correctly. On the other hand, she was able to learn the gestures relatively quickly.
When I asked her what she thought of the user experience, Esther said:
"It was okay, I didn't think it was very easy, I thought it could be easier, but once I saw how it worked a couple of times, it wasn't hard to remember the simple gestures".
A messy fast food stand
Next we tried Vacation Simulator. The VR game has had experimental support for hand tracking since 2020, which has become a regular feature since the last update.
Unlike the studio's latest project, Vacation Simulator was not designed from the ground up for hand tracking, which was noticeable in the experiment. Esther's ten minutes at the beach food stand were quite exhausting, and in the end she couldn't even make the hot dog I asked her to. Most of the ingredients were scattered on the ground.
What Esther wanted to do with her hands and what the game interpreted as an interaction were often two different things. She later described the experience as "frustrating," but said that she could imagine returning to this world and enjoying it more the second and third time around, as she was still a "complete novice" in this area. Watching her, it felt like the hand tracking was more of a hindrance than an aid to the enjoyment of the game.
The Midas problem and other issues
Esther liked the puzzle game Cubism better, but again she had difficulties that had more to do with the hand tracking technology than the game itself. After a few tries, she found it relatively easy to grab the puzzle pieces with her fingers. It was more difficult to release them in the right place, as the headset did not always detect the movement and the puzzle pieces "stuck" to her fingers, as she put it.
Hand tracking is not something that just works out of the box, to some extent it has to be learned. It is not enough to just grab an object and let go. You have to pick it up and release it in a certain way. This is a general hurdle for hand tracking input that has yet to be overcome.
One problem is that computer vision algorithms do a poor job of recognizing subtle movements. Another is that the system may over-interpret hand movements as interactions when they are not. VR developer Antony Vitillo aptly described this phenomenon as the Midas touch problem (the greedy Midas wanted everything he touched to turn to gold. When even food and drink turned to gold, and he was in danger of starving to death, he asked the god Dionysus to take back his wish).
The experiment ended with Hand Physics Lab, which brought a delighted smile to Esther's face. More than the other apps, the hand tracking experiment kit thrives on the shortcomings of the technology, which makes it easier to look past them.
Hand tracking: a long way from "fine" to "perfect"
In the end, I realized that hand tracking is still an experimental technology. VR enthusiasts are aware of its shortcomings and know how to work around them. Beginners who are spending a lot of money on a device have a different requirement: it just has to work, especially if it will be their primary VR and MR input method.
First, hand tracking will only be mature when it feels like we are actually interacting with our hands, rather than having to learn how to use them in order for the system to interpret them correctly. Second, it has to work almost flawlessly. It's not enough, for example, to be able to recognize a button press 90 percent of the time. That would be frustrating.
And for engineers, the task is getting harder, not easier: going from 90 to 99.9 percent correct pose and interaction recognition in the next few years is likely to be much harder than going from 0 to 90 percent.
With that in mind, I don't see Meta selling Quest headsets without controllers anytime soon. And on the content side, there is also still a lot of work to be done before hand tracking can be considered mainstream. The vast majority of VR applications still require controllers, and for good reason.
Note: Links to online stores in articles can be so-called affiliate links. If you buy through this link, MIXED receives a commission from the provider. For you the price does not change.
