MIXED readers ask: How important is eye tracking for VR graphics?
MIXED reader Cynox wanted to know how important eye tracking will be for better graphics in VR and MR. Here is our answer.
What is MIXED readers ask?
Every Monday, we will answer a question from our readers.
Write your question in the comments or send it to hallo at mixed.de with the subject line "MIXED readers ask" and with a bit of luck we'll answer it the following Monday.
Question from Cynox: How relevant is eye tracking for higher resolutions, graphics, and performance?
"Hi Cynox
Eye tracking has many benefits for VR and MR. It enables more realistic looking avatars and eye contact in social apps, it can be used as an elegant input method as demonstrated by Apple Vision Pro, and it saves a lot of processing power through Dynamic Foveated Rendering (DFR), which then can be used for higher resolutions and better graphics.
With DFR, eye tracking determines which area of the image the eye is currently focused on. This allows the processing power to be applied to that limited area rather than the entire field of view.
Playstation VR 2 demonstrates the positive effect this can have with the example of No Man's Sky, which saw a huge leap in graphics quality thanks to the implementation of DFR.
Eye tracking on standalone headsets is more difficult to implement, as such devices have strict limits on compute, power consumption, heat generation, and, in the case of Meta Quest, must be inexpensive to produce.
Meta's CTO has explained many times why eye tracking is not yet a standard for Quest devices. This was also the case in his last Q&A on Instagram, when he was asked what happened to the DFR Meta talked about in 2016. Below is his answer.
"[...] To do Dynamic Foveated Rendering at really high level, which we've been talking about for a long time as an industry, you need really accurate eye tracking. An interesting thing is tracking the eye as accurately as you need to do the foveated Rendering currently takes more processing power than you gain by doing the foveated rendering. Now that obviously won't always be true, and there's a bunch of innovation that we can do in eye tracking that will make that much cheaper. So eventually, this will become the normal standard, especially for mixed reality, where you want to drive super high graphic fidelity without paying this quadratic tax of having to do it across the entire field of view [...]. So we haven't given up on it, it's still in the future, but that is one of the reasons it hasn't happened sooner."
The Meta Quest Pro is the only Meta headset that offers eye tracking, and according to Meta, its DFR is not a game changer (yet) in terms of performance gains. If eye tracking comes to one of the more affordable Quest headsets, and I think it could with Quest 4, it probably won't be in the form we've seen from Playstation VR 2 and professional PC VR headsets, with IR emitters and cameras.
Meta may rely on a new, proprietary eye tracking technology that is cheaper to produce and may track eye movements more accurately than previous solutions. Meta's AR prototype Orion also features eye tracking, and AR glasses have even more stringent requirements than VR headsets in terms of compute, power consumption, and heat generation.
The startup company Inseye is currently working on an eye tracking upgrade module for Quest 3 that uses inexpensive photo sensors instead of cameras for eye tracking, offering a higher sampling rate and lower latency than regular eye tracking. As you can see: We haven't seen this kind of technology fully mature yet.
Back to your question: Eye tracking is and will be much more important in the future in terms of resolution, graphics, and performance, even more so in standalone headsets than in wired devices where there is much less processing power available. The question is not whether eye tracking will one day be standard and enable better graphics, but in what form and at what price.
Kind regards,
Tomislav"
Note: Links to online stores in articles can be so-called affiliate links. If you buy through this link, MIXED receives a commission from the provider. For you the price does not change.
