What does mixed reality need to become truly amazing?
MIXED reader Wouter Zelle wanted to know what features are missing to take mixed reality to the next level. Here is our answer.
Every Monday, we answer a question from our readers. Send your question to tomislav@mixed.de with "Question" in the subject line, and with any luck, we'll answer it the following Monday.
This week's question comes from Wouter Zelle:
What mixed reality feature do you think is missing from the Meta Quest and would be very useful?
Hello Wouter,
Meta Quest 3 was the first headset that has made mixed reality affordable for the masses, and with the cheaper Meta Quest 3S, even more people will be able to experience mixed reality for the first time.
Keep in mind that both devices are 1st generation mixed reality headsets, so they are still very limited technically. Some mixed reality features will be improved over time through software updates, while others will require better hardware.
There is little information available at this time about what Meta Quest 4 might bring in terms of improved mixed reality capabilities, but the following paragraphs will give you a glimpse of where the journey is going. For Meta Quest and mixed reality headsets in general.
The path to mixed reality 2.0 is more gradual than you might think
Mixed reality stands or falls on its ability to map physical reality and to see and categorize the objects within it. The better it understands the world, the better it can augment it or mix it with digital elements in meaningful ways.
Meta Quest 3 has very limited knowledge of your room. When using the space setup feature, the sensors scan the environment very roughly and only take a snapshot. If a piece of furniture is moved, the room has to be scanned again, which is very cumbersome.
3D scan of a living room with automatically recognized boundaries of individual furnishings. | Image: MIXED
Besides this imperfect topographical knowledge, Meta Quest lacks a lot of semantic knowledge. It can distinguish between a door, a window, and a table, but it doesn't go much deeper.
Taking mixed reality to the next level will require a combination of more powerful sensors, real-time 3D scanning in the background, and better object recognition. In other words, advances that involve both hardware and software.
Mixed reality will benefit from these technological advances in many ways. People would need to draw boundaries and scan their room far less often, and would even feel safer while playing fully immersive games. Analog and digital reality would merge more closely, while digital objects would be realistically occluded by physical objects. And there would be many more practical mixed reality applications for everyday life.
The next steps
In this context, passthrough camera access for mixed reality apps is considered a real game changer for mixed reality. Currently, mixed reality apps can only access abstract environmental data, such as the 3D mesh of the room and boundary boxes that represent individual objects. This is due to privacy concerns. With access to the passthrough cameras, mixed reality apps could better recognize the environment and individual objects and integrate them into the experience, enabling more immersive and useful applications.
Left: The image of a living room as seen through the passthrough cameras. Right: The 3D mesh of the same environment available to mixed reality apps. | Image: Coastermania / 4th Wall Breakers
At the recent Meta Connect, Meta announced a new API that will allow apps to access the camera for the first time, although it is not yet clear how far this permission will go. Expect Meta to tread carefully.
Another area where we can expect progress is of course in passthrough quality. There is plenty of room for improvement in resolution, distortion correction, light sensitivity, and latency.
As you can see, mixed reality is still in its infancy, and it will be exciting to see how the technology develops over the coming years.
Kind regards,
Tomislav
Interested in VR and AR? Then follow us on Facebook or X or share your experience in the comments. For feedback, topic suggestions, or other ideas, please email us at hello@mixed-news.com.
Note: Links to online stores in articles can be so-called affiliate links. If you buy through this link, MIXED receives a commission from the provider. For you the price does not change.
