Project Cambria: Meta explains new passthrough technology
Meta's head of engineering reveals new details about Project Cambria, confirming previous rumors about Meta's new VR headset.
The most outstanding new feature of Cambria is its high-quality passthrough mode. Outward-facing cameras capture the environment and pass it on to the displays, where the image can be enhanced with digital elements.
Passthrough technology has been around for a long time. What sets Project Cambria's passthrough mode apart is the quality of the image. Thanks to better cameras, the image is sharp and in color. The grainy black-and-white image of Meta Quest 2 is thus a thing of the past.
The high-quality passthrough mode has two advantages: First, it enhances augmented reality by making it look more realistic, and second, it makes the physical world visible in its entirety, thus increasing user comfort. The latter makes Cambria's Passtrough mode the best feature of the device.
Reconstructing the world: four sensors work in unison
Over the weekend, Meta's head of engineering Andrew Bosworth answered questions from VR fans on Instagram. He also talked about the Passthrough mode and revealed new details about the Passtrough technology.
"We've got way better cameras, so both RGB and black and white cameras working in conjunction to do color passthrough. We also use active rather than passive depth detection with infrared beams, so we're using infrared beams to make sure that we have depth-sensing that allows us to have a better reconstruction of the image," Bosworth said.
Bosworth then explained how the video image is created, "When you do color passthrough, you have to kinda reproject the world because the cameras don't line up, both not with the pupils in line, but also the depth is offset, so you do a reprojection, so there's a lot of work that goes into making it feel good. If you've used passthrough with Quest or Quest 2: This is something different, it's exciting."
Why so many cameras?
This is the first time Meta has confirmed a combination of black and white and color cameras for passthrough mode. How many, we don't know for sure.
Code findings in the Quest firmware suggest that there are two black-and-white cameras and one color camera. The former should be responsible for outlines and movements, the latter for the color space. The depth sensor also plays a role according to Bosworth.
Quest 2 firmware shows how Quest 2 Pro color passthrough works
2 "glacier" cameras on the front left and right
1 "teton" high resolution RGB camera in the centerTeton is only used to colorize the stereo Glacier input cameras which are fast and only provide Luma composite
- Brad Lynch (@SadlyItsBradley) April 11, 2022
The final video image of the physical environment is created from a fusion of all this sensor data, controlled by algorithms. But why so complicated? Wouldn't it be easier to use two RGB cameras?
According to Brad Lynch, the sensors are arranged as follows: two black-and-white cameras on the sides and a color camera in the middle. | Image: Marcus Kane / Brad Lynch
The complex sensor technology should give Meta more flexibility and allow software-based optimization over time. Meta will likely improve the passthrough mode incrementally without having to change the hardware. Quest Hand Tracking is a good example and has made big leaps in quality on the same hardware since its initial release.
For all the info and rumors on the new VR headset so far, check out our Cambria info article.
Note: Links to online stores in articles can be so-called affiliate links. If you buy through this link, MIXED receives a commission from the provider. For you the price does not change.
