Stereo VR180 is the standard for immersive media — but what about VR150 and VR135?
The VR180-3D format has been used for immersive media for several years. In this article, I look at new patent applications for stereo lenses from Canon with a different field of view and their potential impact on immersive media for virtual reality.
By Daniel Pohl
In the VR180-3D format, the environment is recorded in a hemisphere (180°× 180°) for each eye. In virtual reality, this format is suitable for high immersion, as head tracking with a simultaneous stereoscopic view brings the image very close to the experience of a real scene. But why exactly 180 degrees?
Content
VR180 — What is it about?
Without virtual reality, stereoscopic VR180 makes not much sense. In VR, however, the format blossoms perfectly to provide both a 3D view and enough viewing angles to see other areas of the image with tracked head movements. Accordingly, the first VR180 cameras, such as Lucidcam, Z-Cam K1 Pro and the Lenovo Mirage Camera, came onto the market in 2017 and 2018 with the emergence of better VR headsets.
The VR180 images are recorded without any stitching, as is the case with 360° cameras. This avoids any stitching artifacts and inconsistencies such as different lighting of the individual images or problems with dynamic objects, as it can happen with 360-degree drone shots.
The images from VR180 cameras taken with round fisheye lenses are usually saved in the manufacturer-independent equirectangular format or converted afterward using tools. This is comparable to transferring a round globe into a rectangular format.
VR180 photo with Canon EOS R5, 5.2mm Dual Fisheye. Above: Image in fisheye format (left/right reversed due to lens). Below: Image in equirectangular format. | Picture: Daniel Pohl
One half of the equirectangular 180-degree 3D photo is a square that exactly represents the 180°×180° hemisphere. It therefore makes sense, if possible, to fill the entire 180 degrees with content in order to avoid black edges in the format and to allow sufficient head movements in VR.
But why not 190° or 200°? It could be done and has been done. The Z-Cam K2Pro offers a variant with 200° fisheye lenses, which can therefore also look into the area behind the camera. However, when looking at the edges at 180°, the neighboring lens of the stereo camera will inevitably be visible in one eye.
In 3D perception, it is extremely disturbing and can lead to eye strain and motion sickness if the left and right eye receive very different and unrealistic information. With the large lenses of the VR180 Canon 5.2mm Dual Fisheye lens, modern VR image viewers such as immerGallery cut off so much of the edge on the left and right part of the photos from this camera that this conflict between the eyes no longer occurs.
Close-up of the neighboring lens in the Canon 5.2mm lens. | Picture: Daniel Pohl
Therefore, the added value of lenses with more than 180° for virtual reality is not necessarily given. If you compare the above images in fisheye format with the equirectangular version, you will notice that the original fisheye image shows more content. The lens itself can actually capture almost 190°. But precisely because of the adjacent lenses in the image and also because the fisheye quality decreases towards the edge due to chromatic aberrations, there is no added value in displaying these additional pixels in VR. In addition, the 3D effect decreases at the edges anyway.
Why 180° may currently be too much
As nice as it is to be able to move your head to a certain extent in virtual reality, there are also some difficulties. To fill 180°× 180° with detailed image data, very high resolutions are required. The first wave of VR180 cameras in 2018 offered around 6000 pixels horizontally and 3000 pixels vertically. In a headset like the Meta Quest 3, this looks stereoscopically impressive, but unfortunately relatively blurred. With 8K photo resolutions, such as those offered by the Canon EOS R5 with a 5.2 mm lens, the image is significantly sharper than before.
But even the 8K VR180 photos taken with this configuration are not as sharp as content shown on a Quest 3 could be today. A comparison with 360-degree images from drones can be made. Such very high-resolution 360° images are created by taking several high-resolution images while the drone and camera on it are rotated. This results in a stitched image with several hundred megapixels. These images would not be loaded directly into the headset in this resolution, but rather scaled down to a resolution that is similar to the photos in 8K-VR180 in terms of the number of pixels. However, the downscaled images appear significantly more detailed and sharper - even with the same number of pixels per degree of field of view.
Why is that? Firstly, you can see from the fisheye image above that not all the pixels of the EOS R5 camera sensor (45 megapixels) are used, as the image of two round fisheyes has to fit into the square camera sensor. The fisheye image already shows some distortions due to the way fisheye lenses work. When converting to the equirectangular format, pixels have to be interpolated, which also results in a certain amount of blurring, as the pixels are no longer used exactly according to their scanned pattern. There are also certain distortions in the final equirectangular format. A (half) sphere cannot be displayed in a rectangle without distortion.
Why does the drone image look so much sharper when scaled down? If good image algorithms are used, a lot of information is retained in the subpixels when down scaling — a very good anti-aliasing or super sampling, so to speak, if you look at it from a rendering perspective. This leads to a higher impression of sharpness.
VR180 is not sharp enough? What now?
If even a VR camera with a professional camera body with 8K and an L lens is not sharp enough for VR180, what is left to do? Increasing the resolution of the sensors in the camera bodies would be a further step. However, this would make the products even more expensive. 10K horizontal is perhaps still conceivable in the photographic sector. But for such a camera to also be able to record 10K video, preferably at 60 fps or more, doesn't seem possible without immense costs, at least not in 2024.
A reasonable goal is to make the creation of immersive media cheaper and more accessible, even with less than a 45 megapixel sensor. Canon has already shown prototypes of a new dual fisheye lens that is equipped for camera bodies with the more cost-effective APS-C sensors instead of full-frame sensors. However, the EOS R7 currently offers the highest resolution of the APS-C camera bodies with 32 megapixels, which is less than the 45 megapixels of the full-frame EOS R5 camera body. Corresponding VR180 photos with a horizontal resolution of around 7K should be good enough for private users. Nevertheless, a loss of sharpness compared to the R5 will be noticeable. What now?
One dimension that has not yet been considered in cameras for immersive media is 180° × 180°. Of course, the aim is to remain well above the experience of a 3D movie theater or the old 3D cameras for "normal 3D" (approx. 70° field of view) and continue to support head movement in VR for immersion. However, it is conceivable that a certain reduction in the field of view is acceptable. This would mean that less of the surroundings would be projected on the same sensor, resulting in a sharper image for our VR photos.
VR150? VR135?
At the scientific conference IEEE VR 2024, a research project by immerVR and the University of Würzburg demonstrated the coupling of two smartphones to form a VR123 camera, which therefore stereoscopically depicts a 123° field of view. This was also intended to show smartphone manufacturers that it is relatively easy to use modern smartphones to capture more than just 3D photos and 3D videos with a small viewing angle, such as with the iPhone 15 Pro.
Rather, by using ultra-wide-angle lenses, ideally two identical ones on the same smartphone at human eye distance, immersive media can already be created that allow a certain amount of head rotation in VR. In this project, it was 123 degrees due to some limitations of the camera interfaces. However, there are already smartphones, such as the realme GT2 Pro, which has an integrated 150-degree ultra-wide-angle camera. It will be exciting to see whether more will happen in this direction in the future.
![[paper.jpg] Forschungsarbeit mit zwei Smartphones zum Erzeugen von VR123 bzw. VR150-Medien.](https://mixed.de/wp-content/uploads/2024/05/paper-860x249.jpg)
Research work with two smartphones to create VR123 and VR150 media. | Image: Scheuerpflug et al.
In the meantime, Canon has filed two new, very remarkable patent applications. It should be noted, of course, that a patent application is no guarantee of a product being released and could merely be experimental designs. In detail, the website CanonRumors reports on patent applications for lenses that are unofficially referred to there as Canon RF-S 3.2mm F2.8 Dual Fisheye and Canon RF-S 3.8mm F3.5 Dual Fisheye.
These are for the APS-C sensor. Half the angle of view is described, which is 79.1° and 72° for both lenses. Doubled, this results in approximately 158° and 144°. In contrast to VR180, the neighboring lens is of course no longer a problem here. However, since you can already see the optical shortcomings of the lens at the edges in the fisheye image above, I assume that the more extreme outer areas will not be used here either. This means that the two stereo lenses could actually deliver the image content of VR150 and VR135 in the converted, equirectangular version.
What does this mean for content creators of immersive media? With the smaller field of view, beginner's mistakes where your own feet or fingers are in the picture disappear, as has probably happened to all photographers with 180°×180°. The use of a tripod also becomes easier. With VR180, the tripod legs were always in the picture without a longer rail on the tripod.
The increased sharpness will make it possible to capture the main subject, for example a person in the middle of the picture, even sharper than it was possible with VR180. What will no longer work so well is to stand close in front of a large building and create a shot where the viewer has to really look up, with neck movements, to see everything. Here, the vertical restriction of about 135° will take effect too quickly. This means there would still be reasons for having a full 180° setup and smaller variations of it.
What does VR150 and VR135 look like?
How big would the restriction of the field of view be at 150° or 135° compared to the normal VR180? I simulated this for you and took the image data from two original Canon VR180 EOS R5 5.2mm photos to compare it with versions at 150° and 135°. This test is only relevant for comparing the impressions of the field of view, not for the sharpness, as all images were generated from the same original VR180 photo in 8K. Since the equirectangular format, as already mentioned, has distortions, the impression in 2D shown here in the article may differ from how it is perceived in VR.
I have therefore made the comparison photos of VR180, VR150 and VR135 available for download as a .zip file so that you can view them yourself in VR. The easiest way to do this is via the immerGallery app, where you can download the images at immervr://vr150. To do this, go in the app to Download, Custom URL, set the drop-down to immervr:// and then simply type vr150.
A VR180-3D image with the main object in the center of the photo. From top to bottom: 180°,150°, 135°. | Picture: Daniel Pohl
A VR180-3D image with the main object in the center of the photo. From top to bottom: 180°,150°, 135°. | Picture: Daniel Pohl
A VR180-3D image in which you are standing close to a tall object. From top to bottom: 180°, 150°, 135°. | Picture: Daniel Pohl
My subjective impression when viewing VR180, VR150 and VR135 with the VR headset is that the restriction to 150° is relatively easy to cope with if you get a gain in sharpness in return. At 135°, the entire field of view is still filled, but even small head movements remind you that you are only looking at an image and are not really in the world being shown. From first impressions, I would therefore prefer the 150° variant.
Summary
Immersive media can convey impressive VR experiences. However, VR180 still lacks the necessary sharpness of the generated content, even with more professional camera setups in the $5,000 range. It would also be helpful for the VR market if it were possible to create sharp and immersive 3D content more cost-effectively.
A new approach could be to slightly restrict the field of view of immersive 3D camera lenses. However, only to the extent that a certain amount of head movement is still possible for greater immersion and the periphery of the field of view is completely filled with image data. This could be a reasonable compromise in order to achieve higher image sharpness in VR. This will also be expected by VR users sooner or later with headsets with a higher resolution than Quest 3, such as Apple Vision Pro.
According to rumors, Canon will hold a major presentation of new products in the week of May 13, 2024. We are excited and hope to see more products in the 3D and VR sector. Perhaps there will also finally be news in the area of the foldable consumer camera for VR180 and 360° 2D.
The author, Daniel Pohl, is CEO and founder of immerVR GmbH. There, Daniel works daily on innovations in the field of immersive media, mostly in the area of VR180 stereo photography. With his app immerGallery, you can experience immersive photo galleries with voice-overs and background music in various VR formats on Meta Quest headsets – also together with friends in multiplayer.
Note: Links to online stores in articles can be so-called affiliate links. If you buy through this link, MIXED receives a commission from the provider. For you the price does not change.
