How to create and see NeRFs in a VR headset

You can create your own NeRFs and view them in your VR headset using free software if your computer and headset are compatible. Here's how. 

You might have seen NeRFs on Twitter, but rather than looking at 3D objects on a flat screen, you can bring them into your VR headset. In Virtual Reality, you can move through a NeRF with your controllers and turn your head to look around in the virtual scene. It makes the experience more immersive.

Currently, there is only one way to view NeRFs in 3D with your VR headset, Nvidia's Instant-NGP app for Windows. This will probably change in the future, since NeRF technology might be a good solution to fill the metaverse with realistic virtual objects quickly.

What is a NeRF?

NeRFs are AI models trained on a collection of images. They can recreate the look of light captured at every point in a scene. That makes it possible to move through this neural radiance field (NeRF) in VR as if you were at the physical location of the photos or video, viewing actual objects. We have a guide that explains in more detail what a NeRF is and how this technology can help VR, AR, and the metaverse.

Create a NeRF: Getting started with Instant-NGP

To begin, you need a VR or AR headset with OpenXR and a Windows PC with an Nvidia graphics card made in the last six years:

• RTX 3000 & 4000 series, RTX A4000–A6000, and other Ampere & Ada cards,
• RTX 2000 series, Titan RTX, Quadro RTX 4000–8000, and other Turing cards
• GTX 1000 series, Titan Xp, Quadro P1000–P6000, and other Pascal cards

You'll need some software, and all links can be found at the end of this guide.

Follow the steps below to test out Instant-NGP:

1. Download the version of Instant-NGP that matches your GPU type and extract the files.
2. Open the Instant-NGP folder and run Instant-NGP.
3. You might see a Microsoft Defender warning. Click More info, then Run anyway to proceed.
4. In Windows Explorer, open the data/nerf folder, then drag the fox folder onto the window of the Instant-NGP app.
5. Wait a few seconds while the model is trained.
6. Click within the Instant-NGP window and drag to see the 3D effect of the fox head on your computer screen. You can also scroll to zoom and middle-click to drag the NeRF within the window.
7. After a minute, the model probably won't improve any further. Click the Stop Training button, which should improve your viewing frame rate.
8. Click the small arrow at the top-left to close the options menu and see more.

Create your own NeRF with video

If you have access to an iPhone, the Record3D app makes it a bit easier to capture video along with the required camera path data to create a NeRF. Unfortunately, there isn't an Android version of this app.

After installing the Record3D app on your iPhone, record a video of the object you want to see in VR as a NeRF. It's best to move slowly and keep the object of interest centered in the viewfinder. Export as an R3D file and transfer that file to your computer.

The rest of the process happens on your computer:

1. R3D is actually a zip archive. Change the extension to .zip, right-click, and select Extract All…
2. A new folder will be created that contains a metadata file and a rgbd folder.
3. If you don't have Python on your computer, you'll need to install it.
4. Open the Instant-NGP folder in Windows Explorer, right-click on the folder name, and select copy address.
5. Tap the Windows key and start typing Command Prompt and open the app.
6. Type in cd
7. Tap the space bar, then paste the address of the folder and hit enter.
8. In Windows Explorer, find the Record3D video folder and copy its address.
9. In Command Prompt, type python scripts/record3d2nerf.py --scene
10. Paste the address of the Record3D folder and hit enter.
11. The Python script should process this data quickly and create a new file called transforms.json.
12. Open Instant-NGP and drag the Record3D folder into the window to start training.

Follow the instructions below to refine the NeRF and view it in virtual reality.

Crop and center the NeRF view

NeRFs often have floaters and a cloudy look at the edges. You can crop to remove these distractions. This is also a good way to center the object you want to see in VR. Cropping also improves performance, so it's a useful option if you find the frame rate is too low, which can be disorienting when viewing in VR.

Here's how to crop a NeRF in Instant-NGP:

1. Scroll down in the options menu and click the arrow beside World transform & Crop box to see more options.
2. Drag the six sliders labeled Min and Max to crop and center the portion of the NeRF you are interested in and remove the clutter.
3. You can also rotate a NeRf by selecting Rotate world, then dragging the three circles that appear.

After cropping, centering, and rotating, you should have a good view of the NeRF on-screen.

See Instant-NGP NeRFs in VR

When your NeRF is looking good on your computer screen, it's time to check it out in VR:

1. Start your VR headset or compatible AR headset/glasses.
2. In the Instant-NGP app on your computer, click Connect to VR/AR headset.
3. Put on your VR headset and look around. You should see the NeRF.
4. With your controllers, click and drag to move through the scene.

If you don't see anything in your VR headset, it might not be set up as the default OpenXR device in Windows. In most cases, you can select your OpenXR device in SteamVR or the Oculus app on your PC. The Quest 2 and Quest Pro are compatible. Most headsets that work with SteamVR and Windows Mixed Reality should also work.

VR enthusiast Russ tweeted a video of his experience with Instant-NGP using a Quest 2.

Using images for NeRF generation

You can also use photos to create a NeRF, which allows more control of the images used to train the model. This method is more involved since several packages need to be installed, and more is done in Windows Command Prompt.

The Decoder has a good tutorial on a no-code method of creating NeRFs from images that you might want to check out.

Whichever method you use to create a NeRF, you should check it out in your VR headset if possible. In VR, you can move into the scene instead of watching through the window of a computer or a phone.

Note: Links to online stores in articles can be so-called affiliate links. If you buy through this link, MIXED receives a commission from the provider. For you the price does not change.