Half-Life: Alyx experiment is a glimpse into metaverse magic
An artist enters City 17 with a VR headset and paints scenes from the game world using another VR app. How does this work?
Artist Liz Edwards has set up shop in Russell's lair. Armed with an easel, paintbrush and palette, she paints a portrait of the underground fighter absorbed in his computer.
Does Half-Life: Alyx support virtual oil painting? No, not at all. Painting is made possible by another VR program that is brought into the world of Half-Life: Alyx as an application overlay: the oil painting simulation Vermillion. So the painting scene shows a VR world that is extended by another VR application in AR fashion.
Painting Russell's portrait inside Half-Life: Alyx using Vermillion ✨ pic.twitter.com/lBP4LgThTe
— Liz Edwards 💛💙 (@lizaledwards) February 11, 2023
The Vermillion overlay is not limited to Half-Life: Alyx and can be used in other SteamVR apps as well. "It is absolutely wild being able to casually oil paint inside any SteamVR game," Edwards writes.
Beta version now available
Vermillion creator Thomas van den Berge programmed the import feature in less than a week out of curiosity whether it was possible.
"It's an overlay that runs on top of SteamVR and can be interacted with separately to the game. I've only seen 2D windows before," Van den Berge wrote on Twitter.
And what happens if you get attacked by Combine or Headcrabs in the middle of your contemplative painting session? Don't worry: According to Van den Berge, you can always switch back to the game's controls to fight off the invaders.
If you want to try out the beta feature, you'll need to enable it in the VR app's launch options. Virtual painting in VR worlds should work smoothly with the HTC Vive and Valve Index, but there may still be stuttering in Quest. Van den Berge is working on this.
A glimpse into the future of the metaverse
The experiment shows the untapped potential of the metaverse concept.
What if you could not only move avatars and objects from one virtual world to another, but also open VR applications in another? Why should a virtual world be a final product rather than infinitely expandable?
The possibilities of extending VR worlds, combining and nesting VR applications are hard to imagine today. Van den Berge's interesting experiment offers a glimpse into such a future.
How the Vermillion overlay works
Thomas van den Berge explained to me how the technology works. See his answer below.
A tiny bit of technical background on how rendering in VR works first: the game world is rendered for each eye, just like it would be rendered once for a flat screen game. After this however, the output for each eye is warped to counteract the distortion of the lenses. This stretches the rendered image, making it blurry. This is why you need to render at a higher resolution than the amount of pixels on your headset to get a crisp image, or why standalone VR games that don’t have the performance budget to do this look blurrier than games on PC.
VR runtimes have the ability to overlay textures directly onto the final image that is sent to the headset's displays, without having to be distorted to match the lenses. They have to be a specific shape: a rectangle, a cylinder, or a 360 panorama. This is the technology behind the very crisp menu panels you see when you open your VR dashboard. Overlays are also used by games to get a more legible in-game menu.
With OpenVR, the VR runtime behind SteamVR, you can go a step further: you can start an application purely as an overlay rather than as a game, so that you can have both a game and one or more overlay applications running simultaneously. This has been used so far by applications such as LIV or Desktop+ to show supplemental information in VR, on floating rectangular overlays. OpenVR also supports capturing inputs in overlays, so that you can override the inputs from the game, preventing game actions from happening when you actually want to be interacting with the overlay.
With Vermillion however, I needed to show an entire 3D scene as an overlay, not just a floating panel. The overlay feature has practically no documentation, and from what I read on Twitter, there wasn’t a way to get a 3D overlay running smoothly. I was about to give up, when someone mentioned Joe Ludwig to me. He’s one of the developers at Valve behind SteamVR, who happened to have created Aardvark XR in 2020. Aardvark is an overlay application that draws a full 3D scene, a user extensible platform for creating spatial widgets. Aardvark is open source, and while it’s a native C++ application and Vermillion is made with Unity, it had all the components I needed to understand how to create a 3D overlay scene.
AdAdIt leverages the 360 panorama overlay mode by warping the overlay’s eye outputs to a stereo panorama. The next challenge lies in making the camera position and rotation match that of the game that’s being rendered. The overlay is completely separate from the game, and has no notion of where the in-game camera is located. If there is a mismatch between the camera movement in-game and in the overlay, the overlay appears to be “juddering” or lagging behind. As soon as the framerate of the overlay doesn’t match that of the game, this becomes very apparent.
So far I have been able to get a very stable result on SteamVR headsets (such as Vive and Index), completely selling the illusion that the Vermillion easel is part of the game. Sadly it seems that the additional steps required for gaming on Quest over (Air) Link cause a disturbance, as there is still judder on there at this time. Fingers crossed I can get this sorted out!
Note: Links to online stores in articles can be so-called affiliate links. If you buy through this link, MIXED receives a commission from the provider. For you the price does not change.
