Mixed Reality Masking: Varjo Brings Real Gaming Steering Wheel into the VR Cockpit
The Varjo Lab Tools enables you to transfer physical objects into virtual reality at lightning speed so how does the real-time masking work?
Finnish high-end VR headset manufacturer, Varjo, presented the Varjo Reality Cloud in June this year. The platform enables real rooms and environments to be digitized using mixed reality headsets and shared with other virtual reality users. Varjo is now introducing new software that will enable users to integrate real objects into VR environments at lighting fast speeds: Varjo Lab Tools.
Real-time Masking: Integrating Real Objects into Virtual Reality
With the Varjo Reality Cloud, users can scan their surroundings via a lidar scanner that is built into the Varjo XR-3 headset, the company’s ultra-high-resolution headset. A virtual reality image of the room is subsequently generated which can be digitally expanded using augmented reality effects.
Using the real-time masking of the Varjo Lab Tools XR software suite, users can bring a real object into the virtual reality environment. The cameras of the XR-3 headset capture the object and stream it into the XR headset, in a functionality that is similar to the passthrough mode found in the Oculus Quest 2 headset.
The Varjo Lab Tools will give Varjo users access to brand new software features. It is a suite of functionality that allows users to make rapid adjustments to both virtual and mixed reality setups.
The Varjo Lab Tools includes new masking modes for real-time masking of objects from a first-person perspective. It will enable developers to intuitively mark out the surface area of physical objects when they have a Varjo XR-3 headset on and look a real-life video passthrough feed.
Developers will be able to use handheld VR controllers to mark surfaces of physical objects set to quickly appear in virtual reality and directly in the video passthrough mode. The surface area of an intended mask is shown by pinpointing directly on the real-world object or areas which generates a polygonal 3D mesh of the object in VR.
In Varjo’s preview video shown below, a developer can be seen sitting at his desk with a gaming steering wheel in front of him. Using the controller, he draws several digital vertices over the steering wheel. A digital mask or a polygonal 3D mesh is subsequently created which appears in the form of a steering wheel.
Digital Mask Becomes the VR Window into Reality
This scenario is subsequently reversed in the VR application. The polygonal mesh created acts as a window for the real object in virtual reality. In Varjo’s example, the end of the game sees the gaming steering wheel physically in front of the user in a VR racing game simulation, embedded in the respective VR cockpit.
By using a Varjo marker, the digital mask can be rotated in VR and apart from a little flickering, it looks like it truly belongs in the VR game. According to Varjo, different masking modes can be selected. This allows for static and dynamic masks to be edited in the mixed reality view. The use of chroma key setups should also be possible.
Varjo plans to publish additional details on the Varjo Lab Tools with the next major update of Varjo Base.