XR (Extended Reality) and VP (Virtual Production) are two terms you’ll hear used to describe professional video capture on special stages equipped with LED volume wall backgrounds. In consultation with the experts in The Studio B&H, here we review the components needed to set up an XR stage and break down some of the specifics of this VP capture technique.
Extended Reality vs. Virtual Production
XR and VP stages both use LED volume walls which are room-size or larger LED video screens made up of modular panels featuring high-density pixel counts (low pixel pitch), high brightness levels, fast refresh rates, wide viewing angles, and support for higher camera frame rates. Both XR and VP techniques enable your talent to perform in front of prerecorded footage, eliminating the need to send the cast and a large crew on location. High-resolution, anti-reflective, bright panels like INFILED Xmk3, Fabulux LED Alpha Wall Kit, Planar Systems, and ROE Visual LED panels provide pixel pitches of 0.9 to 2.84mm that are ideal for building LED volume walls.
XR takes VP a few steps further—providing an immersive setting in which your talent and camera(s) move in coordination with computer-generated environments. So, while VP stages may include curved walls to provide some side views of your background, XR stages will typically have at least two walls forming a corner or use additional extension walls to create a more cave-like environment. LED floor and ceiling panels complete the virtual environment for XR capture whereas VP setups will more often use floor materials that physically mimic those in the video footage and ceiling panels that reproduce lighting levels to match the background. Floor panels like the ROE Black Marble BM5 can be mixed with interactive BM5i panels which use embedded sensors that enable your XR participants to trigger your programmed content.
Cameras
When choosing a video camera for your XR or VR project, look for features like a large sensor with a high resolution of at least 4K, genlock capability, and high frame rates for capturing smooth motion. Select from cine-style cameras with genlock and a fast-readout rolling shutter like the ARRI ALEXA 35 XTREME, ALEXA MINI LF, Blackmagic URSA Cine 12K LF, and Sony VENICE 2. The RED V-RAPTOR XL [X] has a global shutter sensor for crisp imaging with distortion-free fames. LiDAR scanner/modeler cameras like XGrids PortalCam are also useful for creating 3D models for visual effects.
Camera Tracking
Both XR and VP shoots can use software to create parallaxing, aligning your video capture to track with your footage, ensuring that the video content on the LED wall appears to move and have depth that syncs with your camera placement and motion in the real world.
Danielo Garcia, General Manager of The Studio B&H, notes that "ZEISS Cincraft Scenario is a leading inside-out camera tracking system consisting of both hardware and software. There are two types of systems for camera tracking that B&H provides, inside-out and outside-in tracking. Inside-out tracking systems like stYpe RedSpy, and the Sony Ocellus are some other popular software components available for camera tracking. Another is outside-in tracking systems like Optitrack. By using their motion capture cameras and the Optitrack Active Puck, the camera tracking system will live inside the motion capture volume as well.”
MoCap (Motion Capture)
MoCap cameras, sensors, suits, and markers enable you to record your talent’s live motion for rendering or live streaming for real-time animation or applying special effects.
Media Servers and Processors
These dedicated software, media servers, and processors are the workhorses used to tie together the components required for the camera tracking and the ICVFX (In-Camera Visual Effects) used for XR. Garcia, “Go-to solutions for us at The Studio B&H include software platforms like Assimilate Live FX Studio, along with task-specific media servers from Disguise and PIXERA. Processors from Brompton and Megapixel sit between the media server and the LED tiles, helping ensure the content is accurately processed, mapped, and displayed across the LED volume.”
Image Based Lighting
Used for both XR and VP, this technique maps your lighting fixtures’ output to the light level and color of your recorded footage, letting you augment your on-set lighting. Components used in IBL include LED lights like the Kino Flo MIMIK 120, Quasar Science Double Rainbow Array, Creamsource Vortex8 RGB LED Light Panel, and ARRI SkyPanel X21 Modular LED Panel.
Props and Set Dressing
XR and VP can both use physical props but XR setups tend to have additional elements that are created using software and real-time rendering while VP sets incorporate more real-world items and set dressing for the actors to use.
AR (Augmented Reality) and MR (Mixed Reality)
When discussing virtual reality, you’re likely to hear about augmented reality (AR) and mixed reality (MR). In a nutshell, AR uses wearable technology like glasses and goggles to overlay visual elements on your view of your actual surroundings. Applications include gaming, social meetups, tours of historical travel spots, educational use, and retail try-outs of clothing, hairstyles/colors, and furniture. MR uses AR technology and adds real-world integration of both digital and physical locations, users, characters, and more for gaming and other interactive platforms. This means that with MR, you can have both real-world users interacting with generated environments and computer-generated characters appearing and interacting in real-world locations.
AR and MR integration with an XR stage
Both AR and MR can be incorporated with an XR stage for enhanced audience participation in a live show, demonstration, or educational lecture. They can also be combined so that directors and crew members can visualize all the elements in their production – live and computer generated.
Stage Setup
XR stages require some of the same basics as traditional studio setups like a large space, high ceilings, soundproof solid walls without windows, a quiet HVAC system, and large access doors for loading in equipment. A preinstalled grid on that high ceiling is handy for hanging LED panels and an electric grid with sufficient power and backup is a must.
Floor Surface
Consider building a platform to raise your talent so that your camera is less likely to see the edge where the volume wall meets the floor. Durable truss-style platforms are available for rent and smaller, lower platforms can be constructed by your set builders. Alternatively, you can dress the floor or fill in a raised bed using the surface material (soil, vegetation, etc.) that matches the video playing on your volume wall.
Dance Floor
To eliminate the time spent building dolly tracks, your grip department can create an ultrasmooth floor for dollying shots using panels of Masonite, a.k.a. “dance floor”. Be aware that tracks may still be necessary for creating faster dollying shots that can be exactly repeated at will.
Access
Depending on the size of your volume wall, it may be necessary to rent one or more scissor or boom lifts to build and access your volume’s LED panels and supports.
If you’d like to learn more about capturing augmented reality and virtual production, check out our other guides and articles on the B&H Explora page!
