Virtual production (VP) is a professional video technique that combines live-action footage with computer-generated imagery (CGI) in real time. Virtual production typically happens on “volume” stages where seamless interlocked LED panels display rendered photo-real backdrops that change and adapt as the camera moves. This creates impressive in-camera virtual effects (ICVFX) that require little to no post-production work. Let’s jump into the exciting capabilities of on-set virtual production (OSVP).
Virtual production is an umbrella term that can include foreground overlays and scenic extensions, aka augmented reality (AR), plus extended reality (XR) elements like LED volume or green screen-based background replacement. Mixed reality (MR) can include those elements as well as on-camera live action footage captured in real time. This allows real talent to interact with virtual elements like props and sets. In the video production world, all of this work leads to an output stream, a final product if you will, versus the infinitely extending interactive worlds of virtual reality (VR).
A basic VP setup requires rendering software (often Unreal Engine), a media server, LED cabinets/panels, an LED processor, and a camera. More complex workflows might add in 3D LiDAR scanning, detailed camera and lens tracking, motion capture suit tracking, adaptable IBL (image-based lighting), and more. The media server is the destination for the rendering software’s rendered image, which the server then feeds to the LED system (processors and cabinets) and the lighting system which maps output to the displayed imagery. Camera and lens tracking info is also sent to the media server to determine the final image output. Sometimes this image will need more tweaking in post-production, but often it’s already complete.
LED cabinets/panels built for VP are much different and higher spec’d than those made for digital signage, as VP panels typically have a pixel pitch of P3 or smaller, a constant brightness between 1200 and 1500 nits, a ceiling brightness of at least 6000 nits, wide DCI-P3 color gamut coverage (+90%), anti-reflection coating, wide viewing angles, and a refresh rate of 240 Hz or higher. Popular cabinet brands include ROE Visual, Planar, and Absen. Major media server options include PIXERA and disguise. Brompton, Megapixel, and NovaStar make most current VP controllers. For processors and controllers, look for rack mountable options with built-in genlock, SDI, and networking capabilities.
All the equipment in the workflow needs to be VP-ready, so aim for cameras that either have a global shutter or readout speeds of less than 18 ms to prevent image tearing or rolling. Make sure they have timecode and genlock in/outs, and if they’re PTZ cameras, that they are compatible with FreeD protocols. The lenses you use should be able to output tracking data via the mount or an external port. For lighting, look for RGB panels with DMX control. Rigging choices for cabinets will vary based on the size and weight of the completed volume, but most can be stacked from the ground up or hung from trusses.
VP is particularly popular in the high-end cinema and broadcast world but is quickly growing in other content areas as well, including educational institutions and in-house brand studios. The most popular use case remains scripted content requiring difficult locations that can be more easily recreated in a volume, think sci-fi spaceships or dangerous forest fires, as well as productions that need to move fast and capture a variety of locations without relocating the cast and crew. Growing use cases include interactive educational content, dynamic live productions, and high-end content creation.
VP can be more advantageous than a traditional green screen VFX approach, as ICVFX can be refined and adjusted in real time to achieve a better result. Green screen shooting still requires some degree of guesswork and approximation, then a lot of work in post for spill correction and keying refinement. Virtual production, while often requiring more of a lift in pre-production and production, can help the entire cast and crew zero in on the final look day of, achieving more dynamic lighting, blocking, and performances.
If you’d like to learn more about virtual production, check out our other guides and articles on the B&H Explora page!
