Virtual Production & XR

What Is On-Set Virtual Production?

On-Set Virtual Production (OSVP) is a filmmaking method in which actors and physical sets are placed in front of massive LED walls. These screens display photorealistic, real-time 3D environments that automatically adjust to the camera's perspective, capturing the final composite image directly on set without needing green screens. 

What Are In-Camera Visual Effects?

In-Camera Visual Effects (ICVFX) is a filmmaking technique that captures interactive visual effects live on set. It replaces traditional green screens with large LED walls that display photorealistic, real-time 3D environments. As the physical camera moves, the virtual perspective shifts instantly to create realistic depth and perspective (parallax).  

What Are xR, AR, MR, and VR?

Several immersive technologies work alongside virtual production, each serving different purposes. 

• Extended Reality (xR): Extended reality (xR) is the umbrella term for immersive technologies that blend physical and digital environments. In virtual production, xR often refers to combining real performers with virtual environments displayed on LED walls or composited in real time.                                                                                                   

• Augmented Reality (AR): Augmented reality (AR) overlays digital graphics onto the real world. Broadcast television commonly uses AR to display virtual graphics, statistics, weather maps, or studio elements that appear to exist within the physical set. 

• Mixed Reality (MR): Mixed reality (MR) allows digital objects and physical environments to interact with one another. Virtual objects can respond to camera movement, lighting, and even physical obstacles, creating highly believable interactions between real and digital elements. 

• Virtual Reality (VR): Virtual reality (VR) places users inside a completely computer-generated environment using VR headsets. Although primarily used for gaming and simulation, VR also plays an important role in virtual production by allowing directors, designers, and production teams to scout digital sets before filming begins. 

Together, these technologies form the broader immersive ecosystem that supports modern virtual production workflows. 

Which Virtual Production Components Do I Need?

A successful virtual production system consists of several specialized hardware and software components working together, including: 

• Rendering Software: Rendering software creates the real-time 3D environments displayed on the LED volume. It generates lighting, textures, animations, weather, and visual effects while responding instantly to camera movement. 

• Media Server: The media server acts as the central processing hub that manages rendered content and distributes synchronized video signals to the LED walls. It ensures low latency and frame-accurate playback while maintaining synchronization between rendering software, camera tracking, and LED displays. 

• LED Volume: The LED volume forms the virtual environment surrounding the actors. It typically consists of a curved LED wall, ceiling panels, and sometimes an LED floor that collectively display immersive backgrounds with accurate perspective and lighting. 

• GhostFrame: GhostFrame is a virtual production display technology that allows multiple independent images to be shown on the same LED volume simultaneously—even though the human eye sees only one. By synchronizing the LED wall with the camera's shutter, GhostFrame displays different image "slices" on alternating LED refresh cycles. The camera records only the image intended for that camera, while the audience and crew continue to see a separate reference image. 

• Mounting Accessories: Structural mounting systems securely support LED panels while allowing precise alignment. Curved walls, ceilings, corners, and custom shapes all rely on purpose-built mounting hardware to maintain structural integrity and seamless image quality. 

• Controller: The LED controller receives video signals from the media server and manages image processing across thousands of LED modules. It controls brightness, color calibration, synchronization, and overall image quality throughout the volume. 

• 3D Lidar Scanners: Lidar scanners play a vital role in¿“reality capture”¿workflows by creating highly accurate digital representations of real-world environments. Depending on your production's needs, these scanners can be used for¿SLAM (Simultaneous Localization and Mapping)¿to quickly map large indoor or outdoor locations,¿photogrammetry¿to generate photorealistic textured models from overlapping photographs, or¿Gaussian splatting¿to create highly realistic scene reconstructions with exceptional lighting and view-dependent detail.  

• Camera & Lens Tracking: Tracking systems continuously monitor the position, rotation, and lens characteristics of the production camera. As the camera moves, the rendered background updates in real time to maintain accurate perspective and parallax, creating the illusion that the digital environment exists physically behind the actors. 

• Background Plates: In virtual production, a background plate (or backplate) is a high-quality image or video of a real environment that serves as the background behind the actors or physical set. Instead of building an entire location digitally in real time, filmmakers use backplates to represent distant scenery or static environments. Backplates are commonly displayed on an LED wall or composited behind actors during post-production. 

• Motion Capture (MOCAP): Motion capture systems (MOCAP) record body movements and facial expressions, allowing digital characters and animations to mirror live performances. MOCAP is commonly used for virtual characters, digital doubles, and real-time animation. 

• Lighting: Lighting is critical for blending physical subjects with virtual environments. 

• Image-Based Lighting: Image-based lighting (IBL) uses the virtual environment itself as a lighting source. The LED walls emit realistic environmental light onto actors and props, creating natural color matching and reflections. 

• Pixel Mapping: Pixel mapping synchronizes physical lighting fixtures with content displayed on the LED walls. As the virtual environment changes, studio lights automatically adjust brightness, color, and movement to maintain visual consistency. 

• Cameras: Several camera types are commonly used in virtual production, including: 

• Cinema: Cinema cameras provide maximum image quality, dynamic range, and lens flexibility for feature films and high-end commercials. 

• Broadcast: Broadcast cameras are optimized for live television production, sports, and studio broadcasting while supporting virtual sets and real-time graphics. 

• PTZ: PTZ cameras offer remotely controlled pan, tilt, and zoom functionality, making them ideal for automated studios, live events, educational facilities, and houses of worship. 

How Do I Pick an LED Panel?

Selecting the proper LED panel is one of the most important decisions when building a virtual production studio. When shopping for a proper LED panel, keep the following requirements and considerations in mind: 

• Brightness: Different parts of the LED volume require different brightness levels. For example: 

• Background Walls: Between 1,200 and 1,500 nits to provide realistic imagery while minimizing unwanted reflections and camera artifacts. 

• Ceiling Panels: 5,000 nits or greater to produce sufficient illumination for image-based lighting and realistic highlights. 

• LED Floors: Brightness requirements vary depending on the production, but panels must also be durable enough to function as safe standing surfaces for performers and equipment. 

• Pixel Pitch:  A pixel pitch of P3 (3.0 mm) or smaller is generally recommended. Smaller pixel pitches create sharper images and significantly reduce the visibility of individual LEDs when cameras are positioned close to the wall. 

• Refresh Rate: Choose panels with a minimum refresh rate of 240 Hz. Higher refresh rates help minimize flicker, rolling shutter artifacts, and moiré patterns when filming with professional cameras. 

• Viewing Angle:  A viewing angle of at least ±80° horizontally and vertically (160° total) ensures consistent brightness and color accuracy from multiple camera positions throughout the studio. 

• Anti-Reflection Treatment: LED panels should feature an anti-reflective surface treatment to reduce glare from studio lighting and improve overall image quality.  

What Is Unreal Engine?

Unreal Engine is a real-time 3D creation platform originally developed for video games that has become one of the industry's leading virtual production tools. 

In virtual production, Unreal Engine generates photorealistic environments that update instantly as cameras move through the scene. Production teams use it to build digital sets, animate environments, control lighting, create visual effects, and render cinematic-quality graphics at interactive frame rates. 

Its ability to produce near-final visuals in real time has fundamentally changed how films, television programs, commercials, and live broadcasts are produced. 

What Is Photogrammetry?¿ 

Photogrammetry is the process of creating accurate 3D digital models from hundreds or thousands of overlapping photographs. The resulting models preserve real-world geometry, textures, and surface details with exceptional accuracy. 

Photogrammetry is widely used across numerous industries, including film and television, virtual production, architecture, construction, video game development, geographic mapping, and many more. 

Within virtual production, photogrammetry enables filmmakers to recreate real-world locations as highly detailed digital environments that can be explored, modified, and rendered inside real-time engines. 

How Is Gaussian Splatting Used in Virtual Production?

Gaussian splatting is an emerging 3D scene reconstruction technique that creates highly realistic digital environments using millions of tiny Gaussian "splats" instead of traditional polygon meshes. 

By capturing a real location with multiple photographs or video, Gaussian splatting can reconstruct scenes with exceptional realism while preserving fine details such as vegetation, fabric, transparent materials, and complex lighting. 

In virtual production, Gaussian splatting offers several advantages: 

• Faster creation of photorealistic environments. 

• Highly realistic digital location capture. 

• Efficient rendering of complex scenes. 

• Improved realism for reflections and lighting. 

• Reduced asset creation time compared to traditional 3D modeling. 

As rendering technology continues to mature, Gaussian splatting is expected to become an increasingly valuable tool for creating digital sets that are nearly indistinguishable from real-world locations, further expanding the creative possibilities of virtual production.