This article is an introduction to virtual production using photogrammetry and other reality-capture technologies. In this article I will discuss what photogrammetry is, and how advances in lidar scanning allow you to use it in pre-viz as well as production.
What Is Photogrammetry?
Simply put photogrammetry is the process of creating a 3D image, or gleaning 3D information about a location from 2D images. Essentially you use a series of 2D images to create a 3D “map” of a surface, area, or room.
Why Use Photogrammetry?
Photogrammetry enables you to make realistic and interactive backgrounds/foregrounds. Whether you are using still photos, meticulous physical measurements, or specialized gear to gather the raw data, the information passes through digital computer/mobile software to create a visible, 3D version of the data, also known as a virtual set. This allows you to create a realistic environment, one that your camera can move around and through while capturing realistic perspectives, as opposed to using a flat painted backdrop that does not have the same freedoms of movement with perspective, often forcing you to keep the camera static.
Historical Perspective
Sometimes it is not possible or within budget to travel to a location and build out entire sets of the “canals” of Mars, a post-apocalyptic New York City, an abandoned and sand-covered Las Vegas, or half the Statue of Liberty on the shoreline. This is where composite shots come into play, a fundamental starting point for the world of photogrammetry and virtual productions.
Since the earliest days of filmmaking, painted backdrops, as well as matte painting (glass panels with some painted sections and some clear areas that the camera could shoot through to realistically mix live action with a fake background), were used to create these fantastical locations.
The issue with painted backdrops and matte paintings is that the camera is locked, as creating physically moving a camera outside of the shot would destroy the illusion. Virtual production sets allow your actors to interact with the virtual environment and free your camera from a locked down position, as both can move within the space with no fear of ruining the illusion.
Light Detection and Ranging (LiDAR)
Lidar is a useful methodology for creating realistic environments in a digital space. Lidar scanners fire lasers at objects, timing how long it takes for the laser beam to return, which allows the scanner to take extremely accurate distance measurements. This process is nearly instantaneous, with some systems taking up to one million measurements a second, allowing you to scan areas incredibly fast without compromising on accuracy. Essentially, you and your lidar system are using specialized camera systems, or a smartphone app, to create the 3D map of an environment utilizing these measurements. This allows your software to create a 3D model to scale.
Your pre-viz software can then use this information to create a digital twin of the environment, allowing you to plan out your shots and try camera moves before production begins. Likewise, you can scan objects, creating digital twins of those to place within your digital environment, so you can see how your shots will look before shooting them. Additionally, virtual environment information is extremely valuable when planning and creating 3D composites, as it allows you to save your time on crafting the important stuff: the project itself!
Being able to send a small crew to a far-off location to accurately scan it so it can be recreated digitally on a sound stage can be an immense cost saver, and it doesn't sacrifice the image quality. It also allows you to build relevant portions of the 3D environment, whichever you need, at any time, for the actors to interact with, saving you time and materials. The lidar information can be used to extend out the physically built sections of the set, for seamless blending of real footage and digital models created from measurements and photographs that would otherwise be difficult to edit into a cohesive narrative.
Gaussian Splat
So how is it that lidar can do all this? How does all this information get from scans to a tangible, working model? Gaussian splatting! Gaussian splatting creates a real-time radiance field from the 2D images brought in by scans (though you're scanning 3D objects, the actual product produced could be printed on a piece of paper, making it a 2D image of a 3D object!) that builds an environment with depth, height, and visual perspective instead of reconstituting the information as a flat image. It is entirely perspective dependent. It takes gathered spatial and visual data and layers them together, finding commonalities and creating photorealistic renderings of environments. One advantage of using lidar information + camera reconstruction data to create Gaussian splats is real-time, photorealistic rendering with accurate view-dependent lighting. Gaussian splat fields can also be used to create images of the environment as seen from different angles, without having to go and capture all that information from different angles. It already has all the information it needs from the hyper accurate scanning! Move the camera, and the perspective of the environment changes, making it seem like you're looking at a real place you might walk right into, rather than turning off screen and spying the soundstage itself.
Shooting it Clean
In productions with special effects, it is common to shoot a background plate, a fancy way of saying a basic version of the environment. You clear out the actors, cameras, lights, and other physical components of a filmset out of the space and shoot a clean pass of the empty set. This provides you with a clean background that you can use to fill in any areas from the original frame that need editing to remove any rigs, wires, or other physical gags. This also makes it easier to add in smoke and other environmental effects later. Using a tripod with encoders allows you to track the camera’s movements so you can repeat that for your clean pass, which makes things much easier in postproduction when editors are putting all the pieces together.
Scanning your set when you are done shooting, but before it is taken down, has become a popular choice for gathering this footage, since sometimes the production has to go back and reshoot part of a scene after a space has been reset or removed altogether. With this background plate, there's no need to try and recreate magic, you can just pull up the virtual file and you have all the details in place to use. Some productions even scan between takes, to capture any slight changes to the set/environment.
Scanning has become so quick that it has little negative impact on your shooting schedule and it saves quite a bit of money when it comes time for reshoots. You can encode camera, lens, and tripod information, so that when following your subjects as they move within the virtual set, the encoded camera information allows the virtual environment to move with you, providing a fluid and natural flow to the shot the same way it would if you were physically in the space you've built.
Keeping all this information available and accessible is vital to the success of a virtual production. It saves everyone's time!
Using Photogrammetry, LiDAR and Gaussian Splatting
Here is the really nice thing about these tools: they are so accessible that it feels like you don’t even have to think about them. Photogrammetry and Gaussian splatting are what make the rapid pace of today’s virtual production possible, and lidar is one of the tools for gathering information. It does it all for you! No abacus, paint, woodworking, or mental math required. There are many specialized systems out there to work with like the PortalCam from XGrids, which is an advanced camera tool with multiple built-in cameras for capturing your environment quickly without sacrificing detail and as mentioned earlier. Though the PortalCam doesn't rely on photogrammetry in the classic sense, it utilizes a SLAM-assisted Gaussian splatting workflow that allows users to rapidly create high-quality spatial assets and digital twins without the complexity of traditional reconstruction pipelines. There are even apps for your smartphone that can help you get started creating accurate storyboards (previz), seamless composites, and effects that appear to happen within the environment and not just on top of it.
It is an exciting time right now for virtual production. Concepts and practices that were once labor intensive and limited, especially in the realm of special effects, now more commonly found on all types of productions with access to photogrammetry and like processes. I hope you’ve enjoyed this introduction and that it encourages you to keep up with the new advances while you begin to create your own virtual productions.
