This article is an introduction to virtual production using photogrammetry and lidar. In this article I will discuss what photogrammetry is, and how advances in lidar scanning allow you to use it in pre-viz as well as production.
What Is Photogrammetry?
Simply put photogrammetry is the process of creating a 3D image, or gleaning 3D information about a location from 2D images. Essentially you use a series of 2D images to create a 3D “map” of a surface, area, or room.
Why Use Photogrammetry?
Photogrammetry enables you to make realistic and interactive backgrounds/foregrounds. Whether you are using still-photos or meticulous physical measurements, or specialized gear to gather the raw data, it is going through computer software, or even an app on your phone to create your 3D map/environment aka virtual set. This allows you to create a realistic environment, one that your camera can move around and through while capturing realistic perspectives, unlike using a flat painted backdrop which does not have realistic perspective as you move the camera, often forcing you to keep the camera static.
Historical Perspective
Sometimes it is not possible or just cost prohibitive to travel to a location to shoot or build out entire sets of the “canals” of Mars, a post-apocalyptic New York City, an abandoned and sand-covered Las Vegas, or half the Statue of Liberty on the shoreline. This is where composite shots come into play.
Since the earliest days of filmmaking painted back drops as well as matte painting (glass panels with some painted sections and some clear areas that the camera could shoot through to realistically mix live action with a fake background) were used to create these fantastical locations.
The issue with painted back drops and matte paintings is that the camera is locked off, and creating camera moves would destroy the illusion. This is where virtual production sets come into play as it allows your actors to interact with the virtual environment, while freeing your camera from a locked down position.
Light Detection and Ranging (LiDAR)
How does lidar get used to help you with pre-visualization or even production and postproduction? Lidar scanners fire lasers at objects, timing how long it takes for the laser beam to return, allowing the scanner to take extremely accurate distance measurements. This process is extremely fast with some systems taking up to one million measurements a second, allowing you to scan areas incredibly fast, without the accuracy of the measurements being compromised. Essentially you are using specialized camera systems, or a smartphone app, to create the 3D map of an environment. This allows your software to create an extremely accurate 3D model with which to work from.
Your Pre-viz software can then use this information to create a “digital twin” of the environment, allowing you to plan out your shots and try camera moves before production begins, saving you valuable on-set time. Likewise, you can scan objects, creating digital twins of those to place within your digital environment, so you can see how your shots will look before shooting them. Additionally, virtual environment information is extremely valuable when planning and creating 3D composites.
Being able to send a small crew to a far-off location to accurately scan it, so it can be recreated digitally on a sound stage can be an immense cost saver, without sacrificing the image quality. It also allows you to only build portions of the 3D environment for the actors to interact with, saving you time and materials as the lidar information can be used to extend out the physically built sections of the set, making for a seamless and perfect blending of real footage and digital models that were formerly created from measurements and photographs that would not be an exact match.
Gaussian Splat
And how is it that lidar can do all this? Gaussian Splatting, which takes in all the lidar information and creates a real-time radiance field instead of reconstituting the information as an image. The advantage of creating radiance fields from 2D images or lidar information is that Gaussian Splat files are significantly smaller than traditional imaging stitches while providing better final image quality and more accurate spatial depictions. Gaussian Splat fields can also be used to create images of the environment as seen from different angles, without having to go and capture all that information from different angles. Move the camera and the view of the environment changes, providing you with a realistic perspective of the environment. PortalCam from XGrids.
Shooting it Clean
In productions with special effects, it is common to shoot what is known as a background plate. So, after the actors have performed, you clear out the actors and any rigs to make the physical components happen and shoot a clean pass of the empty set. This provides you with a clean background that you can use to fill in any areas from the original frame that have to be replaced to remove any rigs, wires, or other physical gags. This also makes it easier, though not necessarily easy, to add in smoke effects later. Using a tripod with encoders allows you to track the camera’s movements so you can repeat that for your clean pass, which makes things much easier in postproduction.
Scanning your set when you are done shooting, but before it is taken down has become popular in case the production has to go back and reshoot part of a scene after a set has been taken down. No need to hope to rebuild the set and light it to match, just pull up the virtual file and you are all set. Some productions even scan between takes, to capture any slight changes to the set/environment.
Scanning has become so quick that it can have little negative impact on your days schedule, while saving quite a bit in costs when it comes time for the inevitable reshoots.
Keeping all this information available and accessible is important for numerous reasons. One is that you can encode camera, lens, and tripod information, so that when following your subjects as they move within the virtual set, the encoded camera information allows the virtual environment to move providing you with a fluid and not locked down camera and proper realism of your background.
Using Photogrammetry, LiDAR and Gaussian Splatting
So here is the really nice thing about these tools: You don’t even have to think about them to use them. Photogrammetry and gaussian splatting are what make the rapid pace of today’s virtual production possible, and lidar is one of the tools for gathering information. The nice thing is that you don’t have to gather, interpret, and render out your virtual environment using an abacus and pantagraph. There are many specialized systems out there to work with such as PortalCam from XGrids, which is an advanced camera tool with multiple built-in cameras for capturing your environment quickly without sacrificing detail and as mentioned earlier, there are even apps for your smartphone so you can get started easily creating accurate storyboards (previz), seamless composites, and effects that appear to happen within the environment and not just on top of it.
It is an exciting time right now, with tools that were once labor intensive and limited almost entirely to the realm of special effects, now more commonly found on all types of productions. I hope you’ve enjoyed this introduction and that it encourages you to keep up with the new advances while you begin to create your own virtual productions.
