Why 'The Mandalorian' uses virtual sets over green screens

• Virtual sets provide certain functions that green and blue screens may lack, such as providing realistic colors and reflections on metallic surfaces.
• A recent development to this technology applies Unreal Engine, a powerful video-game-creation tool that allows filmmakers complete control over the set and surrounding environment at a moment's notice.
• These methods were put to the test on Disney's "The Mandalorian," of which over half was filmed indoors on a virtual set.

Following is a transcript of the video.

Narrator: While filming this scene from Disney's "The Mandalorian," the actors could see their surroundings, but the surroundings weren't actually there. All of this is just LED screens displaying backgrounds pre-made in a video game engine.

Compare that with this fight scene from "Avengers: Endgame." Where actors jumped around in a sea of green, imagining how VFX artists would make this planet look once filming had ended.

"The Mandalorian" is one of the first major productions to choose LED walls over green screens. And the benefits for the actors are just the tip of the iceberg. LED walls make the lighting better, filming smoother, and in certain cases, cost a lot less than using green screens. But to understand why the team behind "The Mandalorian" chose these LED screens, we have to understand just how they work.

"Mandalorian" showrunner Jon Favreau revolutionized virtual production while directing "The Lion King" and "The Jungle Book." However, the process for these two remakes still relied heavily on blue screen and post-production work. For "The Mandalorian," LED wall technologies seemed like the next logical step given the show's production budget and time frame.

Now, you may be thinking, 'This isn't so new, I've seen something like this before.' And you're right, kind of. The predecessor to what we see on "The Mandalorian" is a driving scene like this one, from "Dr. No." You've got the actor in the car and behind them, a screen with footage of the road they've traveled. But the technology was limited. Say you want to move the camera angle during the scene. That projected footage can't move with the camera.

But by using Unreal Engine, tech borrowed from the video game field, that problem is solved. Artists can create a photorealistic 3D background that moves strictly with the camera's field of view, known as the frustum. So if the camera swings around and changes angles, the background shifts in precisely the same way. This allows motion-tracked cameras to execute traditional cinematography techniques within the virtual set, achieving cinematic movements like the parallax effect, where an object in the foreground moves at a different speed than the background, amplifies the illusion of filming at an actual location.

Now that LED screens can move with the camera's eye, virtual sets can solve a bunch of green screen problems. The biggest one probably being lighting.

Richard Bluff: We've all seen movies, TV shows, where the lighting on the day didn't necessarily match the post-production work that was added many, many months later.
Narrator: That's Richard Bluff, Industrial Light and Magic's VFX supervisor for "The Mandalorian." He says lighting is one of the key benefits of working with virtual sets. The light coming from the LEDs provide realistic colors and reflections on the actors and props, something you simply can't achieve with green screen. They are also able to completely avoid the problem of spill. To better understand that, we talked to Kim Libreri at Epic Games, creators of Unreal Engine, which was used to build the virtual sets for "The Mandalorian."

Kim Libreri: The problem with the green screen is that you have to have a green screen behind you. And what that does is that instead of projecting beautiful lighting for what the environment would be behind you, it basically puts a lot of green light on you. And we call that "spill." If you wrap an actor with a big 360 LED wall, you can light in a way that you would never be able to do on a green screen. So you can really make it feel like the characters are embedded in their environments.

Narrator: This green screen spill would've been an even bigger problem than usual for "The Mandalorian," whose main character gets a shiny new suit of armor early on in the season. So realistic lighting from the LEDs makes the show look better. But virtual sets help with practical concerns too. Take this scene in the office of The Client, played by Werner Herzog. In a scene like this, surely they just built a simple set, right? Wrong. All of this is that same LED screen. According to Richard, only the desk, the floor, and one or two columns were really there.
Richard: Everything else was displayed on the screen. But this allowed us to limit the amount of stage space that we used in shooting season one of "Mandalorian," which has, understandably, a big impact on time and budget.

Narrator: To be fair, the initial investment in a virtual set like this is huge. However, the creators claim that money saved from traveling to locations, building new sets, and costly post-production, can make the investment worth it.

Along with saving time and money, the team developed a system that allowed artists to make changes and control the world on the day of shooting. Settings like exposure, color, animation playback, and fill lighting are available to the filmmakers at a moment's notice. If they want to move a mountain from one side of the virtual set to the other, they can, right there on set. Not only is it helpful for the actor, who no longer has to imagine where that mountain is, but it's a game changer for the director of photography.
Richard: Because in the distance, if there's a mountain range that is a story point, then the DP's no longer guessing where those elements are, he can frame up to them.

Narrator: The virtual set constructed for season one was 75 feet in diameter, 21 feet high, and also had a roof composed of LEDs. The cast and crew referred to this space as "The Volume."
Richard: And we had doors that can kinda close in to almost give you full 360-degree coverage of LED screens, minus the floor. It's driven by a small group of machines running Unreal and our Stagecraft technologies.

Narrator: At the end of the day, the big question is, do virtual sets really mean the end for green screens? Well, even "The Mandalorian," with it's technological advances, used a version of a green screen, but via the LED wall. Another capability of virtual sets is to choose pinpointed areas of the wall and turn those green.
Richard: Now, we now have the ability to limit the amount of green screen that's visible behind the character. So now it doesn't wrap around the entire stage, it's just a little piece, but also, they gain the lighting from that environment. Which gives us a more seamless result in post-production.
Narrator: Additionally, scenes involving explosives are best kep to traditional methods. This is to prevent LEDs from getting damaged.

Narrator: So will green screens survive this wave of tech innovation?

Richard: Eventually, of course, we hope never to use green screen. But, I still see that there will be a future for it in the short-term, because there's likely always a need to remove people, to add additional action behind them. But we are getting to the point where the amount of green screen that's being used is massively reduced and the sky's the limit right now.