Virtual Shadow Maps in Fortnite Battle Royale Chapter 4

Andrew Lauritzen, Senior Rendering Programmer and Ola Olsson, Senior Rendering Engineer |
26. Januar 2023
Hello, we are Andrew Lauritzen and Ola Olsson, rendering engineers who work primarily on shadowing in Unreal Engine 5. Since we started working on UE5, our focus has been on Virtual Shadow Maps (VSMs). VSMs are designed to pair well with Nanite’s virtualized geometry and efficiently render accurate shadows ranging from small geometric detail near the camera all the way to the horizon.

In this post, we’ll cover some of the details of how Virtual Shadow Maps are used in Fortnite Battle Royale Chapter 4, with a specific focus on recent technical improvements as well as content considerations. VSMs were designed specifically with Nanite in mind and integrate tightly with the Nanite rasterization process; in many cases, optimizing shadows and Nanite performance are directly related. Therefore, we recommend reading the Nanite in Fortnite Battle Royale Chapter 4 blog post before continuing, as many of the techniques mentioned there are as important—or in some cases primarily for—Virtual Shadow Maps.

Given the heavy use of Nanite in Chapter 4, Virtual Shadow Maps are a natural choice; they both perform better and have better quality when compared to conventional shadow maps. Ray-traced shadows are not currently a good fit as they are not performant when forced to rebuild acceleration structures to account for the significant dynamic deformation and animation in Fortnite. Additionally, ray-traced shadows use a less detailed version of the Nanite geometry, which works well for some effects, but lacks fine details when applied to direct shadowing.

Beyond just performance and quality, we had several more specific goals for shadowing in Chapter 4 that we will discuss in the following sections.
 

Sun shadows (directional light)

Being primarily an outdoor game, the majority of shadows that players see in Fortnite are from the sun. The sun is also one of the most challenging lights, as ideally, the shadows need to be visually consistent from a few centimeters from the player all the way to the horizon. In Fortnite Battle Royale, this scale difference is not only present, it is exhibited by the initial sky-diving sequence.

Previously, Fortnite has used a variety of shadow techniques to try to cover all of these cases while balancing performance. There are several conventional shadow map cascades near the player. Beyond these, there is a mix of distance field shadows and a single “far” cascade that covers most, but not all of the map. The player character gets a dedicated per-object shadow map. Screen-space contact shadows are layered on top of all of this to try to recover some of the detail missed by the low-resolution techniques and fill in any missing shadows in the distance.

While this mix of shadow techniques has worked well enough in the past, it can only really produce low-resolution, blurry shadows in the distance, and the transition zones between the different techniques are often noticeable. A lot of the visual benefit of Nanite comes from the temporal stability that it brings to level-of-detail transitions. If shadows disappear, pop, or switch techniques at different distances, this undercuts that temporal stability.

Virtual Shadow Maps replace all of these techniques with a single, unified path.

Virtual Shadow Map caching considerations

Virtual Shadow Maps can fulfill the quality requirements, but our past Lumen in the Land of Nanite and The Matrix Awakens: An Unreal Engine 5 Experience demos were able to rely heavily on caching shadow map pages to achieve good performance. Fortnite not only targets higher performance levels than these previous demos, but two additional constraints heavily undercut the ability to cache shadow maps: a significant amount of animated deformation—chiefly trees—and a continuously moving sun.

Initially, we implemented a bunch of improvements to help reduce invalidations from animated geometry, including respecting the “Optimized WPO” flags (discussed in the Nanite blog). This helped, but was not enough to hit our performance targets.

We tried directly throttling the invalidations with some simple priority schemes, but this produced too many artifacts at page boundaries. While these schemes can work well for conventional shadow maps where the biggest costs are rendering distant geometry, Nanite and Virtual Shadow Maps better match the expense to on-screen pixels. The result is that distant shadows are relatively inexpensive and some of the worst cases are actually things like walking up close to a tree and looking up into it; in this case the geometry is incoherent enough to decrease the efficiency of the shadow pages and reduce Nanite performance due to the heavy overdraw.
Simply throttling or skipping shadow page updates produces obvious artifacts close to the camera. We experimented with drawing some objects into the shadow page cache using a “base pose” and then reprojecting the shadow lookups back to that pose; this results in an effect similar to baked lighting where the shadow follows the animated object around as if it were glued to the surface. Unfortunately, there is no easy way to separate out the self-shadowing parts (shadows from leaves onto other leaves) from the shadows of other objects onto the tree, which looked poor in practice.

The second constraint turned out to be even more of a problem. Fortnite features a time-of-day system with a continuously animating sun direction. To be fully correct, we need to throw out all cached shadow map data when the light direction changes, but it is fairly common to “cheat” a bit on this as long as the sun is not moving too quickly. However, several factors conspire to make this more difficult for Virtual Shadow Maps.

First, it is not just the shadow data rotating, it is the entire parameterization of where the pages fall. This effect can be quite pronounced even for visibly slow sun movement for pages near the camera:
Sun movement in Fortnite causes quite significant shadow page table changes frame to frame.
This in turn means that we must at least reproject the cache data from one frame to the next, adding additional overhead and eating into our savings from caching in the first place. Reprojecting dense shadow maps is relatively straightforward; while some data is lost at the edges, usually the minor issues can be hidden by the relatively blurred shadow output.

Unfortunately, with Virtual Shadow Maps we can only reconstruct a new page if all source pages that it covers were mapped in the previous frame, otherwise we would end up caching a partially invalid shadow page. VSMs gain much of their efficiency by being sparse, with significant “holes” in the conceptually dense shadow data. This further reduces the amount of cached pages we can reuse.

VSMs also are typically higher resolution and use sharper filtering than conventional shadow maps, so issues and inaccuracies are more visually apparent and less likely to be hidden by broad blurring. Finally, the worst cases for invalidation are also bad cases for this reprojection: looking into a noisy object like a tree has so many holes in the data that very few pages can be reconstructed.

These issues, together with the common incidence of bad cases, led us to entirely shift our strategy for Fortnite’s sun shadows.

Uncached sun shadows

Since we cannot avoid cases where caching is not possible in Fortnite, these cases must fit into the performance budget. To that end, and to meet the 60 fps targets, we dropped the effective resolution target of the sun shadows to approximately half of what it was in previous demos. This does have a visual impact, but the shadows are still much higher resolution than with previous techniques (as the previous comparisons demonstrate). We also added a control to reduce some bookkeeping and invalidation overhead by assuming we will never cache directional light pages (r.Shadow.Virtual.Cache.ForceInvalidateDirectional).

The tradeoffs involved in this decision will differ for other games. There is certainly a visual impact from giving up on shadow map caching and dropping the shadow resolution that may not be an optimal choice for other games. That said, we are reasonably happy with the level of quality we are able to achieve in Fortnite even with no directional light shadow caching at 60 fps, as it is still a noticeable improvement over previous solutions.

Non-Nanite geometry

Given the decision to stop chasing caching for the sun, the raw performance of rendering into the shadow map becomes paramount. The most important first step to getting good VSM rendering performance is to make everything Nanite. Non-Nanite objects can pose significant performance problems, especially when they have high polygon counts, are physically large, or overlap many Virtual Shadow Map pages.

Since Fortnite has a ton of content, we created some quick and dirty debug output to help find the worst offenders. By setting r.Shadow.Virtual.NonNanite.NumPageAreaDiagSlots -1, the game will output debug text identifying non-Nanite instances that are covering a large number of shadow map pages:
One of the biggest initial offenders was the Landscape. Thus we added simple Nanite Landscape support primarily to improve Virtual Shadow Map performance, which effectively eliminated it as a problem.

In Fortnite Battle Royale Chapter 4, we were able to convert the vast majority of geometry to Nanite, greatly improving shadow performance. There are still some non-Nanite shadow casting objects (in particular, the player models), but they are few and small enough that they generally do not cause performance problems.

To support various volumetric effects, Virtual Shadow Maps also maintain a very low-resolution map that covers the whole frustum, called “coarse pages.” The amazing LOD of Nanite makes rendering coarse pages efficient. Non-Nanite geometry, however, has caused large and variable overhead in previous releases, mainly due to needless vertex processing. To mitigate this issue, we added small-object filtering to the coarse pages, which has enabled us to ship Fortnite with this feature enabled. This is also enabled by default in Unreal Engine 5.1.

Foliage

Improved foliage was one of the major visual goals for Fortnite Battle Royale Chapter 4, but it was also one of the main performance concerns.

Optimizing uncached shadow rendering performance boils down to many of the same things as for the main view, and is covered in detail in the Nanite blog post. Most impactful for shadow performance was removing alpha test and making the World Position Offset function as cheap as possible (largely pre-computed in Fortnite). The cost of these features is already higher with these very detailed Nanite meshes, but we may also have to rasterize mesh clusters a few times when they span multiple shadow clipmap levels.

We implemented the World Position Offset distance culling for the shadow pass (relative to the main camera position, not the light) in part for consistency with the main view, but also for a modest performance improvement.

Fortnite already used a “shadow proxy” system before Chapter 4, and early on, we were unsure if we would be able to hit our shadow performance budget with the full 300k+ polygon tree meshes in the shadow pass, since shadows are typically allocated a smaller frame budget than the primary view despite rendering similar (or in some cases more) pixel counts. Shadow proxies are implemented by having two static mesh components on a tree actor: one for the main view with “cast shadow” disabled and a hidden one for the shadow proxy with “cast hidden shadow” enabled.

In the case of Chapter 4, the shadow proxy meshes are still fairly complicated: 60k+ triangles is typical.
(Left) Base tree mesh (300k+ polygons) | (Right) Tree shadow proxy mesh (60k+ polygons)
The proxies were tweaked to have a minimal visual impact on the shaded trees, as dropping too much detail heavily reduces the sense of lighting depth in the tree canopy. As shipped, the shadow proxies lose a little bit of detail, but it is not particularly noticeable unless you are directly comparing the two images.
(Left) Tree with shadow proxy mesh | (Right) Tree without shadow proxy mesh
It is worth noting that we evaluated shadow proxies before the Nanite “Preserve Area” option was added, and before many of the Nanite programmable rasterization optimizations were implemented. We did not have time to do a thorough re-evaluation later in the process. Additionally, a shadow proxy system was already present in Fortnite and will continue to be needed for non-Nanite platforms.

That said, we are not entirely convinced that shadow proxies are necessary for performance going forward. Contrary to conventional intuition, they make no significant difference to the performance of distant (or even mid-range) foliage because Nanite will still target the same triangle densities for both meshes at those ranges. The only place where the Nanite shadow proxies matter is when trees are very close to the camera, in which case, the proxies slightly reduce how much geometric detail will be rendered in the shadow pass.

Based on some proof of concept work done after Chapter 4, we are fairly confident that we no longer need artist-generated shadow proxy meshes, and the most we might need to get similar performance and quality is the ability to clamp Nanite’s level-of-detail selection on these meshes. We will continue to evaluate if that capability is even required, given all the other optimization work that has been done since shadow proxies were initially evaluated.

Another quality improvement we implemented primarily for the trees was an alternate way to evaluate the shadow term on subsurface materials (which are used for the leaves in Fortnite trees). Conventionally, these materials use a standard “hard” shadow term on the leaves that face the light, and a softer falloff on back faces as a rough approximation of transmission through the leaves.

With previous shadow techniques, the shadows were so blurry that the details here did not make a huge difference. With Virtual Shadow Maps, several new artifacts were apparent related to the fact that the leaf normals on these trees are manipulated towards a more “spherical” normal rather than their geometric normals to soften the shading effect. This resulted in discontinuities where the shading normals flipped from back- to front-facing, switching the associated shadow terms.

To address this and achieve a softer overall look in the shadow term as well, we implemented an alternative subsurface shadow mode that uses a single term that includes the rough transmission falloff regardless of the shading normal. Additionally, we arbitrarily widen the shadow filtering cone based on the material opacity to give the impression of internal scattering. This new mode can be enabled with r.Shadow.Virtual.SubsurfaceShadowMode 1; it will likely become the default in a future engine version.
(Left) Classic subsurface mode | (Right) Improved subsurface mode (r.Shadow.Virtual.SubsurfaceShadowMode 1)

Grass

We took a different approach for grass than for trees and bushes in Fortnite for both visual and performance reasons.

As is the usual case in other games, grass was not previously rendered into the shadow map in Fortnite and instead relied purely on screen-space “contact shadows.” Conventionally, this is at least partially because the resolution of typical shadow maps is not sufficient to capture the detail of the grass anyway.

With the new Nanite geometry-based grass and Virtual Shadow Maps, we re-evaluated this, but in the end decided to stick to contact shadows only for a few different reasons.

First, the artists wanted control over how dark the shadows cast from grass are, which is not possible to provide with Virtual Shadow Maps. The goal of this is to simulate some amount of transmission through the grass, but also to compensate for the grass blades being somewhat larger than reality. Second, the new grass already provided a nice visual quality bump from its nature of being real geometry, and thus providing a better depth buffer for the contact shadows to work from. Finally, the performance cost of rendering the grass into the Virtual Shadow Map was manageable but not small.

There are a few cases in which having full Virtual Shadow Maps looked better, particularly with the larger flowers and ferns. Overall, though, the result was similar, and the artists preferred the softer intensity control provided by the screen-space method. Given the visual quality comparison, we decided we could put this performance to better use for more visual impact elsewhere.
(Left) Virtual Shadow Map grass shadows are more expensive to render and too harsh for the art style. | (Right) Screen-space contact shadows capture a similar effect and can be lowered in intensity to simulate more transmission.

Water

One other area that needed attention was the shadows on the surface of water. Virtual Shadow Maps determine which pages are needed by analyzing the camera’s depth buffer and projecting those sampling positions into the lights’ coordinate frames. Single layer water and other conventionally “forward rendered” techniques do not contribute to the depth buffer, and thus from certain camera views there might not be high-resolution shadow data available for lookups on the water surface.

Together with the water rendering team, we implemented a “depth prepass” mode for the water to provide the necessary data to Virtual Shadow Maps (and additionally integrate better with some other systems), which can be enabled via the console variable r.Water.SingleLayer.DepthPrepass 1. With this mode enabled, the water surface will also mark pages to ensure high-quality Virtual Shadow Maps.

Local lights

Fortnite also has a fair number of local lights (a mix of spot and point lights). They are used for a broad variety of purposes ranging from large ceiling lights to lamps to “effects” lights for things like the treasure chests. These are more visually obvious during the night portion, but they are always present alongside the sun/moon directional light, and thus both together must fit into the shadow budget. In a given frame, there are generally around a dozen shadowed local lights after culling.
Fortnite uses “effects” lights to draw attention to treasure chests.
From a quality perspective, Virtual Shadow Maps immediately offered a lot over previous solutions. First, they simply capture a lot more detail while removing biasing artifacts that plague conventional shadow maps.
(Left) Shadow map has overly blurry penumbra and biasing artifacts on the wall. | (Right) Virtual Shadow Map fixes the artifacts and has variable penumbra.
Second, they respect the “source radius” of the light and produce more physically plausible shadow penumbras that sharpen at contact points and soften away from them. Since this parameter was ignored previously, we needed to review and set reasonable source radiuses on many of the lights in Fortnite.
(Left) Shadow map has uniformly sharp shadows, while missing shadowing from smaller details. | (Right) Virtual Shadow Map has area light penumbra and contact hardening.
Achieving high performance for these local lights involves two aspects of the frame: rendering into them, and projecting and filtering them back onto the scene. Both of these areas required some work in Fortnite Battle Royale Chapter 4.

Local light caching

Unlike with the sun, shadow map caching works well for local lights in Fortnite. Local lights are often immobile, with relatively static geometry surrounding them. Additionally, while it is important to evaluate shadowing for distant lights to avoid them leaking through walls, it is not as important to update them every frame, since the visual impact is much smaller. The basic machinery used to exploit sparseness for high-resolution, near-field shadows becomes expensive when we need to run this for dozens of lights that each contribute a small number of shadowed pixels.

To that end, we added support for categorizing local lights as “distant lights” when they are small enough on screen that they can only cover a single shadow map page (128 x 128 texels). This mode can be enabled via the console variable r.Shadow.Virtual.DistantLightMode 1. For distant lights, we elide most of the page table machinery since they are effectively “dense” shadow maps. Furthermore, we throttle updates to these lights (r.Shadow.Virtual.MaxDistantUpdatePerFrame), relying on cached pages between updates even if geometry has moved.
Visualization of local lights. The yellow light spheres are small enough to become “distant” lights, while the blue ones get a full Virtual Shadow Map.
In Fortnite, we were able to update just a single distant light each frame with minimal visual issues. Lights nearer the camera still get full Virtual Shadow Maps with pages that are updated and invalidated in the usual manner.

Shadow projection performance

The other important piece of shadow performance occurs in the light loop, when shadowing from each light is being evaluated and applied. Small local lights have long been a pain point here: they affect relatively few pixels, but run several passes for each light with dependencies between them, leading to an underutilized GPU.

While this problem is not new, with Virtual Shadow Maps we have some tools to combat it. Unreal Engine 5 had an optional “one-pass projection” mode that helped, but it was tied to using clustered shading. Unfortunately, enabling clustered shading was a performance regression in Fortnite, to the point that it removed most of the benefit of the mode. For Unreal Engine 5.1, we have decoupled the two, allowing one-pass projection to be enabled without clustered shading.

When “one-pass projection” is active (r.Shadow.Virtual.OnePassProjection 1), shadowing for local lights is done in a single pass up front and the results stored in a temporary buffer rather than interleaving shadowing calculations with lighting. We additionally refactored some other passes (translucency volume injection) to happen up front instead of in the inner loop.

The result is that if a given light only needs a Virtual Shadow Map (and does not use a light function or other pass that contributes to the shadow mask), it can now be done with a single, pipelined draw call, resulting in significantly improved performance for small local lights.
Conventional light loop (1.56ms) has several interleaved passes per light with GPU barriers, wasting performance while the GPU idles.
Optimized “one-pass projection” light loop (1.08 ms) computes all the shadowing up front so that each light can be a single pipelined draw with no barriers.

Conclusion

In addition to the improvements described in this post, we implemented several other general improvements in the engine that apply broadly beyond just Fortnite. We will continue to file down the remaining rough edges in the future.

Overall, despite some initial fears around caching and performance, we are quite happy with how Virtual Shadow Maps look and perform in Fortnite Battle Royale Chapter 4. It is great to see people enjoying the game and the new graphics, and we look forward to seeing how other developers use these features in their own Unreal Engine games.
For more detailed technical information, see the Virtual Shadow Maps documentation.

    Get Unreal Engine today!

    Get the world’s most open and advanced creation tool.
    With every feature and full source code access included, Unreal Engine comes fully loaded out of the box.