Image courtesy of Goldenvoice LLC, all rights reserved ©2022.
Coachella and Flume go all in on live concert AR
May 12, 2022
If you were shocked by what you saw while watching Coachella’s YouTube livestream this year, you wouldn’t be the only one. From giant cockatoos rising above the stage to golden flowers hovering over the crowd, Flume’s headlining performance was filled with awe-inspiring psychedelic visuals, leaving fans asking, “Was that real?”
The answer? No.
Coachella’s organizers—along with artists at &Pull, Six Degrees of Freedom, and All of it Now— added broadcast-quality augmented reality (AR) to their YouTube livestream, kicking off a new era for musicians that want to add real-time 3D visuals to their performances. But unlike other shows, these treats were only for the people at home, turning what could have been ‘just another thing to watch’ into must-stream-TV.
“Our brief was to create proof-of-concept AR moments for the YouTube livestream; something that would feel special, and new, to the millions watching at home,” says Sam Schoonover, Innovation Lead at Coachella.
While on-site performances get most of the attention, the silent story behind the scenes at Coachella is the rising rates of home viewers. At this point, more people watch Coachella at home than attend the festivals (millions more, to be exact), effectively segmenting the experience of the festival into two camps. This also creates a unique challenge for the organizers. How do you make each “version” of Coachella feel special, and befitting of the stature of an event that is almost expected to do something big?
Experiment!
Thankfully, Flume’s team was totally game. Working with legendary artist Jonathan Zawada, Flume's creative team, and show designer John McGuire, Coachella incorporated all the energy and atmosphere of the festival into AR designs that built on Flume’s pre-existing tour visuals / music videos.
The answer? No.
Coachella’s organizers—along with artists at &Pull, Six Degrees of Freedom, and All of it Now— added broadcast-quality augmented reality (AR) to their YouTube livestream, kicking off a new era for musicians that want to add real-time 3D visuals to their performances. But unlike other shows, these treats were only for the people at home, turning what could have been ‘just another thing to watch’ into must-stream-TV.
Making cockatoos for the Mainstage
To long-time Coachella watchers, this might not be a surprise. Big events and Coachella go hand in hand. Since 1999, the festival has become known for creating unique experiences that make people talk, whether that’s reuniting major bands; creating massive art installations; or bringing Tupac back to “life.” But this year, Coachella’s organizers wanted to do something different.“Our brief was to create proof-of-concept AR moments for the YouTube livestream; something that would feel special, and new, to the millions watching at home,” says Sam Schoonover, Innovation Lead at Coachella.
While on-site performances get most of the attention, the silent story behind the scenes at Coachella is the rising rates of home viewers. At this point, more people watch Coachella at home than attend the festivals (millions more, to be exact), effectively segmenting the experience of the festival into two camps. This also creates a unique challenge for the organizers. How do you make each “version” of Coachella feel special, and befitting of the stature of an event that is almost expected to do something big?
Experiment!
Thankfully, Flume’s team was totally game. Working with legendary artist Jonathan Zawada, Flume's creative team, and show designer John McGuire, Coachella incorporated all the energy and atmosphere of the festival into AR designs that built on Flume’s pre-existing tour visuals / music videos.
Unreal Engine was used as the basis for a state-of-the-art content creation pipeline. First, animated models were imported from Blender, Cinema 4D, and 3ds Max into the engine, which enabled the team to test each visual and make tweaks on site before the show. Approved graphics were then turned into detailed final-pixel sequences, which were composited into Flume’s livestreamed performance in real time thanks to media servers running Unreal Engine, the StypeLand compositing plugin, and the stYpe tracking data coming in from three real-world broadcast cameras.
“Using shaders in Unreal Engine, we developed a 3D object occlusion system that helped us scale 3D models to real-world dimensions and composite them behind the stage,” says Berto Mora, Producer at All of it Now. “We then created a trigger so that we could automatically switch graphics in Unreal Engine based on the band’s timecode.”
Happy accidents and impressive results
And the results were impressive! Viewers could marvel at the colorful, psychedelic visuals that surrounded Flume as he performed songs like Never Be Like You, Say Nothing, and Tiny Cities on Coachella’s Mainstage. As well as the giant cockatoos and surrealist flowers during the performance, there were also the eye-catching interstitials—including deforming doughnuts bouncing off the stage and the rainbow towers of Spectra City rising from the ground—that entertained audiences during the least exciting part of any concert: waiting for the next song.
Image courtesy of Goldenvoice LLC, all rights reserved ©2022.
“We also had some happy accidents,” remembers Juan Santillan, Executive Producer at Six Degrees. “For instance, we didn’t know Beck was going to be joining Flume on-stage, and yet, the flower graphic was centered around him perfectly in the frame, like it was made just for him.”
Meanwhile on stage, all video feeds were color corrected live before going to the show’s AR truck, where AR visuals were composited into the live broadcast feed. Thanks to Unreal Engine’s DMX integration, the compositing was photoreal: the team could even synchronize all virtual lighting to the stage. Once composited, the video feed was then sent back to the main broadcast truck, where Flume’s show director was able to cut the final result from the AR cameras live, just as he would in a traditional performance. The result was a seamless, perfectly-timed visual experience that kept fans immersed in the music.
“The festival experience is about fully immersing yourself in the music and atmosphere to the point you get lost in the moment,” adds Eric Wagliardo, Live AR Producer at Coachella. “With live AR, we transported fans at home to that magical place, so they could experience Flume and Zawanda’s surreal imagination. A place where 300 ft. tall trippy cockatoos beckon and surreal flowers hover in the air.”
The future of concert visuals
Beyond Flume’s set, Coachella’s innovative AR pipeline has paved the way for a future where augmented reality visuals aren’t just seen by fans watching at home, but by those in the real world, too. “Engagement went crazy on the stream when the birds came out,” remembers Schoonover. “People weren’t sure if it was real or CG.”For him, that rise in engagement represents a turning point for live show visuals: one that could even serve as a template for future hybrid events and festivals in the metaverse.
“As online audiences grow, it’s crucial that digital events bring something unique to the table. But it’s also important to think about how new tech could affect the experience on site. We hope this project will usher in the next era, where AR glasses and virtual worlds allow fans to experience a completely new dimension of music,” he adds.
“To us, this transition from 2D to 3D will be as revolutionary as the shift from mono to stereo, and Unreal Engine is a key part of making it happen. From start to finish, Unreal Engine created an environment where we could test ideas, iterate on concepts, and deliver an unforgettable experience for Coachella fans that spells the beginning of a new future. We’re excited to do even more next year,” said Wagliardo.
Image courtesy of Goldenvoice LLC, all rights reserved ©2022.
Extending the Coachellaverse
While AR might have been the main event, Coachella organizers didn’t stop there, extending their real-time ambitions into Fortnite and giving at-home fans another way to tap into the Coachella experience. Clothing styles inspired by Indio trends became part of the Fortnite universe, along with new, music-reactive accessories that play off the fun aesthetic of the festival. Coachella also took over the in-game airwaves with an official Icon Radio station featuring songs from over 30 2022 Coachella artists. Coachellaverse engage!
Get Unreal Engine today!
Get the world’s most open and advanced creation tool.
With every feature and full source code access included, Unreal Engine comes fully loaded out of the box.
With every feature and full source code access included, Unreal Engine comes fully loaded out of the box.