The rise of the virtual singer: the making of Fox’s Alter Ego

Creating the next generation of pop stars

If Zinman was going to keep attention on the avatars, he needed the mocap locked in and offstage. For that, he brought in Silver Spoon Animation, who would not only handle the capture process, but the characters, as well. Twenty characters would be needed in total, each of which needed to emotionally connect to an audience and withstand live judging, all in a matter of six weeks. 

To start, a team of 50 artists began by working with each contestant to design a character that accurately reflected their unique stage persona with four different wardrobe options and individual visual effects.

The result was a virtual avatar that would always accurately reflect the motion-captured performance behind the scenes. When a contestant cried, the avatar would cry too. And blush. And run their fingers through their hair. All final performances could then be seen directly through on-set monitors, making it easy for the live audience to engage with the final characters and be immersed in their journey, rather than feeling removed from the story due to the computer-generated imagery.

“In order to pull off a show like Alter Ego, you need Unreal Engine’s DMX capabilities; this is the vocabulary and protocol we all speak and the only way to bring in a network competition show on schedule,” says Zinman. “When lighting directors can program virtual and real studio lights on one console, and the characters can be programmed on another, you run through rehearsals much faster. This is how we were able to do 10 unique performances each day.”

In tandem, Silver Spoon was tapping Unreal Engine’s Live Link capabilities to stream data from over ten mocap instances into the avatars. With virtually no latency, Unreal Engine is able to render high-quality animation captured from the actors onto the avatars in real time—as it happens. Live Link is also used for facial animation via the dedicated Live Link Face app, which enables facial performances to be captured via an iPhone or iPad and rendered in real time in the engine. Using Live Link removed the need for Alter Ego to write a dedicated plugin for the Vicon data packets, giving the team a free-flowing way to animate their skeletal meshes.

A more inclusive future

By the time production was complete, Silver Spoon and Lulu AR had helped create more than 80 performances, or 12 hours’ worth of content, all with no post-production required.

“The pressure on this show was huge. Nothing like it has been done at this scale and in a prime time capacity ever before,” Pack explains. “Unreal Engine is at the point where we can do real-time particle simulation, real-time hair simulation, and real-time lighting directly within the engine. We can pull off these really incredible effects, effects that normally would have to be done in post-processing, in real time.”