Upper image, production shot of live action actor James Franco (Left) and Andy Serkis, with digital character superimposed (Right). Lower image, the finished shot of live action and the CG character Cesar.
Over the last two decades, the entertainment industry has experienced a turn to what Lucy Suchman termed virtualization technologies in film and videogame production (Suchman 2016). In addition, production studies scholars have described authorship as linked to control and ownership, sharpening distinctions between “creative” and “technical” work, a divide with significant economic repercussions (Caldwell 2008). These ideas are useful in understanding film studio workspaces, where visual effects (VFX) workers and actors collaborate in creating believable virtual characters, using three-dimensional (3D) modeling software and motion-capture (mo-cap) systems to capture the attributes and movements of human bodies and transfer them to digital models. Once captured, digital performances become data, to be manipulated and merged seamlessly with those of live actors and environments in the final film. The introduction of virtualization technologies and computer graphics tools have surfaced tensions over creative control, authorship, and labor. British actor Andy Serkis has been a high-profile apologist for the human actor’s central role in bringing virtual characters to life for film. Serkis, who Rolling Stone called “the king of post-human acting,” is known for using motion capture (mo-cap) to breathe life into digitally-created, non-human characters. His notable performances include the creature Gollum in the Lord of the Rings trilogy (2001-2003), the ape Cesar in Rise of the Planet of the Apes (2011), as well as Supreme Leader Snoke in Star Wars: The Force Awakens (2015), and work on several characters in the 2018 Mowgli: Legend of the Jungle, which he also directed. While Serkis’ performances have made him highly visible to audiences, digital labor historians have begun documenting the often-invisible film workers creating 3D models and VFX (Curtin and Sanson, 2017). The tensions between mo-cap performers and VFX workers reveal the contours of an emerging hybrid workspace that combines actors’ physical bodies and movements with VFX workers’ manipulations of digital geometry and data.
(Left) Actor Andy Serkis performs in a motion capture suit, animating the CG character Gollum (right).
Serkis, who founded his own mo-cap company, The Imaginarium, performs wearing a body suit dotted with markers, with his face covered in markers as well. His body and facial movements are captured by sensors and become data, to be mapped to points on the body of a 3D character. For his performance of Gollum, one of the first virtual characters to be convincingly brought to life using these techniques, Serkis positioned himself as the author of Gollum’s performance, with the 3D model as his digital shadow. The team of VFX artists working on Gollem vehemently disagreed. In interviews and online discussions they argued that Serkis’ movements simply generated raw material—the data—which they extensively manipulated to bring the character’s geometry to life. Mo-cap transforms the virtual body of the 3D model into a space of intimate interaction, bridging the human actor and computer-generated model. The linked bodies of the actor and the digital character form an extended hybrid physical/virtual space. The location of the performance becomes unclear as actors, animators, directors, and film companies each claim the space, and the labor, as their own.
Mingling the work of humans and algorithms
In the entertainment industry, skilled craft work has traditionally been performed in an assembly-line-style system of tasks, with boundaries reinforced by union rules and definitions. VFX work—primarily performed by non-union computer workers—and digital technologies disrupt this system by blurring the boundaries between tasks, and introducing computer simulations into the mix. Sophisticated algorithms for simulating fur, flesh and physical phenomena mingle with skilled hand animation work in the final film. Film studio managers have embraced the use of algorithms to automate visual effects and digital character animation, in an effort to cut costs. Though VFX workers have protested moves to replace their creative work with code, they have had limited success in resisting these and other cost-cutting practices in the highly-competitive entertainment industry.
The virtualization of Serkis for the Lord of the Rings, began with high-resolution 3D-scans to capture the surfaces of the actor’s face and body. Using Serkis’ physical characteristics as a starting point, VFX workers digitally sculpted Gollum, incorporating computer-simulated skin, hair, muscles and lighting. Serkis’ body generated the performance data, which was transferred to Gollum’s body. But in a far-from-automated process, VFX workers painstakingly manipulated the data to reshape and edit the performance. Where did the act of performance take place? For Serkis, it was a creative, mental, and physical process of acting, performed with his mind and body. VFX workers’ view was that they performed the model in the computer, using Serkis’ data as raw material.
Serkis’ captured performance became a transportable, transmutable computer database—an archive of his movements and facial expressions that can be reinterpreted and mapped to any digital character at any point in the future. VFX modeling lead artist Bay Raitt created over 900 control points, to shape the 2,600 polygons of Gollum’s face, and the actor’s body was transformed into a data generator. The boundaries of the VFX workspace included both Serkis, the VFX workers and the surfaces of the 3D model—a hybrid physical/virtual space that acted as a third skin upon which the performance was enacted.
Who authors a virtual performance?
Virtualized labor can include transferring human attributes to an animal—as in Serkis’ performance of the ape Cesar; or animal characteristics to a human—as with the humanoid creature Golem. In the studio, mo-cap technologies have elided some of the boundaries between creative and technical work, though creative work, categorized as “above the line” in film budgets, remains more highly-compensated than below-the-line production work. In interviews Serkis maintains his creative status by claiming authorship (and ownership) of Gollum’s performance, describing Weta Studio workers as skilled technicians and copyists.
…they honor the performances that are given by the actors on set. …It’s a given that they absolutely copy [the performance] to the letter, to the point in effect what they are doing is painting digital makeup onto actors’ performances.
Many VFX workers on the project expressed their disgust at the idea that they merely copy a performance, and took exception to the phrase “digital makeup” in online discussions.
The problem is, the more Andy Serkis tries to take ownership of his performances, the more he minimizes the work of the animators who actually do all the hard stuff. As an animator friend of mine said, ‘Without the VFX guys, he’s just a British guy in a leotard pretending to be a magical creature.’
Animator Ferdinand Engländer explains the complex skilled VFX labor needed to shape the digital Cesar, which incorporated algorithms for simulating fur, musculature and flesh.
… You need a team of programmers, riggers, technical directors and probably even biologists and anthropologists to create an ape rig that behaves absolutely realistic and believable and translates the mo-cap input accurately. If there is no animator to correct mistakes, the skinning must be flawless, the muscles must behave super realistically and whoever defines what the actor’s facial expression translates to on the digital model must deliver a spot on job with countless combinations in mind. …There is still someone required to make decisions about how a human smile looks on an ape puppet.
Engländer refers to the “mistakes” made by algorithms simulating skin and muscles, corrected by animators. As Engländer points out, transferring human expressions to creatures is subtle, skilled, creative work. Why does authorship matter? Authorship determines who is eligible for an Oscar, and directly impacts a film’s bottom line. Anxiety over ownership and authorship are threads running through industry discussions—VFX workers describe the struggle to practice their craft as managers push for ways to automate production, while some actors express fears that entirely virtual “synthetic thespians” are on the horizon.
Conclusions
VFX workers protest working conditions in the entertainment industry outside of the 2014 Oscars in Hollywood, California.
Andy Serkis’ performances demonstrate that as mo-cap technologies enter the studio, VFX work and the physical movements of the actor are shifted to the digital body of the virtual character, a performance space that connects actors and VFX workers. The human body is recast as a data generator, which VFX workers manipulate to suit filmmakers’ goals. Curtin and Sanson refer to the “respatialization of production,” which captures mo-cap’s disruption of craft practices and traditional ideas of performance. Serkis asserts his own visibility as a performer, though only his digital selves are seen onscreen. Film studios and the actor himself promote his work through interviews, behind-the-scenes videos, production trailers, etc. VFX workers have remained largely invisible to audiences, even as the elaborate visual effects they create are used to market their films. Questions of creative authorship and performance are complex and subjective, and will continue to be debated as technologies like mo-cap blur workplace boundaries by increasing complexity and facilitating increased automation through simulation. Foregrounding the labor and economic aspects of production work (as production studies scholars including Caldwell have begun to do), can help clarify these debates as mo-cap and other virtualization technologies increasingly forge out-of-body workspaces that merge physical and digital bodies.
References
Caldwell, John Thornton. Production Culture: Industrial Reflexivity and Critical Practice in Film and Television (Durham: Duke University Press, 2008).
