The rise of hyper-realistic virtual characters in AAA games, sci-fi blockbusters, and animation has left audiences amazed. One key technology behind this evolution is motion capture, which breathes “life” into virtual figures.
Take the domestic AAA game “Black Myth: Wukong,” for instance. How do developers create the realistic expressions and movements of characters like “Heaven’s Mandate”? The answer lies in advanced motion capture technology, which is increasingly employed to produce lifelike performances in films, animations, and games alike. Iconic productions such as “Avatar,” “Spider-Man,” “The Avengers,” and “Detective Chinatown,” as well as popular games like “Peacekeeper Elite” and “Genshin Impact,” have successfully utilized this technique. By capturing the physical actions and expressions of actors and leveraging algorithms, developers create dynamic and captivating virtual characters that come alive on screen.
Motion capture has its roots in the animation industry, having been utilized since the 1970s. Its appeal comes from the efficiency and visual quality it offers compared to traditional hand-drawn and 3D animation methods. Recently, domestic companies have rapidly advanced in this field, enhancing the quality of Chinese-produced works with robust technological support. But how is motion captured, and how are domestic enterprises catching up? We sought answers through interviews.
With precision reaching sub-millimeter levels, the motion capture process has sophisticated requirements. In a 60-square-meter room, high-definition cameras are strategically mounted to capture motion. A young tester dressed in a black motion capture suit—complete with a hood, gloves, and 50 optical markers—stands at the center, ready for action.
“We call these markers ‘North Stars,'” explained Qu Nianchen, a motion capture engineer at the computer. “This means that no matter how fast someone moves while wearing them, our self-developed cameras can accurately track the reflection of the North Stars, recording their movement path and creating data for character performances or object motions.”
As the technician assumes a T-pose on command, Qu adeptly checks the 3D reconstruction of their model in motion capture software. This is taking place at YuanKe Vision’s motion capture lab, where they’re testing the FZMotion optical motion capture system.
Leading companies in the industry have achieved impressive sub-millimeter accuracy. This year, the globally popular game “Black Myth: Wukong” implemented the OptiTrack optical motion capture system provided by Leadr Virtual Motion, which can accurately record an actor’s expressions, body posture, and action details with an error margin of less than ±0.1 millimeters and a response time of a mere 2.8 milliseconds.
Cui Chao, the development director at Virtual Motion, noted, “The high precision we strive for in ‘Black Myth: Wukong’ ensures that character expressions are nuanced, movements flow seamlessly, and attacks have impactful tension.” This level of detail aligns with film industry standards, and while combat sequences in games involve looping actions triggered by players, the concept remains the same: high fidelity in motion capture.
In live-action films, action scenes often involve multiple performers, presenting unique challenges during production. The Perception Neuron series of inertial motion capture suits developed by Noiteng excels in these situations. This inertial method requires light, easy-to-wear sensors, making it less precise than optical capture but far more adaptable to varied environments.
Noiteng’s co-founder and CTO, Dai Ruoli, shared insights from a project in Hollywood, revealing, “Every time I heard the director say ‘3-2-1, go,’ my heart raced. The seamless integration of special effects depends on over a dozen technologies working in harmony, and if a company is ever out of sync, it could ruin everything.” To mitigate this risk, Noiteng has invested seriously in refining their motion capture technology to be plug-and-play even after actors have been off stage for an extended period.
“When faced with high-pressure, complex scenarios, these cases compel us to address technical challenges and improve product quality and reliability,” Dai said.
The challenges of motion capture extend to animation productions as well. Chen Pu, solutions director at YuanKe Vision, recounted his experience during the production of the domestic animated feature “The Bounty Hunter” in 2022. They encountered issues such as characters’ hands passing through weapons and the frequent occlusion of optical markers during intense scenes with multiple actors. Achieving their goal of “capturing every frame” demanded tedious data reviews, adjustments, and a pioneering melding of human and rigid body models until they found a resolution.
When asked about his proudest work, Chen stated, “What I find most satisfying today might just be a stepping stone toward an even better achievement tomorrow!” With each technological iteration, motion capture continues to have vast potential for enhancement in both precision and visual aesthetic.
Expanding applications of motion capture also enrich the film production process. It’s not just a tool for on-set shooting; it is revolutionizing how films are made through “virtual previews,” which boost filming efficiency.
In 2021, the sci-fi thriller “Assassination of a Novelist” became a pioneer of this approach, utilizing the OptiTrack system. Cui shared that before filming began, his team worked closely with director Lu Yang and the stunt team over two months to rehearse and capture all dynamic shots, allowing an early preview of the film’s special effects.
An interesting challenge arose during virtual previews when it came time to capture the movement of the ultimate “boss” character with four arms, which are two more than a human possesses. After discussions, the team decided to capture two actors at once: one would perform all body movements, while the other would depict the imaginary arms’ actions—“it was almost like a synchronized performance.”
Similarly, the 2023 Spring Festival Gala segment “When Mythical Beasts Meet” also utilized this technology, with YuanKe Vision providing motion capture support. In this project, performers outfitted with specially designed tails and wings had to portray mythical creatures such as phoenixes and qilins. The team also dressed real animals, like cats and dogs, in custom motion capture suits and affixed tiny optical markers to birds, using a combination of inertial and optical capture techniques for later post-production.
Actually, in virtual content, non-human characters and quadrupeds act as transitional roles between upright humans and fantastic beings. To express narrative imagination and amplify visual impact, quadruped motion capture technology has become a focal point in the industry. As Cui noted, one of the major challenges is that animal behaviors cannot be controlled, making their movements unpredictable. Consequently, motion capture systems must be capable of broad, high-precision captures and innovative anti-occlusion algorithms. “Currently, there are fewer than three companies globally that can effectively capture quadruped motion, and Virtual Motion accounts for one-third of the active output; our effective animal data requires little post-processing,” he remarked.
With the booming development of AI and the metaverse, Chinese motion capture companies are accelerating their contributions to the creation of digital content, integrating digital humans and virtual reality technologies to inject new vitality into various industries.
In 2022, YuanKe Vision helped launch an AI sign language interpretation digital human system, which has since provided service for major events like the Beijing Winter Olympic Games and FIFA World Cup commentary, boasting an accuracy rate of 90%. This innovative solution resolved the challenges of traditional manual sign language interpretation, making international sports events more accessible for individuals with hearing impairments.
Qu explained that over a span of three months, the sign language digital human learned nearly 100,000 sign language phrases, encompassing the national standard sign language dictionary and thousands of sport-specific vocabulary, as well as the necessary facial expressions and body language.
Looking for breakthroughs in development, professionals in the motion capture industry face common challenges, particularly the current inability to capture facial expressions alongside body movements simultaneously. This limitation arises from the subtlety of facial changes, which requires actors to apply facial markers and wear cameras for separate recordings, constraining bodily performances. The existing method of recording body and facial expressions separately and then fine-tuning them adds complexity and extends production timelines. Chen expressed hope for advancements in multi-modal capturing capabilities for motion capture cameras, allowing them to utilize infrared to capture facial markers while conducting visual analysis and information extraction in a single performance.
Another recognized challenge involves finger tracking. Addressing the intricacy of hand movements, Noiteng has released the world’s first consumer-grade motion capture gloves and continues to enhance their accuracy and resistance to interference. “We must leave no technological gaps when it comes to finger tracking,” Dai asserted.
Furthermore, Virtual Motion remains steadfast in its pursuit of precision, planning continued improvements in camera resolution and algorithmic compensation to minimize errors from micro-shake and reduce the manual post-editing workload.
However, the pursuit of perfection doesn’t solve every issue. Developers have realized that different fields have unique demands for motion capture that involve both commonalities and variations. Fields such as sports science, biomedical, virtual simulation, and embodied intelligence push for measurement accuracy and high fidelity. In contrast, audiences of films, animations, and video games crave motions that feel reasonable, comfortable, and aesthetically pleasing. Therefore, there is a necessity to enhance the attractiveness of a performance while ensuring that motion capture maintains fidelity to the original actor’s performance.
Chen openly stated, “Whether a character’s dance moves appear genuine and aesthetically pleasing ultimately rests with the human eye.” Dai added, “We allow virtual characters to flip in ways that may differ from reality or be more exaggerated, but they must still be visually logical.”
Cui concluded by quoting French writer Gustave Flaubert to capture his vision of the ideal motion capture industry: “Art is becoming more technological, and technology is becoming more artistic. These two parted ways at the foothills, but one day they will meet again at the summit.” He looks forward to strengthening collaborations with more filmmakers and creating truly great works together.
The deepening application of motion capture technology reveals a trend: transitioning from precision to natural movement, from reason to aesthetic appeal, and from objective to subjective interpretation. Is this a natural progression of technological development? How do we understand the relationship between motion capture technology and human life?
To this, Dai shared that within the scope of human digitalization, motion capture should be recognized as a bridge between human-machine interactions and the physical and digital realms.
“Only by making technological outcomes more relatable to humans can we preserve existing human-centered design achievements and reinforce human-centric concepts,” Chen emphasized. “Our ultimate goal is to ensure that technology serves and engages with all aspects of human life more effectively.”
