A recent patent for an AI training system could help free Sony animators from some of the constraints of hand-animated motion capture.
News has surfaced that Sony has patented an AI-powered facial performance capture method to ease the workload on animators. It’s no secret that Sony Interactive Entertainment Studios is the industry leader when it comes to capturing real-life performances in the video game medium. With a huge increase in fidelity over the last generation of consoles, in the span of a decade developers like Naughty Dog and Sony Santa Monica have gone to great lengths to build games and characters using hand-drawn facial and motion animation entirely through performance capture. has proceeded with It’s no surprise, then, that Sony has filed a patent for more game-changing performance capture technology.
Filed by Sony Interactive Entertainment LLC on April 1, 2022, the patented system takes mesh data from a human actor captured by a 3D camera to record facial depth. This polygonal mesh is used in game development to create accurate human models and is typically handcrafted by animators using the vertices on a mockup actor’s face as a reference. This method is currently standard in most triple-A videogame development, including Insomniac’s upcoming Spider-Man 2 . The system will automatically generate mesh data, however, using a mix of 3D and 2D cameras to train an artificial intelligence program that can do it accurately. Capture facial expressions and create mesh data with little need for human intervention.
The patent follows an industry trend in the development of AI to perform tasks that were previously performed by human hands. The video game industry has seen a huge increase in the use of AI to do many things, from increasing the resolution of textures to interpolating frames for smoother games. NVIDIA specifically has patented RTX Remix and their deep learning supersampling systems that make remastering classic games a very easy feat for modders and developers.
In fact, this isn’t the first time gamers have seen AI used in the triple-A space. Recently, Rockstar Games came under fire for using AI upscaling in their remaster. GTA TrilogyCauses some unfortunate spelling mistakes on some assets.
As this patent directly states, the presence of AI in this instance helps reduce the workload and skyrocketing costs of facial animation. With hundreds of animators using thousands of composites and shared files to mirror a mockup performance, the process is incredibly time-consuming and expensive. Thus, unlike some artistic AI efforts that attempt to replace actors, the goal is not to completely eliminate the need for an animator or actor, but rather to aid in the capture process. This system, if successfully deployed, can go a long way towards reducing crunch time for animators on particularly demanding games as well as opening the door to freedom of expression.