Speech Animation using Dynamic Visemes

This project will investigate new methods for automatically producing speech animation. For animators in the movie industry this is typically a tedious, iterative process that involves key-framing static lip-poses and then handcrafting a blending function to transition from one key pose to another. In is not uncommon for an animator to spend several hours producing animation for just a few seconds of speech.

We have previously worked on identifying a new dynamic unit for speech animation, termed dynamic visemes, and have shown that these units produce better animation than more traditional phoneme-based units. In this project we will integrate dynamic visemes into state of the art approaches to further improve upon the quality of automated animation that is currently possible. Furthermore, we will investigate how dynamic visemes relate to speech acoustics so that animation can be generated directly from the voice of an actor.

We will build tools that can be implemented in commercial animation pipelines so animation studios can use our tools as a basis for animating any speech on their own models. This will leave their artists free to focus on the overall performance of the character.

The proposed project is ambitious in its aims, proposing a new approaches for producing better speech animation. However, the impact of the work is wide reaching and has the potential to influence the production of speech content in all animated movies and computer games.

During the course of this project we will develop new techniques for producing better quality speech animation than is available using the current state of the art. These techniques will be implemented in easy to use tools that will remove the burden from professional artists in the production of animated speech content. The artist will need only provide the audio assets from the actor and our tools will automatically generate the lip motion synchronised with the spoken words. The artist can then focus on the character performance, e.g. adding the expression and head pose variation to bring the character to life.

The manual effort required to create production-quality animated speech cannot be overstated. It is not unusual for even very skilled artists to spend many hours lip-syncing a character for only a very short scene. We will work with our industrial partner and industrial advisors to develop the research ideas into tools that work with industry standard software so that they can easily fit within animation pipelines currently used by the various studios and VFX houses. These new approaches will allow production-quality animation to be created in real-time, and so will provide significant savings to studios. Furthermore, we have demonstrated that our approach is not dependent on a particular model or rig and so can be applied to all animation, from cartoons to videorealistic characters. This will also mean that high quality speech animation is available to all, and not just the largest studios producing the biggest budget content.

More broadly, this work will impact research areas that involve the analysis of the (visible) speech articulators. For example, our approaches could find application in speech therapy, whereby a patient is shown speech-related exercises on a model of their own face. Our tools could automatically analyse the patients motions and compare them against the expect motion and show errors in their production. This form of assessment might be beneficial to stroke victims, whereby they need to re-train their facial muscles to properly articulate their speech. A face-to-face virtual speech therapist is always at hand to provide useful and immediate feedback, and our analysis tools can be used to log progress and provide progress related information to a real speech therapist.

There has been work showing the effectiveness of computer generated characters as learning aids, and our tools could be developed into a language tutoring system. Speech-related movements can be tracked on the face of a student and the virtual tutor can compare the observed motion with the expected motion of a native speaker. A range of face models, from cartoon-like to video real, will make foreign language learning more 'fun' and hopefully re-engage school children in learning foreign languages.