Dynamically Accurate Avatars

Our bodies move as we speak. Evidently, movement of the jaw, lips and tongue is required to produce coherent speech. Furthermore, additional body gestures both synchronise with the voice and significantly contribute to speech comprehension. For example, a person's eyebrows raise when they are stressing a point, their head shakes when they disagree and a shrug might express doubt.

The goal is to build a computational model that learns the relationship between speech and upper body motion so that we can automatically predict face and body posture for any given audio speech. The predicted body pose can be transferred to computer graphics characters, or avatars, to automatically create character animation directly from speech, on the fly.

A number of approaches have previously been used for mapping from audio to facial motion or head motion, but the limited amount of speech and body motion data that is available has hindered progress. Our research programme will use a field of machine learning called transfer learning to overcome this limitation.

Our research will be used to automatically and realistically animate the face and upper body of a graphics character along with a user's voice in real time. This is valuable for a) controlling the body motion of avatars in multiplayer online gaming, b) driving a user's digital presence in virtual reality (VR) scenarios, and c) automating character animation in television and film production. The work will enhance the realism of avatars during live interaction between users in computer games and social VR without the need for full body tracking. Additionally, we will significantly reduce the time required to produce character animation by removing the need for expensive and time-consuming hand-animation or motion capture.

We will develop novel artificial intelligence approaches to build a robust speech-to-body motion model. For this, we will design and collect a video and motion capture dataset of people speaking, and this will be made publicly available.

The project team is comprised of Dr. Taylor and a PDRA at the University of East Anglia, Norwich, UK.

Economic Impact
The global games audience is estimated at around 2.4 billion people and the global market is expected to grow to an estimated $129 billion by the end of 2020. The UK consumer spend on games was valued at £4.33bn in 2016 with a record of £1.2bn coming from online games sales, so the opportunities for the UK online games industry have never been greater. At present, a player can speak with other players during live gameplay, yet their avatar does not move in sync with their speech. Our software will add significant value to the games industry since it will address this challenge and yield a more compelling gaming experience.

A further £61 million of the UK consumer spend on video games came from the sale of virtual reality (VR) hardware. VR is a fast-developing sector of the creative digital industries, and our technology will allow a user's digital presence to move in sync with their voice without the need for intrusive and expensive full body tracking.

Our methods will significantly reduce the time required to produce character animation by removing the need for expensive and time-consuming hand-animation or motion capture. Consequently, the cost of production will reduce accordingly.

Scientific Impact
Our project introduces a ground-breaking technique for creating highly realistic character animation automatically for any speech. Thus, we envisage computer graphics researchers to shift focus towards improving the fidelity of real-time rendered graphics characters, which will consequently expedite the advancement of human realism in computer graphics.

We expect the academic impact of our work in the field of psychology to be considerable. Our technology allows for the dynamics or the appearance of an animated character to be manipulated in precise ways and, for example, would allow psychologists to conduct experiments on dissociating human behaviour and appearance.

Societal Impact
Our technology will bring an equivalent level of realism to potentially every animated production and every computer game, and it will be available for all game content and not just for cut-scenes. Furthermore, it will be possible to generate character animation dynamically and in response to actions by the player. This will be a significant step forward for an industry that strives for ever more realistic content, and crucially will provide children with characters that are consistently animated with realistic face and body behaviours at a time when their own speech is developing.

The proposed research can also be used in social interaction training tools for people with autism spectrum disorder (ASD), who can use the technology for practising conversations and for learning how to interpret human emotions. This research has potential to positively impact the lives of the 700,000 people in the UK and 3.5 million in the USA alone who have been diagnosed with ASD. We will ensure that colleagues in the relevant faculties and institutions are kept informed of the research, and we will work with them to develop applications through future bids to the research councils

Outreach and Engagement
Dr. Taylor will continue to deliver lectures as part of outreach events at local schools and, since our work will have influenced the content of the computer games that these students play, she will be able to demonstrate that cutting edge computing science research at UEA has practical use. It will help students to understand the way that characters in animated shows are brought to life, and inspire them to get involved with science.

We will interactively demo the work at the Norwich Science Festival and create character animations using voices recorded from members of the public. The video, augmented with the UEA logo, will be emailed to them and they will be encouraged to share it on social media, broadening public awareness of the university and of the research.