Performance based expressive virtual characters

Creating believable, expressive interactive characters is one of the great, and largely unsolved, technical challenges of interactive media. Human-like characters appear throughout interactive media, virtual worlds and games and are vital to the social and narrative aspects of these media, but they rarely have the psychological depth of expression found in other media. This proposal is for the development of research into a new approach to creating interactive characters which identifies the central problem of current methods as being the fact that creating the interactive behaviour, or Artificial Intelligence (AI), of a character is still primarily a programming task, and therefore in the hands of people with a technical rather than an artistic training. Our hypothesis is that the actors' artistic understanding of human behaviour will bring an individuality, subtlety and nuance to the character that it would be difficult to create in hand authored models. This will help interactive media represent more nuanced social interaction, thus broadening their range of application. The proposed research will use information from an actor's performance to determine the parameters of a character's behaviour software. We will use Motion Capture to record an actor interacting with another person. The recorded data will be used as input to a machine learning algorithm that will infer the parameters of a behavioural control model for the character. This model will then be used to control a real time animated character in interaction with a person. The interaction will be a full body interaction involving motion tracking of posture and/or gestures, and voice input.In entertainment this method will enable more social genres and help improve the current limited demographic. It will also enable a number of new applications in education, rehabilitation, media and marketing. Putting actors in charge of creating character AI will also make production pipelines more efficient be requiring less input from programmers This project is timely in that it brings together a number of active and developing research fields including expressive virtual characters, motion capture based animation and machine learning. It has the potential to transform current research in expressive virtual characters and present new research problems for machine learning and motion capture based animation. It is novel in that it proposes a fundamentally new approach to creating interactive characters and it combines disciplines such as animation, statistical machine learning, performance, affective computing, human computer interaction and psychology. The use of both machine learning and performance for virtual characters is particularly novel.

Who will benefit? The primary beneficiaries outside of academia will be the creators of 3D interactive media, particularly computer games. This will also benefit the users of similar technology in specific areas such as education, psychotherapy and the arts. Academics in areas such as computer animation and virtual reality are also likely to benefit. How will they benefit? There are two major benefits of the proposed method: 1. Integrating artists more directly into the content pipeline for character AI, thus improving the efficiency of the production process 2. Creating more expressive and subtle behaviour of characters, thus increasing the range of genres and markets available to interactive media. What will be done to ensure they benefit? Our strategy will be 3 fold: 1. knowledge dissemination: primarily through academic journals and conferences, but also potentially directly through industry through training programmes (Gillies has participated in training schemes for Electronic Arts). 2. Further development: as the proposed research is still new, it is unlikely that the results will be ready for commercial exploitation by the end of this project. Therefore, if the results are positive, further funding will be sought to develop the proposed method in an academic research context 3. Direct exploitation: in the longer term, when the method is sufficiently developed, commercial exploitation will be sought. The most likely route will be through a partnership with an existing company working in games or interactive media via knowledge transfer or commercialization funding. Intermediary software may also be released to the community if this does not conflict with commercial exploitation.