Video-based animation of people

Lead Research Organisation: University of Surrey
Department Name: Vision Speech and Signal Proc CVSSP

Abstract

The goal of this research project is to solve the fundamental problem of re-using multiple-view video capture of people to support interactive animation with the quality of the source video. Research will investigate the resampling of a multiple-view video database for interactive animation of people to enable user control of movement and viewpoint with video-quality rendering. The proposed research will address the underlying problem of representation of articulated and highly dynamic non-rigid structures in video to allow indexing, reuse and manipulation whilst maintaining the visual quality. Ultimately the challenge is to enable video-quality rending of real people by reuse of captured video sequences allowing user control of movement as in conventional animation. Recent research has demonstrated photo-realistic animation of faces and simple objects by resampling video sequences.This proposal aims to take video-based animation to complex articulated objects such as people without full 3D reconstruction which has been shown to result in loss of visual quality in previous work. The new challenge is to synthesise video sequences of novel movements from captured video of different motion. Animating people from video requires several key advances, including: a representation of human posture and movement which supports the generation of sequences of previously unseen movements; a representation of multiple-view video of articulated objects allowing efficient storage and indexing; algorithms to efficiently selecting the most useful examples for the synthesis of a new sequences from a large set of example sequences; and new video synthesis algorithms for articulated objects in previously unseen configurations. Animation of people directly from captured video has the potential to provide enabling technology for next-generation, video-quality content production in film, television and games. Video-based animation of people will allow powerful re-use and manipulation of captured video (e.g., generating novel body movements for an actor), together with seamless compositing within photo realistic scenes.

Publications

10 25 50

publication icon
Huang P (2010) Shape Similarity for 3D Video Sequences of People in International Journal of Computer Vision

publication icon
Starck J (2009) The Multiple-Camera 3-D Production Studio in IEEE Transactions on Circuits and Systems for Video Technology

publication icon
Starck J (2007) Surface capture for performance-based animation. in IEEE computer graphics and applications

 
Description Key findings:

- the first video-based creation of photo-realistic animated content from actor performance
- a new approach to animated content production based on a new surface motion graph representation
- high-accuracy reconstruction of dynamic surface shape of actors from multiple view video
- photo-realistic free-viewpoint rendering of actor performance
- a comprehensive evaluation of methods for measuring shape-similarity of people across different poses and the introduction of new validated methods
- a framework for photo-realistic animated content production and rendering
- fundamental advance demonstrating the principle of reuse of captured dynamic shape and appearance to produce animation by concatenation
Exploitation Route Findings have been taken forward in:
(1) the development of a production pipeline for video-realistic animated content production
(2) interactive real-time animation tools developed in the EU Project RE@CT led by the BBC
(3) exploitation of the technology by SME Artifacto to develop and demonstrate animated content for educational museum applications and cultural heritage
Sectors Creative Economy

Digital/Communication/Information Technologies (including Software)

URL http://cvssp.org/Personal/AdrianHilton
 
Description Video-based animation technology has underpinned the development of systems for free-viewpoint video and video quality animation content production used in the creative industries (film, TV, games, web).
First Year Of Impact 2006
Sector Creative Economy,Education,Culture, Heritage, Museums and Collections
Impact Types Cultural

Economic

 
Description BBC Research and Development 
Organisation British Broadcasting Corporation (BBC)
Country United Kingdom 
Sector Public 
PI Contribution Research in Computer Vision for broadcast production and Audio. Technologies for 3D production, free-view point video in sports, stereo production from monocular cameras, video annotation Member of the BBC Audio Research Partnership - developing the next generation of broadcast technology.
Collaborator Contribution In kind contribution (members of Steering/Advisory Boards) Use of the BBC lab and research/development facilities. Studentships (industrial case) funding and co-supervision of PhD students.
Impact Multi-disciplinary collaboration involves Computer Vision, Video Analysis, Psychoacoustics, Signal Processing and Spatial Audio