i3Dlive: interactive 3D methods for live-action media

Lead Research Organisation: University of Surrey
Department Name: Vision Speech and Signal Proc CVSSP

Abstract

i3Dlive will create methods for enhancing viewer experience of live action in sports broadcast and film production by augmenting conventional 2D production with 3D information. Research will contribute new capabilities in post-production to enable 3D stereo viewing without specialist stereo camera rigs and allow modification of appearance in video for relighting and material editing. University of Surrey research will focus on two challenging open problems:- Extraction of a 2.5D layered depth representation from video for complex dynamic scenes to support stereo viewing such as live sports events.- Perceptually realistic manipulation of scene appearance in video based on local surface orientation and material transfer.Research will focus on live action scenes such as sports events in broadcast and onset production in films. Advances in these areas will contribute to both scientific understanding of complex dynamic scenes and enable technologies to allow 3D stereo viewing and change of appearance in video post-production. The Foundry, an academy award winning post-production tools developer, will lead the Technology Strategy Board collaborative project with BBC Research providing expertise in production technologies together with access to end-users in broadcast for sports, children's and natural history programme making. Research advances will be integrated into prototype post-production tools by The Foundry for industry user evaluation in film and broadcast to inform research and commercialisation.
 
Description - Novel methodology allowing the creation of stereo content for a principal film or broadcast camera without the requirement for a specialist stereo camera rig
- Robust methods for calibration of a moving and zooming principal camera with respect to a set of satellite cameras
- Methods for general scene reconstruction from a set of satellite cameras
- The approach has allows complete creative control of the stereo reproduction (camera separation etc.) in post-production unlike a stereo camera rig where these parameters are fixed at the point of acquisition
- Demonstration in real-production settings including sports and broadcast studio production
Exploitation Route - Commercialisation of stereo production algorithms as tools for the film production industry by industry leader The Foundry
- Use of tools and technologies for stereo production in film and broadcast (BBC)

Subsequent projects using the methods for scene reconstruction form the basis of EU projects SCENE (http://3d-scene.eu) developing a complete toolset for 3D production in film.
Sectors Creative Economy,Digital/Communication/Information Technologies (including Software),Culture, Heritage, Museums and Collections
URL http://www.bbc.co.uk/rd/projects/
 
Description Collaborative project with The Foundry (the foundry.com) and BBC Research focused on the development of technologies for stereo content production. Research at the University of Surrey developed methods for stereo content production without the requirement for a stereo camera rig. The approach used satellite witness cameras to reconstruct the scene allowing the creation of stereo content from the perspective of the principal film or broadcast camera. This approach avoids the requirement for specialist stereo capture technology which is cumbersome and expensive. In additional methods were introduced for robust through-the-lens calibration of the moving principal camera with respect to the witness cameras. The approach was exploited by The Foundry to develop a toolset for the industry leading product NUKE to allow the creation and manipulation of stereo content in film post-production. The approach was also used by the BBC in sports coverage (football/rugby/skiing) to produce stereo content from the principal camera view.
First Year Of Impact 2008
Sector Creative Economy,Digital/Communication/Information Technologies (including Software),Culture, Heritage, Museums and Collections
Impact Types Cultural,Societal,Economic
 
Description Avatarme Ltd 
Organisation Avatarme
Country United Kingdom 
Sector Private 
Start Year 2004
 
Description BBC Research and Development 
Organisation British Broadcasting Corporation (BBC)
Country United Kingdom 
Sector Public 
PI Contribution Research in Computer Vision for broadcast production and Audio. Technologies for 3D production, free-view point video in sports, stereo production from monocular cameras, video annotation Member of the BBC Audio Research Partnership - developing the next generation of broadcast technology.
Collaborator Contribution In kind contribution (members of Steering/Advisory Boards) Use of the BBC lab and research/development facilities. Studentships (industrial case) funding and co-supervision of PhD students.
Impact Multi-disciplinary collaboration involves Computer Vision, Video Analysis, Psychoacoustics, Signal Processing and Spatial Audio
 
Description Sony Broadcast and Professional Europe 
Organisation SONY
Department Sony Broadcast and Professional Europe
Country United Kingdom 
Sector Private 
Start Year 2004