6-DoF VR Video: Towards Immersive 360-degree VR Video with Motion Parallax

Lead Research Organisation: University of Bath
Department Name: Computer Science

Abstract

The goal of my Innovation Fellowship is to create a new form of immersive 360-degree VR video. We are massive consumers of visual information, and as new forms of visual media and immersive technologies are emerging, I want to work towards my vision of making people feel truly immersed in this new form of video content. Imagine, for instance, what it would be like to experience the International Space Station as if you were there - without leaving the comfort of your own home.

The Problem:
To feel truly immersed in virtual reality, one needs to be able to freely look around within a virtual environment and see it from the viewpoints of one's own eyes. Immersion requires 'freedom of motion' in six degrees-of-freedom ('6-DoF'), so that viewers see the correct views of an environment. As viewers move their heads, the objects they see should move relative to each other, with different speeds depending on their distance to the viewer. This is called motion parallax.
Viewers need to perceive correct motion parallax regardless of where they are (3 DoF) and where they are looking (+3 DoF). Currently, only computer-generated imagery (CGI) fully supports 6-DoF content with motion parallax, but it remains extremely challenging to match the visual realism of the real world with computer graphics models. Viewers therefore either lose photorealism (with CGI) or immersion (with existing VR video). To date, it is not possible to capture or view high-quality 6-DoF VR video of the real world.

My Goal:
Virtual reality is a new kind of medium that requires new ways to author content. My goal is therefore to create a new form of immersive 360-degree VR video that overcomes the limitations of existing 360-degree VR video. This new form of VR content - 6-DoF VR video - will achieve unparalleled realism and immersion by providing freedom of head motion and motion parallax, which is a vital depth cue for the human visual system and entirely missing from existing 360-degree VR video.
Specifically, the aim of this Fellowship is to accurately and comprehensively capture real-world environments, including visual dynamics such as people and moving animals or plants, and to reproduce the captured environments and their dynamics in VR with photographic realism, correct motion parallax and overall depth perception. 6-DoF VR video is a significant virtual reality capability that will be a significant step forward for overall immersion, realism and quality of experience.

My Approach:
To achieve 6-DoF VR video that enables photorealistic exploration of dynamic real environments in 360-degree virtual reality, my group and I will develop novel video-based capture, 3D reconstruction and rendering techniques. We first explore different approaches for capturing static and dynamic 360-degree environments, which are more challenging, including using 360 cameras and multi-camera rigs. We next reconstruct the 3D geometry of the environments from the captured imagery by extending multi-view geometry/photogrammetry techniques to handle dynamic 360-degree environments. Extending image-based rendering to 360-degree environments will enable 6-DoF motion within a photorealistic 360-degree environment with high visual fidelity, and will result in detailed 360-degree environments covering all possible viewing directions. We first target 6-DoF 360-degree VR photographs (i.e. static scenes) and then extend our approach to 6-DoF VR videos.

Project partners:
This Fellowship is supported by the following project partners in the UK and abroad: Foundry (London) is a leading developer of visual effects software for film, video and VR post-production, and ideally suited to advise on industrial impact. REWIND (St Albans) is a leading cutting-edge creative VR production company that is keen to experiment with 6-DoF VR video. Reality7 (Hamburg, Germany) is a start-up working on cinematic VR video.

Planned Impact

Immersive technologies in all their forms, including 360-degree video, VR and AR, are a major opportunity for the UK's world-leading creative industries. This is an industry worth over £84bn and employing 2.8 million people. Over $4bn investment in immersive technologies is predicted, with global market size for immersive technologies reaching $150bn by 2020.

Our project partners, are indicative of the creative pipelines within this industry and highlight how different organisation types might benefit from this research. For example, we anticipate our research to inform hardware designers working to create the next generation of 360-degree capture equipment (such as Reality7); software providers for capture editing (Foundry) and commercial end users (REWIND). This encompasses content creation for VR, games, film and broadcast (Foundry, REWIND).

The ability to capture and reproduce a wide range of real-world environments with high visual fidelity is a significant capability that finds applications in many research disciplines such as education, visualisation, anthropology, cultural heritage.
Furthermore, technologies for the creation of new virtual environments will have widespread impact beyond the creative sector, for example in education, medicine, simulation and training. This is an expanding area of UK productivity which relies upon new tools and technologies to drive competitiveness.

All project partners will engage in intense periods of collaboration. Close working will disseminate not only research results, but best practice and novel techniques across several industries. As a result, my team will be trained into researchers with a strong understanding of industry needs and working practices, providing them with a strong foundation for their future careers.

Publications

10 25 50
 
Description We have developed a new method for casually capturing 360° panoramas with a single camera sweep to share and relive experiences with immersive motion parallax rendering. This provides a more realistic perception of panoramic environments which is particularly useful for virtual reality applications. Our approach is the first to enable casual consumers to capture and view high-quality 360° panoramas with motion parallax.
Exploitation Route Our findings provide a strong foundation for our research into 6-DoF VR video techniques, which are the focus towards the second half of this Fellowship. The findings could also be productised by (mobile) camera manufacturers and hence find their way into consumer smartphones.
Sectors Creative Economy,Digital/Communication/Information Technologies (including Software),Culture, Heritage, Museums and Collections

URL http://richardt.name/megaparallax
 
Description BRLSI Talk on Towards more Immersive Panoramas 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact I gave a talk at the Bath Royal Literary and Scientific Institution (BRLSI) on 11 September 2019 on the topic "Towards more Immersive Panoramas". The talk was a ticketed public event and presented some of the research carried out in my team to a lay audience.
Year(s) Of Engagement Activity 2019
URL https://www.brlsi.org/events-proceedings/events/psc-visual-arts-towards-more-immersive-panoramas
 
Description Dagstuhl workshop on Real VR (July 2019) 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact The Dagstuhl seminar brought together researchers and practitioners from academia and industry to discuss the state-of-the-art, current challenges, as well as promising future research directions in Real VR. Real VR, as defined by the seminar participants, pursues two overarching goals: facilitating the import of real-world scenes into head-mounted displays (HMDs), and attaining perceptual realism in HMDs. The vision of Real VR is enabling to experience movies, concerts, even live sports events in HMDs with the sense of immersion of really "being-there", unattainable by today's technologies.

The workshop has led to a book that is currently in press:
Real VR - Immersive Digital Reality
How to Import the Real World into Head-Mounted Immersive Displays
Editors: Magnor, Marcus, Sorkine-Hornung, Alexander (Eds.)
https://www.springer.com/gp/book/9783030418151
Year(s) Of Engagement Activity 2019
URL https://www.dagstuhl.de/en/program/calendar/semhp/?semnr=19272
 
Description Invited talk at IVRPA 360° Virtual Reality Panoramic Photography Conference 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact The annual conference of the International Virtual Reality Photography Association (IVRPA) features four days of talks, workshops, panels and exhibitors, to learn from and meet the key players at the forefront of the 360º VR industry, including photographers, software developers, camera hardware manufacturers and media producers and companies. I was invited to present our latest research to this audience of professional practitioner and give them a glimpse of some of the upcoming 360 photography technologies.
Year(s) Of Engagement Activity 2019
URL https://ivrpa.org/news/megaparallax-towards-6-dof-360-panoramas/
 
Description Organised a course at SIGGRAPH 2019 on Capture for VR 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Dr Christian Richardt organised a course at the leading computer graphics conference that provides a comprehensive overview of the latest progress in bringing photographs and video into VR. In this half-day course, we took the audience on a journey from VR photography to VR video that began more than a century ago but which has accelerated tremendously in the last five years. We discussed both commercial state-of-the-art systems by Facebook, Google and Microsoft, as well as the latest research techniques and prototypes.
Year(s) Of Engagement Activity 2019
URL https://richardt.name/publications/capture4vr/
 
Description Pint of Science talk: VR Photography from the Victorians to today 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact Christian Richardt was invited to give a talk at "Pint of Science". This is a locally run festival that invites researchers to give talks in a very informal atmosphere, direct to the public in a pub. The casual atmosphere promotes discussion and the opportunity for inclusion of members of the public who may not normally engage with complicated scientific research.
Year(s) Of Engagement Activity 2019
 
Description Press release: "AI could make dodgy lip sync dubbing a thing of the past" 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Media (as a channel to the public)
Results and Impact Press release: "AI could make dodgy lip sync dubbing a thing of the past" on 17 August 2018

"You thought fake news was bad? Deep fakes are where truth goes to die", The Guardian https://www.theguardian.com/technology/2018/nov/12/deep-fakes-fake-news-truth
"Can you tell a fake video from a real one?", ABC News (Australia) https://www.abc.net.au/news/2018-09-27/fake-news-part-one/10308638
"The New AI Tech Turning Heads in Video Manipulation", SingularityHub https://singularityhub.com/2018/09/03/the-new-ai-tech-turning-heads-in-video-manipulation-2/
"Deepfake Videos Are Ge?ng Impossibly Good", Gizmodo https://gizmodo.com/deepfake-videos-are-getting-impossibly-good-1826759848
"This new face-swapping AI is scarily realistic", Fast Company https://www.fastcompany.com/90175648/this-new-face-swapping-deep-fakes-ai-is-scarily-realistic
"AI can transfer human facial movements from one video to another" Engadget, https://www.engadget.com/2018/06/05/ai-transfer-facial-movements-from-one-video-to-another/
"Forget DeepFakes, Deep Video Portraits are way better (and worse)", TechCrunch https://techcrunch.com/2018/06/04/forget-deepfakes-deep-video-portraits-are-way-better-and-worse/
Year(s) Of Engagement Activity 2018
URL https://www.bath.ac.uk/announcements/ai-could-make-dodgy-lip-sync-dubbing-a-thing-of-the-past/