6-DoF VR Video: Towards Immersive 360-degree VR Video with Motion Parallax

The goal of my Innovation Fellowship is to create a new form of immersive 360-degree VR video. We are massive consumers of visual information, and as new forms of visual media and immersive technologies are emerging, I want to work towards my vision of making people feel truly immersed in this new form of video content. Imagine, for instance, what it would be like to experience the International Space Station as if you were there - without leaving the comfort of your own home.

The Problem:
To feel truly immersed in virtual reality, one needs to be able to freely look around within a virtual environment and see it from the viewpoints of one's own eyes. Immersion requires 'freedom of motion' in six degrees-of-freedom ('6-DoF'), so that viewers see the correct views of an environment. As viewers move their heads, the objects they see should move relative to each other, with different speeds depending on their distance to the viewer. This is called motion parallax.
Viewers need to perceive correct motion parallax regardless of where they are (3 DoF) and where they are looking (+3 DoF). Currently, only computer-generated imagery (CGI) fully supports 6-DoF content with motion parallax, but it remains extremely challenging to match the visual realism of the real world with computer graphics models. Viewers therefore either lose photorealism (with CGI) or immersion (with existing VR video). To date, it is not possible to capture or view high-quality 6-DoF VR video of the real world.

My Goal:
Virtual reality is a new kind of medium that requires new ways to author content. My goal is therefore to create a new form of immersive 360-degree VR video that overcomes the limitations of existing 360-degree VR video. This new form of VR content - 6-DoF VR video - will achieve unparalleled realism and immersion by providing freedom of head motion and motion parallax, which is a vital depth cue for the human visual system and entirely missing from existing 360-degree VR video.
Specifically, the aim of this Fellowship is to accurately and comprehensively capture real-world environments, including visual dynamics such as people and moving animals or plants, and to reproduce the captured environments and their dynamics in VR with photographic realism, correct motion parallax and overall depth perception. 6-DoF VR video is a significant virtual reality capability that will be a significant step forward for overall immersion, realism and quality of experience.

My Approach:
To achieve 6-DoF VR video that enables photorealistic exploration of dynamic real environments in 360-degree virtual reality, my group and I will develop novel video-based capture, 3D reconstruction and rendering techniques. We first explore different approaches for capturing static and dynamic 360-degree environments, which are more challenging, including using 360 cameras and multi-camera rigs. We next reconstruct the 3D geometry of the environments from the captured imagery by extending multi-view geometry/photogrammetry techniques to handle dynamic 360-degree environments. Extending image-based rendering to 360-degree environments will enable 6-DoF motion within a photorealistic 360-degree environment with high visual fidelity, and will result in detailed 360-degree environments covering all possible viewing directions. We first target 6-DoF 360-degree VR photographs (i.e. static scenes) and then extend our approach to 6-DoF VR videos.

Project partners:
This Fellowship is supported by the following project partners in the UK and abroad: Foundry (London) is a leading developer of visual effects software for film, video and VR post-production, and ideally suited to advise on industrial impact. REWIND (St Albans) is a leading cutting-edge creative VR production company that is keen to experiment with 6-DoF VR video. Reality7 (Hamburg, Germany) is a start-up working on cinematic VR video.

Immersive technologies in all their forms, including 360-degree video, VR and AR, are a major opportunity for the UK's world-leading creative industries. This is an industry worth over £84bn and employing 2.8 million people. Over $4bn investment in immersive technologies is predicted, with global market size for immersive technologies reaching $150bn by 2020.

Our project partners, are indicative of the creative pipelines within this industry and highlight how different organisation types might benefit from this research. For example, we anticipate our research to inform hardware designers working to create the next generation of 360-degree capture equipment (such as Reality7); software providers for capture editing (Foundry) and commercial end users (REWIND). This encompasses content creation for VR, games, film and broadcast (Foundry, REWIND).

The ability to capture and reproduce a wide range of real-world environments with high visual fidelity is a significant capability that finds applications in many research disciplines such as education, visualisation, anthropology, cultural heritage.
Furthermore, technologies for the creation of new virtual environments will have widespread impact beyond the creative sector, for example in education, medicine, simulation and training. This is an expanding area of UK productivity which relies upon new tools and technologies to drive competitiveness.

All project partners will engage in intense periods of collaboration. Close working will disseminate not only research results, but best practice and novel techniques across several industries. As a result, my team will be trained into researchers with a strong understanding of industry needs and working practices, providing them with a strong foundation for their future careers.