EPSRC-SFI: SpheryStream

This proposal addresses the important and not well-studied area in the fast-developing engineering space of the virtual reality and immersive technology. Our final vision is to endow any final user with an unprecedented sense of full immersion in virtual environments for any applicative scenario and with any connectivity level. This requires VR systems that operate at scale, in a personalized manner, remaining bandwidth-tolerant whilst meeting quality and latency criteria. This can be accomplished only by a fundamental revolution of the coding/streaming/rendering chain that has to put the interactive user at the heart of the system rather than at the end of the chain.
Specifically, bringing our final vision to reality requires constant innovation in VR streaming technology to overcome the following main challenges: i) the spherical format of the video content in a scenario in which media codecs are optimized for planar content; ii) the interactive behaviour of users that introduces uncertainty on the popularity of the video content. To overcome the former challenges, the spherical video is currently projected on a bidimensional planar domain, but this introduces noticeable deformation since the sphere is not a developable surface. To address the uncertainty of users' behaviours, predicting models have been investigated recently, but none of them actually work on the spherical (not deformed) domain.

To overcome the above challenges in immersive technology, there is the need to develop an efficient tool for navigation patterns analysis in the spherical domain and leverage on that to predict users' behaviour and build the entire coding-delivery-rendering pipeline as a user- and geometry-centric system. This proposal focuses on develop novel VR delivery strategies, which will rely on the analysis and prediction of the navigation patterns of the omnidirectional content navigation, in a user, content- and application-dependent fashion. Predicting users' behaviour will allow to delivery only the content that will be actually consumed by the users, minimising the transmission resources consumption while maximising the user's Quality of Experience. The key novelty will be to address the above challenges working directly on the spherical domain, rather than on the projected (thus deformed) bidimensional domain. The developed VR system will be tested via a real omnidirectional video streaming testbed that will be implemented within the scope of the project. The outcomes of the project will be three-folds: i) user- content- and application-dependent VR streaming platform, optimized from the source codec to the delivery platform; ii) users analysis tools able to assess users' behaviour (when interactive with the content) in an objective fashion to identify common navigation patterns, and to predict future behaviours; iii) VR streaming testbed to test the proposed technology.

The project is focused on a significant technical challenge for twenty-first century society: how to ensure reliable and high-quality immersive communications for real-world use cases such as healthcare, creative technology, e-education, and high-risk missions. Specific impacts include:

1. People: the project will impact the science and engineering communities by training new researchers that are able to operate at the interface between multimedia systems, video processing, machine learning, and networking hence enhancing the pipeline of highly skilled people required to fuel the autonomy revolution.

2. Knowledge: the project will impact through knowledge creation at the cutting edge of the immersive and multimedia systems roadmap. The investigators have an excellent track record of dissemination and this will continue. In addition to high quality publications in leading international peer review journals and conferences, substantial effort will be dedicated to public communication around VR systems. In addition to high quality publications in leading international peer review journals and conferences, substantial effort will be dedicated to public communication around learning users' behaviour in interactive systems and about immersive communications. Both topics are of broad interest to the public and impact will be sought by: open access dissemination, interdisciplinary collaborations.

3. Economy: Immersive technologies will deliver economic growth and transform the way we communicate, work and play with high impact across several sectors of strong national important, such as manufacturing technologies, creative technology, healthcare and education with a global market size reaching $150bn by 2020 in only the creative technology sector. The analysis recognises immersive's strategic importance and the value of immersive technology underpinning significant and pervasive economic benefits. It is well known that responsible innovation will be key to reaping such economic benefits. This project will deliver fundamental results that can be built into the design and implementation phase for immersive systems. This is essential underpinning work to develop the fundamental technology to enable any future immersive communication system to ensure the full economic benefits of immersive applications can be realised.

3. Society: Robust results in developing a reliable communication platform for VR systems are essential to underpin public engagement in VR systems. The specific architectures that will be developed for VR systems will have societal impact in domains such as e-education, and cultural heritage. The work will also inform and contribute to standards bodies and Government policy on communication standards and policies for novel media systems.