EPSRC-SFI: SpheryStream

Lead Research Organisation: University College London
Department Name: Electronic and Electrical Engineering

Abstract

This proposal addresses the important and not well-studied area in the fast-developing engineering space of the virtual reality and immersive technology. Our final vision is to endow any final user with an unprecedented sense of full immersion in virtual environments for any applicative scenario and with any connectivity level. This requires VR systems that operate at scale, in a personalized manner, remaining bandwidth-tolerant whilst meeting quality and latency criteria. This can be accomplished only by a fundamental revolution of the coding/streaming/rendering chain that has to put the interactive user at the heart of the system rather than at the end of the chain.
Specifically, bringing our final vision to reality requires constant innovation in VR streaming technology to overcome the following main challenges: i) the spherical format of the video content in a scenario in which media codecs are optimized for planar content; ii) the interactive behaviour of users that introduces uncertainty on the popularity of the video content. To overcome the former challenges, the spherical video is currently projected on a bidimensional planar domain, but this introduces noticeable deformation since the sphere is not a developable surface. To address the uncertainty of users' behaviours, predicting models have been investigated recently, but none of them actually work on the spherical (not deformed) domain.

To overcome the above challenges in immersive technology, there is the need to develop an efficient tool for navigation patterns analysis in the spherical domain and leverage on that to predict users' behaviour and build the entire coding-delivery-rendering pipeline as a user- and geometry-centric system. This proposal focuses on develop novel VR delivery strategies, which will rely on the analysis and prediction of the navigation patterns of the omnidirectional content navigation, in a user, content- and application-dependent fashion. Predicting users' behaviour will allow to delivery only the content that will be actually consumed by the users, minimising the transmission resources consumption while maximising the user's Quality of Experience. The key novelty will be to address the above challenges working directly on the spherical domain, rather than on the projected (thus deformed) bidimensional domain. The developed VR system will be tested via a real omnidirectional video streaming testbed that will be implemented within the scope of the project. The outcomes of the project will be three-folds: i) user- content- and application-dependent VR streaming platform, optimized from the source codec to the delivery platform; ii) users analysis tools able to assess users' behaviour (when interactive with the content) in an objective fashion to identify common navigation patterns, and to predict future behaviours; iii) VR streaming testbed to test the proposed technology.

Planned Impact

The project is focused on a significant technical challenge for twenty-first century society: how to ensure reliable and high-quality immersive communications for real-world use cases such as healthcare, creative technology, e-education, and high-risk missions. Specific impacts include:

1. People: the project will impact the science and engineering communities by training new researchers that are able to operate at the interface between multimedia systems, video processing, machine learning, and networking hence enhancing the pipeline of highly skilled people required to fuel the autonomy revolution.

2. Knowledge: the project will impact through knowledge creation at the cutting edge of the immersive and multimedia systems roadmap. The investigators have an excellent track record of dissemination and this will continue. In addition to high quality publications in leading international peer review journals and conferences, substantial effort will be dedicated to public communication around VR systems. In addition to high quality publications in leading international peer review journals and conferences, substantial effort will be dedicated to public communication around learning users' behaviour in interactive systems and about immersive communications. Both topics are of broad interest to the public and impact will be sought by: open access dissemination, interdisciplinary collaborations.

3. Economy: Immersive technologies will deliver economic growth and transform the way we communicate, work and play with high impact across several sectors of strong national important, such as manufacturing technologies, creative technology, healthcare and education with a global market size reaching $150bn by 2020 in only the creative technology sector. The analysis recognises immersive's strategic importance and the value of immersive technology underpinning significant and pervasive economic benefits. It is well known that responsible innovation will be key to reaping such economic benefits. This project will deliver fundamental results that can be built into the design and implementation phase for immersive systems. This is essential underpinning work to develop the fundamental technology to enable any future immersive communication system to ensure the full economic benefits of immersive applications can be realised.

3. Society: Robust results in developing a reliable communication platform for VR systems are essential to underpin public engagement in VR systems. The specific architectures that will be developed for VR systems will have societal impact in domains such as e-education, and cultural heritage. The work will also inform and contribute to standards bodies and Government policy on communication standards and policies for novel media systems.
 
Description This main goal of this proposal has been to develop novel VR delivery strategies, which will rely on the analysis and prediction of the navigation patterns of the omnidirectional content navigation, in a user, content- and application-dependent fashion. This goal has been met by the following advances:

* We have discovered key methods to study and understand users in virtual reality (VR) settings.
* The above findings have been extended to augmented reality systems
* The understanding of users behavior derived above has been shown to help the prediction of users behavior in VR, allowing us to move a step forward in the direction of personalised VR streaming.
* This is now investigated in the direction of streaming strategies and bandwidth gain that our finding can achieve.

This has led to the following outcomes
i) user-centric VR streaming platform (UCL/Trinity College)
ii) users analysis tools able to assess users' behaviour in an objective fashion to identify common navigation patterns, and to predict future behaviours (UCL)
iii) VR streaming testbed to test the proposed technology (Trinity College).
Exploitation Route Our works have been already shown as cited and used in the community to bring the study forward and better understand how to improve current VR systems. Our dataset and methodologies are all open-sources and can be improved by the community.
Sectors Creative Economy

Digital/Communication/Information Technologies (including Software)

Education

Culture

Heritage

Museums and Collections

URL https://v-sense.scss.tcd.ie/research/3dof/
 
Description * Significant impact in the research community, leading to a new research direction on user-centri XR systems. This grant and the work carried out within has led to strong contributions in this direction. * Associated to this grant, I was able to get awarded an impact accelerator (IAA) grant, that delivered a strong contribution on the BBC R&D dataset. Specifically, leveraging on our knowledge of personalisation systems, we carried out an IAA a study toward personalisation systems in BBC iPlayer. * part of the research carried out from the project partners as also led to developing novel dataset to improve cultural heritage at Trinity College Library.
First Year Of Impact 2023
Impact Types Cultural

Societal

 
Description DREAM - Data Communication, Processing, and Delivery in the Era of Machines
Amount £65,741 (GBP)
Funding ID SRF\R1\231074 
Organisation University College London 
Sector Academic/University
Country United Kingdom
Start 08/2023 
End 08/2024
 
Description EPSRC Impact Acceleration Account (IAA) award to UCL 2020-22
Amount £13,647 (GBP)
Organisation University College London 
Sector Academic/University
Country United Kingdom
Start 03/2021 
End 06/2022
 
Description UCL-SJTU Strategic Partner Funding Scheme
Amount £10,000 (GBP)
Organisation University College London 
Sector Academic/University
Country United Kingdom
Start 12/2021 
End 11/2022
 
Title Novel tools for User centric systems in VR 
Description The research funded with this grant allowed us to develop a novel set of tools (specifically metrics for the community to use) to understand users behaviours in virtual reality systems. Especially, we proposed the first set of tools to make this analysis not only from a qualitative aspects but also from a quantitive perspective. This research led to awards and citations, to highlight the impact. 
Type Of Material Improvements to research infrastructure 
Year Produced 2021 
Provided To Others? Yes  
Impact Best TOMM paper 2021 award for: S. Rossi, C. Ozcinar, A. Smolic, L. Toni, "Do Users Behave Similarly in VR? Investigation of the User Influence on the System Design", ACM Transactions on Multimedia Computing, Communications, and Applications. Invitation as invited speakers * Keynote Speaker, MMSPS 2023, "Graph Based Machine learning for the MetaVerse". * Keynote Speaker, WebMedia 2020, "Understanding user interactivity for next-generation immersive systems: from 3DoF to 6DoF", November 2021. * Invited talk at Spring School on Social XR, CWI, "The Power of Graphs Learning in Immersive Communications", 2024. * Invited talk at Symposium Human-Centered Multimedia "Understanding Users' Behaviour in Immersive Communications and its Challegne in 6DoF", TU Delf, 2022 
 
Title First metric to evaluate users' similarity in 6-degree of freedom systems 
Description We proposed the first metric to measure users similarity in XR systems with 6DoF. The model has been validated on existing dataset and social XR settings developed by our collaborators in CWI. 
Type Of Material Data analysis technique 
Year Produced 2023 
Provided To Others? No  
Impact * Keynote Speaker, MMSPS 2023, "Graph Based Machine learning for the MetaVerse". * Keynote Speaker, WebMedia 2020, "Understanding user interactivity for next-generation immersive systems: from 3DoF to 6DoF", November 2021. * Invited talk at Spring School on Social XR, CWI, "The Power of Graphs Learning in Immersive Communications", 2024. * Invited talk at Symposium Human-Centered Multimedia "Understanding Users' Behaviour in Immersive Communications and its Challegne in 6DoF", TU Delf, 2022 
 
Description Collaboration with BBC R&D London 
Organisation British Broadcasting Corporation (BBC)
Department BBC Research & Development
Country United Kingdom 
Sector Public 
PI Contribution The EPSRC project deals with personalisation in VR communications, and this topic is strictly linked to personalisation of recommender systems. This created a synergy between the PI and BBC R&D London. Our main contribution was to create the connection between the two frameworks (VR and recommendation) offer our knowledge to recommendation systems
Collaborator Contribution BBC provided their experience and background from studying iPlayer real users and providing us such dataset.
Impact This collaboration led to an EPSRC accelerator grant in which BBC was a partner and contributed financially.
Start Year 2022
 
Description Collaboration with CWI (national research institute for mathematics and computer science in the Netherlands) -- prof. P. Cesar (partner of the SpheryStream project) leading to an ERCIM Fellowship as follow up of this collaboration 
Organisation Netherlands Organisation for Scientific Research (NWO)
Department National Research Institute for Mathematics and Computer Science
Country Netherlands 
Sector Academic/University 
PI Contribution This collaboration between two highly complementarity team has been extremely fruitful. UCL research is aimed at better understanding how people interact with immersive content, advancing human-centric computing solutions for next-generation immersive multimedia systems. This is highly aligned (and yet complementary) with the research objectives of the Distributed and Interactive Systems (DIS) group at CWI (http://www.dis.cwi.nl). UCL contribution has been to develop novel metrics and theoretical studies to understand users' behaviour in VR. This has been a key piece to improve VR communications.
Collaborator Contribution One of the main research direction of the CWI group is to improve the way people use interactive systems and how people remotely communicate with each other. The DIS group, led by Prof. Pablo Cesar, is recognized by the community as a leader in developing real-time human-centric communication systems, with outstanding outputs such as strong publications and demos/datasets in top-tier conferences (e.g., ACM MM, ACM MMSys, and ACM IMX) and international peer-review journals, as well as many involvements in European and national projects (e.g., VRTogether). Beyond academia, the groups impacted also the industry sector, with constant participation in standardization bodies such as MPEG and ITU. The DIS group has contributed to this collaboration by providing a complementary full-stack, experimental, and human-centric approach, providing real-word testbeds and dataset.
Impact publications associated to this grant are in part output of such a collaboration.
Start Year 2021
 
Description Collaboration with Politecnico BARI 
Organisation Polytechnic University of Bari
Country Italy 
Sector Academic/University 
PI Contribution As follow-up of the work developed in this project (specifically related to volumetric content) we have collaborated with Politecnico of Bari to develop a novel dataset based on new modalities - mmWave sensor data. This is now going on. UCL contribution has been to propose ML based technique to process the data
Collaborator Contribution Politecnico of Bari contribution has been on the acquisition data side.
Impact a new dataset provided to the community and accepted as proceeding to the ODS track at MMSys 2024. A full journal publication just submitted to ACM TOMM.
Start Year 2023
 
Description BBC Periodic meetings 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Other audiences
Results and Impact BBC R&D Team attended the periodic meetings to discuss innovations on VR systems
Year(s) Of Engagement Activity 2023
 
Description EEE Festival Research 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Other audiences
Results and Impact A formal presentation of the activities carried out within this project were presented at the EEE Festival day at UCL - there are 4 line up speakers (I was one of the 4) to present as keynote during the day. The festival is not for UCL members only but it is actually a day to show our research to a wide audience: students, policy makers, entrepreneurs, industry, etc.
Year(s) Of Engagement Activity 2022
URL https://www.ucl.ac.uk/electronic-electrical-engineering/news/2022/may/celebrating-eee-excellency-fes...