BBC Prosperity Partnership: Future Personalised Object-Based Media Experiences Delivered at Scale Anywhere
Lead Research Organisation:
University of Surrey
Department Name: Vision Speech and Signal Proc CVSSP
Abstract
Personalisation of media experiences for the individual is vital for audience engagement of young and old, allowing more meaningful encounters tailored to their interest, making them part of the story, and increasing accessibility. The goal of the BBC Prosperity Partnership is to realise a transformation to future personalised content creation and delivery at scale for the public at home or on the move.
Evolution of mass-media audio-visual 'broadcast' content (news, sports, music, drama) has moved increasingly towards Internet delivery, which creates exciting potential for hyper-personalised media experiences delivered at scale to mass audiences. This radical new user-centred approach to media creation and delivery has the potential to disrupt the media landscape by directly engaging individuals at the centre of their experience, rather than predefining the content as with existing media formats (radio, TV, film). This will allow a new form of user-centred media experience which dynamically adapts to the individual, their location, the media content and producer storytelling intent, together with the platform/device and the network/compute resources available for rendering the content.The BBC Prosperity Partnership will position the BBC at the forefront of this 'Personalised Media' revolution enabling the creation and delivery of new services, and positioning the UK creative industry to lead future personalised media creation and intelligent network distribution to render personalised experiences for everyone anywhere.
Realisation of personalised experiences at scale presents three fundamental research challenges: capture of object-based representations of the content to enable dynamic adaption for personalisation at the point of rendering; production to create personalised experiences which enhance the perceived quality of experience for each user; and delivery at scale with intelligent utilisation of the available network, edge and device resources for mass audiences. The BBC Prosperity Partnership will address the major technical and creative challenges to delivering user-centred personalised audience experiences at scale. Advances in audio-visual AI for machine understanding of captured content will enable the automatic transformation of captured 2D video streams to an object-based media (OBM) representation. OBM will allow adaptation for efficient production, delivery and personalisation of the media experience whilst maintaining the perceived quality of the captured video content. To deliver personalised experiences to audiences of millions requires transformation of media processing and distribution architectures into a hybrid and distributed low-latency computation platform, allowing flexible deployment of compute-intensive tasks across the network. This will achieve efficiency in terms of cost and energy use, while providing optimal quality of experience for the audience within the technical constraints of the system.
Evolution of mass-media audio-visual 'broadcast' content (news, sports, music, drama) has moved increasingly towards Internet delivery, which creates exciting potential for hyper-personalised media experiences delivered at scale to mass audiences. This radical new user-centred approach to media creation and delivery has the potential to disrupt the media landscape by directly engaging individuals at the centre of their experience, rather than predefining the content as with existing media formats (radio, TV, film). This will allow a new form of user-centred media experience which dynamically adapts to the individual, their location, the media content and producer storytelling intent, together with the platform/device and the network/compute resources available for rendering the content.The BBC Prosperity Partnership will position the BBC at the forefront of this 'Personalised Media' revolution enabling the creation and delivery of new services, and positioning the UK creative industry to lead future personalised media creation and intelligent network distribution to render personalised experiences for everyone anywhere.
Realisation of personalised experiences at scale presents three fundamental research challenges: capture of object-based representations of the content to enable dynamic adaption for personalisation at the point of rendering; production to create personalised experiences which enhance the perceived quality of experience for each user; and delivery at scale with intelligent utilisation of the available network, edge and device resources for mass audiences. The BBC Prosperity Partnership will address the major technical and creative challenges to delivering user-centred personalised audience experiences at scale. Advances in audio-visual AI for machine understanding of captured content will enable the automatic transformation of captured 2D video streams to an object-based media (OBM) representation. OBM will allow adaptation for efficient production, delivery and personalisation of the media experience whilst maintaining the perceived quality of the captured video content. To deliver personalised experiences to audiences of millions requires transformation of media processing and distribution architectures into a hybrid and distributed low-latency computation platform, allowing flexible deployment of compute-intensive tasks across the network. This will achieve efficiency in terms of cost and energy use, while providing optimal quality of experience for the audience within the technical constraints of the system.
Organisations
- University of Surrey (Lead Research Organisation)
- SONY (Collaboration)
- Intel (United States) (Collaboration, Project Partner)
- BT Group (Collaboration)
- Boris FX (Collaboration)
- FIGMENT PRODUCTIONS LIMITED (Collaboration)
- Audioscenic (Collaboration, Project Partner)
- Telefonica S.A (Collaboration)
- Foundry (Collaboration)
- Dimensional Imaging Limited (Collaboration)
- British Broadcasting Corporation (BBC) (Collaboration)
- Mirriad (Collaboration)
- Imagination Technologies (Collaboration)
- BT Group (United Kingdom) (Project Partner)
- Synthesia (Project Partner)
- To Play For Ltd (Project Partner)
- Dimension Studios (Project Partner)
- Network Media Communications (Project Partner)
- Telefonica Research and Development (Project Partner)
- Foundry (United Kingdom) (Project Partner)
- Boris FX (United Kingdom) (Project Partner)
- SalsaSound (Project Partner)
- Framestore (Project Partner)
- Imagination Technologies (United Kingdom) (Project Partner)
- Figment Productions (Project Partner)
- Sony (Europe) (Project Partner)
- British Broadcasting Corporation (United Kingdom) (Project Partner)
- Mirriad (United Kingdom) (Project Partner)
Publications
Berghi D
(2024)
Leveraging Visual Supervision for Array-Based Active Speaker Detection and Localization
in IEEE/ACM Transactions on Audio, Speech, and Language Processing
Bridgeman L
(2021)
Dynamic Appearance Modelling from Minimal Cameras
Einabadi F
(2021)
Deep Neural Models for Illumination Estimation and Relighting: A Survey
in Computer Graphics Forum
Kim H
(2021)
Acoustic Room Modelling Using 360 Stereo Cameras
in IEEE Transactions on Multimedia
Lyko T
(2022)
QoE Assessment for Multi-Video Object Based Media
Lyko T
(2023)
Improving quality of experience in adaptive low latency live streaming
in Multimedia Tools and Applications
Mustafa A
(2022)
4D Temporally Coherent Multi-Person Semantic Reconstruction and Segmentation
in International Journal of Computer Vision
Nadeem A
(2023)
SEM-POS: Grammatically and Semantically Correct Video Captioning
Nadeem A
(2023)
CAD -- Contextual Multi-modal Alignment for Dynamic AVQA
Nadeem A
(2023)
SEM-POS: Grammatically and Semantically Correct Video Captioning
Pesavento M
(2021)
Attention-based Multi-Reference Learning for Image Super-Resolution
Pesavento M
(2021)
Super-Resolution Appearance Transfer for 4D Human Performances
Description | Frameworks for Understanding Personalisation - BBC White Paper WHP 404 |
Geographic Reach | Multiple continents/international |
Policy Influence Type | Contribution to new or improved professional practice |
Impact | Future of media services |
URL | http://www.bbc.co.uk/rd/pubs/whp |
Description | Ofcom Object-based Media Working Group |
Geographic Reach | National |
Policy Influence Type | Participation in a guidance/advisory committee |
Impact | Influence on media communication and service regulation |
URL | https://www.ofcom.org.uk |
Description | The Role of the Audience in Media - BBC White Paper WHP 396 |
Geographic Reach | Multiple continents/international |
Policy Influence Type | Contribution to new or improved professional practice |
Impact | White paper on future media |
URL | https://www.bbc.co.uk/rd/publications/role-of-audience-in-media-how-culture-framing-narration-shape-... |
Title | Audio-visual production dataset for weather forecaster personalisation use case |
Description | Audio and video recording of presenters for production resarch |
Type Of Material | Database/Collection of data |
Year Produced | 2023 |
Provided To Others? | Yes |
Impact | Use in production demonstrators January 2023 |
URL | http://cvssp.org |
Description | AudioScenic |
Organisation | Audioscenic |
Country | United Kingdom |
Sector | Private |
PI Contribution | Advisor/collaboration on spatial audio research and development for personalised media |
Collaborator Contribution | Participation in industry events |
Impact | Research advice in spatial audio |
Start Year | 2021 |
Description | BBC Research and Development |
Organisation | British Broadcasting Corporation (BBC) |
Country | United Kingdom |
Sector | Public |
PI Contribution | Research in Computer Vision for broadcast production and Audio. Technologies for 3D production, free-view point video in sports, stereo production from monocular cameras, video annotation Member of the BBC Audio Research Partnership - developing the next generation of broadcast technology. |
Collaborator Contribution | In kind contribution (members of Steering/Advisory Boards) Use of the BBC lab and research/development facilities. Studentships (industrial case) funding and co-supervision of PhD students. |
Impact | Multi-disciplinary collaboration involves Computer Vision, Video Analysis, Psychoacoustics, Signal Processing and Spatial Audio |
Description | BT |
Organisation | BT Group |
Country | United Kingdom |
Sector | Private |
PI Contribution | research collaboration in object-based/personalised media delivery |
Collaborator Contribution | support of research activities |
Impact | ongoing collaboration |
Start Year | 2021 |
Description | British Broadcasting Corporation |
Organisation | British Broadcasting Corporation (BBC) |
Country | United Kingdom |
Sector | Public |
Start Year | 2004 |
Description | Dimension Studios |
Organisation | Dimensional Imaging Limited |
Country | United Kingdom |
Sector | Private |
PI Contribution | Access to expertise and data for volumetric capture |
Collaborator Contribution | Collaboration partner in volumetric performance capture |
Impact | Research collaboration |
Start Year | 2021 |
Description | Figment Productions |
Organisation | Figment Productions Limited |
Country | United Kingdom |
Sector | Private |
PI Contribution | Advice/collaboration on personalisation in XR experiences |
Collaborator Contribution | research collaboration and attendance at industry meetings |
Impact | ongoing research |
Start Year | 2021 |
Description | Foundry |
Organisation | Foundry |
Country | United Kingdom |
Sector | Private |
PI Contribution | Collaboration on Computer Vision and AI tools for film and VFX production |
Collaborator Contribution | Novel computer vision methods for segmentation, reconstruction, tracking and representation of people from video. Used for actor performance capture and visual effects production. |
Impact | Novel Computer vision methods for video analysis of actor performance |
Start Year | 2008 |
Description | Foundry |
Organisation | Foundry |
Country | United Kingdom |
Sector | Private |
PI Contribution | Collaboration on Computer Vision and AI tools for film and VFX production |
Collaborator Contribution | Novel computer vision methods for segmentation, reconstruction, tracking and representation of people from video. Used for actor performance capture and visual effects production. |
Impact | Novel Computer vision methods for video analysis of actor performance |
Start Year | 2008 |
Description | Imagination Technologies |
Organisation | Imagination Technologies |
Country | United Kingdom |
Sector | Private |
PI Contribution | Contribution to advise on personalisation for the mobile market |
Collaborator Contribution | Industry presentation and collaboration events |
Impact | Ongoing collaboration |
Start Year | 2021 |
Description | Imagineer Systems |
Organisation | Boris FX |
Country | United Kingdom |
Sector | Private |
PI Contribution | Personalised media production tools |
Collaborator Contribution | Advice/collaboration on production tools |
Impact | ongoing collaboration |
Start Year | 2021 |
Description | Intel |
Organisation | Intel Corporation |
Country | United States |
Sector | Private |
PI Contribution | Research collaboration in personalised media |
Collaborator Contribution | collaboration/PhD Sponsorship |
Impact | ongoing collaboration |
Start Year | 2021 |
Description | Mirriad |
Organisation | Mirriad |
Country | United Kingdom |
Sector | Private |
PI Contribution | Personalisation in advertising advice |
Collaborator Contribution | Industry partner briefing on requirements for personalisation |
Impact | Ongoing collaboration |
Start Year | 2021 |
Description | Sony Broadcast and Professional Europe |
Organisation | SONY |
Department | Sony Broadcast and Professional Europe |
Country | United Kingdom |
Sector | Private |
Start Year | 2004 |
Description | Telefonica |
Organisation | Telefonica S.A |
Department | Telefonica Research |
Country | Spain |
Sector | Private |
PI Contribution | Research collaboration in personalised/object-based media |
Collaborator Contribution | media delivery |
Impact | ongoing research |
Start Year | 2021 |
Description | BBC Sounds Amazing |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | Industry/academic forum for research and production industry professionals in audio and sound hosted by the BBC |
Year(s) Of Engagement Activity | 2021,2022 |
URL | https://www.bbc.co.uk/academy/events/sounds-amazing-2022/ |
Description | BBC Wing Watch public trial |
Form Of Engagement Activity | A broadcast e.g. TV/radio/film/podcast (other than news/press) |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | If so you can you could include the public trial of the Wing Watch service during Winterwatch https://www.bbc.co.uk/rd/blog/2023-01-wing-watch-ai-bird-wildlife-video This allowed us to experiment with how AI-generated data can be used to drive a flexible media experience, giving the user the freedom to navigate the media in the way that most interest then. In doing so we were also experimenting with production tools to manage that AI-data to make sure it was suitable for presenting to the audience. |
Year(s) Of Engagement Activity | 2022,2023 |
URL | https://www.bbc.co.uk/rd/blog/2023-01-wing-watch-ai-bird-wildlife-video |
Description | CVPR |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Industry/Business |
Results and Impact | Primary international forum for computer vision and AI/machine learning research in audio-visual media. Research dissemination through papers, key-note invited talks and workshop organisation |
Year(s) Of Engagement Activity | 2021,2022,2023 |
URL | https://cvpr2023.thecvf.com |
Description | European Conference on Visual Media Production |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | Presentation of research advances at industry-academic forum |
Year(s) Of Engagement Activity | 2021,2022 |
URL | https://www.cvmp-conference.org/ |