Multisource audio-visual production from user-generated content

The pervasiveness of amateur media recorders either embedded in smartphones or as stand-alone devices is revolutionizing the way events are captured and reported. The aim of this project is to devise intelligent editing and production algorithms based on new signal processing techniques for processing multi-view user-generated content.

The explosion of shared video content offers the opportunity for new ways of not only analysing but also timely reporting stories, ranging from disaster scenes and protests to music concerts and sports events. However, the large amount of data increasingly available and their varying quality makes the selection and editing of appropriate multimedia items in a timely manner very difficult thus strongly limiting the opportunity to harvest this data for security, cultural and entertainment applications. There is an urgent need to investigate and develop new ways to help or replace what used to be the role of a producer/director in this rapidly changing landscape. In particular, there is the need to automate production tasks and to generate new and high-quality content from multiple views.

The key aspect of the project is the integration of audio and visual inputs that support each other in reaching objectives that would otherwise be impossible using only one modality. We will focus on a set of relevant event-types: sports, music shows and crowd scenes. We will devise novel multisource processing techniques to improve audiovisual production and to enable synchronisation processing. This will in turn allow generation of novel and higher quality audio-visual rendering of captured events.

This project will have major impact on the use and quality of user-generated videos. It will lead to new, intelligent content production algorithms that enable user-generated content to be seamlessly and rapidly integrated into multimedia items. It also supports the development of video-based citizen journalism and other participatory media, where members of the public play an active role in the process of news reporting and community-based content creation.

Dissemination will be through journal and conference publications as well as an up-to-date project website, which will also provide a service for uploading and producing content. We will build on existing links and engage with new beneficiaries and user groups by publishing the research at conferences (especially those with strong industry presence) and in high impact journals, visits to other groups and institutions.

Videos related to published results will also be disseminated through the Investigators' YouTube channels (that have already attracted 10,000 views to date) and websites. Our tools and accomplishments will be promoted to the multimedia signal processing community via press releases sent to mailing lists and magazines. Halfway through the project, we will host a one day Workshop on multisource signal processing, bringing together leaders in this emerging field. Near the end of the project, a Workshop on systems for intelligent production of user generated content will also be held, in which we will discuss and promote the outcomes of our research.

To enhance engagement with the public and improve communication of project results, the Queen Mary EECS Public Engagement team will advise and mentor the research team to deliver high impact public engagement activities, assist with writing and editing articles for web and magazine, integrating messages from the project in events for schools and the general public and contribute towards distribution costs of magazines including project results. We will participate in communication activities by exhibiting prototypes developed in the project, contributing our technologies for use in creative projects (c4dmpresents.org) and disseminating results through the school magazines Audio! and CS4Fun (Computer Science for Fun).

The Investigators will lead Impact-related activities, although all researchers will be involved. We will use the services and expertise available from the QMUL Communications Officer for drafting press releases and promotional materials, and from the QM Innovation Marketing Manager for producing materials aimed at industry and user groups with a commercial interest. Descriptions of the project and its outcomes will be included in promotional material from the Investigators' research groups, the Computer Vision group and the Centre for Digital Music