Multimodal Video Search by Examples (MVSE)

How to effectively and efficiently search for content from large video archives such as BBC TV programmes is a significant challenge. Search is typically done via keyword queries using pre-defined metadata such as titles, tags and viewer's notes. However, it is difficult to use keywords to search for specific moments in a video where a particular speaker talks about a specific topic at a particular location. Most videos have little or no metadata about content in the video, and automatic metadata extraction is not yet sufficiently reliable. Furthermore, metadata may change over time and cannot cover all content. Therefore, search by keyword is not a desirable approach for a comprehensive and long-lasting video search solution.

Video search by examples is a desirable alternative as it allows search for content by one or more examples of the interested content without having to specify interest in keyword. However, video search by examples is notoriously challenging, and its performance is still poor. To improve search performance, multiple modalities should be considered - image, sound, voice and text, as each modality provides a separate search cue so multiple cues should identify more relevant content. This is multimodal video search by examples (MVSE). This is an emerging area of research, and the current state of the art is far from desirable so there is a long way to go. There is no commercial service for MVSE.

This proposal has been co-created with BBC R&D through the BBC Data Science Partnership via a number of online meetings and one face to face meeting involving all partners. The proposal has been informed by recent unpublished ethnographic research on how current BBC staff (producers, journalists, archivists) search for media content. It was found that they were very interested in knowledge retrieval from archives or other sources but they required richer metadata and cataloguing of non-verbal data.

In this proposal we will study efficient, effective, scalable and robust MVSE where video archives are large, historical and dynamic; and the modalities are person (face or voice), context, and topic. The aim is to develop a framework for MVSE and validate it through the development of a prototype search tool. Such a search tool will be useful for organisations such as the BBC and British Library, who maintain large collections of video archives and want to provide a search tool for their own staff as well as for the public. It will also be useful for companies such as Youtube who host videos from the public and want to enable video search by examples. We will address key challenges in the development of an efficient, effective, scalable and robust MVSE solution, including video segmentation, content representation, hashing, ranking and fusion.

This proposal is planned for three years, involving three institutions (Cambridge, Surrey, Ulster) and one partner (the BBC) who will contribute significant resources (estimated at £128.4k) to the project (see Letter of Support from the BBC).

The project's objective is to provide scalable next-generation 'search by example' functionality across national video archives. The project will develop beyond the state of the art in video segmentation, content representation/matching/ranking functionality and these outputs are intended to provide positive, disruptive impact in multimedia search capability across the media industry nationally and internationally.

The beneficiaries of this project's outputs will include academics, journalists, broadcasters, TV viewers, multimedia companies and organisations hosting and managing large video or multimedia repositories.

Journalists and broadcasters will directly benefit by time efficiency savings and the rapid discovery of relevant content when using this new technology. This will in turn provide better, more relevant and more enriched TV programming in less time thus having economic savings. This will have a benefit to TV viewers who will enjoy more relevant TV programmes by the effective repurposing of content within big media archives. As a key partner, the immediate beneficiary will be the BBC who will likely adopt and integrate the new technology within their workflows to improve the discovery of media content when producing TV programmes. However, the technologies developed are transferable to other broadcasters and indeed major online companies such as Youtube who rely on semantically enriched search technologies.

Academics will benefit by the dissemination and inspiration of the project's new research findings and search technologies for rapidly discovering relevant video/multimedia content based on new intelligent algorithms.

The pathways to impact document provides an outline of a series of activities including co-creation workshops and licensing to increase the likelihood of research impact and adoption of the novel, disruptive technologies produced in this project.