Visual Sense. Tagging visual data with semantic descriptions

Recent years have witnessed an unprecedented growth in the number of image and video collections, partially due to the increased popularity of photo and video sharing websites. One such website alone (Flickr) stores billions of images. And this is not the only way in which visual content is present on the Web: in fact most web pages contain some form of visual content. However, while most traditional tools for search and retrieval can successfully handle textual content, they are not prepared to handle heterogeneous documents. This new type of content demands the development of new efficient tools for search and retrieval.


The large number of readily accessible multi-media data-collections pose both an opportunity and a challenge. The opportunity lies in the potential to mine this data to automatically discover mappings between visual and textual content. The challenge is to develop tools to classify, filter, browse and search such heterogeneous data. In brief, the data is available, but the tools to make sense of it are missing.

The Visual Sense project aims to automatically mine the semantic content of visual data to enable "machine reading" of images. In recent years, we have witnessed significant advances in the automatic recognition of visual concepts. These advances allowed for the creation of systems that can automatically generate keyword-based image annotations. However, these annotations, e.g. "man" and "pot", fall far short of the sort of more meaningful descriptive captions necessary for indexing and retrieval of images, for example,"Man cooking in kitchen". The goal of this project is to move a step forward and predict semantic image representations that can be used to generate more informative sentence-based image annotations, thus facilitating search and browsing of large multi-modal collections. It will address the following key open research challenges:

1) Develop methods that can derive a semantic representation of visual content. Such representations must go beyond the detection of objects and scenes and also include a wide range of object relations.
2) Extend state-of-the-art natural language techniques to the tasks of mining large collections of multi-modal documents and generating image captions using both semantic representations of visual content and object/scene type models derived from semantic representations of the textual component of multi-modal documents.
3) Develop learning algorithms that can exploit available multi-modal data to discover mappings between visual and textual content. These algorithms should be able to leverage 'weakly' annotated data and be robust to large amounts of noise.

Thus, the main focus of the Visual Sense project is the development of machine learning methods for knowledge and information extraction from large collections of visual and textual content and for the fusion of this information across modalities. The tools and techniques developed in this project will have a variety of applications. To demonstrate them, we will address three case studies: 1) evaluation of generated descriptive image captions in established international image annotation benchmarks, 2) re-ranking for improved image search and 3) automatic illustration of articles with images.

To address these broad challenges, the project will build on expertise from multiple disciplines, including computer vision, machine learning and natural language processing (NLP). It brings together four research groups from University of Surrey (Surrey, UK), Institut de Robotica i Informatica Industrial (IRI, Spain), Ecole Centrale de Lyon (ECL, France), and University of Sheffield (Sheffield, UK) having each well established and complementary expertise in their respective areas of research.

Innovations in rich annotation of visual data directly impact the European ICT, Digital Media and Creative industries as well as having broader societal impact for any individual seeking to make sense of the huge amount of valuable, yet unstructured, image and video published online. Specifically, the project will:

Reinforce the position of the European ICT and Digital Media research, widening marketing opportunities especially for technology-providing SMEs who consume, produce or have need to search or aggregate visual data.
Stimulate greater creativity through technologies and tools to search professional and user-generated digital media content.
Provide digital media/service search engines with innovative offers for interactive and personalised digital media.
Enhance opportunities for education through illustrative images and videos complementing text descriptions.
This project technologies focus on visual data search, and metadata is the key to improving image and video search capability. The development of metadata and associated tools is in its infancy, and here lies an opportunity within both the ICT and Digital Media sectors. Although metadata frequently accompanies image and video sources, it is often patchy and limited to technical data on media capture rather than on content itself. When editorial metadata is present, this is often interest-specific and in a non-standardised form. As the NEM note within their 2009 Strategic Research Agenda (NEM-SRA), "without metadata [visual data] content is almost valueless". Our core innovations include the automated annotation of visual data to enrich source-supplied metadata with additional metadata derived automatically from visual and text content, so enabling search and aggregation. NEM-SRA notes this topic of "automatic video indexing" as one of five key promising topics for further research; another being the fusion of visual, aural and meta-data in multimedia search. These topics are explicitly addressed by WP2-WP5. Not only will enhanced techniques for semantic metadata annotation add value to Digital Media, but visual recognition also benefit other areas of ICT e.g. for surveillance, robotics, sports and medical analysis, automated manufacturing, and assisted living.

ViSen is timely in that the opportunities to exploit its outcomes are just beginning to appear. Professional customers are starting to use textual metadata tools for searching archives and managing stored content; they are already acutely aware of the need for similar and more advanced tools for live content. Consumers are becoming accustomed to searching social network sites for visual content. Other professional users
have expressed interest in the ability to search live feeds for specified content or contexts. Market conditions are favourable and will be actively monitored during the final year of ViSen through a living website reflecting routes to market and exploitation opportunities.
Project partners are in close collaboration with the BBC and SMEs, e.g. Omniperception UK, which offers multimedia search tools. We will also investigate commercial exploitation possibilities arising from ViSen with the assistance of the University's Commercialisation of IP Team and via its agreement with Fusion IP plc (http://www.fusionip.co.uk), whose mission is to seek commercial avenues to exploit IP generated in research projects by the University of Sheffield.