Learning to Recognise Dynamic Visual Content from Broadcast Footage

This research is in the area of computer vision - making computers which can understand what is happening in photographs and video. As humans we are fascinated by other humans, and capture endless images of their activities, for example home movies of our family on holiday, video of sports events or CCTV footage of people in a town center. A computer capable of understanding what people are doing in such images would be able to do many jobs for us, for example finding clips of our children waving, fast forwarding to a goal in a football game, or spotting when someone starts a fight in the street. For Deaf people, who use a language combining hand gestures with facial expression and body language, a computer which could visually understand their actions would allow them to communicate in their native language. While humans are very good at understanding what people are doing (and can learn to understand special actions such as sign language), this has proved extremely challenging for computers.Much work has tried to solve this problem, and works well in particular settings for example the computer can tell if a person is walking so long as they do it clearly and face to the side, or can understand a few sign language gestures as long as the signer cooperates and signs slowly. We will investigate better models for recognising activities by teaching the computer by showing it many example videos. To make sure our method works well for all kinds of setting we will use real world video from movies and TV. For each video we have to tell the computer what it represents, for example throwing a ball or a man hugging a woman . It would be expensive to collect and label lots of videos in this way, so instead we will extract approximate labels automatically from subtitle text and scripts which are available for TV. Our new methods will combine learning from lots of approximately labelled video (cheap because we get the labels automatically), use of contextual information such as which actions people do at the same time, or how one action leads to another ( he hits the man, who falls to the floor ), and computer vision methods for understanding the pose of a person (how they are standing), how they are moving, and the objects which they are using.By having lots of video to learn from, and methods for making use of approximate labels, we will be able to make stronger and more flexible models of human activities. This will lead to recognition methods which work better in the real world and contribute to applications such as interpreting sign language and automatically tagging video with its content.

The proposed research has potential impact for three communities: 1. The computer vision and machine learning communities in academia 2. The potential end users of tools for automatic categorisation and searching of content such as organisations like the BBC 3. The Deaf community, by providing tools for automatic recognition of sign language The computer vision and machine learning communities will benefit from new knowledge and new techniques and the creation of new and challenging datasets for use by the wider research community. Dissemination to the research community will be via publishing in the major national and international conferences/journals and via a project website. The principal search and media companies have a significant presence at these conferences. Additionally the PIs will organise an international workshop in conjunction with one of the major international conferences to disseminate the outputs of the project and there is a possibility to run a technical meeting in association with the British Machine Vision Association (BMVA). There is potential to contribute to other scientific disciplines and one of our project partners, DCAL (see letter of support), have indicated that any tools for automatic recognition, annotation and search of sign language would be immensely beneficial to their research on Sign linguistics and the EPSC Corpus project. By automatically categorising, labelling and providing search facilities, the research has immense benefits to the broadcast and media industry. Much of the BBC's digital video archive material from the 1970s and 1908s has only the name of the programme and transmission data. Enabling search and annotation of such material will bring vast improvements in accessibility. BBC achieves are another project partner, specifically for this reason. However, with the ever growing quantities of digital media from personal devices such tools have far wider reaching applications to the general public. Finally the research could have a considerable impact on Deaf-hearing communication, providing tools to automatically translate sign into spoken English. Such tools could provide benefits to organisations that provide such services to the Deaf community such as Significan't, another project partner. They are also well placed to commercialise such technology and the wider application of tools that allow the categorisation and searching of sign have applications for online or web based resources for the Deaf community.