Audio-Visual Media Research Platform

The strategic objective of this platform grant is to underpin Audio-Visual Media Research within the Centre for Vision, Speech and Signal Processing (CVSSP) to pursue fundamental research combining internationally leading expertise in understanding of real-world audio and visual data, and to transfer this capability to impact new application domains. Our goal is to pioneer new technologies which impact directly on industry practice in healthcare, sports, retail, communication, entertainment and training. This builds on CVSSP's unique track-record of world-leading research in both audio and visual machine perception which has enabled ground-breaking technology exploited by UK industry. The strategic contribution and international standing of the centres research in audio and visual media has been recognised by EPSRC through two previous platform grant awards (2003-14) and two programme grant awards in 2013 and 2015. Platform Grant funding is requested to reinforce the critical mass of expertise and knowledge of specialist facilities required to contribute advance in both fundamental understanding and pioneering new technology. In particular this Platform Grant will catalyse advances in multi-sensory machine perception building on the Centre's unique strengths in audio and vision. Key experienced post-doctoral researchers have specialist knowledge and practical know-how, which is an important resource for training new researchers and for maintaining cutting edge research using state-of-the-art facilities. Strategically the Platform Grant will build on recent independent advances in audio and visual scene analysis to lead multi-sensory understanding and modelling of real-world scenes. Research advances will provide the foundation for UK industry to lead the development of technologies ranging from intelligent sensing for healthcare and assisted living to immersive entertainment production. Platform Grant funding will also strengthen CVSSP's international collaboration with leading groups world-wide through extended research secondments US (Washington, USC), Asia (Tsinghua, Tianjin, Kyoto, Tokyo, KAUST) and Europe (INRIA, MPI, Fraunhofer, ETH, EPFL, KTH, CTU, UPF).

Platform Grant support for audio-visual media processing will support a critical-mass of joint research expertise in multi-sensory machine perception. This is critical towards achieving machines which can hear and see to understand and interact with real-world dynamic scenes. Recent advances in both the audio and vision research communities, with the introduction of deep learning methodologies, have achieved a step change in the capability of machine understanding enabling for the first time automatic interpretation of real-world complexity audio and visual data. Our research seeks to capitalise on these advances developing both the next generation of research leaders in multi-sensory machine perception and realising the capabilities for autonomous systems capable of combining sensing modalities to robustly understand and interpret audio-visual media. Research will address the open-challenge of machine understanding of complex dynamic real-world scenes combining the complementary information available from audio and visual sensor to achieve robust interpretation. These research advances are of central interest to both the audio and vision research communities and will bring together advances in machine perception. Joint audio-visual processing is essential to overcome the inherent ambiguities in either sensing modality such as occlusion, limited field of view and uniform appearance in visual sensing which commonly occur and can result in failure of visual understanding. Audio cues can overcome these limitations providing wide-area information and allowing the continuous sensing of objects which are visually obscured. For example both audio and visual cues are essential for non-contact monitoring of people in healthcare and assisted living applications.

Research in audio-visual signal processing and machine perception enabling robust interpretation of complex real-world dynamic scenes will have wide-spread impact on application domains through collaborative research with UK industry. These include:
- Medical diagnosis: to easily measure and monitor moving people from audio and video sensing for biomechanical analysis and to understand the relationship of body shape to health problems such as obesity and sleep patterns
- Sports performance analysis: to allow non-invasive performance analysis of player movement and team interaction in both training and live matches
- Veterinary science: to monitor animal behaviour for improved welfare and early detection of abnormalities
- Film, games and VR production: to support creativity through interpretation and editing of real-world content and to create digital doubles that appear and behave like a real person based on 4D audio-visual capture of actor performance
- Security and surveillance: to monitor individual human behaviour using visual biometrics based on 4D audio-visual analysis of the face and body in motion
- Robotics: to enable robots to safely navigate domestic environments with people and pets
- Human-computer communication: to understand individual behaviour for natural interaction
- Manufacturing industry: new methods for contactless measurement and analysis of surfaces using joint audio-visual sensing.
- Retail industry: methods to capture representations of both the audio and visual properties of objects to allow customers to experience these remotely.
- Cultural heritage: techniques to digitise artefacts for scientific study and to make them available to a wider audience.

Advances in these areas will be of direct benefit to the general public in healthcare, assisted living, entertainment and training.
Platform Grant funding will support underpinning research and pilot studies to build collaborative research in these domains.