Integrating sound and context recognition for acoustic scene analysis

The amount of audio data being generated has dramatically increased over the past decade, spanning from user-generated content, recordings in audiovisual archives, to sensor data captured in urban, nature or domestic environments. The need to detect and identify sound events in environmental recordings (e.g. door knock, glass break) as well as to recognise the context of an audio recording (e.g. train station, meeting) has led to the emergence of a new field of research: acoustic scene analysis. Emerging applications of acoustic scene analysis include the development of sound recognition technologies for smart homes and smart cities, security/surveillance, audio retrieval and archiving, ambient assisted living, and automatic biodiversity assessment.

However, current sound recognition technologies cannot adapt to different environments or situations (e.g. sound identification in an office environment, assuming specific room properties, working hours, outdoor noise and weather conditions). If information about context is available, it is typically characterised by a single label for an entire audio stream, not taking into account complex and ever-changing environments, for example when recording using hand-held devices, where context can consist of multiple time-varying factors and can be characterised by more than a single label.

This project will address the aforementioned shortcomings by investigating and developing technologies for context-aware sound recognition. We assume that the context of an audio stream consists of several time-varying factors that can be viewed as a combination of different environments and situations; the ever-changing context in turn informs the types and properties of sounds to be recognised by the system. Methods for context and sound recognition will be investigated and developed, based on signal processing and machine learning theory. The main contribution of the project will be an algorithmic framework that jointly recognises audio-based context and sound events, applied to complex audio streams with several sound sources and time-varying environments.

The proposed software framework will be evaluated using complex audio streams recorded in urban and domestic environments, as well as using simulated audio data in order to carefully control contextual and sound properties and have the benefit of accurate annotations. In order to further promote the study of context-aware sound recognition systems, a public evaluation task will be organised in conjunction with the public challenge on Detection and Classification of Acoustic Scenes and Events (DCASE).

Research carried out in this project targets a wide range of potential beneficiaries in the commercial and public sector for sound and audio-based context recognition technologies, as well as users and practitioners of such technologies. Beyond acoustic scene analysis, we believe this new approach will advance the broader fields of audio and acoustics, leading to the creation of context-aware systems for related fields, including music and speech technology and hearing aids.

By investigating and developing technologies for sound and audio-based context recognition and by creating new datasets of sound scenes in various environments, this project holds benefits to several groups beyond the academic community, both in the UK and internationally. These include:

Commercial Private Sector:

* Internet of Things companies creating technologies and devices for smart homes and ambient assisted living, through access to sound and environment recognition technologies adaptable to various contexts.
* Audiovisual archiving companies, through technologies for automatic annotation of continuous audio streams with respect to both context and sounds present.
* Companies in the security/surveillance sector, through access to novel algorithms for sound detection and identification in complex and noisy environments.
* Acoustics and sound engineering companies, through new acoustic measurement methods for automatic environment recognition and noise measurement.
* Composers and sound artists, who will benefit from a new set of digital tools for exploring soundscapes and sounds, and using them in their creative output.

Public and third sectors:

* Libraries, museums and public archives hosting, preserving and curating sound collections, through new methods for automatic organisation, exploration and annotation of audio streams and recordings.
* Urban planning authorities and smart city developers, through new methods and tools for automated environmental and noise monitoring.
* Independent organisations promoting STEM (science, technology, engineering and maths) to young people and underrepresented groups, by using sound and audio to generate interest in STEM careers.

Wider public:

* Users of sound recognition technologies, through smart home applications or mobile devices which enable sound capture and processing.
* Users of public and private sound collections and archives, through new technologies for exploring and annotating sound data.
* Residents of urban areas, as beneficiaries of audio-based smart city technologies resulting in lower noise pollution.
* Audiences of artistic output that involves the use of audio technologies related to sound recognition and exploration.
* Teachers and students in technical and creative fields related to sound, involving the use of audio technologies or sound examples.