CHIME: Computational Hearing in Multisource Environments

In everyday environments it is the norm for there to exist multiple sound sources competing for the listener's attention. Understanding any one of the jumble of sounds arriving at our ears requires being able to hear it separately from the other sounds arriving at the same time. For example, understand what someone is saying when there is a television on in the same room requires separating their voice from the television audio. The lack of an adequate computational solution to this problem prevents hearing technologies from working reliably in typical noisy human environments -- often the situations where they could be most useful. Computational hearing algorithms designed to operate in multisource environments would enable a whole range of listening applications: robust speech interfaces, intelligent hearing aids, audio-based monitoring and surveillance systems.The CHIME project will develop a framework for computational hearing in multisource environments. Our approach operates by exploiting two levels of processing that combine to simultaneously separate and interpret sound sources. The first processing level exploits the continuity of sound source properties, such as location, pitch, and spectral profile, to clump the acoustic mixture into pieces (`fragments') belonging to individual sources. Such properties are largely common to all sounds and can be modelled without having to first identify the sound source. The second processing level uses statistical models of specific sound sources expected to be in the environment. These models are used to separate fragments belonging to the acoustic foreground (i.e. the `attended' source) from fragments belonging to the background. For example, in a speech recognition context, this second stage will recruit sound fragments which string together to form a valid utterance. This second stage both separates foreground from background and provides an interpretation of the foreground.The CHIME project aims to investigate and develop key aspects of the proposed two-level hearing framework: we will develop statistical models that use multiple signal properties to represent sound source continuity; we will develop approaches for combining statistical models of the attended `foreground' and the unattended `background' sound sources; we will investigate approximate search techniques that allow acoustic scenes containing complex sources such as speech to be processed in real-time; we will investigate strategies for trying to learn about individual sound sources directly from noisy audio data. The results of this research will be built into a single demonstration system simulating a home-automation application with a speech-driven interface that will operate reliably in a noisy domestic environment.