Understanding speech in the presence of other speech: Perceptual mechanisms for auditory scene analysis in human listeners

We take it for granted that we can converse with other people in daily life and be understood with little, if any, noticeable effort. However, it is fairly unusual to hear the speech of a particular talker in isolation; speech is typically heard in the presence of interfering sounds, such as the voices of other talkers. The human auditory system, which is responsible for our sense of hearing, therefore faces the challenge of identifying which parts of the sounds reaching our ears have originated from the same environmental source. This involves separating those sound elements coming from one source (e.g., the voice of one talker) from those arising from other sources, and grouping them in ways that can be interpreted by higher-level processes in the brain (such as those involved in our understanding of speech). Without a solution to this "auditory scene analysis" problem, our perceptions of speech (and other sounds) would not correspond to the events that produced them. Humans have been exposed to a variety of complex listening environments over the course of evolution, and so we are generally very successful at understanding the speech of one person in the presence of other talkers. This contrasts with attempts to develop listening machines, which often fail catastrophically when confronted with complex listening environments, such as an open-plan office or a crowded party. Human listeners with hearing impairment also find these environments very difficult, even when using the latest developments in hearing-aid or cochlear-implant technology.

So far, most research on auditory scene analysis has focussed on relatively simple sounds and has identified a number of general principles for the perceptual grouping and separation of sound elements. However, at least as currently understood, these principles seem inadequate to explain fully the perceptual grouping of speech. This is because the speech signal consists of a diverse and rapidly changing stream of sounds. The speech of our native language is also a highly familiar stimulus, and so by adulthood our auditory system has had many years to learn about its speech-specific properties. These properties may also assist in the successful perceptual grouping of speech.

Much of the information necessary to understand speech is carried by the changes in frequency over time of a few broad peaks in the frequency spectrum of the speech signal, known as formants. The aim of this project is to investigate how human listeners presented with speech sound mixtures are able to group together the appropriate formants, and to reject others, such that the speech of the talker we want to listen to can be understood. We will do so using perceptual experiments with human listeners, in which we measure how the intelligibility of target speech (measured, for example, as the number of words reported correctly) changes under a variety of conditions. The project will explore the roles of general-purpose grouping factors (i.e., those that apply to a wide variety of sounds) and of speech-specific grouping factors, including higher-level constraints associated with the articulation of speech (i.e., the way our tongue, lips, and jaw move when we speak) and with the rules of our language. Our approach is to generate artificial speech-like stimuli with precisely controlled properties, to mix target speech with carefully designed "competitors" that offer alternative grouping possibilities, and to measure how manipulating the properties of these competitors affects the ability of human listeners to recognise the target speech in the mixture. The results of this project will not only improve our understanding of how human listeners separate speech from interfering sounds, but will also help to refine computer models of listening. Such refinements will in turn provide ways of improving the performance of devices such as hearing aids and automatic speech recognisers when they operate in noisy environments.

Private-sector companies who develop robust automatic speech recognition (ASR) devices and techniques for speech enhancement:

Robust performance in unpredictable noise remains a key problem for ASR and mobile telephony. The project will provide perceptual data needed by researchers in computational auditory scene analysis (CASA) to develop further models and algorithms for separating speech from interfering sounds. This in turn offers the prospect of improved front-end processors for ASR and speech enhancement systems, which is likely to improve the performance of commercial systems. Our industrial advisor, Audience, is one of the leading commercial developers of speech enhancement technology. Our research findings will be communicated to Audience via email and at regular review meetings, timed to coincide with jointly attended international conferences. A report will also be made available to other key companies in ASR and speech enhancement technology.

Private-sector companies who develop hearing aids and cochlear implants:

Effective separation of a target voice from interfering sounds is one of the key problems facing designers of hearing aids and cochlear implant (CI) processors. Our findings on the perceptual grouping of formants will potentially improve the coding of speech in CI processors. The first benefits of enhanced CASA solutions for improving hearing aids and CI processors are likely to emerge after about 5 years, which is typical for development lead-time of such devices. Our findings will be communicated to hearing aid and CI manufacturers by sending them a bespoke report, by visits to selected companies, and meetings with their representatives at conferences.

Charities that support the hearing impaired:

Our research findings will be communicated to major UK charities that support the hearing impaired (Action on Hearing Loss, Deafness Research UK). The improved understanding of the perceptual processes underpinning successful auditory grouping will encourage the development of collaborative psychophysical and modelling studies. Such cross-disciplinary interaction is likely to stimulate the funding by these charities of research projects on advanced hearing aids and CI algorithms. Our findings will be communicated to hearing charities via a report tailored to their interests, and by conference presentations.

The general public:

Indirectly, the impact on the beneficiaries listed above will also benefit the general public within a timescale of 5-10 years. Improvements to hearing aid technology will benefit the estimated 9 million deaf and hard-of-hearing people in the UK. Similarly, there are about 180,000 CI users worldwide who would benefit from better techniques for encoding noisy speech in CI processors. Improved ASR and speech enhancement for mobile telephony, which may arise from our project via its impact on enhanced CASA solutions, will impact on quality of life through improved speech-based communication with machines, and enhanced electronically-mediated vocal communication between individuals. The outcomes of the project will also be communicated directly to the public via a website, which will include a lay summary of our findings, and by public lectures.

The research fellow:

Summers will receive communication skills training from the Royal Society to prepare him for public lectures, report writing and company visits. This will provide transferable skills that are relevant to many professional careers. His involvement in writing technical reports and visits to leading hearing-technology companies will foster links with the commercial sector likely to enhance the prospect of future employment either in academic or industrial research.

The project team has the expertise needed to implement this impact plan. We have already established contacts with CASA researchers and with major commercial organisations with interests in ASR, mobile communications, and hearing prostheses.