Multi-modal cue integration for auditory spatial location by normal-hearing and hearing-impaired listeners.

Lead Research Organisation: University of Nottingham
Department Name: School of Medicine

Abstract

A sharp sense of space is essential to making the world appear natural, real and joined-up. People with normal hearing can do exceptionally well in experimental tasks that measure the sense of space - someone's "spatial hearing". Yet hearing impairment generally worsens spatial hearing, leaving only an impoverished impression of auditory space. Hearing aids hardly ever help spatial hearing. Worse, they can often add further problems. But despite decades of research the science of spatial hearing is still not solved, especially for complex situations with multiple sounds happening simultaneously or in understanding the effects of impairment or aiding. This programme will help answer these puzzles.

There are two overall benefits to listening from having acute spatial hearing. Both take advantage of that fact that in everyday listening sound sources are almost always in distinct physical locations. First, it allows sound sources to be perceived in the locations that they really are. Second, it allows sounds to be separated by virtue of being in different positions. These two together give a sense of realism to the auditory world and help identifying, recognizing or understanding sounds. These reduce the "clutter" of hearing in busy, noisy situations: without any spatial hearing at all, everything would be more likely to be heard together as a jumble.

The situation is complicated by there being multiple cues to the direction and distance of a sound source. One is time: the sound from a source to the left will arrive a fraction of a second later at the right ear compared to the left as it has to travel some 50 cm further around the head. Another is intensity: the sound is less powerful as your head casts an acoustic "shadow". Yet despite these multiple cues we generally hear one location for a sound, not many. Further, what we see can affect where we hear a sound -- a famous example of this is the ventriloquist effect. The problem of generating accurate perceptions of auditory space is, overall, akin to solving a three-dimensional jigsaw of multiple overlapping sounds, time delays, and power differences. Somehow our perceptual systems seamlessly and apparently effortlessly solves the puzzle of putting all the pieces together properly.

In this programme, we will use cutting-edge auditory experiments to answer two key questions. First, how does the auditory system join the multiple cues to location in complex, dynamic, multi-sound, audio-visual listening situations? Second, how does hearing impairment and aided listening affect this? We expect that the insights gained in this programme will help us to understand better how spatial hearing works in real, everyday listening, and will help inform how future hearing aids might be designed to improve spatial hearing

Technical Summary

A faithful perceptual representation of the spatial position of sound sources is fundamental to making sense of the acoustic world. It is required to give a real appearance to the auditory scene, and helps considerably in separating sounds and enhancing listening. But hearing impairment worsens spatial perception, reducing or removing these benefits, and hearing aids can actively interfere with it. The perceptual problem of generating the representation is particularly complicated as there are multiple separate cues to location - not only auditory but also visual and vestibular - yet somehow these are combined seamlessly. Quite how is unclear. The overarching purposes of this programme are therefore to determine (1) how this integration of information, across frequency, time, space and audio or visual modality is done and (2) how it is affected by hearing impairment and aided listening.

We will use innovative auditory & audio-visual psychophysical experiments to recreate the real world in the laboratory, emphasizing complex scenes with multiple objects. These will exploit figure-ground separation, so with the same stimuli we can quantify the contribution of spatial cues to both a figure's auditory position and its distinctiveness. We will also exploit front-back indeterminacy to study the effects of vision. Data will be collected using methods such as discrimination of position, subjective reporting of position, and orientation tracking to changes in position.

The research will help deliver our long-term aim of developing a fully detailed scientific account of spatial hearing in everyday, noisy listening: i.e., how someone's perception of a sound source's position is determined from the source's actual position in the world. Such a theory will help identify new phenomena which future hearing devices can exploit or mitigate.

Planned Impact

We will seek to continue IHR's excellent track record of disseminating our scientific results and knowledge to wider communities (eg http://www.100yearsofamplifiedmusic.org/). In this programme we plan to deliver two non-academic impact goals: (1) increase public awareness of hearing and hearing impairment in general and spatial-hearing in particular, and (2) interact with hearing-device developers so that our science can lead to improved future devices that benefit spatial hearing instead of interfering with it.

To do these we will engage three audiences:

A. Health Engagement: We expect that the insights found in this programme will be of interest both to many of those with hearing impairment and to many health-professionals in audiology and ENT. Though the results will not lead to an immediate improvement in hearing aids (but see Industry Engagement below for downstream effects) we expect that they will be of background benefit through increasing their understanding of hearing and the emphasis on its problems (goal #1).

B. Industry Engagement: Our results will be of great interest to the hearing-device industry, as both improving the spatialness of sounds delivered by hearing aids and developing visually-aware hearing aids are current foci of industrial research. We confidently hope that this programme will feed into improvements in future hearing devices, by identifying new phenomena which they can exploit or mitigate and indicating how hearing aids might be designed to improve spatial hearing (goal #2)

C. Public Engagement: Our research will give new scientific knowledge of how spatial hearing works and how it is affected by hearing impairments or hearing aids. Our research will add to the wider public's knowledge of hearing. It will also encourage public awareness and discussion of just how important hearing impairment is as a health issue (goal #1).
 
Description Challenges To Revolutionise Hearing Device Processing
Amount £251,509 (GBP)
Funding ID EP/S031308/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 08/2019 
End 08/2024
 
Title Binaural cross-correlogram tookbox 
Description This is a re-release of a suite of software for supporting binaural research 
Type Of Material Improvements to research infrastructure 
Year Produced 2018 
Provided To Others? Yes  
Impact its used by quite a few groups around the world 
URL https://rdmc.nottingham.ac.uk/handle/internal/324
 
Title Hearing in Adults 
Description The 1011-page book, Hearing in Adults, published in 1995, contains the fullest report of the United Kingdom's Medical Research Council National Study of Hearing. It was designed to determine the prevalence and distribution in Great Britain of audiometrically measured hearing loss as a function of age, gender, occupation, and noise exposure. The study's size, quality, and breadth made it unique when it was done in the 1980s. These qualities remain, and its data are still the primary U.K. source for the prevalence of auditory problems. However, only 550 copies were printed, and the book is essentially unobtainable today. We have made a fully searchable, open-access, digital (PDF) "reprinting" of Hearing in Adults, summarizing the study's design and the book's contents, together with a brief commentary in the light of subsequent developments. 
Type Of Material Database/Collection of data 
Year Produced 2020 
Provided To Others? Yes  
Impact not yet 
URL https://journals.sagepub.com/doi/full/10.1177/2331216519887614?url_ver=Z39.88-2003&rfr_id=ori:rid:cr...
 
Description CLARITY 
Organisation University of Sheffield
Country United Kingdom 
Sector Academic/University 
PI Contribution To create and run three competitions to revolutionise hearing-aid processsing
Collaborator Contribution Running hearing tests
Impact none yet
Start Year 2019
 
Description COG-MHEAR 
Organisation Edinburgh Napier University
Country United Kingdom 
Sector Academic/University 
PI Contribution This is a new interdisciplinary collaboration to research the next generation of hearing aids
Collaborator Contribution Chairing ther user group
Impact None yet
Start Year 2020
 
Description Honorary appt - NAL 
Organisation National Acoustics Laboratory
Country Australia 
Sector Academic/University 
PI Contribution We have started a long-term collaboration with leading scientists at NAL, Sydney
Collaborator Contribution Expertise in hearing-aid devices, fitting and services
Impact none yet
Start Year 2017
 
Description Multimodal spatial hearing 
Organisation Newcastle University
Country United Kingdom 
Sector Academic/University 
PI Contribution I am leading a collaboration looking at how cross-modal influences affect auditory spatial location. This has resulted in a new MRC programme grant
Collaborator Contribution My collaborators' contributions are in expertise of vision, selective attention, and spatial hearing
Impact None yet
Start Year 2017
 
Description Multimodal spatial hearing 
Organisation Vanderbilt University
Department Vanderbilt Medical Center
Country United States 
Sector Academic/University 
PI Contribution I am leading a collaboration looking at how cross-modal influences affect auditory spatial location. This has resulted in a new MRC programme grant
Collaborator Contribution My collaborators' contributions are in expertise of vision, selective attention, and spatial hearing
Impact None yet
Start Year 2017
 
Description Institute of Acoustics visit 
Form Of Engagement Activity Participation in an open day or visit at my research institution
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Professional Practitioners
Results and Impact Regional members of the Institute of Acoustics visited the lab for an evening of talks and demos.
Year(s) Of Engagement Activity 2020
 
Description IoA - London 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Industry/Business
Results and Impact I gave a presentation at the London branch of the Institute of Acoustics on my research
Year(s) Of Engagement Activity 2019
 
Description Laurel vs Yanny 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact I was interviewed by BBC Coventry & Warwickshire local radio about the "Laurel vs Yanny" phenomena
Year(s) Of Engagement Activity 2018
 
Description invited talk at Birkbeck 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact A gave an invited lecture on a "Masterclass" to UK audiology students
Year(s) Of Engagement Activity 2018
 
Description invited talk at Bruges 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact I gave an invited talk at an international audiology workship, mainly aimed at practiconers
Year(s) Of Engagement Activity 2018