Using touch to enhance auditory perception

Difficulty in following a single voice or conversation in noisy situations is one of the main frustrations for people with hearing impairment, even when fitted with modern hearing aids. Tactile (touch) sensation combined with sound is widely used in commonplace technology like mobile phones and video game controllers. This research investigates whether tactile stimulation (e.g. vibrations on the wrist), if synchronised with the sound around us, could improve our ability to separate out different sound sources (e.g. voices) in noisy situations. The research aims to demonstrate the benefit of this approach to help people with hearing loss, which leads to significantly reduced quality of life, impacts productivity in working life, affects 1 in 6 individuals in the UK and costs the NHS of 450 million pounds per year.

Sound sources can be separated based on differences in e.g. their pitch, location and timing: people talk at different speeds and start/stop stop talking at different times. One of the main features used to distinguish voices is their so-called fundamental frequency, e.g. male voices tend to sound lower than female voices. We are able to distinguish tactile vibrations in this same range of fundamental frequencies. If some important sound information, perhaps missing due to hearing loss, is automatically transformed into tactile stimulation, could this help to separate one voice from another? This research will use behavioural experiments with carefully designed combinations of sound and touch stimuli to answer this question. It aims to demonstrate the constraints for how this approach works best, firstly for normal-hearing listeners and then for people with hearing loss. The research will be supported by computational modelling to simulate processing in the brain that combines information from our senses of hearing and touch. This will help to link our own findings from behavioural experiments to research with brain imaging and other approaches used to understand how sound is processed with information from other senses.

In summary, the project investigates how tactile sensation affects sound source separation, how its effect is modified with impaired hearing and how information across the senses is combined through computations in the brain. These advances will provide a foundation of knowledge to improve strategies and technology to help with people impaired hearing by supporting the development of hearing aids that are enhanced by touch stimulation, e.g. on the wrist.