Designing better hearing aids using physiologically inspired speech enhancement

Lead Research Organisation: University of Southampton
Department Name: Faculty of Engineering & the Environment

Abstract

Hearing aids can profoundly transform the lives of people with hearing impairments but in the UK alone about 6 million such people do not use them. An important reason for this is that conventional hearing aids don't make it easy to distinguish meaningful signals such as speech from background noise. The combination of a fully functioning ear and the brain are fantastically good at this job and easily outperform the best computerized speech-recognition apps when even just a small amount of noise is present. A conventional hearing aid amplifies both the speech and noise indiscriminately, so even though the neural pathways of the brain may be unimpaired the task of distinguishing speech from noise becomes much harder. Various approaches to automatic speech enhancement have been tried but none comes close to what nature can do.
Despite the demand for better solutions, recent progress in research and development has been slow and no breakthrough technology has yet emerged. Speech enhancement strategies have been generally developed on the basis of mathematical, but not physiological principles. These methods, although based on fundamentally different strategies, have two things in common: first, they operate on signal features that rely primarily on the signals' local energy, and second, they have not improved speech intelligibility. Here, we propose overcoming this conceptual barrier by developing engineering solutions to the speech-in-noise problem that are based on physiological principles. The same technology will also be of benefit for automatic speech recognition systems, since the problems of both are similar.
We expect to see direct applications of our work to be implemented in hearing aids within the next 5 years. Two of the biggest companies in the world in their field (Siemens for hearing aids and Google for signal processing) demonstrate the interest, support and the confidence toward this approach. We have substantial experience with the whole development cycle: we have invented, designed, evaluated and implemented a noise reduction scheme that will be part of the next generation of Cochlea Ltd. cochlear implants.
Our central hypothesis in this proposal is that the brain uses sparse coding when distinguishing meaningful signals from noise and it uses a dynamic dictionary for sound representation. We are going to investigate this coding mechanism in individual neurons in the auditory brainstem, and based on the results, will develop novel signal-processing strategies. We expect that these algorithms will be better than conventional algorithms and consequently can help hearing impaired. An animal model is essential to this project, because it is impossible to study responses of individual neurons from the auditory brainstem in humans. Sparse
Neuron adapt their response because they have a limited dynamic rate which they constantly optimize in response to the environment in order to reduce redundancy and to maximise the information flow. In this project, we are going to extend the description of static neural response patterns to include a time varying and context sensitive components and we will measure these dynamic responses in single neurons in the brain stem. Knowing how neuronal responses change in noise will enable us to create a dynamic dictionary that can be used for sparsification. We expect that in this representation speech and noise is separable.
We will use these dynamic response pattern as the basis of a novel transformation and sparsification in order to enhance the components of speech that are relevant for understanding, thus improving speech intelligibility without reducing the quality.
We will evaluate the algorithm in substantial clinical trials.

Planned Impact

The total annual cost of hearing impairment in the EU has been estimated to be Euro224 Billion (EuroQol: Euro16K/QALY). In the UK alone 6 million more people could benefit from a hearing aid (HA) (£18 Billion p.a.). These costs are mainly a consequence of economical inactivity due to hearing loss.
We have designed a programme that will lead to better hearing aids. We present a programme that integrates physiological recordings, signal processing and evaluation in a coherent work plan. Letters from two of the biggest companies in the world in their field (Siemens for hearing aids and Google for signal processing) demonstrate the interest and support that the world leading industry has toward this approach. We therefore expect to see direct applications of our work to be implemented in hearing aids within the next 5 years.
Results of this research are designed to break this deadlock and create the impulse for a step change in the signal processing research. Application of the results of this research can therefore lead to improved signal processing algorithms that will enable hearing aid users to better communicate. The results of this project can directly improve the quality of life for 250 million people around the world.
Hearing aids are a market worth around billions world-wide and progress in signal processing is closely watched by the hearing aid industry. Siemens, one of the biggest hearing aid company in the world has expressed the confidence in our leadership and their interest in this work. Siemens, as well as other hearing aid companies have a direct interest to improve hearing aids to remain competitive in the market and they have a track record of quickly implementing novel signal processing strategies. We are aware of IP-issues, and we have established good communication channels to avoid conflicts in future. It is planned to patent results as soon as possible. IP will remain with the University of Southampton and the University of Cambridge.
The proviion of hearing aids is a big part of the NHS budget and attempts are made in various ways to reduce costs. The policy of central tendering by the NHS supply chain enables the cost effective provision of huge numbers of hearing aids, however, a big burden on the cost in this package is that large numbers of aids are not used by wearers after disappointing results soon after consultation and allocation. There is therefore a huge potential for higher efficiency by improving the quality of aids with the side effect of increased hearer satisfaction and time of wearing.
The programme is aligning technical developments with matching experimentation. These experiments will result in useful data for the development for novel signal processing algorithms and can be used in many labs all over the world. The following research streams are also interesting for a variety of researchers:
- The sparse coding algorithm is interesting for several interest groups. It can be used in hearing aids, but also in cochlear implants, signal compression schemes and communication devices like phones. Google, one of the biggest companies in the world describes in their letter of support how important new research approaches in this area are.
- The neural data collected will be extremely interesting to neuroscientists interested in sensory neural coding and these experiments will be closely scrutinized by the neuroscience community worldwide.
This programme is of an essential interdisciplinary nature. The success depends crucially on the demonstrated quality and variety of the background of its members. The success of the programme will be measurable by the number of publications in peer reviewed journals and conference presentations of results in national and international conferences.
 
Description The key findings in short are:
1) We have developed a deep neural network, based on our understanding of the human auditory periphery, that can enhance speech for hearing impaired people in noisy situations. We have tested the network it in CI users and hearing impaired. Both experiments were successful and published in peer reviewed papers. Our industrial partner is investigating using the network in their products, potentially helping millions of CI users in the future.
2) we have gained a better understanding of the neural responses in the auditory brainstem where we gained not only insights into single neural responses to complex sounds, but we also suggested a novel classification scheme based on these insights. The findings were submitted to a peer reviewed journal.
3) we have investigated a novel method of speech enhancement based on sparse coding that was directly informed by responses of auditory brain stem neurons. This algorithm was tested and proved not as successful as we hoped.
Exploitation Route We have developed a much more systematic understanding of the responses in the auditory brain stem to complex sounds. Our methods of analysing the responses in form of spatio-temporal receptive fields has been published and is already influencing the way other labs are describing neuronal responses. Furthermore, we have established a novel way of characterising neuronal response classes in the auditory brainstem based on a new technique using intervals rather than averaged spike times. We expect that this method will gain track in future and will become the (or at least part of the) standard way of classifying neuronal responses.
Sectors Digital/Communication/Information Technologies (including Software),Healthcare

 
Description Our development of a deep neural network that can enhance speech in noisy environments has been shown to help cochlear implant users. This method has been recognised as the most promising signal processing method to help hearing impaired people to communicate better in noise. Hearing aid companies, cochlear implant companies and producers of 'audibles' are investigating this method now, and we helped improving and publicizing this.
First Year Of Impact 2015
Sector Healthcare
 
Description AVRITH TRAVEL GRANT SCHEME
Amount £500 (GBP)
Organisation University of Cambridge 
Sector Academic/University
Country United Kingdom
Start 02/2014 
End 02/2014
 
Description Deafness Research UK ARO TRAVEL FELLOWSHIPS
Amount £500 (GBP)
Organisation Action on Hearing Loss 
Sector Charity/Non Profit
Country United Kingdom
Start 02/2013 
End 02/2013
 
Title Open Development Platform 
Description In auditory signal processing research, we are often interested in algorithms that modify audible sounds, for example to increase speech intelligibility. Oftentimes, it is interesting and sufficient in an experiment to process such sounds offline and play them back to participants at a later stage. This approach has the advantage of full control over all parameters including background noise. However, in many situations it would be beneficial to have a more direct approach, by being able to manipulate sounds in real time. Real time processing of environmental sounds means giving up on controlling all environmental parameters, but gaining highest possible ecological validity, that is to test the participant in unpredictable situations as they would face in real life. This approach is important for example for hearing aid noise reduction algorithms, where the wearers outside of the lab encounter unpredictable situations. In a typical development cycle, an algorithms' benefit must first be demonstrated in laboratory situations, before going real time. However, only when knowing the interaction between algorithm and participant in real live situations, the algorithm can be fully evaluated for it's usefulness. In fact, most algorithms that have shown benefit in laboratory situations were either never evaluated in real time environments or failed to live up to their promise. However, in order to develop algorithms that have potential to be implemented in the next generation of auditory devices, they must be tested in all situations, but testing real time algorithms in real live situations is difficult for many reasons. Firstly, programming them is difficult, as the necessary programming skills are not the same as for offline programming. Secondly, it often requires specialised hardware that is suited and fast enough to do the processing. For signal processing research it would be beneficial to have a platform that helps to bridge the development from existing offline algorithms to real time. proposed model: We present here a comprehensive solution: a platform that enables current users of matlab to quickly port their algorithms to real time. We provide an open library that can be used freely and that will grow in future. The algorithms can run on standard PC hardware or on special hardware when required. The platform is the result of a European Training network (www.icanhear.eu), and all seven network partners have contributed to the library. The software is distributed freely via soundsoftware.ac.uk, where the library as well as supporting documentation can be downloaded for free under a fair use licence. The library is open, that is, everybody (who is registered on the web page) can add algorithms. 
Type Of Material Computer model/algorithm 
Year Produced 2015 
Provided To Others? Yes  
Impact The platform is publicly available on the internet (www.soundsoftware.co.uk) Other groups have tested our platform and investigate at the moment if they are going to use it. 
URL https://code.soundsoftware.ac.uk/projects/odp-open-developer-platform-for-auditory-research
 
Description Industrial collaboration with Cochlear 
Organisation Cochlear Ltd.
Country Global 
Sector Private 
PI Contribution we have established a collaboration with the Cochelar implant company "cochlear". We are developing signal processing algorithms that can reduce noise and improve speech. They are interested in finding new ways to improve speech intelligibility for CI users. Under the leadership of me and our PostDoc in Southampton (Dr Jessica Monaghan) we have developed a neural network based algorithm that is now tested in CI patients. This might lead to a new system and patents.
Collaborator Contribution we gave Cochlea a novel algorithm Cochlea is undertaking the Psychophysical measurements
Impact several reviewed conference papers (see publication list).
Start Year 2014
 
Description British Science Festival 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? Yes
Geographic Reach Regional
Primary Audience Schools
Results and Impact more than 100 school children attended at our stall at the festival.
We organized three interactive hands on workshops where people could experience hearing loss, hearing aids or cochlear implants.

many kids were astonished to hear about the impact of hearing loss on their lives or lives of relatives.
Year(s) Of Engagement Activity 2014
 
Description University of Southampton Science and Engineering day 
Form Of Engagement Activity Participation in an open day or visit at my research institution
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact this is an outreach activity where all of our postdoctoral staff participates. We reach >100 people, mostly children and inform about the dangers of hearing loss.
Children and adults alike are often very interested and report a change in behaviour in that they will be more careful with noise exposure.
Year(s) Of Engagement Activity 2014,2015,2016