Restoring the sense of sound: deep-learning based compensation strategies for the electro-neural transmission of sound by cochlear implants

Cochlear implants provide a sense of sound to people who are severely or profoundly deaf, by electrically stimulating the auditory nerve in place of the damaged sensory hair cells. Cochlear implants are life changing devices and more than half a million people are using a cochlear implant globally. Cochlear implants work well for helping users to understand speech in quiet situations but in noisy environments, such as a busy street or a restaurant, someone using a cochlear implant is likely to struggle to comprehend speech. This is an important problem with negative implications for the quality of life of cochlear implant users. I will help to address this problem by developing a strategy that can identify and enhance speech in background noise. The strategy will build on a multi-disciplinary approach that combines computational models simulating the auditory nerve response to electrical stimulation, with machine-learning algorithms trained to reduce interfering noise. I will use audio data consisting of realistic speech and noise recordings to generate large amounts of training data for teaching the algorithm how to detect and preserve speech in noise. Importantly, the strategy will be optimised for the specific requirements of cochlear-implant users with a computational model that simulates the cochlear implant processing and its stimulation of the auditory nerve. I will design the strategy so that it can be integrated into the external speech processor of a cochlear implant without the need for surgical re-implantation. This project will contribute to a better understanding of the transmission of sound by cochlear implants and help to overcome the communication challenges that cochlear implant users face in their daily lives.

Over 800,000 severe-to-profound hearing-impaired individuals use a cochlear implant (CI) worldwide. Compromised speech perception in noisy environments is a major problem for CI users and can have a negative impact on their quality of life and mental health. Noise-reduction algorithms, that process the corrupted speech signal before being presented, have produced some benefits for CI users but they still struggle in even moderate levels of noise and there remains considerable variability in the benefits across CI users. I will develop a compensation strategy to overcome the limitations of previous approaches by taking into account CI-specific and user-specific effects. The compensation strategy consists of a computational model to simulate the electro-neural transmission of CIs and a machine-learning algorithm trained to reduce the noise component of a speech-in-noise signal. I will develop the strategy in three stages: (I) by incorporating a computational model of the CI processing, (II) by incorporating a computational model of the electrode-nerve interface and (III) by adjusting these models with user-specific parameters measured with electro-physiological and psycho-physical tests. Each stage will be used to generate training data for a noise-reduction algorithm based on deep recurrent neural networks. Hereby, real-world speech and noise recordings will be processed by the computational model to generate labels for the supervised training of the neural networks. Listening experiments will be performed with CI users to evaluate the strategy and its effect on the perception of speech in noise by measuring speech reception thresholds and quality ratings. If successful, the strategy could be integrated into the external speech processor of a cochlear implant without the need for surgical re-implantation. This CI-specific approach has the potential to overcome the current limitations and to provide benefits for CI users.

Who will benefit?

This project will have a major impact on academic and industrial researchers investigating speech-in-noise perception with cochlear implants in the short term. This is an important problem and one of the main challenges for cochlear implant users in day-to-day life. The research will develop new technology based on computational models and machine learning that has the potential to overcome previous limitations and to generate new important knowledge. Along the way, several aspects of the electro-neural transmission of cochlear implants will be investigated that are also of interest to other researchers working on cochlear implants, but who are not directly working on noise-reduction algorithms, such as coding strategies, electrophysiological and psychophysical measures or speech and music perception and appreciation. Outside the field of cochlear implants, the findings will be of interest to clinicians working with cochlear implants or other hearing devices and researchers from related fields such as from auditory science, neuroscience, artificial intelligence and machine learning.
In the longer term, cochlear implant manufacturers will benefit by being able to adjust and improve their products based on the research findings. Most importantly, cochlear implant users will receive direct and indirect benefits from this research.

How will they benefit?

Academic researchers from the auditory field as well as industrial developers and researchers in the hearing industry will be interested in the outcomes of the research as it provides new knowledge about one of the most challenging aspects of listening with a cochlear implant. This is an ongoing research problem in the growing field of cochlear implants which has recently gained new momentum due to the advances in computational modelling and machine learning techniques. Industrial researchers from other fields such as hearing aids, machine perception or speech recognition will be able to draw inspiration from the research findings for their own work. Therefor this research is of strong interest to both academic and industrial researchers that aim to advance the field and to achieve better outcomes for the users. Researchers from the auditory field and the wider research community will also further benefit from the generated knowledge and the sharing of research data and software models. Clinicians and researchers from technical fields will benefit from this research as it will advance the understanding of the limitations of cochlear implants and about the applicability of computational models and machine learning to hearing science. In addition, the anticipated improvement of listening performance by cochlear implant users will help advance the field as a whole and increase the potential for uptake and demand for these devices. Cochlear implant manufacturers (e.g. Advanced Bionics, Cochlear Corp., MedEl, Oticon Medical) will be interested in refining their products in light of the study outcomes to achieve better performance for their users in terms of speech-in-noise perception. Finally, assuming the strategy developed in this project will be successful, child and adult CI users (together with related carers of CI users, including parents and teachers, relatives and friends) will benefit from better speech performance in professional and social situations with background sounds, compared to their performance before the project. A direct benefit will be obtained by the users with the provision of an updated next-generation speech processor (external part of the cochlear implant), without the need for a surgical re-implantation and avoiding the health risk associated with these. Indirect benefits will be received through the advancement of the research field and better insights into how to overcome the limitations.