Pathways and mechanisms underlying the visual enhancement of hearing in challenging environments.

The perception of speech is of paramount importance for social communication and the quality of our daily life. Yet, many individuals have difficulties in speech comprehension, especially so under conditions where the speech signal is acoustically distorted (e.g. by background noise). Visual information from seeing the speakers face or lip movements can greatly enhance speech intelligibility under such challenging circumstances. Yet, the specific brain mechanisms that render hearing susceptible to such challenges and those that implement the visual facilitation of hearing remain poorly understood.

We here use two critical manipulations of natural sounds, the addition of background noise and changes in natural speech rhythm, to study where and how along auditory pathways the neural encoding of sounds is affected by these acoustic challenges. We then use these manipulations to study where and how proper sound encoding is restored by informative simultaneous visual information. To this end we use a combination of behavioural tests with high-density electric neuroimaging (Magnetoencephalography) and advanced computational signal analysis in adult human listeners, hence taking a systems approach studying large-scale brain activations during sensory perception. Our specific hypothesis and questions are based on recent insights demonstrating the importance of rhythmic patterns of neural activity for auditory perception. We directly investigate how specific signatures of speech encoding in the human brain are affected by degradation of the speech signal and how they are restored by visual information.

Hearing difficulties are a highly common disability in the UK and a global burden to society (WHO), making this an important topic for the biosciences underpinning live-long health. The knowledge gained here is of interest for our basic understanding of the physiological processes underlying hearing and to understand how the brain merges visual and acoustic information to achieve a percept that can withstand challenging sensory environments. The insights gained here will also enhance our understanding of the neural mechanisms responsible for impaired auditory comprehension and hence could provide critical knowledge to facilitate the improvement of auditory prosthetic devices or the training of patients in using them. Finally, understanding the interplay between speech signals and their representation in the brain is of socio-economic interest as it is important for the design of human-computer voice-based interfaces, which are becoming ubiquitous in everyday life.

Speech is paramount for social communication and daily life. Yet, for many individuals hearing is challenged when speech is distorted by background noises or other sound irregularities. Seeing the speakers face greatly enhances intelligibility under such conditions. Yet, it remains unclear where and how visual information enhances auditory processing. Recent work described multiple rhythmic processes along auditory pathways as critical markers for speech encoding. Based on these we use a systems-approach combining psychophysics with measurements of brain activity (magnetic-neuroimaging; MEG) in adult human listeners to study this timely problem. We implement two critical sound manipulations that challenge intelligibility: addition of background noises and changes in speech rhythm. Using data-driven computational analysis we contribute to a more detailed understanding of those cortical mechanisms that allow vision to enhance hearing. We test specific questions relating to the overall hypothesis that vision facilities hearing by restoring the tracking and encoding of acoustic features by specific rhythmic auditory processes that are affected by our realistic manipulations of sound intelligibility. Using high-density MEG we localize these processes to specific areas and study their functional interactions and relation to perception. Hearing difficulties are a frequent disability in the UK and a global burden to society. The proposed work provides basic knowledge about the physiology of hearing, the neural processes relevant for speech intelligibility, and the mechanisms of multisensory perception. Our work will provide knowledge to better understand hearing in challenging environments, the mechanisms involved in hearing deficits or to improve the function of auditory prosthesis and speech-based human-computer interfaces. In addition we enhance computational tools and analysis methods for data-driven neuroimaging studies.

Academic impact:
The proposed work is fundamental neuroscience research and will produce key insights into the functional organisation of sensory pathways and the mechanisms underlying auditory perception and multisensory integration. While the proposed research addresses basic issues with little immediate potential for direct commercial exploitation, it offers critical knowledge relevant for academic researchers in neuroscience and neurophysiology, psychology, systems and clinical neuroscience and researchers and engineers focusing on the development of auditory prosthetic devices or human-computer interfaces. Specifically, we see the following academic impact:
1. Enhancing the knowledge economy by dissemination of results at national and international meetings and publishing papers in peer-reviewed journals. We target the relevant subject areas in neuroscience and engineering and we have a track record in high impact publications and review journals, making this impact very likely.
2. Enhancing collaborations for multidisciplinary research. This project will yield a huge amount of data that can be used for additional analysis and to inspire and inform computational and theoretical neuroscience. We are continuously collaborating with computational neuroscientists to refine our analysis and to share data. Both data and algorithms will be made available on request to others.
3. Delivering a highly skilled researcher. This project will require both advanced experimental and data handling skills. By training a research assistant at the postdoc level, this project will deliver a highly skilled researcher for future neuroscience, clinical research or bioinformatics.
4. Delivering tools for bioinformatics and computational approaches. Systems and data driven approaches to neuroscience provide considerable challenges for data mining and interpretation. Our work continuously refines existing and develops novel methods for data analysis in computational neuroscience. These will be described in open access publication (where possible), shared with others on request or may be posted on our laboratory website.

Economic and Societal Impact:
1. Engaging with industry and clinical science
This project can create links to markets of neurotechnology, especially the development of neuroprosthetic implants for hearing deficits. We have already engaged with the newly formed 'Scottish Hearing Research Network' and we are envisaging a closer engagement with clinically-oriented scientists at the Scottish branch of the MRC Hearing Research Institute. This provides multiple points of contact for the exchange of knowledge with clinically and engineering oriented scientists and potential industrial partners.
2. Enhancing public awareness of hearing
Because this project and other ongoing projects in our laboratory contribute to a better understanding of hearing, we will contribute to enhancing public awareness of the importance of hearing and hearing disorders. Our university is regularly participating in events on public engagement (e.g. Brain Awareness Week by the Dana Foundation, visits to Schools). In addition the PI has a track record of publications in general peer review journals (Nature Reviews Neuroscience, Trends in Neuroscience) and public science journals (Scientific American), which further aids to this impact. Together with media relation officers of Glasgow University we seek the best strategies to disseminate our results to the press and general public.
3. Contributing to 3Rs
Combining this work based on non-invasive neuroimaging in humans in combination with our other work using animal models can contribute to the 'reduction' of the 3Rs. A better validation and understanding of non-invasive tools can further reduce the use of animals in the long term.