Cerebral processing of affective nonverbal vocalizations: a combined fMRI and MEG study.

Recognizing and interpreting emotions in other persons is crucial for social interactions. In particular, people of all cultures are able to recognize emotions in vocalizations without speech such as laughs, cries or screams of fear. But how our brain analyses emotion in voices remains poorly understood, compared to how we perceive emotion in faces, for example. In this project we combine a range of advanced techniques to precisely map the brain network involved in recognizing emotions from the voice and determine its exact time-course. We will first use recent morphing technology to manipulate a database of affective voices in order to generate new vocalizations with more or less intense, possibly ambiguous expressions (e.g., pleasure mixed with fear). We will then measure a number of parameters in the vocalizations thus generated. A large group of listeners will be asked to rate each vocalization on what emotion they think is expressed, on how intense and on how positive/negative they think it is. We will also precisely measure important physical properties of the sounds such as their intensity and pitch. In parallel we will use state-of-the art, complementary brain imaging techniques (functional magnetic resonance imaging and magneto-encephalography) to measure brain activity in a smaller number of participants while they listen to the affective vocalizations and perform simple tasks-a Male/Female gender categorisation task, and a Fear/Anger/Pleasure emotion categorization task. The combination of these two brain imaging techniques will allow measurements of cerebral activity with high time (millisecond) and space (millimetre) accuracy. Analysis of this high-resolution, high density dataset will use the most recent algorithms to address three important, unresolved questions. First we want to differentiate the part of the brain that reacts to the acoustics in the sounds - a vocalization of pleasure sounds different from an angry shout-from that part of the brain that reflects genuine affective value-these two vocalizations express different affective states in the speaker. This important distinction has generally not been adequately addressed in past studies. Second, we want to better understand, in those parts of the brain genuinely related to emotional processing, exactly to which parameter they react: To the emotional category, e.g., a response to fear but not to pleasure? To the negative/positive dimension, e.g., a response to all threatening sounds but not happy or joyful sounds? Or to the task being performed by the subject, e.g., a response during an emotional task but not a gender task? Third, we want to better understand the time-course of processing of affective information at different nodes of the network of brain areas involved in processing these vocalizations. If what has been observed with facial expressions of emotion also applies to voices, then we should observe a very fast, probably unconscious reaction to affective vocalizations, in a 'fast route' that bypasses detailed analysis for the sake of a fast reaction. Advanced algorithms will allow us to determine the precise time course of neuronal activity in different parts of the brain network and understand how different emotional parameters affect the speed of brain reaction. Overall, the results of this project will allow us to understand how the brain processes a socially central dimension of voices-the emotion they carry-and what are the key parameters. They will contribute to the advancement of knowledge, but also in the longer term to a better understanding of impairments of emotion processing in pathologies such as autism or schizophrenia. They also have high potential importance for the growing industry of automated voice processing, as engineers need to know not only how to best automatically recognize emotions in people but also how to best generate realistic emotions in artificial voices.

Recognizing and interpreting emotions in other persons is crucial for social interactions. People of all cultures are able to recognize emotions in affective nonverbal vocalizations such as laughs, cries or screams of fear, but the underlying cerebral mechanisms remain poorly understood compared to those of facial expression processing, for example. In this project we combine functional magnetic resonance imaging (fMRI) and magneto-encephalography (MEG) to precisely map the brain network involved in detecting and recognizing emotions from the voice and determine its exact time-course. We will first use voice morphing to generate, from a validated database, a set of vocalizations with large variation in perceived affective properties, including ambiguous expressions. We will characterize this set of stimuli both acoustically by detailed analyses, and perceptually through an extensive rating study. In parallel we will measure cerebral activity with MEG and fMRI (in separate sessions) in a group of normal adult volunteers while they listen to the affective vocalizations and perform either an implicit (gender) or explicit (affect) categorisation task. Analysis of this high-resolution, high density dataset will address three important, unresolved questions. First we want to dissociate the contributions of acoustical vs. perceptual information on activity at different nodes of the cerebral network involved in processing affective nonverbal vocalizations. Second we want to better understand the influence of several parameters on the activity in different parts of this network: perceived valence and arousal; emotional category and categorical ambiguity; conscious attention to affective content. Third, we want to characterize the time-course of activity at different nodes of this network. In particular we will test the hypothesis of fast processing of affective information suggested by studies of facial expression processing.

Outside of academia, the research is likely to have positive impact on several user groups in the longer term. Person with auditory perception deficits, such as cochlear implant patients and persons with hearing aids experience difficulties recognizing affective states in voice, with negative impact on their social interactions. As good social interactions are a primary determinant of healthy ageing, contributing to a better perception of affective states is likely to have positive outcomes on interactions and quality of life in older populations. Unfortunately, manufacturers of hearing aids and cochlear implants have largely concentrated so far on enhancing speech intelligibility - for obvious reasons. But other, socially-central aspects of vocal communication, such as the detection and recognition of affective states, have been comparatively neglected. The research we conduct in normal participants has the potential to be translated into clinical practice by providing mechanisms to enhance algorithms for auditory decoding in cochlear implants, or optimizing training and rehabilitation strategies. We have experience enhancing the impact of our work through links with industry. We have had funded collaborations with France -Telecom and with Cochlear, one of the leading cochlear implant manufacturers, and this research could lead to further similar collaborations in the longer term. Another potential pathway to impact in this research is to develop links with the growing industry of 'social computing'. Our results have the potential to be of interest for designers of software for automated extraction of affective states. Automated recognition of speech is now at a commercial stage, but automated recognition of affective states for more socially rewarding human-computer interactions is still poor. Information on the solutions to this problem found by our brain over millions of years of evolution could potentially give important clues to the design of more parsimonious, and especially more robust to degradation, recognition systems. Links will the industry in this domain will be specifically sought for with assistance from the Research and Enterprise office e at university of Glasgow. Another important user group potentially affected by the research is the wider public. There is an enormous public and media interest in the voice and social interactions. Therefore we have had in general an excellent response to our activities to engage the public in our work. The laboratory has engaged with the public at several occasions such as at the Glasgow Science Centre during Brain Awareness Week. In addition to publishing in academic journals and presenting at Scientific Conferences, we engage with the press to improve the impact of our findings. The PI has received specific training with the media thanks to the BBSRC-organized Media Training Course, which has been instrumental in enhancing the profile of the laboratory's research (several journal and radio appearances in the past year, including the Sunday Telegraph and BBC4). The research will have an important impact on the career of Dr Patricia Bestelmeyer, the co-investigator research assistant. Dr Bestelmeyer has a growing international reputation in visual and auditory perception and cognition, and she has gained valuable new skills in design and analysis of fMRI data in the PI's laboratory. The proposed project will extend her expertise into a new domain (auditory affective processing) and the use of a new technique (MEG). The project should allow Dr Bestelmeyer to acquire additional experience, research skills and publications, and she should be in a position to apply to faculty positions and start her own research programme as an independent researcher.