The neural representation of vocal emotion: representational similarity analysis and information-theoretic approaches

Recognizing and interpreting emotions in other persons is crucial for social interactions. In particular, people of all cultures are able to recognize emotions in vocalizations without speech such as laughs, cries or screams of fear. But how our brain analyses emotion in voices remains poorly understood, compared to how we perceive emotion in faces, for example.
In this project we bring together several expert co-investigators and external collaborators to address two new streams of unanswered questions on the way our brain processes emotional information in voice. We will use several complementary neuroimaging techniques to measure cerebral activity with the best precision currently possible, and advanced analysis techniques to ask two main series of questions:
(1) Does the brain processes emotions as different categories (e.g., fear, anger, pleasure) or rather as a continuum emotions each characterized by different amount of activation (arousal) or how negative/positive the expression is (valence)?
(2) How do different nodes of the oscillatory network of brain regions responsible for analysing emotion in voice interact with one another? Do they share information between nodes in a synergistic way, or is there redundancy between the different nodes? And if yes, how, where and when is emotional information combined to arrive at the recognition of a specific emotion?
The results from this project will bring important new knowledge on how the normal brain works, with longer-term impact on the treatment and diagnosis of communication disorders and the development of better neural prostheses such as cochlear implants. The project will also contribute to consolidating and preserving UK's leading position in advanced neuroscience research.

A growing number of studies have contributed to an increasingly detailed picture of the network of cerebral regions involved in this task. Yet, the exact nature of information processing at the different nodes of the network and their associated representations remain largely unexplored.
In this project we bring together several expert co-investigators and external collaborators to address two new streams of unanswered questions on the way our brain processes emotional information in voice. We will use several complementary neuroimaging techniques to measure cerebral activity with the best precision currently possible, and advanced analysis techniques to ask two main sets of questions:
(1) How does the representation of the emotional stimuli evolve from acoustical-based at early levels of processing to more emotion-based allowing conscious emotional categorization at higher processing stages?
(2) How do different nodes of the oscillatory network of brain regions responsible for analysing emotion in voice interact with one another? Do they share information between nodes in a synergistic way, or is there redundancy between the different nodes? And if yes, how, where and when is emotional information combined to arrive at the recognition of a specific emotion?
To address question 1, we propose to use Representational Similarity Analysis (RSA) as a powerful framework for bringing together multiple types of measures (acoustical, perceptual, cerebral) and models (computational, theoretical) in a comprehensive evaluation of representations at different stages of cerebral processing.
To address question 2, We propose to use novel information-theoretic methods to first individuate network nodes and oscillatory components of activity carrying affective information, then to assess whether combinations of those information-bearing elements carry affect information that cannot possibly obtained by a single element alone (synergy).

Outside of academia, the research is likely to have positive impact on several user groups in the longer term. Person with auditory perception deficits, such as cochlear implant patients and persons with hearing aids experience difficulties recognizing affective states in voice, with negative impact on their social interactions. As good social interactions are a primary determinant of healthy ageing, contributing to a better perception of affective states is likely to have positive outcomes on interactions and quality of life in older populations. Unfortunately, manufacturers of hearing aids and cochlear implants have largely concentrated so far on enhancing speech intelligibility - for obvious reasons. But other, socially-central aspects of vocal communication, such as the detection and recognition of affective states, have been comparatively neglected. The research we conduct in normal participants has the potential to be translated into clinical practice by providing mechanisms to enhance algorithms for auditory decoding in cochlear implants, or optimizing training and rehabilitation strategies. We have experience enhancing the impact of our work through links with industry. We have had funded collaborations with France -Telecom and with Cochlear, one of the leading cochlear implant manufacturers, and this research could lead to further similar collaborations in the longer term.
Another potential pathway to impact in this research is to develop links with the growing industry of 'social computing'. Our results have the potential to be of interest for designers of software for automated extraction of affective states. Automated recognition of speech is now at a commercial stage, but automated recognition of affective states for more socially rewarding human-computer interactions is still poor. Information on the solutions to this problem found by our brain over millions of years of evolution could potentially give important clues to the design of more parsimonious, and especially more robust to degradation, recognition systems. Links will the industry in this domain will be specifically sought for with assistance from the Research and Enterprise office e at university of Glasgow.
Another important user group potentially affected by the research is the wider public. There is an enormous public and media interest in the voice and social interactions. Therefore we have had in general an excellent response to our activities to engage the public in our work. The laboratory has engaged with the public at several occasions such as at the Glasgow Science Centre during Brain Awareness Week. In addition to publishing in academic journals and presenting at Scientific Conferences, we engage with the press to improve the impact of our findings. The PI has received specific training with the media thanks to the BBSRC-organized Media Training Course, which has been instrumental in enhancing the profile of the laboratory's research (several journal and radio appearances in the past year, including the Sunday Telegraph and BBC4).
Finally, the research will have an important impact on the career of Drs Emilie Salvia and Bruno Giordano. Both will greatly enhance their knowledge, CV and visibility as part of the project and would be in a competitive position on the academic market at the end of the grant period.