The neural representation of vocal emotion: representational similarity analysis and information-theoretic approaches

Lead Research Organisation: University of Glasgow
Department Name: College of Medical, Veterinary &Life Sci

Abstract

Recognizing and interpreting emotions in other persons is crucial for social interactions. In particular, people of all cultures are able to recognize emotions in vocalizations without speech such as laughs, cries or screams of fear. But how our brain analyses emotion in voices remains poorly understood, compared to how we perceive emotion in faces, for example.
In this project we bring together several expert co-investigators and external collaborators to address two new streams of unanswered questions on the way our brain processes emotional information in voice. We will use several complementary neuroimaging techniques to measure cerebral activity with the best precision currently possible, and advanced analysis techniques to ask two main series of questions:
(1) Does the brain processes emotions as different categories (e.g., fear, anger, pleasure) or rather as a continuum emotions each characterized by different amount of activation (arousal) or how negative/positive the expression is (valence)?
(2) How do different nodes of the oscillatory network of brain regions responsible for analysing emotion in voice interact with one another? Do they share information between nodes in a synergistic way, or is there redundancy between the different nodes? And if yes, how, where and when is emotional information combined to arrive at the recognition of a specific emotion?
The results from this project will bring important new knowledge on how the normal brain works, with longer-term impact on the treatment and diagnosis of communication disorders and the development of better neural prostheses such as cochlear implants. The project will also contribute to consolidating and preserving UK's leading position in advanced neuroscience research.

Technical Summary

A growing number of studies have contributed to an increasingly detailed picture of the network of cerebral regions involved in this task. Yet, the exact nature of information processing at the different nodes of the network and their associated representations remain largely unexplored.
In this project we bring together several expert co-investigators and external collaborators to address two new streams of unanswered questions on the way our brain processes emotional information in voice. We will use several complementary neuroimaging techniques to measure cerebral activity with the best precision currently possible, and advanced analysis techniques to ask two main sets of questions:
(1) How does the representation of the emotional stimuli evolve from acoustical-based at early levels of processing to more emotion-based allowing conscious emotional categorization at higher processing stages?
(2) How do different nodes of the oscillatory network of brain regions responsible for analysing emotion in voice interact with one another? Do they share information between nodes in a synergistic way, or is there redundancy between the different nodes? And if yes, how, where and when is emotional information combined to arrive at the recognition of a specific emotion?
To address question 1, we propose to use Representational Similarity Analysis (RSA) as a powerful framework for bringing together multiple types of measures (acoustical, perceptual, cerebral) and models (computational, theoretical) in a comprehensive evaluation of representations at different stages of cerebral processing.
To address question 2, We propose to use novel information-theoretic methods to first individuate network nodes and oscillatory components of activity carrying affective information, then to assess whether combinations of those information-bearing elements carry affect information that cannot possibly obtained by a single element alone (synergy).

Planned Impact

Outside of academia, the research is likely to have positive impact on several user groups in the longer term. Person with auditory perception deficits, such as cochlear implant patients and persons with hearing aids experience difficulties recognizing affective states in voice, with negative impact on their social interactions. As good social interactions are a primary determinant of healthy ageing, contributing to a better perception of affective states is likely to have positive outcomes on interactions and quality of life in older populations. Unfortunately, manufacturers of hearing aids and cochlear implants have largely concentrated so far on enhancing speech intelligibility - for obvious reasons. But other, socially-central aspects of vocal communication, such as the detection and recognition of affective states, have been comparatively neglected. The research we conduct in normal participants has the potential to be translated into clinical practice by providing mechanisms to enhance algorithms for auditory decoding in cochlear implants, or optimizing training and rehabilitation strategies. We have experience enhancing the impact of our work through links with industry. We have had funded collaborations with France -Telecom and with Cochlear, one of the leading cochlear implant manufacturers, and this research could lead to further similar collaborations in the longer term.
Another potential pathway to impact in this research is to develop links with the growing industry of 'social computing'. Our results have the potential to be of interest for designers of software for automated extraction of affective states. Automated recognition of speech is now at a commercial stage, but automated recognition of affective states for more socially rewarding human-computer interactions is still poor. Information on the solutions to this problem found by our brain over millions of years of evolution could potentially give important clues to the design of more parsimonious, and especially more robust to degradation, recognition systems. Links will the industry in this domain will be specifically sought for with assistance from the Research and Enterprise office e at university of Glasgow.
Another important user group potentially affected by the research is the wider public. There is an enormous public and media interest in the voice and social interactions. Therefore we have had in general an excellent response to our activities to engage the public in our work. The laboratory has engaged with the public at several occasions such as at the Glasgow Science Centre during Brain Awareness Week. In addition to publishing in academic journals and presenting at Scientific Conferences, we engage with the press to improve the impact of our findings. The PI has received specific training with the media thanks to the BBSRC-organized Media Training Course, which has been instrumental in enhancing the profile of the laboratory's research (several journal and radio appearances in the past year, including the Sunday Telegraph and BBC4).
Finally, the research will have an important impact on the career of Drs Emilie Salvia and Bruno Giordano. Both will greatly enhance their knowledge, CV and visibility as part of the project and would be in a competitive position on the academic market at the end of the grant period.

Publications

10 25 50
 
Description Over several months of dedicated work we have acquired a unique dataset that consists of brain responses to morphed emotional voicing in the same set of participants recorded with different brain imaging methods (fMRI, MEG, EEG). The functional brain data is complemented with behavioural data. This data will be used to investigate the processing of emotional stimuli in the brain especially with respect to the question if these stimuli are processed along continuous dimensions (valence, arousal) or categorical (by emotional categories).
We have already fully characterised the behavioural response (paper is written and we will submit soon).
We have also developed the fMRI and MEG analysis pipeline using state-of-the-art representational similarly analysis (that involved new methods development and validation). Our first results indicate that categorical models explain MEG data better at early time points following stimulus onset than continuous models.This is a key finding that addresses a long-standing debate in cognitive neuroscience. The corresponding manuscript has been submitted to 'Science'.
Exploitation Route Our published results allow to derive new hypothesis that can be experimentally tested. Also, the analysis pipeline is novel and will be of interest to others.
Sectors Other

 
Title Design and implementation of RSA pipeline 
Description Design and implementation of pipeline for the statistical analysis of fMRI and MEG data based on Representational Similarity Analysis (RSA) methods. 
Type Of Material Improvements to research infrastructure 
Provided To Others? No  
Impact The state-of-the art pipeline will be available to additional projects beyond the current one. Portions of the improvements are being shared with the community of developers of open-source software for the analysis of brain imaging data. 
 
Title Design of whole-brain information-theoretic network pipeline 
Description Design of information-theoretic pipeline for the whole-brain network analysis of stimulus encoding in fMRI and MEG data. 
Type Of Material Improvements to research infrastructure 
Provided To Others? No  
Impact Likely impact on community researching brain stimulus encoding at large. 
 
Title Methods for the fusion of MEG and fMRI data. 
Description Development and implementation for the measurement of cerebral representations in combined fMRI and MEG data. 
Type Of Material Improvements to research infrastructure 
Year Produced 2018 
Provided To Others? No  
Impact Novel approach to the measurement of brain encoding with multimodal imaging data. 
 
Title Multimodal MEG/fMRI dataset on encoding of vocal emotions. 
Description A large database comprising data from 10 individuals each participating in 4 fMRI, 4 MEG and 3 behavioral sessions part of a study on the cortical/perceptual processing of emotions in vocal communication. 
Type Of Material Database/Collection of data 
Provided To Others? No  
Impact We plan to publish the database at the end of the project. 
 
Description Development of information theoretic methods 
Organisation Italian Institute of Technology (Istituto Italiano di Tecnologia IIT)
Country Italy 
Sector Public 
PI Contribution Development of information theoretic methods for analysis of fMRI and MEG data
Collaborator Contribution Development of information theoretic methods for analysis of fMRI and MEG data
Impact Novel methods for quantifying mutual information, redundancy and synergy in MEG and fMRI data
Start Year 2015
 
Description Development of materials and methods for study of cortical processing of vocal affect 
Organisation Aix-Marseille University
Country France 
Sector Academic/University 
PI Contribution Acquisition of pilot behavioral and neuroimaging data. Optimization of sound stimuli. Experimental design.
Collaborator Contribution Sound stimuli. Experimental design.
Impact Experimental protocol and software for implementing them.
Start Year 2015
 
Description Grant collaborations 
Organisation Aix-Marseille University
Country France 
Sector Academic/University 
PI Contribution We are leading the collaboration that is part of our grant. Collaborators are Co-Is or collaborators on the grant. We are providing most of the data and supervise the RAs on the project.
Collaborator Contribution Partners bring essential world-leading expertise to the project. Specifically, expertise in analysing data (cambridge, IIT) and expertise in the neuroscientific aspects of the project (Marseille, Maastricht, Geneva).
Impact There is an EBR paper that has resulted from the grant (listed in output). We are still in the phase of collecting essential data before we will start with the analysis.
Start Year 2015
 
Description Grant collaborations 
Organisation Italian Institute of Technology (Istituto Italiano di Tecnologia IIT)
Country Italy 
Sector Public 
PI Contribution We are leading the collaboration that is part of our grant. Collaborators are Co-Is or collaborators on the grant. We are providing most of the data and supervise the RAs on the project.
Collaborator Contribution Partners bring essential world-leading expertise to the project. Specifically, expertise in analysing data (cambridge, IIT) and expertise in the neuroscientific aspects of the project (Marseille, Maastricht, Geneva).
Impact There is an EBR paper that has resulted from the grant (listed in output). We are still in the phase of collecting essential data before we will start with the analysis.
Start Year 2015
 
Description Grant collaborations 
Organisation Maastricht University (UM)
Country Netherlands 
Sector Academic/University 
PI Contribution We are leading the collaboration that is part of our grant. Collaborators are Co-Is or collaborators on the grant. We are providing most of the data and supervise the RAs on the project.
Collaborator Contribution Partners bring essential world-leading expertise to the project. Specifically, expertise in analysing data (cambridge, IIT) and expertise in the neuroscientific aspects of the project (Marseille, Maastricht, Geneva).
Impact There is an EBR paper that has resulted from the grant (listed in output). We are still in the phase of collecting essential data before we will start with the analysis.
Start Year 2015
 
Description Grant collaborations 
Organisation Medical Research Council (MRC)
Department Medical Research Council (MRC) Centre Cambridge
Country United Kingdom 
Sector Public 
PI Contribution We are leading the collaboration that is part of our grant. Collaborators are Co-Is or collaborators on the grant. We are providing most of the data and supervise the RAs on the project.
Collaborator Contribution Partners bring essential world-leading expertise to the project. Specifically, expertise in analysing data (cambridge, IIT) and expertise in the neuroscientific aspects of the project (Marseille, Maastricht, Geneva).
Impact There is an EBR paper that has resulted from the grant (listed in output). We are still in the phase of collecting essential data before we will start with the analysis.
Start Year 2015
 
Description Grant collaborations 
Organisation University of Geneva
Country Switzerland 
Sector Academic/University 
PI Contribution We are leading the collaboration that is part of our grant. Collaborators are Co-Is or collaborators on the grant. We are providing most of the data and supervise the RAs on the project.
Collaborator Contribution Partners bring essential world-leading expertise to the project. Specifically, expertise in analysing data (cambridge, IIT) and expertise in the neuroscientific aspects of the project (Marseille, Maastricht, Geneva).
Impact There is an EBR paper that has resulted from the grant (listed in output). We are still in the phase of collecting essential data before we will start with the analysis.
Start Year 2015
 
Description MEG representation of computational models of audio-visual integration 
Organisation Bielefeld University
Country Germany 
Sector Academic/University 
PI Contribution Application of RSA pipeline to MEG dataset.
Collaborator Contribution Experiment design. Results interpretation.
Impact A journal submission is expected by the end of the year.
Start Year 2017
 
Description MEG representation of computational models of audio-visual integration 
Organisation University of Oxford
Country United Kingdom 
Sector Academic/University 
PI Contribution Application of RSA pipeline to MEG dataset.
Collaborator Contribution Experiment design. Results interpretation.
Impact A journal submission is expected by the end of the year.
Start Year 2017
 
Description Optimization of sound stimulation system for MEG 
Organisation Medical Research Council (MRC)
Department MRC Cognition and Brain Sciences Unit
Country United Kingdom 
Sector Public 
PI Contribution Provided measurements of frequency response of audio stimulation system in Glasgow
Collaborator Contribution Provided new components of audio stimulation system
Impact Optimized sound stimulation system for MEG
Start Year 2015
 
Description RSA methods development 
Organisation Medical Research Council (MRC)
Department MRC Cognition and Brain Sciences Unit
Country United Kingdom 
Sector Public 
PI Contribution Development of methods for the multivariate analysis of fMRI and MEG data
Collaborator Contribution Development of methods for the multivariate analysis of fMRI and MEG data
Impact Data analysis methods; refinement of acquisition protocols for fMRI data.
Start Year 2015
 
Description Representation of natural sounds in MEG and fMRI cerebral responses. 
Organisation Aix-Marseille University
Country France 
Sector Academic/University 
PI Contribution Experimental design; data collection; data analysis.
Collaborator Contribution Development of analysis methods; interpretation of results.
Impact Journal submission by the end of the year.
Start Year 2015
 
Description Representation of natural sounds in MEG and fMRI cerebral responses. 
Organisation Bielefeld University
Country Germany 
Sector Academic/University 
PI Contribution Experimental design; data collection; data analysis.
Collaborator Contribution Development of analysis methods; interpretation of results.
Impact Journal submission by the end of the year.
Start Year 2015
 
Description Representation of natural sounds in MEG and fMRI cerebral responses. 
Organisation Columbia University
Country United States 
Sector Academic/University 
PI Contribution Experimental design; data collection; data analysis.
Collaborator Contribution Development of analysis methods; interpretation of results.
Impact Journal submission by the end of the year.
Start Year 2015
 
Description Invited talk: Department of Psychology, University of Oxford. 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Postgraduate students
Results and Impact Invited talk at Department of Experimental Psychology, University of Oxford, Oxford, UK. Title: "Information-theoretic and RSA measurement of cerebral stimulus encoding". Date: 22 March 2017.
Year(s) Of Engagement Activity 2017
 
Description MEG UK 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Postgraduate students
Results and Impact Talk at national conference (MEG UK)
Year(s) Of Engagement Activity 2016
 
Description Royal Society of Medicine 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Industry/Business
Results and Impact Prof Joachim Gross presented at the joint Royal Society of Medicine/Institute of Engineering & Technology event on 6th May 2015 that included industry representatives, practitioners and scientists.

I was contacted after the presentation to give a talk at another University and to collaborate in a grant application.
Year(s) Of Engagement Activity 2015
 
Description Talk at MEG UK 2017. 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Postgraduate students
Results and Impact Talk at MEG UK 2017, Oxford, UK. Title: "Cerebral encoding of vocal emotions in fMRI and MEG multivariate response patterns". Date: 23 March 2017.
Year(s) Of Engagement Activity 2017
 
Description Talk: Institut de Recherche et Coor- dination Acoustique/Musique (STMS-IRCAM-CNRS), Paris, France 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Presentation of research results at a CNRS center that groups psychologists, acousticians, sound engineers and artists focusing on sound. Discussion of research directions in the field of auditory perception and neuroscience.
Year(s) Of Engagement Activity 2015
 
Description Talk: Institute for Computer Music and Sound Technology, Zurich University of the Arts, Zurich, Switzerland 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Presentation of research results at a workshop on musical practice. Extended debates about research methods in experimental psychology.
Year(s) Of Engagement Activity 2016
 
Description Talk: Institut de Neurosciences de la Timone, Marseille, France 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Postgraduate students
Results and Impact Presentation of grant results.
Year(s) Of Engagement Activity 2016
 
Description Talk: Institut de Neurosciences des Systèmes, Marseille, France 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Postgraduate students
Results and Impact Presentation of project results.
Year(s) Of Engagement Activity 2017
 
Description Talk: NIHR Nottingham Hearing Biomedical Research Unit, Nottingham, UK 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact Presentation of research results to students and staff of NIHR and MRC centers in Nottingham involved in the health care sector (hearing deficit patients such as cochlear implant -- CI -- patients). Discussion of research directions in auditory perception and neuroscience. Discussion with staff about improvements to ongoing research in CI patients.
Year(s) Of Engagement Activity 2016
 
Description Talks 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Postgraduate students
Results and Impact Preliminary outcomes from this grant were presented in four different talks reaching postgraduates and other scientists.
These are the talks:

Joachim Gross
Presentation at MEG UK conference 2015
Seminar Talk at UCL, London 2015
Seminar Talk in Geneva, 2015

Bruno Giordano
Seminar Talk in Nottingham, 2016
Year(s) Of Engagement Activity 2015,2016
 
Description Zurich 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Postgraduate students
Results and Impact Workshop for postgraduate students
Year(s) Of Engagement Activity 2017