The neural representation of vocal emotion: representational similarity analysis and information-theoretic approaches
Lead Research Organisation:
University of Glasgow
Department Name: College of Medical, Veterinary, Life Sci
Abstract
Recognizing and interpreting emotions in other persons is crucial for social interactions. In particular, people of all cultures are able to recognize emotions in vocalizations without speech such as laughs, cries or screams of fear. But how our brain analyses emotion in voices remains poorly understood, compared to how we perceive emotion in faces, for example.
In this project we bring together several expert co-investigators and external collaborators to address two new streams of unanswered questions on the way our brain processes emotional information in voice. We will use several complementary neuroimaging techniques to measure cerebral activity with the best precision currently possible, and advanced analysis techniques to ask two main series of questions:
(1) Does the brain processes emotions as different categories (e.g., fear, anger, pleasure) or rather as a continuum emotions each characterized by different amount of activation (arousal) or how negative/positive the expression is (valence)?
(2) How do different nodes of the oscillatory network of brain regions responsible for analysing emotion in voice interact with one another? Do they share information between nodes in a synergistic way, or is there redundancy between the different nodes? And if yes, how, where and when is emotional information combined to arrive at the recognition of a specific emotion?
The results from this project will bring important new knowledge on how the normal brain works, with longer-term impact on the treatment and diagnosis of communication disorders and the development of better neural prostheses such as cochlear implants. The project will also contribute to consolidating and preserving UK's leading position in advanced neuroscience research.
In this project we bring together several expert co-investigators and external collaborators to address two new streams of unanswered questions on the way our brain processes emotional information in voice. We will use several complementary neuroimaging techniques to measure cerebral activity with the best precision currently possible, and advanced analysis techniques to ask two main series of questions:
(1) Does the brain processes emotions as different categories (e.g., fear, anger, pleasure) or rather as a continuum emotions each characterized by different amount of activation (arousal) or how negative/positive the expression is (valence)?
(2) How do different nodes of the oscillatory network of brain regions responsible for analysing emotion in voice interact with one another? Do they share information between nodes in a synergistic way, or is there redundancy between the different nodes? And if yes, how, where and when is emotional information combined to arrive at the recognition of a specific emotion?
The results from this project will bring important new knowledge on how the normal brain works, with longer-term impact on the treatment and diagnosis of communication disorders and the development of better neural prostheses such as cochlear implants. The project will also contribute to consolidating and preserving UK's leading position in advanced neuroscience research.
Technical Summary
A growing number of studies have contributed to an increasingly detailed picture of the network of cerebral regions involved in this task. Yet, the exact nature of information processing at the different nodes of the network and their associated representations remain largely unexplored.
In this project we bring together several expert co-investigators and external collaborators to address two new streams of unanswered questions on the way our brain processes emotional information in voice. We will use several complementary neuroimaging techniques to measure cerebral activity with the best precision currently possible, and advanced analysis techniques to ask two main sets of questions:
(1) How does the representation of the emotional stimuli evolve from acoustical-based at early levels of processing to more emotion-based allowing conscious emotional categorization at higher processing stages?
(2) How do different nodes of the oscillatory network of brain regions responsible for analysing emotion in voice interact with one another? Do they share information between nodes in a synergistic way, or is there redundancy between the different nodes? And if yes, how, where and when is emotional information combined to arrive at the recognition of a specific emotion?
To address question 1, we propose to use Representational Similarity Analysis (RSA) as a powerful framework for bringing together multiple types of measures (acoustical, perceptual, cerebral) and models (computational, theoretical) in a comprehensive evaluation of representations at different stages of cerebral processing.
To address question 2, We propose to use novel information-theoretic methods to first individuate network nodes and oscillatory components of activity carrying affective information, then to assess whether combinations of those information-bearing elements carry affect information that cannot possibly obtained by a single element alone (synergy).
In this project we bring together several expert co-investigators and external collaborators to address two new streams of unanswered questions on the way our brain processes emotional information in voice. We will use several complementary neuroimaging techniques to measure cerebral activity with the best precision currently possible, and advanced analysis techniques to ask two main sets of questions:
(1) How does the representation of the emotional stimuli evolve from acoustical-based at early levels of processing to more emotion-based allowing conscious emotional categorization at higher processing stages?
(2) How do different nodes of the oscillatory network of brain regions responsible for analysing emotion in voice interact with one another? Do they share information between nodes in a synergistic way, or is there redundancy between the different nodes? And if yes, how, where and when is emotional information combined to arrive at the recognition of a specific emotion?
To address question 1, we propose to use Representational Similarity Analysis (RSA) as a powerful framework for bringing together multiple types of measures (acoustical, perceptual, cerebral) and models (computational, theoretical) in a comprehensive evaluation of representations at different stages of cerebral processing.
To address question 2, We propose to use novel information-theoretic methods to first individuate network nodes and oscillatory components of activity carrying affective information, then to assess whether combinations of those information-bearing elements carry affect information that cannot possibly obtained by a single element alone (synergy).
Planned Impact
Outside of academia, the research is likely to have positive impact on several user groups in the longer term. Person with auditory perception deficits, such as cochlear implant patients and persons with hearing aids experience difficulties recognizing affective states in voice, with negative impact on their social interactions. As good social interactions are a primary determinant of healthy ageing, contributing to a better perception of affective states is likely to have positive outcomes on interactions and quality of life in older populations. Unfortunately, manufacturers of hearing aids and cochlear implants have largely concentrated so far on enhancing speech intelligibility - for obvious reasons. But other, socially-central aspects of vocal communication, such as the detection and recognition of affective states, have been comparatively neglected. The research we conduct in normal participants has the potential to be translated into clinical practice by providing mechanisms to enhance algorithms for auditory decoding in cochlear implants, or optimizing training and rehabilitation strategies. We have experience enhancing the impact of our work through links with industry. We have had funded collaborations with France -Telecom and with Cochlear, one of the leading cochlear implant manufacturers, and this research could lead to further similar collaborations in the longer term.
Another potential pathway to impact in this research is to develop links with the growing industry of 'social computing'. Our results have the potential to be of interest for designers of software for automated extraction of affective states. Automated recognition of speech is now at a commercial stage, but automated recognition of affective states for more socially rewarding human-computer interactions is still poor. Information on the solutions to this problem found by our brain over millions of years of evolution could potentially give important clues to the design of more parsimonious, and especially more robust to degradation, recognition systems. Links will the industry in this domain will be specifically sought for with assistance from the Research and Enterprise office e at university of Glasgow.
Another important user group potentially affected by the research is the wider public. There is an enormous public and media interest in the voice and social interactions. Therefore we have had in general an excellent response to our activities to engage the public in our work. The laboratory has engaged with the public at several occasions such as at the Glasgow Science Centre during Brain Awareness Week. In addition to publishing in academic journals and presenting at Scientific Conferences, we engage with the press to improve the impact of our findings. The PI has received specific training with the media thanks to the BBSRC-organized Media Training Course, which has been instrumental in enhancing the profile of the laboratory's research (several journal and radio appearances in the past year, including the Sunday Telegraph and BBC4).
Finally, the research will have an important impact on the career of Drs Emilie Salvia and Bruno Giordano. Both will greatly enhance their knowledge, CV and visibility as part of the project and would be in a competitive position on the academic market at the end of the grant period.
Another potential pathway to impact in this research is to develop links with the growing industry of 'social computing'. Our results have the potential to be of interest for designers of software for automated extraction of affective states. Automated recognition of speech is now at a commercial stage, but automated recognition of affective states for more socially rewarding human-computer interactions is still poor. Information on the solutions to this problem found by our brain over millions of years of evolution could potentially give important clues to the design of more parsimonious, and especially more robust to degradation, recognition systems. Links will the industry in this domain will be specifically sought for with assistance from the Research and Enterprise office e at university of Glasgow.
Another important user group potentially affected by the research is the wider public. There is an enormous public and media interest in the voice and social interactions. Therefore we have had in general an excellent response to our activities to engage the public in our work. The laboratory has engaged with the public at several occasions such as at the Glasgow Science Centre during Brain Awareness Week. In addition to publishing in academic journals and presenting at Scientific Conferences, we engage with the press to improve the impact of our findings. The PI has received specific training with the media thanks to the BBSRC-organized Media Training Course, which has been instrumental in enhancing the profile of the laboratory's research (several journal and radio appearances in the past year, including the Sunday Telegraph and BBC4).
Finally, the research will have an important impact on the career of Drs Emilie Salvia and Bruno Giordano. Both will greatly enhance their knowledge, CV and visibility as part of the project and would be in a competitive position on the academic market at the end of the grant period.
Organisations
- University of Glasgow (Lead Research Organisation)
- UNIVERSITY OF OXFORD (Collaboration)
- Maastricht University (UM) (Collaboration)
- University of Geneva (Collaboration)
- Columbia University (Collaboration)
- Italian Institute of Technology (Istituto Italiano di Tecnologia IIT) (Collaboration)
- Bielefeld University (Collaboration)
- Medical Research Council (MRC) (Collaboration)
- Aix-Marseille University (Collaboration)
Publications
Cao Y
(2018)
Causal inference in the multisensory brain
Cao Y
(2019)
Causal Inference in the Multisensory Brain.
in Neuron
Cao Y
(2016)
The dominance of haptics over audition in controlling wrist velocity during striking movements.
in Experimental brain research
Giordano BL
(2017)
Contributions of local speech encoding and functional connectivity to audio-visual speech perception.
in eLife
Giordano BL
(2021)
The representational dynamics of perceived voice emotions evolve from categories to dimensions.
in Nature human behaviour
Ince RA
(2017)
A statistical framework for neuroimaging data analysis based on mutual information estimated via a gaussian copula.
in Human brain mapping
Ince RA
(2015)
Tracing the Flow of Perceptual Features in an Algorithmic Brain Network.
in Scientific reports
Kotz SA
(2018)
The Evolution of Rhythm Processing.
in Trends in cognitive sciences
Description | Over several months of dedicated work we have acquired a unique dataset that consists of brain responses to morphed emotional voicing in the same set of participants recorded with different brain imaging methods (fMRI, MEG, EEG). The functional brain data is complemented with behavioural data. This data will be used to investigate the processing of emotional stimuli in the brain especially with respect to the question if these stimuli are processed along continuous dimensions (valence, arousal) or categorical (by emotional categories). We have already fully characterised the behavioural response (paper is written and we will submit soon). We have also developed the fMRI and MEG analysis pipeline using state-of-the-art representational similarly analysis (that involved new methods development and validation). Our first results indicate that categorical models explain MEG data better at early time points following stimulus onset than continuous models.This is a key finding that addresses a long-standing debate in cognitive neuroscience. The corresponding manuscript has been submitted to 'Science'. |
Exploitation Route | Our published results allow to derive new hypothesis that can be experimentally tested. Also, the analysis pipeline is novel and will be of interest to others. |
Sectors | Other |
Title | Design and implementation of RSA pipeline |
Description | Design and implementation of pipeline for the statistical analysis of fMRI and MEG data based on Representational Similarity Analysis (RSA) methods. |
Type Of Material | Improvements to research infrastructure |
Provided To Others? | No |
Impact | The state-of-the art pipeline will be available to additional projects beyond the current one. Portions of the improvements are being shared with the community of developers of open-source software for the analysis of brain imaging data. |
Title | Design of whole-brain information-theoretic network pipeline |
Description | Design of information-theoretic pipeline for the whole-brain network analysis of stimulus encoding in fMRI and MEG data. |
Type Of Material | Improvements to research infrastructure |
Provided To Others? | No |
Impact | Likely impact on community researching brain stimulus encoding at large. |
Title | Methods for the fusion of MEG and fMRI data. |
Description | Development and implementation for the measurement of cerebral representations in combined fMRI and MEG data. |
Type Of Material | Improvements to research infrastructure |
Year Produced | 2018 |
Provided To Others? | No |
Impact | Novel approach to the measurement of brain encoding with multimodal imaging data. |
Title | Data from: The perception of caricatured emotion in voice |
Description | Data for Whiting et al. (2020, Cognition): "The perception of caricatured emotion in voice". Raw behavioural data and sound stimuli. |
Type Of Material | Database/Collection of data |
Year Produced | 2020 |
Provided To Others? | Yes |
URL | http://datadryad.org/stash/dataset/doi:10.5061/dryad.18931zcsn |
Title | Multimodal MEG/fMRI dataset on encoding of vocal emotions. |
Description | A large database comprising data from 10 individuals each participating in 4 fMRI, 4 MEG and 3 behavioral sessions part of a study on the cortical/perceptual processing of emotions in vocal communication. |
Type Of Material | Database/Collection of data |
Provided To Others? | No |
Impact | We plan to publish the database at the end of the project. |
Description | Development of information theoretic methods |
Organisation | Italian Institute of Technology (Istituto Italiano di Tecnologia IIT) |
Country | Italy |
Sector | Academic/University |
PI Contribution | Development of information theoretic methods for analysis of fMRI and MEG data |
Collaborator Contribution | Development of information theoretic methods for analysis of fMRI and MEG data |
Impact | Novel methods for quantifying mutual information, redundancy and synergy in MEG and fMRI data |
Start Year | 2015 |
Description | Development of materials and methods for study of cortical processing of vocal affect |
Organisation | Aix-Marseille University |
Country | France |
Sector | Academic/University |
PI Contribution | Acquisition of pilot behavioral and neuroimaging data. Optimization of sound stimuli. Experimental design. |
Collaborator Contribution | Sound stimuli. Experimental design. |
Impact | Experimental protocol and software for implementing them. |
Start Year | 2015 |
Description | Grant collaborations |
Organisation | Aix-Marseille University |
Country | France |
Sector | Academic/University |
PI Contribution | We are leading the collaboration that is part of our grant. Collaborators are Co-Is or collaborators on the grant. We are providing most of the data and supervise the RAs on the project. |
Collaborator Contribution | Partners bring essential world-leading expertise to the project. Specifically, expertise in analysing data (cambridge, IIT) and expertise in the neuroscientific aspects of the project (Marseille, Maastricht, Geneva). |
Impact | There is an EBR paper that has resulted from the grant (listed in output). We are still in the phase of collecting essential data before we will start with the analysis. |
Start Year | 2015 |
Description | Grant collaborations |
Organisation | Italian Institute of Technology (Istituto Italiano di Tecnologia IIT) |
Country | Italy |
Sector | Academic/University |
PI Contribution | We are leading the collaboration that is part of our grant. Collaborators are Co-Is or collaborators on the grant. We are providing most of the data and supervise the RAs on the project. |
Collaborator Contribution | Partners bring essential world-leading expertise to the project. Specifically, expertise in analysing data (cambridge, IIT) and expertise in the neuroscientific aspects of the project (Marseille, Maastricht, Geneva). |
Impact | There is an EBR paper that has resulted from the grant (listed in output). We are still in the phase of collecting essential data before we will start with the analysis. |
Start Year | 2015 |
Description | Grant collaborations |
Organisation | Maastricht University (UM) |
Country | Netherlands |
Sector | Academic/University |
PI Contribution | We are leading the collaboration that is part of our grant. Collaborators are Co-Is or collaborators on the grant. We are providing most of the data and supervise the RAs on the project. |
Collaborator Contribution | Partners bring essential world-leading expertise to the project. Specifically, expertise in analysing data (cambridge, IIT) and expertise in the neuroscientific aspects of the project (Marseille, Maastricht, Geneva). |
Impact | There is an EBR paper that has resulted from the grant (listed in output). We are still in the phase of collecting essential data before we will start with the analysis. |
Start Year | 2015 |
Description | Grant collaborations |
Organisation | Medical Research Council (MRC) |
Department | Medical Research Council (MRC) Centre Cambridge |
Country | United Kingdom |
Sector | Public |
PI Contribution | We are leading the collaboration that is part of our grant. Collaborators are Co-Is or collaborators on the grant. We are providing most of the data and supervise the RAs on the project. |
Collaborator Contribution | Partners bring essential world-leading expertise to the project. Specifically, expertise in analysing data (cambridge, IIT) and expertise in the neuroscientific aspects of the project (Marseille, Maastricht, Geneva). |
Impact | There is an EBR paper that has resulted from the grant (listed in output). We are still in the phase of collecting essential data before we will start with the analysis. |
Start Year | 2015 |
Description | Grant collaborations |
Organisation | University of Geneva |
Country | Switzerland |
Sector | Academic/University |
PI Contribution | We are leading the collaboration that is part of our grant. Collaborators are Co-Is or collaborators on the grant. We are providing most of the data and supervise the RAs on the project. |
Collaborator Contribution | Partners bring essential world-leading expertise to the project. Specifically, expertise in analysing data (cambridge, IIT) and expertise in the neuroscientific aspects of the project (Marseille, Maastricht, Geneva). |
Impact | There is an EBR paper that has resulted from the grant (listed in output). We are still in the phase of collecting essential data before we will start with the analysis. |
Start Year | 2015 |
Description | MEG representation of computational models of audio-visual integration |
Organisation | Bielefeld University |
Country | Germany |
Sector | Academic/University |
PI Contribution | Application of RSA pipeline to MEG dataset. |
Collaborator Contribution | Experiment design. Results interpretation. |
Impact | A journal submission is expected by the end of the year. |
Start Year | 2017 |
Description | MEG representation of computational models of audio-visual integration |
Organisation | University of Oxford |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | Application of RSA pipeline to MEG dataset. |
Collaborator Contribution | Experiment design. Results interpretation. |
Impact | A journal submission is expected by the end of the year. |
Start Year | 2017 |
Description | Optimization of sound stimulation system for MEG |
Organisation | Medical Research Council (MRC) |
Department | MRC Cognition and Brain Sciences Unit |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | Provided measurements of frequency response of audio stimulation system in Glasgow |
Collaborator Contribution | Provided new components of audio stimulation system |
Impact | Optimized sound stimulation system for MEG |
Start Year | 2015 |
Description | RSA methods development |
Organisation | Medical Research Council (MRC) |
Department | MRC Cognition and Brain Sciences Unit |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | Development of methods for the multivariate analysis of fMRI and MEG data |
Collaborator Contribution | Development of methods for the multivariate analysis of fMRI and MEG data |
Impact | Data analysis methods; refinement of acquisition protocols for fMRI data. |
Start Year | 2015 |
Description | Representation of natural sounds in MEG and fMRI cerebral responses. |
Organisation | Aix-Marseille University |
Country | France |
Sector | Academic/University |
PI Contribution | Experimental design; data collection; data analysis. |
Collaborator Contribution | Development of analysis methods; interpretation of results. |
Impact | Journal submission by the end of the year. |
Start Year | 2015 |
Description | Representation of natural sounds in MEG and fMRI cerebral responses. |
Organisation | Bielefeld University |
Country | Germany |
Sector | Academic/University |
PI Contribution | Experimental design; data collection; data analysis. |
Collaborator Contribution | Development of analysis methods; interpretation of results. |
Impact | Journal submission by the end of the year. |
Start Year | 2015 |
Description | Representation of natural sounds in MEG and fMRI cerebral responses. |
Organisation | Columbia University |
Country | United States |
Sector | Academic/University |
PI Contribution | Experimental design; data collection; data analysis. |
Collaborator Contribution | Development of analysis methods; interpretation of results. |
Impact | Journal submission by the end of the year. |
Start Year | 2015 |
Description | Invited talk: Department of Psychology, University of Oxford. |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Postgraduate students |
Results and Impact | Invited talk at Department of Experimental Psychology, University of Oxford, Oxford, UK. Title: "Information-theoretic and RSA measurement of cerebral stimulus encoding". Date: 22 March 2017. |
Year(s) Of Engagement Activity | 2017 |
Description | MEG UK |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Postgraduate students |
Results and Impact | Talk at national conference (MEG UK) |
Year(s) Of Engagement Activity | 2016 |
Description | Royal Society of Medicine |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Industry/Business |
Results and Impact | Prof Joachim Gross presented at the joint Royal Society of Medicine/Institute of Engineering & Technology event on 6th May 2015 that included industry representatives, practitioners and scientists. I was contacted after the presentation to give a talk at another University and to collaborate in a grant application. |
Year(s) Of Engagement Activity | 2015 |
Description | Talk at MEG UK 2017. |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Postgraduate students |
Results and Impact | Talk at MEG UK 2017, Oxford, UK. Title: "Cerebral encoding of vocal emotions in fMRI and MEG multivariate response patterns". Date: 23 March 2017. |
Year(s) Of Engagement Activity | 2017 |
Description | Talk: Institut de Recherche et Coor- dination Acoustique/Musique (STMS-IRCAM-CNRS), Paris, France |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Presentation of research results at a CNRS center that groups psychologists, acousticians, sound engineers and artists focusing on sound. Discussion of research directions in the field of auditory perception and neuroscience. |
Year(s) Of Engagement Activity | 2015 |
Description | Talk: Institute for Computer Music and Sound Technology, Zurich University of the Arts, Zurich, Switzerland |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Presentation of research results at a workshop on musical practice. Extended debates about research methods in experimental psychology. |
Year(s) Of Engagement Activity | 2016 |
Description | Talk: Institut de Neurosciences de la Timone, Marseille, France |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Regional |
Primary Audience | Postgraduate students |
Results and Impact | Presentation of grant results. |
Year(s) Of Engagement Activity | 2016 |
Description | Talk: Institut de Neurosciences des Systèmes, Marseille, France |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Regional |
Primary Audience | Postgraduate students |
Results and Impact | Presentation of project results. |
Year(s) Of Engagement Activity | 2017 |
Description | Talk: NIHR Nottingham Hearing Biomedical Research Unit, Nottingham, UK |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | Presentation of research results to students and staff of NIHR and MRC centers in Nottingham involved in the health care sector (hearing deficit patients such as cochlear implant -- CI -- patients). Discussion of research directions in auditory perception and neuroscience. Discussion with staff about improvements to ongoing research in CI patients. |
Year(s) Of Engagement Activity | 2016 |
Description | Talks |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Postgraduate students |
Results and Impact | Preliminary outcomes from this grant were presented in four different talks reaching postgraduates and other scientists. These are the talks: Joachim Gross Presentation at MEG UK conference 2015 Seminar Talk at UCL, London 2015 Seminar Talk in Geneva, 2015 Bruno Giordano Seminar Talk in Nottingham, 2016 |
Year(s) Of Engagement Activity | 2015,2016 |
Description | Zurich |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | Regional |
Primary Audience | Postgraduate students |
Results and Impact | Workshop for postgraduate students |
Year(s) Of Engagement Activity | 2017 |