Neural Representation of the Identities and Expressions of Human Faces

Lead Research Organisation: Royal Holloway University of London
Department Name: Psychology


Abstracts are not currently available in GtR for all funded research. This is normally because the abstract was not required at the time of proposal submission, but may be because it included sensitive information such as personal details.
Description The brain is composed of "representations": neural responses that signal information about the outside world, or about our perception of it. One of the main goals of neuroscience is to map out the locations of representations of the brain, to infer the mechanisms that create and manipulate these representations and to understand how they relate to behaviour. Neuroscience has discovered that, in parts of the visual system, visual representations seem laid out and divided into separate "pathways": connected networks of areas that represent information and perform an important function. After basic information is represented in low-level visual cortex, different types of higher-level (more complex visual patterns) are analysed and represented by specialised strings of areas that form pathways.

The main objective of our ESRC grant was to test theories about how these pathways work. Are there distinguishable brain pathways responsible for recognition of facial identities versus expressions (different complex visual patterns)? Or are there different pathways that are distinguished instead by processing, analysing and representing facial forms versus facial movements?

We hypothesised that an analysis of brain responses to dynamic videos of facial movements would reveal new insight into this pathway structure in the brain. We also hypothesised that we could discover representations of form and motion using (1) computational models (e.g., dynamic causal modelling, DCM); (2) multivariate "decoding" analysis of functional magnetic resonance imaging (fMRI) and magnetoencephalography data; (3) analyses of oscillations measured using MEG.

As described below, we learned a great deal about the pathway structure in the higher visual system using all three of these approaches, as planned. Please note that part of this research was conducted at the MRC Cognition Brain Sciences Unit (CBU) in Cambridge and part of it was performed at Royal Holloway, University of London (RHUL). The work at these institutions derived from one grant project but are listed on ResearchFish as separate grants and so we have reported separate outputs for each. Here, we describe the details of outputs resulting from work at RHUL (ES/I01134X/2). Please see ResearchFish report ES/I01134X/1 for details of outputs resulting from the work conducted at the CBU at the CBU.

Here, we focus our description on some of the work done in part or wholly at RHUL. Three of our research outputs associated with our work at the CBU (ES/I01134X/1) applied a type of connectivity modelling, DCM, to reveal the interactions associated with brain areas responding to form and motion. At RHUL, we continued to develop this work by using DCM to learn more about these pathways in "face-blind" individuals, who are less able than others to recognise faces (developmental prosopagnosics). We discovered that the human pathway structure best resembles a model were brain responses that are specific to faces (compared to non-faces) arise because of the influence of connections between early visual cortex (where simple and local visual features are processed and represented) and areas dedicated to processing faces (such as fusiform face area and superior temporal sulcus). Most importantly, the influence of these connections was reduced in developmental prosopagnosics (See 2). Therefore, we successfully showed deficits in how brain areas are connected, suggesting a facial identity-related pathway has been disrupted in these individuals.

Lohse M., Duchaine B, Garrido L, Driver J, Dolan R & Furl N. 2016. Effective Connectivity from Early Visual Cortex to Posterior Occipito-temporal Face Areas Predicts Developmental Prosopagnosia. J Neurosci 36:321-328.

DCM is a type of "effective" connectivity analysis. This means that DCM measures how functional responses in different brain areas affect each other. However, other methods can measure the physical "wiring" that connects different brain areas. We used one such method, diffusion tensor imaging (DTI) to measure how the brain areas that respond to faces are physically (structurally) connected. Moreover, we tested this "wiring structure" in typical individuals and developmental prosopagnosics, those who show deficient face recognition (of facial identities, at least). We found that one brain area, the fusiform face area, was not as well-connected in developmental prosopagnosics as in typical controls. This area may be key to recognising facial identity.

Song S, Garrido L, Nagy Z, Mohammadi S, Steel A, Driver J, Dolan RJ, Duchaine B, Furl N. 2015. Local but not long-range microstructural differences of the ventral temporal cortex in developmental prosopagnosia. Neuropsychologia 78:195-206.

To better integrate both the DCM findings from our MRC CBU portion of the grant and the DCM and DTI connectivity pathway results from the RHUL portion of the grant, we wrote a review paper which covered applications of this type of connectivity modelling to understanding pathway organisation in the brain in the occipitotemporal cortex of the brain.

Furl N. 2015. Structural and effective connectivity among high-level visual areas. Frontiers in Human Neuroscience 9:253.

We also examined neural oscillations - a key neural mechanism for representing information in the brain. Our work at MRC CBU had indicated (using a variant of DCM applied to MEG data) that a certain low frequency range of oscillations was involved in communicating information between brain regions about static forms and shapes of faces . To follow up this work, we used MEG to test whether this frequency range related to representations of form and motion in dynamic faces. Using a special type of "multivariate decoding" method called "representational similarity analysis", we were indeed able to show that low frequency oscillations indeed signal information about both static form and facial dynamics. We also linked these representations to a brain area called the fusiform face area. And, even further, we developed some new methods for determining facial form and motion from facial videos and our low-frequency brain responses were related to this information too.

Furl N, Lohse M, Pizzorni-Ferrarese F. 2017. Low-frequency oscillations employ a general coding of the spatio-temporal similarity of dynamic faces. Neuroimage 157:486-499.

Our most recent 2020 paper built on our advances stemming from this grant in 2017 for quantifying facial motion from facial movements. We built a stimulus set of caricatured facial movements. This involves tracking landmarks of videos of facial expression movements. Computer animated head models (with normalised head shape) could then be animated with these movements. And the distinctiveness of the movements could be exaggerated (caricatured) or reduced (anti-caricatured). This advance is important because, in the past, photographic caricature of face form and shape has been used to test theories that the brain uses a type of representation called a "face space". Evidence for face spaces based on shape and form has previously been observed in behavioural studies and localised in the brain to the expected ventral pathway. Because of our new expression movement caricatures, we could repeat these methods: but instead of testing for faces spaces based on form and shape we could test for movement-based face space. In two behavioural studies we validated our stimuli, showing that the expressions are convincing and that, in many ways, people perceive as predicted by face space theory. We then scanned participants with fMRI and used a combination of traditional and multivariate (representational similarity analysis) methods to localise motion based face spaces with BOTH dorsal and ventral pathways. In the end, it appears facial movements are represented in a much more widespread network that previously believed before our ESRC funded project. Also, our methods for quantifying and animating facial movements open new doors for studying dynamic facial information. There is little reason for continuing to limit studies of face perception to static stimuli.

Furl N, Begum F, Sulik J, Ferrarese FP, Jans S, Woolley C. 2020. Face space representations of movement. Neuroimage 212:116676. doi: 10.1016/j.neuroimage.2020.116676

Our papers studying DCM and DTI in developmental prosopagnosics and typical individuals used new analyses of data that existed before our grant. However, Furl et al. (2017) involved a new data collection (from MRC CBU) funded by this grant that was later analysed and published while the grant was held at RHUL. These data are right now freely available for download from the RESHARE archive at the UKDataService. This data archive joins even more archives or for data for previous published papers from the MRC CBU portion of the grant. Published data funded by this grant is freely available using this service. fMRI data from the 2020 Neuroimage paper mentioned above is available in BIDS format at Openneuro
Exploitation Route Our research has applications in developing artificial visual recognition systems for video information as well as developing clinical models of brain disorders. These studies suggest ways that abstract visual information can be coded by neurons as well as the computations these neurons perform when coding. This knowledge can be used to devise and improve artificial visual recognition systems. Particularly, our results using dynamic facial stimuli can help develop software which can visually recognise video. Our research using the macaque monkey will help develop animal models of visual function. Our research on oscillatory communication between brain regions is a first step towards developing sophisticated models of disorders such as schizophrenia.
Sectors Education,Security and Diplomacy