Neural correlates of learning and confidence during decision making and their utility in developing "intelligent" information technologies
Lead Research Organisation:
University of Glasgow
Department Name: College of Medical, Veterinary, Life Sci
Abstract
Consider an image intelligence analyst trying to distinguish between numerous targets of interest in a large array of noisy CCTV or satellite images to pinpoint only those that might pose a real security threat. Her ability to do this reliably depends on her years of training in interpreting such images and on her overall level of confidence that the target selection is correct. This example highlights two important phenomena underpinning human cognition. Firstly, training and experience can induce long-lasting improvements in our ability to detect, identify and make decisions based on ambiguous sensory information; a phenomenon commonly known as perceptual learning. Secondly, decision confidence - the degree of belief that one's choice is likely to be correct - can inform our decisions further by providing a graded assessment of expected outcome. Despite the prevalence and obvious utility of these phenomena in everyday life (e.g. learn about our surroundings to make better predictions and plan future actions), their neural substrates and how they can be used to build intelligent machines to improve human decision making when solving important socioeconomic problems remain elusive.
Modern neuroimaging methods made it possible to "take a peak" into the human brain. Human electroencephalography (EEG) is a non-invasive method whereby electrodes placed on the surface of the scalp measure the brain's electrical activity. This technique provides precise information on when the brain processes information in time. In contrast, functional magnetic resonance imaging (fMRI) measures (non-invasively in an MRI scanner) changes in the local blood flow in the brain as we process information. This in turn allows one to identify where in the brain this processing takes place. Here we propose to combine these two methods to simultaneously obtain information on when and where the brain processes information relating to perceptual learning and decision confidence while human participants engage in various decision making tasks.
Crucially, after deciphering the behavioural and neurobiological origins of learning and confidence during decision making we will develop a brain-computer interface system (computer program controlled in part by human brain signals) that exploits these decision correlates to optimise human performance when dealing with problems relying on inconclusive or partially ambiguous evidence in areas such as security (e.g. optimise systems for threat detection), global economy (e.g. optimise marketing and investment strategies) and health and wellbeing (e.g. identify prognostic precursors of cognitive ageing and disorders known to compromise ones decision-making faculties).
The main translational output of our work will be a prototype system for fast and intelligent image triaging of large image/video repositories (e.g. CCTV, satellite, advertising image databases). The sheer volume, diversity and sparsity of items of interest in these databases render a manual search for targets of interest very ineffective. Our system will improve human performance (i.e. targets detected per unit time) by presenting a large amount of images in rapid succession (i.e. increase throughput relative to manual search) and re-prioritise the image sequence, placing those that are more likely to contain a target of interest in the front of an image stack. Re-prioritising will be achieved in real-time, by intercepting (non-invasively) brain signals that are indicative of ones decision about a potential target of interest while exploiting signatures of learning and confidence as identified above.
This work will be performed collaboratively with our international partners at Columbia University in New York. Together, we started to use cutting-edge technology and analysis tools that allow us to characterise and exploit neural signatures associated with human decision making (this work was recently featured in a BBC Horizon documentary).
Modern neuroimaging methods made it possible to "take a peak" into the human brain. Human electroencephalography (EEG) is a non-invasive method whereby electrodes placed on the surface of the scalp measure the brain's electrical activity. This technique provides precise information on when the brain processes information in time. In contrast, functional magnetic resonance imaging (fMRI) measures (non-invasively in an MRI scanner) changes in the local blood flow in the brain as we process information. This in turn allows one to identify where in the brain this processing takes place. Here we propose to combine these two methods to simultaneously obtain information on when and where the brain processes information relating to perceptual learning and decision confidence while human participants engage in various decision making tasks.
Crucially, after deciphering the behavioural and neurobiological origins of learning and confidence during decision making we will develop a brain-computer interface system (computer program controlled in part by human brain signals) that exploits these decision correlates to optimise human performance when dealing with problems relying on inconclusive or partially ambiguous evidence in areas such as security (e.g. optimise systems for threat detection), global economy (e.g. optimise marketing and investment strategies) and health and wellbeing (e.g. identify prognostic precursors of cognitive ageing and disorders known to compromise ones decision-making faculties).
The main translational output of our work will be a prototype system for fast and intelligent image triaging of large image/video repositories (e.g. CCTV, satellite, advertising image databases). The sheer volume, diversity and sparsity of items of interest in these databases render a manual search for targets of interest very ineffective. Our system will improve human performance (i.e. targets detected per unit time) by presenting a large amount of images in rapid succession (i.e. increase throughput relative to manual search) and re-prioritise the image sequence, placing those that are more likely to contain a target of interest in the front of an image stack. Re-prioritising will be achieved in real-time, by intercepting (non-invasively) brain signals that are indicative of ones decision about a potential target of interest while exploiting signatures of learning and confidence as identified above.
This work will be performed collaboratively with our international partners at Columbia University in New York. Together, we started to use cutting-edge technology and analysis tools that allow us to characterise and exploit neural signatures associated with human decision making (this work was recently featured in a BBC Horizon documentary).
Planned Impact
Economic, social and public policy impact: Being able to tap into the neural processes of human decision making can provide the platform for developing technologies that help inform a variety of socioeconomic problems. The key translational aspect of our work (to be delivered at project completion) is the design of a prototype system for brain-informed image triaging. Image and video repositories are growing at an exponential rate and as a result the efficient searching of these databases, given their size, diversity and sparsity of items of interest, has become a major problem. Crucially, this problem arises in domains as diverse as intelligence image analysis (e.g. identifying targets of interest and exposing security threats) and marketing (e.g. informing advertisement strategies and product design).
To date, search technologies in intelligence image analysis rely heavily on computer vision programs to explore the relevant databases despite being less effective than human image analysts (IAs) who are limited in number relative to the enormous amount of imagery available. Similarly, multimedia search for identifying effective advertisement material rely mainly on cumbersome market research tools (e.g. surveys/questionnaires). Our work will have direct impact on these and related tasks by utilising neural correlates of decision making to optimise image search and improve IAs performance relative to manual search.
Our system will dramatically increase throughput by presenting an IA with a large amount of images in rapid succession and re-prioritising the order in which the images are later scrutinised (using real-time, non-invasive brain signals), by placing those that are more likely to contain task-relevant information in the front of an image stack. The system can have a wide range of applications in remote sensing, multimedia search, neuromarketing, advanced gaming interfaces etc. We aim to further develop this technology through well-established links with industrial partners (see Pathways to Impact). We believe the system can have direct benefits for a number of organisations in government/MoD (e.g. remote sensing and intelligence image analysis) and the private sector (e.g. marketing and multimedia search).
As optimal decision making is at the heart of strategic planning, providing a mechanistic account of human decision making can also have wider, long-term, socioeconomic impact on problems ranging from policy making and risk management to informing individual decisions on health behaviours and savings strategies. Furthermore, identifying reliable markers of decision confidence in particular can help inform problems that rely on inconclusive or partially ambiguous evidence (e.g. medical diagnosis and treatment) by providing a probabilistic assessment of expected outcome such that alternative actions are better evaluated.
Health and Wellbeing: In normal and abnormal cognitive ageing, even simple decision making is affected, as evidenced by changes in behavioural measures such as accuracy and response time. In addition, a large number of disorders known to compromise ones decision-making faculties (e.g. ADHD, Schizophrenia, autism, personality disorders) can also lead to similar behavioural changes. The imaging tools proposed here could provide a useful tool for characterising the neural changes that are causal to these behavioural changes and lead to the development of new methods for identifying diagnostic and prognostic indicators of cognitive deficits and more directed behavioural and pharmacological treatments at a much earlier stage. This is another very tangible goal, which we plan to pursue as soon as we roll out the first results from this study. To this end we already established links with local colleagues - experts in ageing and mental disorders (see Pathways to Impact) - to formally test whether such prognostic indicators can be used in the clinic.
To date, search technologies in intelligence image analysis rely heavily on computer vision programs to explore the relevant databases despite being less effective than human image analysts (IAs) who are limited in number relative to the enormous amount of imagery available. Similarly, multimedia search for identifying effective advertisement material rely mainly on cumbersome market research tools (e.g. surveys/questionnaires). Our work will have direct impact on these and related tasks by utilising neural correlates of decision making to optimise image search and improve IAs performance relative to manual search.
Our system will dramatically increase throughput by presenting an IA with a large amount of images in rapid succession and re-prioritising the order in which the images are later scrutinised (using real-time, non-invasive brain signals), by placing those that are more likely to contain task-relevant information in the front of an image stack. The system can have a wide range of applications in remote sensing, multimedia search, neuromarketing, advanced gaming interfaces etc. We aim to further develop this technology through well-established links with industrial partners (see Pathways to Impact). We believe the system can have direct benefits for a number of organisations in government/MoD (e.g. remote sensing and intelligence image analysis) and the private sector (e.g. marketing and multimedia search).
As optimal decision making is at the heart of strategic planning, providing a mechanistic account of human decision making can also have wider, long-term, socioeconomic impact on problems ranging from policy making and risk management to informing individual decisions on health behaviours and savings strategies. Furthermore, identifying reliable markers of decision confidence in particular can help inform problems that rely on inconclusive or partially ambiguous evidence (e.g. medical diagnosis and treatment) by providing a probabilistic assessment of expected outcome such that alternative actions are better evaluated.
Health and Wellbeing: In normal and abnormal cognitive ageing, even simple decision making is affected, as evidenced by changes in behavioural measures such as accuracy and response time. In addition, a large number of disorders known to compromise ones decision-making faculties (e.g. ADHD, Schizophrenia, autism, personality disorders) can also lead to similar behavioural changes. The imaging tools proposed here could provide a useful tool for characterising the neural changes that are causal to these behavioural changes and lead to the development of new methods for identifying diagnostic and prognostic indicators of cognitive deficits and more directed behavioural and pharmacological treatments at a much earlier stage. This is another very tangible goal, which we plan to pursue as soon as we roll out the first results from this study. To this end we already established links with local colleagues - experts in ageing and mental disorders (see Pathways to Impact) - to formally test whether such prognostic indicators can be used in the clinic.
Publications
Arabadzhiyska D
(2021)
A common neural currency account for social and non-social decisions
Arabadzhiyska DH
(2022)
A Common Neural Account for Social and Nonsocial Decisions.
in The Journal of neuroscience : the official journal of the Society for Neuroscience
Balsdon T
(2023)
Secondary motor integration as a final arbiter in sensorimotor decision-making.
in PLoS biology
Delis I
(2016)
Space-by-time decomposition for single-trial decoding of M/EEG activity.
in NeuroImage
Delis I
(2022)
Neural Encoding of Active Multi-Sensing Enhances Perceptual Decision-Making via a Synergistic Cross-Modal Interaction
in The Journal of Neuroscience
Diaz J
(2017)
Perceptual learning alters post-sensory processing in human decision-making
in Nature Human Behaviour
Faller J
(2019)
Regulation of arousal via online neurofeedback improves human performance in a demanding sensory-motor task.
in Proceedings of the National Academy of Sciences of the United States of America
| Title | Short video animation about reward learning |
| Description | An animation video presenting our recent work on decision making and learning for a lay audience. |
| Type Of Art | Film/Video/Animation |
| Year Produced | 2017 |
| Impact | The video was added to our lab's YouTube channel and was made publically available to engage the non-experts with the work we carry out in the lab. |
| URL | https://www.youtube.com/watch?v=CasPtVhDW2E&t=5s |
| Description | Consider an image intelligence analyst inspecting a large array of noisy CCTV or satellite images in order to identify targets that might pose a real security threat. Her ability to perform this task successfully depends on her years of experience in interpreting such images. This example highlights that training and experience are required to induce long-lasting improvements in our ability to make decisions based on ambiguous sensory information a phenomenon commonly referred to as perceptual learning. Despite the prevalence and obvious utility of this phenomenon in everyday life (e.g. learning in an ever-changing environment to make better predictions and plan future actions), its neural substrates and how these affect decision-making remain elusive. An emerging view in perceptual learning is that improvements in perceptual sensitivity are not only due to enhancements in early sensory representations but also due to changes in post-sensory decision processing. In humans, however, direct neurobiological evidence of the latter account remains scarce. Here, we trained participants on a visual categorization task over three days and used multivariate pattern analysis of the electroencephalogram to identify two temporally-specific components encoding sensory (Early) and decision (Late) evidence, respectively. Importantly, the single-trial amplitudes of the Late, but not the Early component, were amplified in the course of training and these enhancements predicted the behavioural improvements on the task. Correspondingly, we modelled these improvements with a reinforcement learning mechanism, using a reward prediction error signal to strengthen the readout of sensory evidence used for the decision. We validated this mechanism through a robust association between the model's decision variables and our Late component's amplitudes indexing decision evidence (see Diaz et al., Nature Human Behaviour, 2017). We are currently following up these experiments by coupling single-trial electroencephalography (EEG) with simultaneously acquired functional magnetic resonance imaging (fMRI). An EEG machine records brain activity with high temporal precision ("when" things are happening in the brain) while functional MRI provides information on the location of this activity ("where" things are happening in the brain). To date, "when" and "where" questions have largely been studied separately, using each technique in isolation. By combining the two techniques we plan to map out the full spatiotemporal dynamics of the brain networks associated with perceptual learning. More specifically, we designed motion discrimination tasks designed to enable perceptual learning via an adaptive adjustment of stimulus difficulty to maintain a constant performance (~65% accuracy) throughout the experiment. To capture decision confidence we used a post-decision wagering manipulation (i.e. betting on the decision to double the payout/penalty in case of success/error). Trials with and without feedback were randomly interleaved to probe the role of confidence during different learning contexts. Participants' neural responses were recorded through simultaneous EEG-fMRI. We found robust learning effects as indicated by a systematic increase in stimulus difficulty during the experiment. Confidence effects were also evident as indicated by improved accuracy and faster responses on confident compared to less-confident trials ("bet"-vs-"no-bet" trials). EEG results revealed outcome-locked activity scaling with outcome value, but not confidence, during feedback trials. In contrast, the same sensors revealed a modulation by confidence but not outcome value during no-feedback trials. These results are consistent with the notion that in the absence of explicit feedback, confidence might act as an implicit reward, with higher confidence being a proxy for success. We hypothesise that confidence signals can be integrated into a reinforcement-guided mechanism to drive learning and we are currently finalizing the analysis of the simultaneously acquired fMRI data to validate this interpretation and identify the relevant neural networks. The other aim of our study was to study the role of subjective confidence in decision making. Our everyday lives involve frequent situations where we must make judgments based on noisy or incomplete sensory information, such as whether it is safe to cross the street on a foggy morning, when visibility is poor. Being able to rely on an internal estimate of whether our perceptual judgments are accurate is therefore fundamental to adaptive behaviour. Correspondingly, recent years have seen a growing interest in understanding the neural basis of confidence judgments. Most human studies to date have used fMRI to identify the regions of the brain encoding confidence at the time of metacognitive evaluation (i.e., while subjects actively judge their performance following a choice). In this work we collected simultaneous EEG-fMRI data to map out the full spatiotemporal dynamics of the role of confidence in decision making, starting as early as the decision process itself. In doing so we first analysed the EEG data to temporally characterise perceptual confidence, and identified an early component, which reliably discriminated between High vs. Low confidence choices. Next, we exploited the trial-by-trial variability in this neural signature to inform the analysis of the fMRI data in order to identify the brain regions involved, thereby characterising decisional confidence with both high temporal and high spatial precision. Importantly, these endogenous neural signatures of confidence represent a graded neural response that hold a more accurate representation of internal confidence than can be captured by behavioural ratings alone, and thus have the potential to reveal latent brain states that may not otherwise be visible using conventional methods. Using this analysis framework, we identified a positive correlation with decisional confidence within the ventromedial prefrontal cortex (vmPFC), a region not previously associated with perceptual confidence, and one that was not visible with a conventional analysis of the fMRI data. Furthermore, a functional connectivity analysis revealed a confidence-level dependent link between the activity in the vmPFC and a region in the prefrontal cortex previously implicated in metacognitive evaluation. This suggests that signatures of confidence evolve early in the decision process and are subsequently passed onto to a separate network for metacognitive assessment (see Gherman and Philiastides, 2018). Furthermore, this work suggests that these confidence representations can causally affect subsequent choices and learning. Specifically, we identified sequential trial dependencies between our neural estimates of confidence and subjects' behavioural performance - for example more confident trials led to an increased likelihood of repeating the same choice in the following trial. Finally, we utilised the neural signatures of learning and confidence we uncovered above to develop a brain-computer-interface (BCI) system to inform and ultimate influence behaviour in a number of high throughput tasks. For example, we looked at pilot induced oscillations (PIOs) that are potentially catastrophic events that occur during flight when pilots attempt to control an aircraft close to a performance or physical boundary. PIO-like behaviour is typically observed in boundary avoidance tasks (BAT), which simulate tight performance or physical boundaries and induce high cognitive workload. Our previous research linked the occurrence of PIO-like behaviour to network level activity in the brain, where higher states of arousal reduce the flexibility of decision making networks such that less environmental information was incorporated to dynamically adjust action. This led us to hypothesize that down regulating arousal via closed-loop audio feedback of a user state could improve piloting performance by enabling learning of environmental contingencies and thereby increased decision flexibility. We implemented a hybrid brain computer interface (hBCI) to dynamically provide feedback to a "pilot" that facilitates their ability to reduce their state of arousal. We conduct a systematic comparison relative to control and sham conditions and test to see if this feedback increases the time a "pilot" can fly before a catastrophic PIO. We find that hBCI feedback, which includes central nervous system components consistent with theta activity in the anterior cingulate cortex (ACC), enables prolonged flight relative to closed-loop control and sham feedback. We also find that this feedback induces changes in pupil diameter, which are absent in open-loop conditions and closed-loop conditions when feedback is not veridical. Pupil diameter has been reported as a surrogate measure of activity in the locus coeruleus-norepinephrine (LC-NE) system, which is also linked to a circuit that includes the ACC. We conclude that the feedback we induce with our hBCI provides preliminary evidence that self-regulation of LC-NE/ACC is possible and can be used to dynamically increase decision flexibility when under high cognitive workload (see Saproo et al., 2016). Many real-world decisions rely on active sensing, a dynamic process for directing our sensors (e.g. eyes or fingers) across a stimulus to maximize information gain. Though ecologically pervasive, limited work has focused on identifying neural correlates of the active sensing process. In tactile perception, we often make decisions about an object/surface by actively exploring its shape/texture. As part of this ESRC grant we also investigated the neural correlates of active tactile decision-making by simultaneously measuring electroencephalography (EEG) and finger kinematics while subjects interrogated a haptic surface to make perceptual judgments. Since sensorimotor behaviour underlies decision formation in active sensing tasks, we hypothesized that the neural correlates of decision-related processes would be detectable by relating active sensing to neural activity. Novel brain-behaviour correlation analysis revealed that three distinct EEG components, localizing to right-lateralized occipital cortex (LOC), middle frontal gyrus (MFG), and supplementary motor area (SMA), respectively, were coupled with active sensing as their activity significantly correlated with finger kinematics. To probe the functional role of these components, we fit their single-trial-couplings to decision-making performance using a hierarchical-drift-diffusion-model (HDDM), revealing that the LOC modulated the encoding of the tactile stimulus whereas the MFG predicted the rate of information integration towards a choice. Interestingly, the MFG disappeared from components uncovered from control subjects performing active sensing but not required to make perceptual decisions. By uncovering the neural correlates of distinct stimulus encoding and evidence accumulation processes, this study delineated, for the first time, the functional role of neural circuits in active tactile decision-making. |
| Exploitation Route | Our general research approach (simultaneous EEG-fMRI and corresponding analysis methods) as well as our scientific findings open up new avenues for the investigation of the neural systems underlying decision making in humans. Crucially, our findings have the potential to further our understanding of how everyday responses to new information can affect our capacity to make optimal decisions, as well as facilitate the study of how mental disorders - such as chronic stress, obsessive-compulsive-disorder, post-traumatic disorder and depression - affect learning and strategic planning. Correspondingly, our work can lead to the development of new methods for identifying diagnostic and prognostic indicators of cognitive deficits in disease, as well as more directed behavioural and pharmacological treatments. It is also likely that our results can have significant impact on how artificial neural networks are trained and adapted to learn to deal with an ever-increasing number of online "big data" applications. Correspondingly, our ability to integrate neural signatures of human cognition into the development of brain-computer interfaces promises to be a valuable tool for monitoring and ultimately augmenting human performance during tasks requiring high throughput of information and high levels of cognitive load. We have made two full EEG-fMRI datasets fully available on openneuro.org to inspire the development of future fusion methodologies for EEG and fMRI data and to encourage others to either re-analyse the data (aiming at reproducibility) or use them to address new and novel questions. Furthermore, we utilised the outcome of this work to inform the development of novel biomarkers in Psychiatry. Specifically, we have used the paradigms and neural signatures developed through this work (neural correlates of "learning" in particular) to demonstrate that brain activation patterns in a sample of depressed patients undergoing scanning prior to commencing treatment (Cognitive Behavioural Therapy) were predictive of the subsequent success or failure of the treatment. Our findings offer a paradigm shift in Psychiatry and enable a potentially significant and far-reaching impact on the way patients are triaged and treated for mental health problems. This work has only recently been completed and we are currently in the process of replicating and validating the findings in a larger sample of patients. Completing this next step successfully would pave the way for introducing our predictive biomarkers in a day-to-day clinical practice. |
| Sectors | Aerospace, Defence and Marine,Communities and Social Services/Policy,Creative Economy,Digital/Communication/Information Technologies (including Software),Education,Healthcare,Government, Democracy and Justice,Pharmaceuticals and Medical Biotechnology,Security and Diplomacy,Transport |
| Description | As we had originally planned, one of the practical contributions of this work is a brain-computer interface (BCI) platform that uses information in the EEG to generate a neurofeedback signal that dynamically adjusts an individual's arousal and decision confidence states when they are engaged in a boundary-avoidance task (BAT; e.g. controlling a fighter jet). Specifically, the BAT is a demanding high throughput aerial navigation task we developed in virtual reality and which creates cognitive conditions that escalate arousal and quickly results in task failure (e.g., missing or crashing into the boundary). We demonstrate that task performance, measured as time and distance over which the subject can navigate before failure, is significantly improved when veridical neurofeedback is provided. Our work has practical implications in that it demonstrates a BCI system that uses online neurofeedback can lead to better learning and increased task performance. This approach is potentially applicable to different task domains and/or for clinical applications that utilize self-regulation as a targeted treatment, such as in mental illness. We are currently exploring these new avenues and their potential utility for future applications. Furthermore, we utilised the outcome of this work to inform the development of novel biomarkers in Psychiatry. Specifically, we have used the paradigms and neural signatures developed through this work (neural correlates of "learning" in particular) to demonstrate that brain activation patterns in a sample of depressed patients undergoing scanning prior to commencing treatment (Cognitive Behavioural Therapy) were predictive of the subsequent success or failure of the treatment. Our findings offer a paradigm shift in Psychiatry and enable a potentially significant and far-reaching impact on the way patients are triaged and treated for mental health problems. This work has only recently been completed and we are currently in the process of replicating and validating the findings in a larger sample of patients. Completing this next step successfully would pave the way for introducing our predictive biomarkers in a day-to-day clinical practice. This work has the potential to transform clinical practice in the field of Psychiatry and in particular the way depressed patients are being assessed and assigned to treatment. Specifically, our approach has the potential to predict the efficacy of different treatment protocols *prior* to the deployment of treatment via the use of predictive imaging biomarkers. Critically, these biomarkers offer additional predictive power, over and above what could be inferred by standard clinical assessment tools alone). As such, our approach can optimise both the outcome and time-to-outcome, while at the same time minimize the use of valuable NHS resources. Finally, since our approach allows us to model the mechanisms behind treatment response, it can lead to the identification of novel target sites for future drug development. |
| First Year Of Impact | 2019 |
| Sector | Aerospace, Defence and Marine,Communities and Social Services/Policy,Healthcare,Government, Democracy and Justice,Pharmaceuticals and Medical Biotechnology |
| Impact Types | Societal,Economic,Policy & public services |
| Description | Chair of the IEEE Brain Initiative |
| Geographic Reach | Multiple continents/international |
| Policy Influence Type | Contribution to a national consultation/review |
| URL | http://brain.ieee.org |
| Description | Participant in US Army 2040 Vision |
| Geographic Reach | Multiple continents/international |
| Policy Influence Type | Participation in a guidance/advisory committee |
| Description | Summer School on Brain Imaging |
| Geographic Reach | Multiple continents/international |
| Policy Influence Type | Influenced training of practitioners or researchers |
| Impact | We have been developing an international summer school on advanced brain imaging methods that will run on a yearly basis at the Centre for Cognitive Neuroimaging at the University Glasgow starting in 2016. |
| Description | Dynamic Network Reconstruction of Human Perceptual and Reward Learning via Multimodal Data Fusion |
| Amount | € 2,000,000 (EUR) |
| Funding ID | 865003 |
| Organisation | European Research Council (ERC) |
| Sector | Public |
| Country | Belgium |
| Start | 09/2020 |
| End | 08/2025 |
| Title | EEG-informed fMRI analysis tool |
| Description | We developed a multivariate single-trial analysis framework for integrating simultaneously acquired EEG and whole-brain fMRI data. Specifically, we exploit trial-to-trial variability of electrophysiologically derived measures to inform the subsequent analysis of the simultaneously acquired fMRI data, to tease apart the cascade of constituent cortical processes involved in value-based decision making. |
| Type Of Material | Improvements to research infrastructure |
| Provided To Others? | No |
| Impact | Our analysis techniques can be used in way that supersedes what is currently considered common analysis practice in the human neuroimaging literature. Our analysis framework has the potential to open new avenues in analysing human neuroimaging data in ways that help identify latent brain states that would otherwise be missed using more conventional analysis tools to independently analyse EEG and fMRI data. We plan to make our analysis tools freely available upon completion of the project and submission of all relevant work for publication. |
| Title | Combined EEG/fMRI datasets - Confidence & Learning |
| Description | Combined measurements of electroencephalographic (EEG) and functional magnetic resonance imaging (fMRI) data from 25 healthy participants performing two different tasks designed to probe the neural correlates of perceptual learning and the role of confidence during perceptual decisions. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2017 |
| Provided To Others? | Yes |
| Impact | Combined EEG and fMRI measurements offer an opportunity to gain important new insights into human brain function that cannot be obtained using either modality alone (e.g. revealing latent brain states). Our ESRC funded work clearly demonstrated this in the context of studying perceptual learning and confidence during perceptual decisions (publications in preparation). Importantly, collecting this database allowed us to develop novel techniques to analyse the acquired data in way that supersedes what is currently considered common analysis practice in the human neuroimaging literature. We plan to make the database and our analysis tools fully available upon completion of the project and submission of all relevant work for publication. |
| URL | https://openneuro.org/datasets/ds001512/versions/2.0.0 |
| Title | Combined EEG/fMRI datasets - Value & Reward Learning |
| Description | Combined measurements of electroencephalographic (EEG) and functional magnetic resonance imaging (fMRI) data from 24 healthy participants performing two different tasks designed to probe the neural correlates of value-based decision making and the role of learning during reward-based decisions. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2017 |
| Provided To Others? | No |
| Impact | Combined EEG and fMRI measurements offer an opportunity to gain important new insights into human brain function that cannot be obtained using either modality alone (e.g. revealing latent brain states). Our current BBSRC funded work clearly demonstrated this in the context of studying learning during value- and reward-based decision making. Importantly, collecting this database allowed us to develop novel techniques to analyse the acquired data in way that supersedes what is currently considered common analysis practice in the human neuroimaging literature. The database and our analysis tools are fully available upon request. |
| URL | https://openneuro.org/datasets/ds001219/versions/1.0.0 |
| Description | Brain computer interfaces for augmenting decision making |
| Organisation | Columbia University |
| Country | United States |
| Sector | Academic/University |
| PI Contribution | Agreement established in October 2014. Collaboration will commence in early 2015. We will be providing insights into the behavioural and neurobiological origins of decision making and learning as these have been (and continue to be) developed through our BBSRC (BB/J015393/1-2) and ESRC (ES/L012995/1) awards. The collaboration will focus on the development of cognitive interfaces (brain-computer interfaces) that exploit decision correlates to optimise human performance in dealing with problems relating to security (e.g. optimise target/threat detection), global economy (e.g. optimising marketing strategies) and health and wellbeing (e.g. identifying prognostic indicators of cognitive ageing and disorders known to compromise ones decision-making faculties). |
| Collaborator Contribution | Our partner at Columbia University is an expert in the development of brain computer interface (BCI) systems and will take the lead in developing the actual BCI platform required for the project as highlighted above. |
| Impact | Disciplines involved: neuroscience, psychology, engineering, computer science |
| Start Year | 2014 |
| Description | Closed-loop BCI for Boundary Avoidance Tasks |
| Organisation | US Army Research Lab |
| Country | United States |
| Sector | Public |
| PI Contribution | We are working iwith the Army Research Lab to building brain-computer interface (BCI) demonstrations which use adaptive neurofeedback to dynamically regulate arousal during demanding sensory-motor tasks. This includes new ways of merging human decision makers with autonomy so as to maximize task performance under substantial levels of stress and fatigue. |
| Collaborator Contribution | ARL has helped us develop a paradigm and contributed to the design and testing of a system for BCI-based human-agent teaming (HAT). |
| Impact | Demonstrations, co-authored papers, funding. |
| Start Year | 2015 |
| Description | Decision making in DBS patients |
| Organisation | National Hospital for Neurology and Neurosurgery |
| Department | Unit of Functional Neurosurgery |
| Country | United Kingdom |
| Sector | Hospitals |
| PI Contribution | Analysed LFP data from patients implanted with deep brain electrodes. Future plans for additional experiments are being made. |
| Collaborator Contribution | Provided LFP data. |
| Impact | Journal Publication in preparation (title: Direct recordings from human dorsal anterior cingulate cortex show prediction error signalling but not reaction to stimulus parameters) |
| Start Year | 2016 |
| Description | Modelling of confidence and decision making |
| Organisation | University of Leuven |
| Department | Faculty of Psychology and Educational Sciences |
| Country | Belgium |
| Sector | Academic/University |
| PI Contribution | With our collaborators at KU Leuven, we develop computational models that are informed (and validated) by neural data in order to understand the role of confidence in decision making. We provide all neural data related to this work. |
| Collaborator Contribution | Our collaborators are leading the computational modelling aspects of this work. |
| Impact | This work is multi-disciplinary in nature and requires expertise in multimodal neuroimaging and analysis, computational modelling and advanced statistical modelling. A manuscript is currently under development. |
| Start Year | 2016 |
| Description | Reward learning in depression |
| Organisation | University of Dundee |
| Country | United Kingdom |
| Sector | Academic/University |
| PI Contribution | Developed a new framework for understanding changes in neuronal activity during reward learning in humans that are characterised both in time and space. |
| Collaborator Contribution | Offering access to a clinical group (depression) with surgical lesions to the mid-cingulate cortex to formally test the causal role of this region in reward learning using the framework developed by our research team. |
| Impact | MRC grant application submitted in early 2016. |
| Start Year | 2015 |
| Description | Berlin School of Mind and Brain 2015 |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Other audiences |
| Results and Impact | Workshop with an international audience of academic peers (academic staff, students and research assistants), on current state of the art in multimodal brain imaging. Presentation followed by questions and discussion and , hand-on exercises . |
| Year(s) Of Engagement Activity | 2015 |
| Description | CBU Seminar Cambridge University |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Other audiences |
| Results and Impact | Research talk on value based decision making and learning, perceptual learning and decision confidence. |
| Year(s) Of Engagement Activity | 2018 |
| Description | Cutting edge EEG Workkhop |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Postgraduate students |
| Results and Impact | I organised a four day event in Glasgow to discuss and showcase the most up-to-date developments in EEG research. The event included plenary talks and hands-on workshops. 300 people attended the event (postgraduate students, postdocs, faculty) from more than 100 institutions worldwide. In addition to organising the event I also presented our lab's work during one of the plenary sessions. |
| Year(s) Of Engagement Activity | 2017 |
| URL | http://cuttingeeg.ccni.gla.ac.uk |
| Description | FENS Brain Conference Talk |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Other audiences |
| Results and Impact | Research talk on value based decision making and learning, perceptual learning and decision confidence. |
| Year(s) Of Engagement Activity | 2018 |
| URL | https://www.fens.org/Meetings/The-Brain-Conferences/The-Computational-Neuroscience-of-Prediction/ |
| Description | Invited Lecture at USC Institute for Creative Technology |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Professional Practitioners |
| Results and Impact | Lecture to USC Institute for Creative Technologies and ARL West. Follow-on meeting focused on ways we could collaborating on projects in this area. |
| Year(s) Of Engagement Activity | 2017 |
| Description | Keynote Lecture at IEEE SMC Brain Machine Interface Workshop |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Goal was to provide an overview of research activities focused on integrating brain-computer interface technology with augmented and virtual reality |
| Year(s) Of Engagement Activity | 2017 |
| Description | Keynote at IEEE International Conference on Consumer Electronics (ICCE) |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | I gave a Keynote at ICCE which is part of the Consumer Electronics Show (CES) in Las Vegas. This attracts about 200,000 participants per year. My talk was recorded and is on Youtube (see below). |
| Year(s) Of Engagement Activity | 2017 |
| URL | https://youtu.be/fn9eBJFvSuA |
| Description | Keynote lecture at University of Tuebingen |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Postgraduate students |
| Results and Impact | A research talk on decision making and learning to a group of academic peers (research staff, students) followed by questions and discussion. |
| Year(s) Of Engagement Activity | 2017 |
| URL | http://meg.medizin.uni-tuebingen.de/2017/ |
| Description | Neuroeconomics Society, Annual Conference 2013 |
| Form Of Engagement Activity | Scientific meeting (conference/symposium etc.) |
| Part Of Official Scheme? | Yes |
| Type Of Presentation | poster presentation |
| Geographic Reach | International |
| Primary Audience | Other academic audiences (collaborators, peers etc.) |
| Results and Impact | Shared initial results of BBSRC funded work on the role of learning in economic decision (see abstract below) with a wide range of academic attendees (including post-graduate students and junior postdoctoral fellows). Objective: Adaptive decisions depend on accurate reward representations associated with potential choices. These representations can be acquired with reinforcement learning mechanisms that use the prediction error (PE) - the difference between expected and actual rewards - as a learning signal to update reward expectations. To date, most studies used fMRI and EEG independently to either identify brain regions or activation latencies associated with PE signals. The goal of this study is to couple high temporal resolution, single-trial EEG with simultaneously acquired fMRI to infer the full spatiotemporal dynamics of the brain networks involved in PE processing. Methods: Twenty-four subjects participated in the study. They performed a probabilistic reversal-learning task while we simultaneously collected concurrent EEG/fMRI data. Single-trial multivariate discriminant analysis was used to identify linear spatial weightings of the EEG sensors to find speci?c temporal windows, which optimally discriminated between trials conditioned along different PE valence and magnitude dimensions. The resulting single-trial discriminant component amplitudes were used to build EEG-informed fMRI regressors to identify brain regions correlating with the trial-by-trial variance in PE valence and magnitude. For comparison, a more conventional fMRI analysis was carried out whereby behaviorally derived, model-based regressors were used instead. Results: We confirmed our previous, stand-alone, EEG work by providing a temporal account of PE processing. Specifically, 220 ms after feedback, outcomes are initially evaluated categorically with respect to their valence (positive vs. negative). Around 300 ms and parallel to a second valence-evaluation, the brain also represents quantitative information about PE magnitude. We are currently using trial-by-trial variability intrinsic to these temporally-distinct neural representations to map out the spatial characteristics of PE processing and provide a mechanistic and causal account of the neural networks involved in learning reward contingencies and guiding future actions. Conclusions: Our results suggest that the temporal brain dynamics of PE processing can be inferred reliably using single-trial EEG acquired inside an MR-scanner. Crucially, to provide a complete spatiotemporal characterization of the underlying networks, single-trial EEG estimates associated with PE valence and magnitude can be used to inform the analysis of simultaneously acquired fMRI data. We propose that EEG-informed fMRI has the potential to expose latent brain states - that could otherwise go astray using conventional model-based fMRI - by exploiting trial-by-trial variability in electrophysiologically- rather than behaviorally-derived measures. Presenting at this conference sparked later exchange with academic colleagues in related disciplines (e.g. in the form of requests for additional information on the work and invitation for seminar talks for the PI and lead postdoctoral fellow). |
| Year(s) Of Engagement Activity | 2013 |
| Description | Neuroeconomics Society, Annual Conference 2014 |
| Form Of Engagement Activity | Scientific meeting (conference/symposium etc.) |
| Part Of Official Scheme? | Yes |
| Type Of Presentation | poster presentation |
| Geographic Reach | International |
| Primary Audience | Other academic audiences (collaborators, peers etc.) |
| Results and Impact | Shared initial results of BBSRC funded work on the role of the role of spatial attention (eye-movements) on consumer decision making (see abstract below) with a wide range of academic attendees (including post-graduate students and junior postdoctoral fellows). Objective: Product branding is a crucial dimension of consumer choices. Recent work has suggested that branding information and subjective product preference may be integrated into a single source of evidence in the decision-making process. Here we investigate how exactly these two sources of information are combined, by employing the attentional drift-diffusion model (aDDM) to relate choices and reaction times to the relative gaze time on the two products and their brands. Methods: We carried out an experiment in which subjects made a series of hypothetical preference decisions between two items of clothing paired with different designer brands. In control trials subjects also made preference-based clothing decisions, but with phase-scrambled brand images. While subjects made these choices, we tracked their eye-movements. Beforehand we also collected separate individual ratings for each clothing item and brand. We then used subjects' ratings and gaze patterns as inputs to the aDDM to test whether these measures alone could account for subjects' choices and reaction times. Results: Using the aDDM we were able to accurately predict the influence of gaze time on the probability of choosing the left or right item. Comparing the intact brand trials to the scrambled control trials, we find that subjects spent more time looking at the brand information, took longer to make their decisions, and were more likely to choose an item if it was paired with a preferred brand. Furthermore, we were able to use the aggregate fraction of time spent looking at the brands to predict the average influence of the brand ratings on subjects' choices. This relationship was further established with a significant across-subject correlation between brand gaze time and brand weight in their utility functions. Finally, consistent with previous aDDM findings, we observed no correlation between item or brand ratings and gaze duration. Conclusions: Our results indicate that branding information and subjective product preference are combined together in a multi-attribute drift-diffusion model, where the relative weights on the two attributes are determined by the gaze time on the product vs. brand. These findings lend further support to the aDDM as a common mechanism underlying value-based decisions and are consistent with the hypothesis that in binary choice, attention leads to preference, and not vice-versa. Presentation sparked later exchange with academic colleagues in related disciplines (e.g. in the form of requests for additional information on the work and requests for potential collaborators). |
| Year(s) Of Engagement Activity | 2014 |
| Description | Open Days, Studying decision making using brain imaging, 2016 |
| Form Of Engagement Activity | Participation in an open day or visit at my research institution |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Public/other audiences |
| Results and Impact | Open Days, University of Glasgow. The event is designed to inform prospective students and their families about ongoing research activities in our department. We used this opportunity to present our work to the general public. |
| Year(s) Of Engagement Activity | 2016 |
| Description | Organisation of Human Brain Mapping, Annual Meeting 2014 |
| Form Of Engagement Activity | Scientific meeting (conference/symposium etc.) |
| Part Of Official Scheme? | Yes |
| Type Of Presentation | poster presentation |
| Geographic Reach | International |
| Primary Audience | Other academic audiences (collaborators, peers etc.) |
| Results and Impact | Shared initial results of BBSRC funded work on the role of learning in economic decision (see abstract below) with a wide range of academic attendees (including post-graduate students and junior postdoctoral fellows) as well as a small number of the general public. Introduction: Adaptive decisions depend on accurate reward representations associated with potential choices. These representations can be acquired with reinforcement learning mechanisms that use the prediction error (PE) - the difference between expected and actual outcomes - as a learning signal to update expectations [1]. To date, most studies used fMRI and EEG independently to either identify brain regions or activation latencies associated with PE signals [2, 3]. The goal of this study is to couple high temporal resolution, single-trial EEG with simultaneously acquired fMRI to infer the full spatiotemporal dynamics of the brain networks involved in PE processing. Methods: Twenty-four subjects participated in the study (9 males, 24yrs±3). Data were acquired on a 3T Philips MR scanner and 64-channel Brain Products EEG system. Subjects performed a probabilistic reversal-learning task while we collected, simultaneously, EEG and fMRI data. Single-trial multivariate discriminant analysis was used to identify linear spatial weightings of the EEG sensors to find speci?c temporal windows, which optimally discriminated between trials conditioned along different PE valence and magnitude dimensions [4]. EEG-informed fMRI regressors were built from the resulting single-trial discriminant component amplitudes and employed in a random-effects GLM analysis to identify brain regions correlating with the trial-by-trial variance in PE valence and magnitude. For comparison, a more conventional fMRI analysis was carried out whereby behaviorally derived, model-based regressors were used in the analysis instead. Results: We provide additional support to our previous, stand-alone, EEG work [3] by characterizing the temporal dynamics of PE processing. Specifically, 220 ms after feedback, outcomes are initially evaluated categorically with respect to their valence (positive vs. negative). Around 300 ms and parallel to a second valence-evaluation, the brain also represents quantitative information about PE magnitude, which is partially correlated with the trial-by-trial changes in model-based PE magnitude (r ¯ = 0.3, p<.001). Importantly, we also show that unaccounted variance in single-trial EEG amplitudes has additional predictive power in explaining fMRI data and we have begun to identify brain regions correlating with each of our temporally-specific EEG components. Additionally, we expect that activations traditionally absorbed by a single regressor in a model-based analysis (e.g. for PE valence) will now be distributed across regressors from temporally distinct components, thereby allowing us to begin making causal inferences about the underlying network. Conclusions: Our results suggest that the temporal dynamics of PE processing can be inferred reliably using single-trial EEG during simultaneous fMRI. Crucially, to provide a complete spatiotemporal characterization of the underlying networks, single-trial EEG estimates associated with PE valence and magnitude can be used to inform the analysis of the fMRI data. These results extend previous findings and allow us to begin mapping the spatiotemporal profile of PE processing, providing the first instance of a mechanistic and causal account of the network involved in learning reward contingencies. We propose that EEG-informed fMRI has the potential to expose latent brain states - that could otherwise go astray using conventional model-based fMRI - by exploiting trial-by-trial variability in electrophysiologically- rather than behaviorally-derived measures. Presenting at this conference sparked later exchange with academic colleagues in related disciplines (e.g. in the form of requests for additional information on the work and invitation for seminar talks for the PI and lead postdoctoral fellow). |
| Year(s) Of Engagement Activity | 2014 |
| Description | Plenary Lecture, IEEE ANTBI Workshop, Shanghai |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Lecture at largest IEEE Neural Engineering Conference. Report research and make outreach to research and technical communities. |
| Year(s) Of Engagement Activity | 2017 |
| Description | Presentation at Human Brain Project Summit |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Public/other audiences |
| Results and Impact | Presented our recent work on value-based decision making and learning and the importance of multimodal neuroimaging in enabling new neuroscientific discoveries. The event was designed to showcase recent neuroscience developments to the general public and expert professional. The audience was comprised of about 300 people of all ages and ranging in expertise as expected (general public to the neuroscience professional). |
| Year(s) Of Engagement Activity | 2017 |
| URL | https://www.humanbrainproject.eu/en/follow-hbp/news/5th-annual-human-brain-project-summit/ |
| Description | Science Slam, Studying decision Making using brain imaging, 2016 |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Public/other audiences |
| Results and Impact | Science Slam IV, Glasgow. The event encourages PhD students to present their research in creative ways and to do something the audience has never seen before. We used this opportunity to present our ongoing research activities to the general public. |
| Year(s) Of Engagement Activity | 2016 |
| Description | Scottish Neuroscience Meeting, 2014 |
| Form Of Engagement Activity | Scientific meeting (conference/symposium etc.) |
| Part Of Official Scheme? | Yes |
| Type Of Presentation | keynote/invited speaker |
| Geographic Reach | National |
| Primary Audience | Other academic audiences (collaborators, peers etc.) |
| Results and Impact | Presented a summary of our recent work on the neurobiology of decision making to academic colleagues (neuroscientists) from throughout Scotland as well as a small number of members of the general public. Increased in requests for additional information, further participation in our study and scientific involved in our lab (e.g. postdoc and studentship requests). |
| Year(s) Of Engagement Activity | 2014 |
| Description | TEDx Talk 2020 |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Public/other audiences |
| Results and Impact | Invited to give a TEDx talk in Glasgow on our recent work on multimodal brain imaging of human decision making and learning. The talk focused on the latest technologies used for brain imaging and their application on mental health - more specifically, their role in diagnostic stratification and predicting treatment response in depression, enabling early intervention and more favourable patient outcomes. TED is one of the most recognised brands worldwide, focusing on "ideas worth sharing" that have potential for significant impact. TED talks freely available online on TED and TEDx youtube channels, totalling more than 40 million subscribers across the globe. Therefore the impact of this talk is presumably far reaching. |
| Year(s) Of Engagement Activity | 2020 |
| Description | Talk at Oxford University - Functional Neurosurgery Unit 2016 |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Other audiences |
| Results and Impact | A research talk on decision making and learning to a group of academic peers (research staff, students) and clinicians followed by questions and discussion. Plans for future collaborations (looking at clinical groups) made. |
| Year(s) Of Engagement Activity | 2016 |
| Description | Talk at the British Association for Cognitive Neuroscience |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Postgraduate students |
| Results and Impact | A research talk on decision making and learning to a group of academic peers (research staff, students) followed by questions and discussion. |
| Year(s) Of Engagement Activity | 2017 |
| Description | University of Leuven Visit, 2015 |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Other audiences |
| Results and Impact | Research talk on decision making and learning to an audience of academic peers (research staff, students) followed by questions and discussion. |
| Year(s) Of Engagement Activity | 2015 |
| Description | University of Zurich Visit, 2014 |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Other audiences |
| Results and Impact | A research talk on decision making and learning to a group of academic peers (research staff, students) followed by questions and discussion. |
| Year(s) Of Engagement Activity | 2014 |
| Description | invited lecture at Oculus Research |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Industry/Business |
| Results and Impact | Present research efforts that integrate neural and physiological measures to make inferences of cognitive and emotional state while users are immersived in virtual reality (VR). |
| Year(s) Of Engagement Activity | 2017 |