Closed-Loop Multisensory Brain-Computer Interface for Enhanced Decision Accuracy

Lead Research Organisation: Imperial College London
Department Name: Electrical and Electronic Engineering


The goals of our interdisciplinary effort are to develop new methodologies for modeling multimodal neural activity underlying multisensory processing and decision making, and to use those methodologies to design closed-loop adaptive algorithms for optimized exploitation of multisensory data for brain-computer communication. We are motivated by the observation that a dismounted soldier or a tank driver routinely makes decisions in time-pressured and stressful conditions based on a multiplicity of multisensory information presented in cluttered and distracting environments. We envision a closed-loop brain-computer interface (BCI) architecture for enhancing decision accuracy. The architecture will collect multimodal neural, physiological, and behavioral data, decode mental states such as attention orientation and situational awareness, and use the decoded states as feedback to adaptively change the multisensory cues provided to the subject, thus closing the loop. To realize such an architecture we will make fundamental advances on four fronts, constituting our research Thrusts: (1) modeling multisensory integration, attention, and decision making, and the associated neural mechanisms; (2) machine-learning algorithms for high-dimensional multimodal data fusion; (3) adaptive tracking of the neural and behavioral models during online operation of the BCI; and (4) adaptive BCI control of multisensory cues for optimized performance. We have assembled a multidisciplinary team with expertise spanning engineering, computer science, and neuroscience. We will take a fully integrated approach to address these challenges by combining rare state-of-the-art experimental capabilities with novel computational modeling. Complementary experiments in rodents, monkeys, and humans will collect multimodal data to study and model multisensory integration, attention, and decision making, and to prototype a BCI for enhanced decision accuracy. Our modeling efforts will span Bayesian inference, stochastic control, adaptive signal processing, and machine learning to develop: novel Bayesian and control-theoretic models of the brain mechanisms; new stochastic models of multimodal data and adaptive inference algorithms for this data; and novel adaptive stochastic controllers of multisensory cues based on the feedback of users' cognitive state.


10 25 50
Description Imperial College Dementia Research Centre
Amount £20,000,000 (GBP)
Funding ID Imperial College Dementia Research Centre 
Organisation Care UK 
Sector Private
Country United Kingdom
Start 10/2019 
End 09/2026
Description Low-cost high-tech concussion assessment and post-fall health monitoring, funded by The Racing Foundation
Amount £241,000 (GBP)
Organisation British Heart Foundation (BHF) 
Sector Charity/Non Profit
Country United Kingdom
Start 10/2018 
End 11/2020
Description Collaboration with Sonion, manufacturer of hearing aid components 
Organisation Sonion
Country Denmark 
Sector Private 
PI Contribution Sonion have provided us with their last generation MEMS microphones and in-ear receivers, which we will use in order to improve our current multimodal sensor
Collaborator Contribution Expert advice and parts, ca 600 GBP
Impact Improved multimodal sensor
Start Year 2017
Title Electrocardiogram Apparatus and Method 
Description This is an appartus and method to record in-ear Electrocardiogram 
IP Reference WO2019077363 
Protection Patent application published
Year Protection Granted 2018
Licensed No
Impact n/a yet
Description Imperial Techno-Festival 
Form Of Engagement Activity Participation in an open day or visit at my research institution
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact We are showcasing our technology at this Imperial open event
Year(s) Of Engagement Activity 2019