Closed-Loop Multisensory Brain-Computer Interface for Enhanced Decision Accuracy
Lead Research Organisation:
Imperial College London
Department Name: Electrical and Electronic Engineering
Abstract
Abstracts are not currently available in GtR for all funded research. This is normally because the abstract was not required at the time of proposal submission, but may be because it included sensitive information such as personal details.
Publications
Adjei T
(2018)
The Female Heart: Sex Differences in the Dynamics of ECG in Response to Stress.
in Frontiers in physiology
Adjei T
(2019)
The ClassA Framework: HRV Based Assessment of SNS and PNS Dynamics Without LF-HF Controversies.
in Frontiers in physiology
Amadori P
(2022)
HammerDrive: A Task-Aware Driving Visual Attention Model
in IEEE Transactions on Intelligent Transportation Systems
Amadori P
(2022)
Predicting Secondary Task Performance: A Directly Actionable Metric for Cognitive Overload Detection
in IEEE Transactions on Cognitive and Developmental Systems
Amadori P
(2020)
Decision Anticipation for Driving Assistance Systems
Davies H
(2022)
Wearable In-Ear PPG: Detailed Respiratory Variations Enable Classification of COPD
in IEEE Transactions on Biomedical Engineering
Davies HJ
(2020)
In-Ear SpO2: A Tool for Wearable, Unobtrusive Monitoring of Core Blood Oxygen Saturation.
in Sensors (Basel, Switzerland)
Georgiou T
(2017)
Adaptive user modelling in car racing games using behavioural and physiological data.
in User modeling and user-adapted interaction
Hemakom A
(2017)
Quantifying team cooperation through intrinsic multi-scale measures: respiratory and cardiac synchronization in choir singers and surgical teams.
in Royal Society open science
| Description | (a) We developed deep learning models that predict the focus of attention of a driver from scene information and telemetry data. Our studies has shown that maneuver-awareness is greatly beneficial for human visual attention prediction and that it can be leveraged using telemetry data to achieve robust and reliable predictions. Being able to anticipate the focus of attention of a driver can have great benefits to human safety, such as inattention prevention. We have also designed a sequential model that uses physiological information, i.e., gaze and head pose, in a sequence to sequence learning paradigm to infer the likelihood of incoming mistakes from a driver on a cognitive task. Our work has shown that gaze patterns and head movements correlate with decision-making mistakes in highly cluttered and dynamic driving environments, and that sequential deep learning models can exploit such correlation to anticipate the likelihood of secondary task mistakes up to 2 seconds in advance. Finally, we have demonstrated that we can utilise physiological and behavioural data from human drivers to develop personalised models for cognitive workload prediction. (b) We have established that SpO2 levels are reliably monitored from our in-ear-canal sensor. The in-ear-canal reading of SpO2 shows faster detection times (12.4 s faster on average) for decreases in whole-body SpO2 than conventional wrist/hand measures. The sensor is designed to be worn for long periods of time, in uncontrolled, real-world environments, without causing discomfort to the user or significantly impeding activity. Due to the ear canal's proximity to the brain's major blood supplying vessels, the in-ear-canal SpO2 reading is highly sensitive to brain metabolism. Our study showed that our in-ear-canal SpO2 sensor can reliably identify varying levels of cognitive workload (higher cognitive workload = higher oxygen consumption = lower in-ear-canal SpO2). |
| Exploitation Route | This project is developing algorithms and technologies for brain-computer interfaces that have the potential to significantly improve decision accuracy and human safety in a variety of domains, such as driving and human-computer interaction. |
| Sectors | Aerospace, Defence and Marine,Digital/Communication/Information Technologies (including Software),Electronics,Healthcare,Security and Diplomacy |
| Description | Non-academic outputs from this research include two granted patents (EP3094235 & US2016331328 on Biosensing Electrodes, and a further patent applied for (WO2019077363 on Electrocardiogram Apparatus and method). Wearable sensors developed during this projects are under trial in the Imperial Dementia Research Centre, and the outcomes are being explored for commercialisation with industrial partners, and for further use in defence applications. |
| First Year Of Impact | 2016 |
| Sector | Aerospace, Defence and Marine,Electronics,Healthcare |
| Impact Types | Economic |
| Description | Bioelectric Signals for Warfighter Lethality |
| Amount | £1,174,655 (GBP) |
| Organisation | US Army |
| Sector | Public |
| Country | United States |
| Start | 10/2020 |
| End | 09/2023 |
| Description | Imperial College Dementia Research Centre |
| Amount | £20,000,000 (GBP) |
| Funding ID | Imperial College Dementia Research Centre |
| Organisation | Care UK |
| Sector | Private |
| Country | United Kingdom |
| Start | 10/2019 |
| End | 09/2026 |
| Description | Low-cost high-tech concussion assessment and post-fall health monitoring, funded by The Racing Foundation |
| Amount | £241,000 (GBP) |
| Organisation | British Heart Foundation (BHF) |
| Sector | Charity/Non Profit |
| Country | United Kingdom |
| Start | 10/2018 |
| End | 11/2020 |
| Description | UKRI Trustworthy Autonomous Systems Node in Trust |
| Amount | £3,056,751 (GBP) |
| Funding ID | EP/V026682/1 |
| Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
| Sector | Public |
| Country | United Kingdom |
| Start | 11/2020 |
| End | 04/2024 |
| Title | Biophysics model for the prediction of ear EEG sensitvity to neural and artifact sources |
| Description | The ear-EEG has emerged as a promising candidate for real-world wearable brain monitoring. While experimental studies have validated several applications of ear-EEG, the source-sensor relationship for neural sources from across the brain surface has not yet been established. In addition, modeling of the ear-EEG sensitivity to sources of artifacts is still missing. Through volume conductor modeling, the sensitivity of various configurations of ear-EEG is established for a range of neural sources, in addition to ocular artifact sources for the blink, vertical saccade, and horizontal saccade eye movements. |
| Type Of Material | Computer model/algorithm |
| Year Produced | 2023 |
| Provided To Others? | Yes |
| Impact | Results conclusively support the introduction of ear-EEG into conventional EEG paradigms for monitoring neural activity that originates from within the temporal lobes, while also revealing the extent to which ear-EEG can be used for sources further away from these regions. The use of ear-EEG in scenarios prone to ocular artifacts is also supported, through the demonstration of proportional scaling of artifacts and neural signals in various configurations of ear-EEG. The results from this study can be used to support both existing and prospective experimental ear-EEG studies and applications in the context of sensitivity to both neural sources and ocular artifacts. |
| URL | https://www.frontiersin.org/articles/10.3389/fnins.2022.997377/full |
| Description | Bioelectric Signals for Warfighter Lethality |
| Organisation | US Army |
| Country | United States |
| Sector | Public |
| PI Contribution | The project is to evaluate Hearables in a realistic environment, relevant to the military. We provide sensors and algorithms. |
| Collaborator Contribution | The collaborators test them on auditory paradigms, on a treadmill and while moving. |
| Impact | Collaborative design of new hardware for the latest hearable device prototype. |
| Start Year | 2020 |
| Description | Collaboration with Imperial Dementia Research Centre |
| Organisation | UK Dementia Research Institute |
| Country | United Kingdom |
| Sector | Charity/Non Profit |
| PI Contribution | We are part of the new Imperial Dementia Research Centre, a 20 million GBP centre funded by DRI Care & Technology Programme, 2019-2015 |
| Collaborator Contribution | Our in-ear EEG system will be further developed and used to detect and predict the progress of dementia, in a 24/7 fashion |
| Impact | No outputs yet, just started |
| Start Year | 2019 |
| Description | Collaboration with SONY company, Japan |
| Organisation | SONY |
| Country | Japan |
| Sector | Private |
| PI Contribution | This is a funded work on the analysis of Ear-EEG data for entertainment systems |
| Collaborator Contribution | SONY provided funding for research into artefact removal from ear-EEG |
| Impact | N/A |
| Start Year | 2020 |
| Description | Collaboration with SONY: Motion artefact removal and artefact classification in ear-EEG |
| Organisation | SONY |
| Country | Japan |
| Sector | Private |
| PI Contribution | Characterised three key sources of artefact in daily activity as measured by our multi-modal artefact removal sensor: head movement, facial expressions, and walking/chewing/speaking. Successfully demonstrated feasibility of removing artefacts from ear-EEG using the multi-modal sensor in conjunction with LMS and RLS methods. |
| Collaborator Contribution | SONY collaborated on design of the experimental protocol, and helped guide the method of evaluation for the proposed motion artefact removal system. |
| Impact | The first stage of the project commenced in July 2020 and finished in September 2020. The aim of this project was to establish feasibility of the proposed motion artefact removal system. Following the success of the first stage of the project, we are now in the process of collaborating on the next stage, which will aim to use develop more advanced methods of artefact removal in conjunction with the multi-modal sensor, as well as collect more data. We are yet to publish results from this project. |
| Start Year | 2020 |
| Description | Collaboration with Sonion, manufacturer of hearing aid components |
| Organisation | Sonion |
| Country | Denmark |
| Sector | Private |
| PI Contribution | Sonion have provided us with their last generation MEMS microphones and in-ear receivers, which we will use in order to improve our current multimodal sensor |
| Collaborator Contribution | Expert advice and parts, ca 600 GBP |
| Impact | Improved multimodal sensor |
| Start Year | 2017 |
| Description | Collaboration with Sonion, manufacturer of hearing aid components |
| Organisation | Sonion |
| Country | Denmark |
| Sector | Private |
| PI Contribution | Sonion have provided us with their last generation MEMS microphones and in-ear receivers, which we will use in order to improve our current multimodal sensor |
| Collaborator Contribution | Expert advice and parts, ca 600 GBP |
| Impact | Improved multimodal sensor |
| Start Year | 2017 |
| Title | BIOSENSING ELECTRODES |
| Description | A dual modality sensor comprises a tissue-contact electrode having a first surface configured for receiving an electrical signal from a user's tissue when attached thereto; and a mechanical sensor overlying the cutaneous electrode and configured to sense a mechanical displacement of the first surface through the electrode. The electrode and the mechanical sensor thereby provide electrical and mechanical signals which originate from precisely the same tissue location. |
| IP Reference | EP3094235 |
| Protection | Patent granted |
| Year Protection Granted | 2016 |
| Licensed | No |
| Impact | A new collaboration with SONY corporation |
| Title | BIOSENSING ELECTRODES |
| Description | A dual modality sensor comprises a tissue-contact electrode having a first surface configured for receiving an electrical signal from a user's tissue when attached thereto; and a mechanical sensor overlying the cutaneous electrode and configured to sense a mechanical displacement of the first surface through the electrode. The electrode and the mechanical sensor thereby provide electrical and mechanical signals which originate from precisely the same tissue location. |
| IP Reference | US2016331328 |
| Protection | Patent granted |
| Year Protection Granted | 2016 |
| Licensed | No |
| Impact | We have established a new collaboration with Sony |
| Title | ELECTROCARDIOGRAM APPARATUS AND METHOD |
| Description | The disclosure relates to a device and method of obtaining an electrocardiogram for a subject. The method comprises receiving electrical signals from at least two head- mounted sensors; and analysing said electrical signals to resolve shape and timing information for each of the P-, Q-, R-, S-, and T-waves available for the subject over a number of cardiac cycles, to derive a composite electrocardiogram, ECG, in which the composite electrocardiogram is derived using signals only from said head-mounted sensors. |
| IP Reference | WO2019077363 |
| Protection | Patent application published |
| Year Protection Granted | 2019 |
| Licensed | No |
| Impact | n/a yet |
| Title | This is an open source Python library for manipulation of big data |
| Description | HOTTBOX is a Python library for exploratory analysis and visualisation of multi-dimensional arrays of data, also known as tensors. It comprises methods ranging from standard multi-way operations through to multi-linear algebra based tensor decompositions and sophisticated algorithms for generalised multi-linear classification and data fusion such as Support Tensor Machine (STM) and Tensor Ensemble Learning (TEL). For user convenience, HOTTBOX offers a unifying API which establishes a self-sufficient ecosystem for various forms of efficient representation of multi-way data and corresponding decomposition and association algorithms. Particular emphasis is placed on scalability and interactive visualisation, to support multidisciplinary data analysis communities working on big data and tensors. HOTTBOX also provides means for integration with other popular data science libraries for visualisation and data manipulation. The source code, examples and documentation ca be found at https://github.com/hottbox/hottbox. |
| Type Of Technology | Software |
| Year Produced | 2021 |
| Open Source License? | Yes |
| Impact | It has been downloaded and used more than 1000 times |
| URL | https://github.com/hottbox/hottbox |
| Description | DSTL Neuroadaptive Systems Workshop |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Industry/Business |
| Results and Impact | Presentation to a DSTL-organised workshop on Neuroadaptive systems, with practitioners from governmental organisations |
| Year(s) Of Engagement Activity | 2019 |
| Description | Imperial Techno-Festival |
| Form Of Engagement Activity | Participation in an open day or visit at my research institution |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Public/other audiences |
| Results and Impact | We are showcasing our technology at this Imperial open event |
| Year(s) Of Engagement Activity | 2019 |
| Description | Tutorial, "Recurrent Neural Networks: From universal function approximation to a Big Data tool" |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Postgraduate students |
| Results and Impact | The tutorial was part of the Winter School on Machine Learning (WISMAL), part of the 3rd International Conference on Applications of Intelligent Systems (APPIS). The students were deeply interested and have formed new professional relationships with the students in our own research group, yielding sharing and development of ideas between the two groups. |
| Year(s) Of Engagement Activity | 2020 |
| URL | http://appis.webhosting.rug.nl/2020/wismal-2020/ |
| Description | Yiannis Demiris, Invited Keynote Speaker, Towards Autonomous Robotic Systems Conference (TAROS-2021), Lincoln, UK, 8-10 Sept 2021 |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Postgraduate students |
| Results and Impact | Yiannis Demiris was an Invited Keynote Speaker at the conference |
| Year(s) Of Engagement Activity | 2021 |
| URL | https://lcas.lincoln.ac.uk/wp/taros-2021/ |
| Description | Yiannis Demiris, Invited Keynote speaker: TransAIR Workshop on Cognitive Architectures, Bremen, Germany (virtual), 22-28 March 2021 |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Postgraduate students |
| Results and Impact | Yiannis Demiris was an Invited Keynote speaker at the workshop |
| Year(s) Of Engagement Activity | 2021 |
| URL | https://www.research-in-germany.org/the-future-of-work/news-and-stories/2021-transatlantic-workshop-... |
| Description | Yiannis Demiris, Invited Speaker: RSS'21 Workshop on Robotics for People (R4P): Perspectives on Interaction, Learning and Safety, at the Robotics: Science and Systems 2021 conference, 12-16 July 2021 |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Yiannis Demiris was an Invited Speaker at the workshop |
| Year(s) Of Engagement Activity | 2021 |
| URL | https://sites.google.com/view/r4p2021/speakers?authuser=0 |
| Description | Yiannis Demiris: Invited Keynote Speaker, IEEE 30th International Conference on Robot and Human Interactive Communication (IEEE RO-MAN 2021), 8-12 August 2021, Vancouver, Canada (virtual). |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Postgraduate students |
| Results and Impact | Yiannis Demiris: Invited Keynote Speaker at the HRI conference |
| Year(s) Of Engagement Activity | 2021 |
| URL | https://ro-man2021.org/program/keynote-speakers/ |