Bridging the gap between eye movements and event-related potentials

Lead Research Organisation: University of Leicester
Department Name: Engineering

Abstract

In this project we will measure the brain activity of subjects searching for hidden targets in crowded scenes (e.g. Waldo in the Where's Waldo game). Neuronal activity will be registered non-invasively with an electroencephalographic (EEG) system. Simultaneously, the subject's gaze will be recorded with an eye tracker. This combined methodology will represent a shift in the way conventional EEG paradigms are performed, where eye movements are minimised and experimental conditions are fully controlled by the experimenter. We propose to combine eye tracking and EEG to gain new insights into the neuronal underpinning of visual search. By merging both technologies, we will be able to study brain activity under naturalistic conditions where the subject decides where and how to explore a scene. Moreover, we will optimise different methodologies to assess and reduce the impact of eye movements on the EEG signal. Overall, this project has multiple areas of application including defence and the health-care sector.

Planned Impact

The proposed research has the potential to have a wide impact over multiple application domains: The proposed methodology to obtain and analyse concurrent EEG and eye movements recordings will be considered for patent submission, directly benefiting the University of Leicester and increasing the economic competitiveness of the United Kingdom. A possible collaboration with NHS will be looked for to apply the methods developed during this project. This would benefit patients and healthcare practitioners. Visual search is a task of great importance appearing in diverse fields. For example, the military industry has multiple needs for aerial reconnaissance. The single-trial analysis of ERPs where subjects are allowed to change their gaze can improve the performance of subjects in time demanding decision-making tasks. This can potentially be relevant for national security. The proposed research's main ideas are easily comprehensible. This key fact will be used for reaching the general audience through several events for public engagement. This will contribute to increasing public awareness and understanding of science.

Publications

10 25 50
 
Description This work aimed to perform a non-invasive study of brain activity under free-viewing conditions. For that purpose novel methodologies were developed to combine simultaneous electroencephalographic (EEG) and eye movements recordings. We designed an eye-movement visual search paradigm that allowed us to concurrently record EEG and eye movements while subjects were asked to find a hidden target face in a crowded scene with distractor faces. We calculated the brain signals aligned to fixations landing on targets and distractors and further compared those signals with the ones obtained in a control (classical) task at fixation.

The original objectives of the proposal were largely achieved. As originally hypothesised, we showed the emergence of robust sensory components associated with the perception of stimuli and cognitive components associated with the detection of target faces, showing qualitative similarities as well as differences in terms of scalp topography and latency between free-viewing and classical tasks.

By using single trial analyses, we also showed that fixations to target and distractors could be decoded from the EEG signals above chance level in 11 out of 12 subjects.

Altogether, our results show that EEG signatures related to cognitive behavior develop across spatially unconstrained exploration of natural scenes and provide a first step towards understanding the mechanisms of target detection during natural search.
Exploitation Route We believe our results will encourage further work to understand the mechanisms of object processing and target detection during visual exploration of natural scenes. The methodology we developed can be further exploited by Neuroscientists and Psychologists who routinely use EEG recordings.
Visual search is a ubiquitous task of great importance appearing in many diverse fields. For example, the military industry has multiple needs for aerial reconnaissance. An expert typically has to quickly detect a structure. The single-trial analyses of ERPs aligned to fixation onset are potentially relevant for national security.
In a wider context, the research grant could other applications in different domains. One example is the detection of drowsiness in drivers, where the simultaneous use of EEG and eye movements recordings could help detect when those drivers start to feel tired and possibly prevent dangerous accidents.
The brain and eye-tracking technology could also be used to develop brain computer interfaces, which aim to restore movement or communication to people with serious movement disabilities. For instance, people with amyotrophic lateral sclerosis, a disease that causes progressive degeneration of motor neurons, maintain good control of their eye movements until late stages of the illness.
Incorporating eye-tracking with EEG control could lead to improved brain computer interfaces. The same principle could also be used for hands-free video games in which players control the action with their eye movements and thoughts.
Sectors Aerospace, Defence and Marine,Creative Economy,Digital/Communication/Information Technologies (including Software),Healthcare,Transport

URL http://www.sciencedirect.com/science/article/pii/S105381191301210X
 
Description The possible applications of our research have been disseminated to the General Public by talks and press releases (receiving wide media coverage from The Huffington Post, theEngineer, CNN "What's Next", among others. It has therefore contributed to increasing public awareness and understanding of science.
First Year Of Impact 2013
Sector Digital/Communication/Information Technologies (including Software),Healthcare
Impact Types Societal

 
Description EPSRC
Amount £323,993 (GBP)
Organisation University of Leeds 
Department Faculty of Engineering
Sector Academic/University
Country United Kingdom
Start 10/2012 
End 09/2015
 
Description PIP 2015-2017
Amount $585,000 (ARS)
Funding ID 11220150100787CO 
Organisation National Scientific and Technical Research Council (Argentina) 
Sector Public
Country Argentina
Start 09/2016 
End 08/2019
 
Description Juan 
Organisation University of Buenos Aires
Department Department of Computer Science
Country Argentina 
Sector Academic/University 
PI Contribution Design, Execution of Research, Data Analyses
Collaborator Contribution Design, Execution of Research, Data Analyses
Impact Lisandro Kaunitz, Juan E Kamienkowski, Alexander Varatharajah, Mariano Sigman, Rodrigo Quian Quiroga & Matias J. Ison Looking for a face in the crowd: fixation-related potentials in an eye-movement visual search task. NeuroImage 89, 297-305 (2014). Juan E Kamienkowski*, Matias J Ison*, Rodrigo Quian Quiroga, Mariano Sigman Fixation-related potentials in visual search: a combined EEG and eye tracking study. J Vis July 9, 12(7): 4 (2012). *Equal Contribution.
Start Year 2010
 
Description Brain Awareness Day (University of Leicester) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Schools
Results and Impact Presented a poster on the research at an open day on "Brain Awareness Day". This was open to the public but had many schools visiting with over 100 potential students asking questions about the research and finding a new interest to pursue. With a students discovering for the first time what neuroscience involved.
Year(s) Of Engagement Activity 2015
URL http://www2.le.ac.uk/departments/npb/news-1/baw/brain-awareness-day-event-for-a-level-students
 
Description European Conference of Eye Movements (Lund) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Study participants or study members
Results and Impact An international conference, where the study I collaborated on was presented by team member Dr. Matias Ison. We were hoping to get opinions from experts in the field, as well as acknowledgement on some of the findings. The feedback was all positive and the questions posed led to other branches of analysis to be performed, and key improvements made that were the foundations for later work. The research presented was later published (can be seen in the publication section).
Year(s) Of Engagement Activity 2013
URL http://ecem2013.eye-movements.org/
 
Description European Conference of Eye Movements (Vienna) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Study participants or study members
Results and Impact International conference where I presented the main findings of the research I have undertaken in the last 3 years. The main aim was to present my work and defend it against scrutiny from leading experts in the field. The work was successfully defended and the result was better than expected; with many researchers in a similar field asking how we performed certain data collection, as well as the resulting analysis. After the presentation I remained after to discuss the findings with many other researchers who were interested in what we had found.
Year(s) Of Engagement Activity 2015
URL http://ecem2015.univie.ac.at/
 
Description Labman 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Undergraduate students
Results and Impact 2nd LATIN AMERICAN BRAN MAPPING NETWORK MEETING. To take place in Buenos Aires (Argentina) in March, 2017.
Year(s) Of Engagement Activity 2017
URL http://2ndmeeting.labman.org/meeting_schedule/
 
Description Postgraduate Festival (University of Leicester) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Industry/Business
Results and Impact Presented a poster on the work to date, where the majority of those attending where industry specialists. A lot of questions on the work were how it applied to the "real world", and what effect it could have. There were quite a few long discussions and engagement with some attending, very interested in the work and where it could progress to. Many of whom I exchanged contact information with.
Year(s) Of Engagement Activity 2014
URL http://www2.le.ac.uk/research/festival