EAR: Eye-based Activity Recognition

Lead Research Organisation: Lancaster University
Department Name: Computing & Communications

Abstract

Activity recognition is concerned with inference of human activity from observation of the user's actions, to facilitate systems that proactively support users in everyday activity. Application areas include health, safety, cognitive assistance, lifestyle monitoring, entertainment, and energy management. In this project, we propose to advance activity recognition by investigating patterns of eye movement as a novel contextual cue. What makes eye movement a very distinct source of activity-related information is their direct link to visual cognition. This means that eye movement data has the potential to provide an online indication not only of what activity occurs but also of underlying cognitive processes, for example attention. Eye movements, including their measurement and analysis, have been studied extensively in clinical ophthalmology and cognitive science, but their use for on-line activity recognition in everyday settings is a novel proposition. Significant challenges include the adaptation of measurement techniques fit for everyday settings, the quality of information that can be mined under realistic conditions, and the demonstration of practical utility of eye-based context in real-world application.This project is designed as an initial investigation of the feasibility and potential of eye-based activity recognition. The research will include an assessment of eye tracking techniques, specifically of optical methods that track features on the eye, and of electrooculography (EOG) measuring eye movement with skin electrodes placed near the eyes. Based on the assessment, we will conduct explorative studies of eye patterns in daily life. The aim here is to gain insight into what eye movement reveals about everyday activity, and to develop methods for extraction of features from eye movement patterns that may be useful for activity and context recognition. A third part of the work will investigate application of eye-based context information in interactive systems. The plan is to develop two practical demonstrators, a first one exploring how longitudinal eye movement analysis may contribute to health and lifestyle monitoring, and a second one showing how eye movements might be used for task assistance (for example, search assistance when the eyes indicate visual search activity).

Publications

10 25 50
 
Description The way our eyes move is strongly linked to what we do. As we engage in certain activities, our eyes will move in quite specific and often predictable ways. That makes the eyes a rich source of information about human activity. Would it then be possible to automatically infer what we do, from analysis of our eye movements in daily life? This is a fundamental question as eye movement analysis had never before been considered for activity inference, and yet there is great application potential for systems that are able to recognize and support their user's activity. Therefore, the aim of the project was to conduct a first exploration of eye-based activity recognition, to establish the technical feasibility, study eye movement in daily life situations, and develop novel methods for eye movement pattern analysis.

The most significant findings of our research are on eye-based activity classification and activity spotting. Activity classification is the problem of predicting what a person is doing, given a predefined set of activities that we anticipate. In an experiment we had users perform five distinct activities in front of a computer screen (reading, writing, copying text, web browsing, and watching video), and demonstrated that these could be classified with good precision and recall solely based on eye movement analysis. This means that a machine can predict an activity by looking the user in the eyes, without knowing what they look at. Moreover, we showed that it is possible to do this in a user-independent manner; meaning that we can train an activity classification system with eye movements of some users, and then readily deploy it with other users. The results have been published in IEEE Trans. Pattern Analysis and Machine Intelligence, the highest impact journal in Computer Science and Artificial Intelligence, and are already cited over 100 times by research groups around the world. In other experiments we demonstrated that it is possible to "spot" an activity of interest with a wearable eye-tracker, even when the activity occurs in different situations including when people are mobile and out in the streets (published in ACM Trans Applied Perception).

We developed a number of novel methods for eye movement pattern analysis. One of the ideas we developed was to code eye movements with symbols that represent direction and amplitude of movement, so that recurring movement patterns can be captured as words of variable length. Using machine learning and minimum redundancy maximum relevance feature selection, we found that "wordbook encoding" contributed significantly to characterization of activities. We also developed novel "shape features" to model the shape of eye movement signals over consecutive time windows, and showed that this enabled robust detection of smooth pursuit eye movements (these movements occur when our eyes follow moving objects). This is a landmark development as smooth pursuits have previously been ignored in research on eye tracking for human-computer interaction.
Exploitation Route The findings can be taken forward in development of eye tracking systems and applications, in particular for mining of eye movement data for patterns that correlate with activities and mental health. They also can inform new creative uses of eye-tracking for human-computer interaction.
Sectors Creative Economy

Digital/Communication/Information Technologies (including Software)

Healthcare

Leisure Activities

including Sports

Recreation and Tourism

 
Description Doctoral Training Award
Amount £70,273 (GBP)
Organisation Lancaster University 
Sector Academic/University
Country United Kingdom
Start 09/2010 
End 06/2014
 
Description Faculty PhD Scholarship
Amount £59,686 (GBP)
Organisation Lancaster University 
Sector Academic/University
Country United Kingdom
Start 09/2010 
End 03/2014
 
Description Marie Curie Initial Training Network
Amount £504,000 (GBP)
Funding ID 264738 
Organisation European Commission 
Sector Public
Country European Union (EU)
Start 01/2011 
End 12/2014
 
Description SENSING AND IMAGING FOR DIAGNOSIS OF DEMENTIAS (SIDD)
Amount £950,403 (GBP)
Funding ID EP/M006255/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 03/2015 
End 09/2015