EAR: Eye-based Activity Recognition
Lead Research Organisation:
Lancaster University
Department Name: Computing & Communications
Abstract
Activity recognition is concerned with inference of human activity from observation of the user's actions, to facilitate systems that proactively support users in everyday activity. Application areas include health, safety, cognitive assistance, lifestyle monitoring, entertainment, and energy management. In this project, we propose to advance activity recognition by investigating patterns of eye movement as a novel contextual cue. What makes eye movement a very distinct source of activity-related information is their direct link to visual cognition. This means that eye movement data has the potential to provide an online indication not only of what activity occurs but also of underlying cognitive processes, for example attention. Eye movements, including their measurement and analysis, have been studied extensively in clinical ophthalmology and cognitive science, but their use for on-line activity recognition in everyday settings is a novel proposition. Significant challenges include the adaptation of measurement techniques fit for everyday settings, the quality of information that can be mined under realistic conditions, and the demonstration of practical utility of eye-based context in real-world application.This project is designed as an initial investigation of the feasibility and potential of eye-based activity recognition. The research will include an assessment of eye tracking techniques, specifically of optical methods that track features on the eye, and of electrooculography (EOG) measuring eye movement with skin electrodes placed near the eyes. Based on the assessment, we will conduct explorative studies of eye patterns in daily life. The aim here is to gain insight into what eye movement reveals about everyday activity, and to develop methods for extraction of features from eye movement patterns that may be useful for activity and context recognition. A third part of the work will investigate application of eye-based context information in interactive systems. The plan is to develop two practical demonstrators, a first one exploring how longitudinal eye movement analysis may contribute to health and lifestyle monitoring, and a second one showing how eye movements might be used for task assistance (for example, search assistance when the eyes indicate visual search activity).
Publications
Bulling A
(2012)
Multimodal recognition of reading activity in transit using body-worn sensors
in ACM Transactions on Applied Perception
Bulling A
(2011)
Eye movement analysis for activity recognition using electrooculography.
in IEEE transactions on pattern analysis and machine intelligence
Bulling A
(2013)
EyeContext
Bulling A
(2010)
Toward Mobile Eye-Based Human-Computer Interaction
in IEEE Pervasive Computing
Turner J
(2011)
Combining gaze with manual interaction to extend physical reach
Vidal M
(2012)
Wearable eye tracking for mental health monitoring
in Computer Communications
Ward J
(2011)
Performance metrics for activity recognition
in ACM Transactions on Intelligent Systems and Technology
| Description | The way our eyes move is strongly linked to what we do. As we engage in certain activities, our eyes will move in quite specific and often predictable ways. That makes the eyes a rich source of information about human activity. Would it then be possible to automatically infer what we do, from analysis of our eye movements in daily life? This is a fundamental question as eye movement analysis had never before been considered for activity inference, and yet there is great application potential for systems that are able to recognize and support their user's activity. Therefore, the aim of the project was to conduct a first exploration of eye-based activity recognition, to establish the technical feasibility, study eye movement in daily life situations, and develop novel methods for eye movement pattern analysis. The most significant findings of our research are on eye-based activity classification and activity spotting. Activity classification is the problem of predicting what a person is doing, given a predefined set of activities that we anticipate. In an experiment we had users perform five distinct activities in front of a computer screen (reading, writing, copying text, web browsing, and watching video), and demonstrated that these could be classified with good precision and recall solely based on eye movement analysis. This means that a machine can predict an activity by looking the user in the eyes, without knowing what they look at. Moreover, we showed that it is possible to do this in a user-independent manner; meaning that we can train an activity classification system with eye movements of some users, and then readily deploy it with other users. The results have been published in IEEE Trans. Pattern Analysis and Machine Intelligence, the highest impact journal in Computer Science and Artificial Intelligence, and are already cited over 100 times by research groups around the world. In other experiments we demonstrated that it is possible to "spot" an activity of interest with a wearable eye-tracker, even when the activity occurs in different situations including when people are mobile and out in the streets (published in ACM Trans Applied Perception). We developed a number of novel methods for eye movement pattern analysis. One of the ideas we developed was to code eye movements with symbols that represent direction and amplitude of movement, so that recurring movement patterns can be captured as words of variable length. Using machine learning and minimum redundancy maximum relevance feature selection, we found that "wordbook encoding" contributed significantly to characterization of activities. We also developed novel "shape features" to model the shape of eye movement signals over consecutive time windows, and showed that this enabled robust detection of smooth pursuit eye movements (these movements occur when our eyes follow moving objects). This is a landmark development as smooth pursuits have previously been ignored in research on eye tracking for human-computer interaction. |
| Exploitation Route | The findings can be taken forward in development of eye tracking systems and applications, in particular for mining of eye movement data for patterns that correlate with activities and mental health. They also can inform new creative uses of eye-tracking for human-computer interaction. |
| Sectors | Creative Economy,Digital/Communication/Information Technologies (including Software),Healthcare,Leisure Activities, including Sports, Recreation and Tourism |
| Description | Doctoral Training Award |
| Amount | £70,273 (GBP) |
| Organisation | Lancaster University |
| Sector | Academic/University |
| Country | United Kingdom |
| Start | 10/2010 |
| End | 06/2014 |
| Description | Faculty PhD Scholarship |
| Amount | £59,686 (GBP) |
| Organisation | Lancaster University |
| Sector | Academic/University |
| Country | United Kingdom |
| Start | 10/2010 |
| End | 03/2014 |
| Description | Marie Curie Initial Training Network |
| Amount | £504,000 (GBP) |
| Funding ID | 264738 |
| Organisation | European Commission |
| Sector | Public |
| Country | European Union (EU) |
| Start | 01/2011 |
| End | 12/2014 |
| Description | SENSING AND IMAGING FOR DIAGNOSIS OF DEMENTIAS (SIDD) |
| Amount | £950,403 (GBP) |
| Funding ID | EP/M006255/1 |
| Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
| Sector | Public |
| Country | United Kingdom |
| Start | 04/2015 |
| End | 09/2015 |