Integrating 'when' and 'where' in models of saccade target selection

Lead Research Organisation: University of Bristol
Department Name: Experimental Psychology

Abstract

The human visual system is limited in its ability to resolve fine detail. In fact, only in the one or two degrees of central vision (an area of approximately the width of two thumbs at an arm's length) are we able to see with high acuity. Therefore, in order to explore the visual environment humans move their eyes very frequently. These eye movements are called saccades, and we make ~10,000 of these movements every hour of our waking lives.These eye movements are critical for our vision, and therefore for the successful interaction with the world around us. Eye movement researchers are interested in the properties of saccades, and the visual signals that are effective in triggering these movements. The critical question in this regard is: How do people decide where to look next and when to move the current point of fixation to the selected location?Theories of saccadic eye movement generation typically focus on one or the other of these two aspects. That is, sophisticated theories have been developed to explain which spatial locations in a visual scene are fixated. These theories do not explain the order in which the various locations are visited, nor do they account for how long the eyes remain stationary before moving on to the next location. Likewise, theories exist that account for the latency, or reaction time, of saccades in response to the appearance of a single visual target on a blank background. Such theories do not account for which locations are selected for detailed visual scrutiny. Therefore, current theories of saccade target selection are incomplete in that they only consider the spatial or the temporal aspects.The aim of this project is to develop a general theory of saccade target selection that incorporates both the spatial and temporal components. Specifically, the theory will be formulated in such a way that it can be implemented, or simulated, in a computer program. Detailed simulations of such kind are often helpful in that it enables one to make more specific, and sometimes counter-intuitive, predictions that can then be tested in experiments.Testing the theory in this way is an important and substantial part of the project. The experiments proposed generally involve having human observers perform some discrimination task on some visual image. For instance, we can present a number of patterns that contain oriented lines, and ask observers to find the one pattern that consists of vertically oriented lines. Using an eye tracker, we can then monitor which patterns are fixated, in what order, and how long it takes observers to look from one place to the next. These eye movement data are used to confirm or reject the theory's predictions.A unified theory of saccade target selection is intrinsically interesting to scientists who investigate the human eye movement and/or visual systems. In addition, its computational implementation may be relevant in more applied settings. For instance, a major problem in robotics is one of representation: a mobile robot simply cannot store all the information it might want to store about its environment (this amount would rapidly grow out of control). Instead, a solution might be to sample information from the environment if and when it is needed for some task, similar to the way humans sample the visual world using eye movements. Knowledge and theories about how humans do this, can be extremely useful in an engineering context.In addition, the empirical findings generated in the project are not only of theoretical relevance. Knowledge of the kinds of representations human observers can use to drive their eye movements can be important in a variety of settings in which people are required to quickly respond to some visual signals. Here you can think about a pilot flying an aircraft or the staff in an airtraffic control room. In these settings it may be very important to taylor the lay-out of the visual scene to the human who will be operating in that environment.

Publications

10 25 50

publication icon
Etchells PJ (2010) The target velocity integration function for saccades. in Journal of vision

publication icon
Evens DR (2010) Dual-task costs and benefits in anti-saccade performance. in Experimental brain research

publication icon
Farrell S (2010) Influence of environmental statistics on inhibition of saccadic return. in Proceedings of the National Academy of Sciences of the United States of America

publication icon
Ludwig CJ (2014) Foveal analysis and peripheral selection during active visual sampling. in Proceedings of the National Academy of Sciences of the United States of America

 
Description Humans pick up visual information from their environment by shifting their gaze a few times every second. This central aim of this project was to understand the decision processes involved in choosing where to look next and when to go there. In particular, this project was focused on saccadic eye movements - these are rapid jumps of the eyes and form the dominant mode of acquiring visual information in a wide range of visually driven activities (e.g. reading, driving).



The first three years of the fellowship were largely devoted to model development and testing with behavioural eye tracking experiments. The modelling is rooted in the idea that at any one point there are multiple possible targets to fixate next. Over the course of a fixation, humans accumulate "evidence" in favour of these targets in parallel. When the evidence for a particular target reaches a critical threshold, that target is fixated next. This type of model provides a coherent and integrated account of the 'when' and 'where' decisions in eye movement control.



The key findings from this phase of the project include:

1. Inclusion of physiologically realistic sensory response functions in the drift-diffusion model of decision-making (Ludwig, 2009, Vision Research).

2. An account for well-known sequential effects in eye movement control (Inhibition of Return; Ludwig et al., 2009, Cognitive Psychology).

3. Experimental evidence that shows that this sequential effect is flexibly adapted to the (learned) statistics of the local environment (Farrell et al., 2010, Proceedings of the National Academy of Sciences; Ludwig et al., 2012, Journal of Experimental Psychology: General).

4. Development of a "front-end" to the accumulator framework that tracks changing environmental statistics in a context-sensitive manner (Ludwig et al., 2012, Journal of Experimental Psychology: General).

5. Experimental and computational analyses of the sources of "internal noise" that need to be included in models in order to account for behavioural variability (Ludwig & Davies, 2011, Cognitive Psychology).



In the final two years of the project, I have considered more ecologically realistic demands faced by the visual and eye movement systems. I will highlight three major bodies of work that were conducted in this phase.



1. Consideration of visual information processing across saccadic eye movements. When you shift gaze, a different part of the visual scene projects to a same part of the retina. Given the frequency with which gaze shifts occur, the visual system is bombarded with such changes in visual input. We have shown experimentally that humans seem particularly sensitive to "trans-saccadic" changes in luminance. The key finding here is that a simple comparison of the luminance at the same retinal location before and after the saccade can act as a "dumb" edge detector (Ludwig, et al., 2012, Journal of Vision). Finding edges is important in scene segmentation and interpretation.



2. Consideration of visual processing in the fovea. You shift gaze in order to analyse an object or region of interest with foveal vision - the part of the retina where visual acuity is maximal. However, a great deal of work in this domain (including much of the work conducted in this project) examines gaze shifting mechanisms in the absence of any foveal processing requirement. We have developed a novel "noise classification" methodology that allows us to identify how visual information is used for the purpose of foveal object identification and selection of the next saccade target. Key findings here are that foveal object identification and target selection occur completely in parallel and independently from each other (Ludwig et al., 2014, Proceedings of the National Academy of Sciences). These findings have strong implications for theories of eye movement control in a range of visual-motor domains (e.g. reading, driving, scene perception).



3. Information foraging. During a fixation you extract information from the currently fixated region. Across multiple fixations, you have gathered information and you typically use that information for some purpose. For example, when you are about to cross the road, you look both ways in one or more cycles in order to estimate how safe it is to cross the road. One way to think of this behaviour is as a form of information foraging. We have studied such foraging behaviour empirically and developed computational models to account for this behaviour (Cassey et al., 2013, PLoS ONE; Ludwig & Evens, 2017, Journal of Experimental Psychology: Human Perception and Performance).
Exploitation Route Knowledge of how humans decide what information to sample at what point in time is valuable in a number of domains, such as (i) Human-Computer Interaction (HCI): eye tracking interfaces such as those developed for quadriplegic patients; (ii) Robotics: overcome sensory bottlenecks through active sensors that focus the available, limited bandwidth selectively on information that is relevant for the task at hand; (iii) Ergonomics: promote the adaptive uptake of visual information through appropriate lay-out and alerting signals in the environment (e.g. cockpit design).



The transfer of knowledge to these more complex and naturalistic domains calls for techniques to study visually guided behaviour on a larger scale: that is, larger environments, richer visual inputs, and a richer repertoire of behaviour that also includes other motor systems (i.e. head and whole body movements). To this end, I have set up the Bristol Vision Institute Movement laboratory (funded by The Wellcome Trust), a facility that allows us to study complex behaviour in more naturalistic situations.
Sectors Digital/Communication/Information Technologies (including Software),Healthcare

URL https://sites.google.com/site/casimirludwig/
 
Description EPSRC cross-disciplinary interfaces
Amount £1,646,369 (GBP)
Funding ID EP/I032622/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 09/2011 
End 09/2015
 
Description Platform Grants
Amount £1,400,000 (GBP)
Funding ID EP/M000885/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 02/2015 
End 01/2020
 
Description User Interaction with ICT
Amount £806,994 (GBP)
Funding ID EP/N013964/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 03/2016 
End 02/2020
 
Description Wellcome Trust infrastructure funding
Amount £390,439 (GBP)
Funding ID 089367/Z/09/Z 
Organisation Wellcome Trust 
Sector Charity/Non Profit
Country United Kingdom
Start 07/2010 
End 07/2014
 
Title Competitive Accumulator Modelling toolbox 
Description Models of evidence accumulation are widely used in cognitive and neuroscience. We have developed a toolbox that allows for straightforward simulation of a wide variety of these so-called "accumulator models", with many different tunable parameters. The toolbox also includes code for quantifying the goodness-of-fit of a model with respect to empirically observed data. 
Type Of Material Computer model/algorithm 
Year Produced 2009 
Provided To Others? Yes  
Impact Informally, I know that the toolbox has been used by a variety of labs, typically as part of the stage of "trying out" a number of different model architectures, prior to the full-blown implementation and fitting of a specific model (the toolbox is meant for simulating data, not necessarily for fitting data---although for complex models, simulation is often the only way to fit the data and the toolbox may be used in that way). 
URL http://casiwiki.pbworks.com/w/page/15355615/Competitive%20accumulator
 
Description Bristol Decision-Making Network 
Organisation University of Bristol
Country United Kingdom 
Sector Academic/University 
PI Contribution A consortium of 7 applicant from Bristol University was awarded £1.6m from the EPSRC Cross-Disciplinary Interfaces Programme, to build a network of scientists working on all aspects of human, biological and machine decision-making. The core group of 7 academics come from Psychology, Computer Science and Mathematics. A wider network involves academics from a host of other disciplines, including Biology and Economics.
Start Year 2011
 
Description Collaboration 
Organisation Cardiff University
Country United Kingdom 
Sector Academic/University 
PI Contribution Collaboration with Dr. Petroc Sumner and his group at the University of Cardiff. I am an external advisor on a recently awarded ESRC project, aiming to develop a modelling toolkit to estimate individual differences in inhibitory control. These estimates are linked to measures of impulsive behaviour.
Start Year 2009
 
Description Collaboration 
Organisation Cardiff University
Country United Kingdom 
Sector Academic/University 
PI Contribution Collaboration with Dr. Simon Rushton, University of Cardiff. Dr. Rushton is an internationally leading expert on optic flow and visual processing during ego-motion. We have begun the explore the allocation of visual attention during simulated self-movement. We have conducted a large series of experiments on this topic. Some preliminary data were reported at the 2008 European Conference on Visual Perception. A manuscript reporting the results of 5 experiments is currently in preparation.
Start Year 2008
 
Description Collaboration 
Organisation Justus Liebig University Giessen
Country Germany 
Sector Academic/University 
PI Contribution Collaboration with Professor Karl Gegenfurtner, University of Giessen in Germany. A novel collaboration that started after being invited to give a departmental seminar at the University of Giessen, in June 2009. This collaboration led to a large body of experimental work on trans-saccadic perception, the results of which have been published in 2012 (Ludwig et al., 2012, Journal of Vision) and presented at two conferences (Vision Sciences Society 2011; European Conference on Eye Movements 2011).
Start Year 2009
 
Description Collaboration 
Organisation University of Bristol
Country United Kingdom 
Sector Academic/University 
PI Contribution Collaboration with computer scientists at the University of Bristol and University of Sheffield. Together with Drs Rafal Bogacz (Department of Computer Science, University of Bristol; now at University of Oxford) and James Marshall (Department of Computer Science, University of Sheffield), we developed an 'ideal observer' model for a particular class of decision problems and tested the model with experiments. We trained a computer science PhD student (modelling) and an experimental psychology PhD student (experiments). Some of the work has been published in PLoS ONE (Cassey et al., 2013).
Start Year 2009
 
Description Collaboration 
Organisation University of California, Santa Barbara
Country United States 
Sector Academic/University 
PI Contribution Existing collaboration with Professor Miguel Eckstein, University of California Santa Barbara. The fellowship involved strengthening of an existing collaboration with Prof Eckstein. In the final two years of the project, we have developed an innovative methodology to assess the uptake of visual information in eye movement control. An empirical paper describing the methodology and some initial results is close to being submitted. At least one other paper based on the current data set is to be expected. A further series of at least 6 experiments has been planned.
Start Year 2005
 
Description Collaboration 
Organisation University of Sheffield
Country United Kingdom 
Sector Academic/University 
PI Contribution Collaboration with computer scientists at the University of Bristol and University of Sheffield. Together with Drs Rafal Bogacz (Department of Computer Science, University of Bristol; now at University of Oxford) and James Marshall (Department of Computer Science, University of Sheffield), we developed an 'ideal observer' model for a particular class of decision problems and tested the model with experiments. We trained a computer science PhD student (modelling) and an experimental psychology PhD student (experiments). Some of the work has been published in PLoS ONE (Cassey et al., 2013).
Start Year 2009
 
Description Invited seminars 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact I have given a number of invited seminars about the work conducted as part of the fellowship: University of Potsdam, Germany, 2017; Paris Descartes University, France, 2017; University of Cardiff, UK, 2016; University of Bielefeld, Germany, 2016; Aston University, UK, 2015; Universidad Torcuato Di Tella, Buenos Aires, Argentina, 2015; University of Wales, Bangor, 2014; University of California Irvine, 2014; University of Southampton, UK, 2014; University of Potsdam, Germany, 2012; University of Giessen, Germany, 2009; University of Birmingham, UK, 2009; University of Strathclyde, UK, 2009; University of Geneva, Switzerland, 2008; University of Cardiff, UK, 2008; DSTL Knowledge Network Meeting, Winfrith Technology Centre, Winfrith, UK, 2008;.

The impact of this activity is primarily academic.
Year(s) Of Engagement Activity 2008,2009,2012,2014,2015,2016,2017
 
Description Public engagement at @Bristol 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? Yes
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact Together with my colleague Dr. Simon Rushton, we ran a public engagement activity in the @-Bristol science museum. Our demonstration involved a stereoscopic display with which we study people's sensitivity to "scene-relative" motion: object motion that is embedded within a global pattern of optic flow of the type that would result from self-movement of the observer.

I would like to think we engaged people's interest in psychological research generally, and human vision science specifically.
Year(s) Of Engagement Activity 2009
URL http://www.bris.ac.uk/publicengagementstories/stories/2009/49.html