Moving to see: the benefits of self---motion for visual perception

Lead Research Organisation: Cardiff University
Department Name: Sch of Psychology

Abstract

Moving to see: the benefits of self-motion for visual perception

A long-standing question is how the brain transforms the light patterns impinging onto the retina into a meaningful world of objects and animates with which the observer can interact. While enormous progress has been made in the understanding of brain functions during the last few decades, the fundamental principles underlying the processing and extraction of visual information remain elusive. This project builds on the observation that perception has been traditionally studied in a passive manner, paying relatively little attention to the observer's motor activity during the acquisition of visual information. Yet, like other species, humans are not passively exposed to the incoming flow of sensory data. Instead, they actively seek useful information by coordinating sensory processing with motor activity. Our motivating hypothesis is that self-movement is a critical component of visual perception. Considered as a problem of simple visual geometry this hypothesis might appear counter-intuitive. Considered as an image-processing problem it might appear counter-intuitive. Considered against decades of work concerned with how the brain "compensates" for self-movement it might also appear counter-intuitive. However, this hypothesis is fully plausible from a biological perspective, because more information about the scene is available when the observer moves. This was pointed out many years ago by ecological psychologists and has more recently been recognised in computer vision - where it has caused a paradigm change. We argue that the visual, motor, and proprioceptive information generated by self-movement is fundamental to visual processing

The project brings together three laboratories from The Netherlands (Dr. Brenner at VU University in Amsterdam), the US (Dr. Rucci at Boston University), and the UK (Dr Rushton at Cardiff University), which have developed critical expertise on the analyses of different types of motor activities in humans. We will systematically investigate the mechanisms by which human observers use motor, proprioceptive and global optic flow signals to accomplish visual tasks. Elucidating the perceptual impact of motor activity is critical to advancing our knowledge of how the visual system functions. Such knowledge can also potentially guide the design of objects and environments, inform the building of machines capable of replicating human visual functions, and it may provide a scientific basis for the development of treatments of visual impairments commonly associated with abnormal motor activity in pathological conditions.

Planned Impact

Perceptual research has an exceedingly broad impact. It underpins the design of the computer this document is being written on (and potentially read on); both the interface and the physical screen. If you were to set this document aside and take a walk outside you would see evidence of the use of perceptual research all around. If you were stood in a street then the influence of perceptual research could be seen in the design of the street markings and traffic lights. If you walked into the nearest shop the capture of your attention would probably be driven in part by research that can be traced back to perceptual psychology and if you admire the colour of an object in the shop, the reason that the colour you perceive corresponds to the colour the designer intended is due to the used of perceptual colour models.

Worldwide scientific advancement: The UK, Netherlands and the US are the primary contributors to research on understanding of perceptual processes. This project brings together researchers from the three countries to attempt to raise understanding of perceptual processes to a new level. We will examine the interaction of vision and movement. Is movement a source of noise, or is it a source of information?

Specific beneficiaries: A number of potential beneficiaries can be identified. There are a number of specific problems associated with perception during movement. These include visual vertigo, oscillopsia and nystagmus. Links to movement perception have also been suggested in autism, dyslexia and developmental coordination disorder. A serious problem experienced during self-movement by the elderly is falls. The basic research conducted here might inform clinical diagnosis and intervention. Sports almost by definition involve movement. An understanding of how vision and movement interact may inform training of individuals. Virtual and remote environments (from computer games to remotely operated robots or vehicles) are becoming increasing common. All require a user to move comfortably, naturally and accurately through an environment. An understanding of the interaction between movement and vision in natural environments could provide constraints and guidelines for the design of such systems.

Training Highly Skilled Researchers: We will train post-doctoral fellows who will have substantial input into the organization and design of the project. These postdoctoral fellows should gain the skills necessary to run their own laboratories and translate the techniques and findings into applied settings.

Inter/Multidisciplinarity: The grant sits at the interface between psychology, computer vision and movement science. It provides an opportunity to bring techniques and researchers together across these disciplines and provide training in the range of disciplines to the post-doctoral fellows.

Innovative methodologies: The project brings together a range of technologies and approaches to allow the interaction of vision and movement to be assessed using a number of innovative and novel techniques.

Each of the three laboratories has existing clinical and industrial contacts and. In particular the UK PI, Rushton is currently collaborating with clinicians in the UK on visual vertigo and has an ongoing transatlantic collaboration investigating motion deficits following brain injury. This work will be informed by this project.

Publications

10 25 50
publication icon
Clark K (2015) Improvement in visual search with practice: mapping learning-related changes in neurocognitive stages of processing. in The Journal of neuroscience : the official journal of the Society for Neuroscience

publication icon
Rushton Simon K. (2016) Does self-movement silence change detection? in PERCEPTION

publication icon
Rushton SK (2018) The Primary Role of Flow Processing in the Identification of Scene-Relative Object Movement. in The Journal of neuroscience : the official journal of the Society for Neuroscience

 
Description The primary aims of the project was to examine the role of self-movement in perception and the impact of self-movement on perception. During the course of the project we collected data that suggest:

[1] When walking: The act of walking does not change the way the brain processes visual motion information, the area over which it is integrated, to optimise estimates of self-movement. The ability to detect changes in objects is impaired when the objects move ("motion silencing"). Motion silencing occurs when object movement is due to movement of the observer, but it is a relatively small effect, unlikely to have any consequence during normal walking.

[2] Detecting the movement of objects within the scene when you are moving: The process that underpins this ability is exceedingly fast. The process relies solely on global motion information ("optic flow") about self-movement, not positional or form cues. The ability to judge object movement is at least as precise as the ability of judge the direction of self-movement.

[3] Using visual information when you are moving to guide walking: Information about where you are in the scene ("allocentric location cues") have an important, and previously unrecognised, role in guiding walking direction. Humans very rapidly acquire a mental representation of the environment in which they are walking and they can make use of it when the visual information is poor. When walking to a target object, its change in direction relative to the body ("target drift) is a surprisingly good cue to tell you if you are currently on a course to the left or right of the target. People who have lost awareness of space on one side due to sensory or attentional impairments have problems walking. We found that the lateralised loss of awareness does not underpin the difficulties they encounter.

[4] Self-generated movements change perception of time, in-line with an internal clock slowing down.

We formed a strong multi-disciplinary, international collaborative team (Brenner, Netherlands; Rucci, USA) during the course of the project. All partners benefited greatly from the knowledge, expertise and support of the other partners. In addition we took the opportunity to forge additional collaborations both internationally (Harris, York University, Toronto) and locally.

The postdoc received training in new techniques and has gone on to take a Lectureship at the University of the West of England. A number of undergraduate and postgraduate students had the opportunity to take part in research associated with the project.
Exploitation Route Academic:
[1] The findings provide the foundation for a newly awarded EPSRC award ("Seeing the Future", EP/R030855/1, Human-like Computing Initiative; Marshall, Rushton and Moran). In the new award we will be applying the idea that, the human brain predicts incoming sensory information as the eye moves through space, to the design of a computer vision system.
[2] The findings provide the foundation for a new ORA project, The Active Observer, led by Katja Fielher (Germany) and including Eli Brenner (Dutch partner on current ORA project).
[3] The finding that the detection of object movement during self-movement ("flow-parsing) is fast, and based solely on global motion ("optic flow") information, provides a foundation for others to explore the computational and neural substrates of flow parsing.
[4] The work on allocentric location cues and target drift provide the foundation for a new line of work on the visual guidance of walking.

Industrial, Healthcare, security
[1] The new EPSRC award provides a bridge between the basic science in this award and application in industrial, healthcare or security settings.
[2] We have shared our findings with industry. At a recent workshop we had participants from the Microsoft Hololens division (Cambridge, UK) and Hewlett Packard Laboratories (Bristol, UK). We discussed our findings and the design of contemporary products.
Sectors Aerospace, Defence and Marine,Digital/Communication/Information Technologies (including Software)

URL http://psych.cf.ac.uk/publications/rushton.php
 
Description [1] We have shared our findings with industry. At a recent workshop we had participants from the Microsoft Hololens division (Cambridge, UK) and Hewlett Packard Laboratories (Bristol, UK). We discussed our findings and the design of contemporary products (including the Microsoft Hololens). [2] Work on this project set the foundations for a newly awarded EPSRC award ("Seeing the Future", EP/R030855/1, Human-like Computing Initiative; Marshall, Rushton and Moran). In the new award we will be applying the idea that the human brain predicts incoming sensory information as the eye movements through space to the design of a computer vision system. This new award provides a potential bridge between the basic science in this award and direct application in industrial, healthcare or security situations.
First Year Of Impact 2017
Sector Digital/Communication/Information Technologies (including Software)
 
Description Human-like Computing
Amount £290,959 (GBP)
Funding ID 513881 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Academic/University
Country United Kingdom
Start 04/2018 
End 04/2020
 
Description The active observer
Amount £849,000 (GBP)
Organisation Economic and Social Research Council 
Sector Public
Country United Kingdom
Start 01/2019 
End 12/2022
 
Title Experimental code 
Description Code for generating experimental stimuli. Code was published with papers. See papers for details. 
Type Of Material Improvements to research infrastructure 
Year Produced 2018 
Provided To Others? Yes  
Impact Only just been made available. 
 
Title Flow as primary cue 
Description Psychophysical data on judgements of object movement and heading. 
Type Of Material Database/Collection of data 
Year Produced 2018 
Provided To Others? No  
Impact Led to paper http://www.jneurosci.org/content/38/7/1737 
URL http://TBC.org
 
Title Heading vs Parsing 
Description Psychophysical data on judgements of the precision of heading vs judgements of parsing 
Type Of Material Database/Collection of data 
Year Produced 2018 
Provided To Others? Yes  
Impact Led to publication. https://arvojournals.org/article.aspx?articleid=2686023 
URL http://doi.org/10.5255/UKDA-SN-853037
 
Title Lateral Occlusion 
Description Walking trajectories to targets during lateral occlusion. 
Type Of Material Database/Collection of data 
Year Produced 2018 
Provided To Others? Yes  
Impact Led to paper https://arvojournals.org/article.aspx?articleid=2702357&resultClick=1 
URL http://TBC.org
 
Title Peripheral Flow 
Description Psychophysical data on role of peripheral flow in flow parsing. 
Type Of Material Database/Collection of data 
Year Produced 2017 
Provided To Others? Yes  
Impact Led to paper https://doi.org/10.1177/2041669517736072 
URL http://doi.org/10.5255/UKDA-SN-852880
 
Title Spatial integration of motion information during the act of walking 
Description Raw data from Kesten staircases for duration discrimination task to compare duration thresholds as a function of stimulus size and walking vs standing. The data will be made publicly available at the time of publication. 
Type Of Material Database/Collection of data 
Year Produced 2017 
Provided To Others? No  
Impact Led to paper - currently under review. 
URL http://not.yet.com
 
Description Active head movements 
Organisation Free University of Amsterdam
Department Faculty of Human Movement Sciences
Country Netherlands 
Sector Academic/University 
PI Contribution We are working on a collaborative project with our ORA partners in Amsterdam on a project looking at whether actions are supported by active head movements to optimise visual perception. We contributed to the design of the experiments and expect to contribute to the analysis and publication.
Collaborator Contribution As noted above, this is a collaborative project. Our ORA partners collaborated on the design of the experiment. They will collect the data and we will collaborate on the analysis and write up.
Impact Multi-disciplinary, psychology and human movement science. Paper submitted for publication.
Start Year 2016
 
Description Effect on self-movement on perception of temporal intervals 
Organisation Cardiff University
Department School of Psychology
Country United Kingdom 
Sector Academic/University 
PI Contribution With Marc Buehner and Richard Fereday I have been looking at the impact of self-movement on perception of temporal intervals. I contributed to the design, analysis and write-up of empirical research.
Collaborator Contribution The partners worked with me to design, execute and write-up empirical work on the impact of self-movement on perception of temporal intervals.
Impact Single discipline: psychology. Paper about to be submitted to Cognition, The role of time perception in temporal binding: Impaired temporal resolution in causal sequences.
Start Year 2016
 
Description Motion silencing 
Organisation York University Toronto
Country Canada 
Sector Academic/University 
PI Contribution In a collaborative project I designed and ran an experiment looking at motion silencing in the presence of full-field optic flow. In collaboration I analysed the data and presented it at ECVP.
Collaborator Contribution My collaborators at York contributed to the design, execution, analysis and write-up stages of the project.
Impact Presentation at ECVP. [insert REF]
Start Year 2016
 
Description Perception of motion in depth 
Organisation Free University of Amsterdam
Department Faculty of Human Movement Sciences
Country Netherlands 
Sector Academic/University 
PI Contribution In collaboration with our ORA partners we designed an experiment to examine whether flow parsing precedes perception of motion-in-depth. We collected pilot data and presented the work at ECVP.
Collaborator Contribution Our ORA partners provided input to the design of the experiment and will provide input to later experimental work as well as analysis and write-up.
Impact Presentation at ECVP [add REF]
Start Year 2016
 
Description Visual Vertigo 
Organisation Cardiff and Vale University Health Board
Country United Kingdom 
Sector Public 
PI Contribution Along with internal collaborators (Sumner) and clinical collaborators (Derry and Rajenderkumar) at the University Hospital Wales, Cardiff I have begun to look at Visual Vertigo. This is a problem with motion perception (sometimes called "supermarket syndrome" or "visual dependency") that can arise following damage to the vestibular system. In collaboration we have collected preliminary data on the relationship between symptoms of visual vertigo and visual processing in the student population. Next we will examine clinical patients.
Collaborator Contribution The partners have collaborated with us on data collection.
Impact Presentation at ECVP (Powell et al) in 2016.
Start Year 2015
 
Title Spatial integration of motion information during the act of walking (analysis and stimulus code) 
Description This software is for generating stimuli for the spatial integration dataset and for analysis of the data generated. 
Type Of Technology Software 
Year Produced 2017 
Impact contribution to open science 
URL http://not.yet.com
 
Description Moving to see, industry workshop, Somerset 2017 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Industry/Business
Results and Impact We had a workshop with attendees from Microsoft (Hololens division) and Hewlett-Packard in which we discussed the work we have been conducting and the results we have obtained as part of the ORA+ project. We also discussed industry developments and issues (such as the technical and practical issues associated with Hololens).
Year(s) Of Engagement Activity 2017