Moving to see: the benefits of self---motion for visual perception
Lead Research Organisation:
CARDIFF UNIVERSITY
Department Name: Sch of Psychology
Abstract
Moving to see: the benefits of self-motion for visual perception
A long-standing question is how the brain transforms the light patterns impinging onto the retina into a meaningful world of objects and animates with which the observer can interact. While enormous progress has been made in the understanding of brain functions during the last few decades, the fundamental principles underlying the processing and extraction of visual information remain elusive. This project builds on the observation that perception has been traditionally studied in a passive manner, paying relatively little attention to the observer's motor activity during the acquisition of visual information. Yet, like other species, humans are not passively exposed to the incoming flow of sensory data. Instead, they actively seek useful information by coordinating sensory processing with motor activity. Our motivating hypothesis is that self-movement is a critical component of visual perception. Considered as a problem of simple visual geometry this hypothesis might appear counter-intuitive. Considered as an image-processing problem it might appear counter-intuitive. Considered against decades of work concerned with how the brain "compensates" for self-movement it might also appear counter-intuitive. However, this hypothesis is fully plausible from a biological perspective, because more information about the scene is available when the observer moves. This was pointed out many years ago by ecological psychologists and has more recently been recognised in computer vision - where it has caused a paradigm change. We argue that the visual, motor, and proprioceptive information generated by self-movement is fundamental to visual processing
The project brings together three laboratories from The Netherlands (Dr. Brenner at VU University in Amsterdam), the US (Dr. Rucci at Boston University), and the UK (Dr Rushton at Cardiff University), which have developed critical expertise on the analyses of different types of motor activities in humans. We will systematically investigate the mechanisms by which human observers use motor, proprioceptive and global optic flow signals to accomplish visual tasks. Elucidating the perceptual impact of motor activity is critical to advancing our knowledge of how the visual system functions. Such knowledge can also potentially guide the design of objects and environments, inform the building of machines capable of replicating human visual functions, and it may provide a scientific basis for the development of treatments of visual impairments commonly associated with abnormal motor activity in pathological conditions.
A long-standing question is how the brain transforms the light patterns impinging onto the retina into a meaningful world of objects and animates with which the observer can interact. While enormous progress has been made in the understanding of brain functions during the last few decades, the fundamental principles underlying the processing and extraction of visual information remain elusive. This project builds on the observation that perception has been traditionally studied in a passive manner, paying relatively little attention to the observer's motor activity during the acquisition of visual information. Yet, like other species, humans are not passively exposed to the incoming flow of sensory data. Instead, they actively seek useful information by coordinating sensory processing with motor activity. Our motivating hypothesis is that self-movement is a critical component of visual perception. Considered as a problem of simple visual geometry this hypothesis might appear counter-intuitive. Considered as an image-processing problem it might appear counter-intuitive. Considered against decades of work concerned with how the brain "compensates" for self-movement it might also appear counter-intuitive. However, this hypothesis is fully plausible from a biological perspective, because more information about the scene is available when the observer moves. This was pointed out many years ago by ecological psychologists and has more recently been recognised in computer vision - where it has caused a paradigm change. We argue that the visual, motor, and proprioceptive information generated by self-movement is fundamental to visual processing
The project brings together three laboratories from The Netherlands (Dr. Brenner at VU University in Amsterdam), the US (Dr. Rucci at Boston University), and the UK (Dr Rushton at Cardiff University), which have developed critical expertise on the analyses of different types of motor activities in humans. We will systematically investigate the mechanisms by which human observers use motor, proprioceptive and global optic flow signals to accomplish visual tasks. Elucidating the perceptual impact of motor activity is critical to advancing our knowledge of how the visual system functions. Such knowledge can also potentially guide the design of objects and environments, inform the building of machines capable of replicating human visual functions, and it may provide a scientific basis for the development of treatments of visual impairments commonly associated with abnormal motor activity in pathological conditions.
Planned Impact
Perceptual research has an exceedingly broad impact. It underpins the design of the computer this document is being written on (and potentially read on); both the interface and the physical screen. If you were to set this document aside and take a walk outside you would see evidence of the use of perceptual research all around. If you were stood in a street then the influence of perceptual research could be seen in the design of the street markings and traffic lights. If you walked into the nearest shop the capture of your attention would probably be driven in part by research that can be traced back to perceptual psychology and if you admire the colour of an object in the shop, the reason that the colour you perceive corresponds to the colour the designer intended is due to the used of perceptual colour models.
Worldwide scientific advancement: The UK, Netherlands and the US are the primary contributors to research on understanding of perceptual processes. This project brings together researchers from the three countries to attempt to raise understanding of perceptual processes to a new level. We will examine the interaction of vision and movement. Is movement a source of noise, or is it a source of information?
Specific beneficiaries: A number of potential beneficiaries can be identified. There are a number of specific problems associated with perception during movement. These include visual vertigo, oscillopsia and nystagmus. Links to movement perception have also been suggested in autism, dyslexia and developmental coordination disorder. A serious problem experienced during self-movement by the elderly is falls. The basic research conducted here might inform clinical diagnosis and intervention. Sports almost by definition involve movement. An understanding of how vision and movement interact may inform training of individuals. Virtual and remote environments (from computer games to remotely operated robots or vehicles) are becoming increasing common. All require a user to move comfortably, naturally and accurately through an environment. An understanding of the interaction between movement and vision in natural environments could provide constraints and guidelines for the design of such systems.
Training Highly Skilled Researchers: We will train post-doctoral fellows who will have substantial input into the organization and design of the project. These postdoctoral fellows should gain the skills necessary to run their own laboratories and translate the techniques and findings into applied settings.
Inter/Multidisciplinarity: The grant sits at the interface between psychology, computer vision and movement science. It provides an opportunity to bring techniques and researchers together across these disciplines and provide training in the range of disciplines to the post-doctoral fellows.
Innovative methodologies: The project brings together a range of technologies and approaches to allow the interaction of vision and movement to be assessed using a number of innovative and novel techniques.
Each of the three laboratories has existing clinical and industrial contacts and. In particular the UK PI, Rushton is currently collaborating with clinicians in the UK on visual vertigo and has an ongoing transatlantic collaboration investigating motion deficits following brain injury. This work will be informed by this project.
Worldwide scientific advancement: The UK, Netherlands and the US are the primary contributors to research on understanding of perceptual processes. This project brings together researchers from the three countries to attempt to raise understanding of perceptual processes to a new level. We will examine the interaction of vision and movement. Is movement a source of noise, or is it a source of information?
Specific beneficiaries: A number of potential beneficiaries can be identified. There are a number of specific problems associated with perception during movement. These include visual vertigo, oscillopsia and nystagmus. Links to movement perception have also been suggested in autism, dyslexia and developmental coordination disorder. A serious problem experienced during self-movement by the elderly is falls. The basic research conducted here might inform clinical diagnosis and intervention. Sports almost by definition involve movement. An understanding of how vision and movement interact may inform training of individuals. Virtual and remote environments (from computer games to remotely operated robots or vehicles) are becoming increasing common. All require a user to move comfortably, naturally and accurately through an environment. An understanding of the interaction between movement and vision in natural environments could provide constraints and guidelines for the design of such systems.
Training Highly Skilled Researchers: We will train post-doctoral fellows who will have substantial input into the organization and design of the project. These postdoctoral fellows should gain the skills necessary to run their own laboratories and translate the techniques and findings into applied settings.
Inter/Multidisciplinarity: The grant sits at the interface between psychology, computer vision and movement science. It provides an opportunity to bring techniques and researchers together across these disciplines and provide training in the range of disciplines to the post-doctoral fellows.
Innovative methodologies: The project brings together a range of technologies and approaches to allow the interaction of vision and movement to be assessed using a number of innovative and novel techniques.
Each of the three laboratories has existing clinical and industrial contacts and. In particular the UK PI, Rushton is currently collaborating with clinicians in the UK on visual vertigo and has an ongoing transatlantic collaboration investigating motion deficits following brain injury. This work will be informed by this project.
People |
ORCID iD |
Simon Rushton (Principal Investigator) |
Publications
Rushton SK
(2018)
Ability to identify scene-relative object movement is not limited by, or yoked to, ability to perceive heading.
in Journal of vision
Evans L
(2020)
Detection of scene-relative object movement and optic flow parsing across the adult lifespan.
in Journal of vision
Rushton Simon K.
(2016)
Does self-movement silence change detection?
in PERCEPTION
Clark K
(2015)
Improvement in visual search with practice: mapping learning-related changes in neurocognitive stages of processing.
in The Journal of neuroscience : the official journal of the Society for Neuroscience
Dunn MJ
(2018)
Lateral visual occlusion does not change walking trajectories.
in Journal of vision
Rogers C
(2017)
Peripheral Visual Cues Contribute to the Perception of Object Movement During Self-Movement.
in i-Perception
De La Malla C
(2019)
The predictability of a target's motion influences gaze, head, and hand movements when trying to intercept it.
in Journal of neurophysiology
Rushton S
(2018)
The Primary Role of Flow Processing in the Identification of Scene-Relative Object Movement
in The Journal of Neuroscience
Description | The primary aims of the project was to examine the role of self-movement in perception and the impact of self-movement on perception. During the course of the project we collected data that suggest: [1] When walking: The act of walking does not change the way the brain processes visual motion information, the area over which it is integrated, to optimise estimates of self-movement. The ability to detect changes in objects is impaired when the objects move ("motion silencing"). Motion silencing occurs when object movement is due to movement of the observer, but it is a relatively small effect, unlikely to have any consequence during normal walking. [2] Detecting the movement of objects within the scene when you are moving: The process that underpins this ability is exceedingly fast. The process relies solely on global motion information ("optic flow") about self-movement, not positional or form cues. The ability to judge object movement is at least as precise as the ability of judge the direction of self-movement. [3] Using visual information when you are moving to guide walking: Information about where you are in the scene ("allocentric location cues") have an important, and previously unrecognised, role in guiding walking direction. Humans very rapidly acquire a mental representation of the environment in which they are walking and they can make use of it when the visual information is poor. When walking to a target object, its change in direction relative to the body ("target drift) is a surprisingly good cue to tell you if you are currently on a course to the left or right of the target. People who have lost awareness of space on one side due to sensory or attentional impairments have problems walking. We found that the lateralised loss of awareness does not underpin the difficulties they encounter. [4] Self-generated movements change perception of time, in-line with an internal clock slowing down. We formed a strong multi-disciplinary, international collaborative team (Brenner, Netherlands; Rucci, USA) during the course of the project. All partners benefited greatly from the knowledge, expertise and support of the other partners. In addition we took the opportunity to forge additional collaborations both internationally (Harris, York University, Toronto) and locally. The postdoc received training in new techniques and has gone on to take a Lectureship at the University of the West of England. A number of undergraduate and postgraduate students had the opportunity to take part in research associated with the project. |
Exploitation Route | Academic: [1] The findings provide the foundation for a newly awarded EPSRC award ("Seeing the Future", EP/R030855/1, Human-like Computing Initiative; Marshall, Rushton and Moran). In the new award we will be applying the idea that, the human brain predicts incoming sensory information as the eye moves through space, to the design of a computer vision system. [2] The findings provide the foundation for a new ORA project, The Active Observer, led by Katja Fielher (Germany) and including Eli Brenner (Dutch partner on current ORA project). [3] The finding that the detection of object movement during self-movement ("flow-parsing) is fast, and based solely on global motion ("optic flow") information, provides a foundation for others to explore the computational and neural substrates of flow parsing. [4] The work on allocentric location cues and target drift provide the foundation for a new line of work on the visual guidance of walking. Industrial, Healthcare, security [1] The new EPSRC award provides a bridge between the basic science in this award and application in industrial, healthcare or security settings. [2] We have shared our findings with industry. At a recent workshop we had participants from the Microsoft Hololens division (Cambridge, UK) and Hewlett Packard Laboratories (Bristol, UK). We discussed our findings and the design of contemporary products. |
Sectors | Aerospace Defence and Marine Digital/Communication/Information Technologies (including Software) |
URL | http://psych.cf.ac.uk/publications/rushton.php |
Description | [1] We have shared our findings with industry. At a recent workshop we had participants from the Microsoft Hololens division (Cambridge, UK) and Hewlett Packard Laboratories (Bristol, UK). We discussed our findings and the design of contemporary products (including the Microsoft Hololens). [2] Work on this project set the foundations for a newly awarded EPSRC award ("Seeing the Future", EP/R030855/1, Human-like Computing Initiative; Marshall, Rushton and Moran). In the new award we will be applying the idea that the human brain predicts incoming sensory information as the eye movements through space to the design of a computer vision system. This new award provides a potential bridge between the basic science in this award and direct application in industrial, healthcare or security situations. |
First Year Of Impact | 2017 |
Sector | Digital/Communication/Information Technologies (including Software) |
Description | Human-like Computing |
Amount | £290,959 (GBP) |
Funding ID | 513881 |
Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
Sector | Public |
Country | United Kingdom |
Start | 03/2018 |
End | 04/2020 |
Description | The active observer |
Amount | £849,000 (GBP) |
Funding ID | ES/S015272/1 |
Organisation | Economic and Social Research Council |
Sector | Public |
Country | United Kingdom |
Start | 01/2019 |
End | 12/2022 |
Title | Experimental code |
Description | Code for generating experimental stimuli. Code was published with papers. See papers for details. |
Type Of Material | Improvements to research infrastructure |
Year Produced | 2018 |
Provided To Others? | Yes |
Impact | Only just been made available. |
Title | Flow as primary cue |
Description | Psychophysical data on judgements of object movement and heading. |
Type Of Material | Database/Collection of data |
Year Produced | 2018 |
Provided To Others? | No |
Impact | Led to paper http://www.jneurosci.org/content/38/7/1737 |
URL | http://TBC.org |
Title | Heading vs Parsing |
Description | Psychophysical data on judgements of the precision of heading vs judgements of parsing |
Type Of Material | Database/Collection of data |
Year Produced | 2018 |
Provided To Others? | Yes |
Impact | Led to publication. https://arvojournals.org/article.aspx?articleid=2686023 |
URL | http://doi.org/10.5255/UKDA-SN-853037 |
Title | Lateral Occlusion |
Description | Walking trajectories to targets during lateral occlusion. |
Type Of Material | Database/Collection of data |
Year Produced | 2018 |
Provided To Others? | Yes |
Impact | Led to paper https://arvojournals.org/article.aspx?articleid=2702357&resultClick=1 |
URL | http://TBC.org |
Title | Peripheral Flow |
Description | Psychophysical data on role of peripheral flow in flow parsing. |
Type Of Material | Database/Collection of data |
Year Produced | 2017 |
Provided To Others? | Yes |
Impact | Led to paper https://doi.org/10.1177/2041669517736072 |
URL | http://doi.org/10.5255/UKDA-SN-852880 |
Title | Spatial integration of motion information during the act of walking |
Description | Raw data from Kesten staircases for duration discrimination task to compare duration thresholds as a function of stimulus size and walking vs standing. The data will be made publicly available at the time of publication. |
Type Of Material | Database/Collection of data |
Year Produced | 2017 |
Provided To Others? | No |
Impact | Led to paper - currently under review. |
URL | http://not.yet.com |
Description | Active head movements |
Organisation | Free University of Amsterdam |
Department | Faculty of Human Movement Sciences |
Country | Netherlands |
Sector | Academic/University |
PI Contribution | We are working on a collaborative project with our ORA partners in Amsterdam on a project looking at whether actions are supported by active head movements to optimise visual perception. We contributed to the design of the experiments and expect to contribute to the analysis and publication. |
Collaborator Contribution | As noted above, this is a collaborative project. Our ORA partners collaborated on the design of the experiment. They will collect the data and we will collaborate on the analysis and write up. |
Impact | Multi-disciplinary, psychology and human movement science. Paper submitted for publication. |
Start Year | 2016 |
Description | Effect on self-movement on perception of temporal intervals |
Organisation | Cardiff University |
Department | School of Psychology |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | With Marc Buehner and Richard Fereday I have been looking at the impact of self-movement on perception of temporal intervals. I contributed to the design, analysis and write-up of empirical research. |
Collaborator Contribution | The partners worked with me to design, execute and write-up empirical work on the impact of self-movement on perception of temporal intervals. |
Impact | Single discipline: psychology. Paper about to be submitted to Cognition, The role of time perception in temporal binding: Impaired temporal resolution in causal sequences. |
Start Year | 2016 |
Description | Motion silencing |
Organisation | York University Toronto |
Country | Canada |
Sector | Academic/University |
PI Contribution | In a collaborative project I designed and ran an experiment looking at motion silencing in the presence of full-field optic flow. In collaboration I analysed the data and presented it at ECVP. |
Collaborator Contribution | My collaborators at York contributed to the design, execution, analysis and write-up stages of the project. |
Impact | Presentation at ECVP. [insert REF] |
Start Year | 2016 |
Description | Perception of motion in depth |
Organisation | Free University of Amsterdam |
Department | Faculty of Human Movement Sciences |
Country | Netherlands |
Sector | Academic/University |
PI Contribution | In collaboration with our ORA partners we designed an experiment to examine whether flow parsing precedes perception of motion-in-depth. We collected pilot data and presented the work at ECVP. |
Collaborator Contribution | Our ORA partners provided input to the design of the experiment and will provide input to later experimental work as well as analysis and write-up. |
Impact | Presentation at ECVP [add REF] |
Start Year | 2016 |
Description | Visual Vertigo |
Organisation | Cardiff and Vale University Health Board |
Country | United Kingdom |
Sector | Public |
PI Contribution | Along with internal collaborators (Sumner) and clinical collaborators (Derry and Rajenderkumar) at the University Hospital Wales, Cardiff I have begun to look at Visual Vertigo. This is a problem with motion perception (sometimes called "supermarket syndrome" or "visual dependency") that can arise following damage to the vestibular system. In collaboration we have collected preliminary data on the relationship between symptoms of visual vertigo and visual processing in the student population. Next we will examine clinical patients. |
Collaborator Contribution | The partners have collaborated with us on data collection. |
Impact | Presentation at ECVP (Powell et al) in 2016. |
Start Year | 2015 |
Title | Spatial integration of motion information during the act of walking (analysis and stimulus code) |
Description | This software is for generating stimuli for the spatial integration dataset and for analysis of the data generated. |
Type Of Technology | Software |
Year Produced | 2017 |
Impact | contribution to open science |
URL | http://not.yet.com |
Description | Moving to see, industry workshop, Somerset 2017 |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Industry/Business |
Results and Impact | We had a workshop with attendees from Microsoft (Hololens division) and Hewlett-Packard in which we discussed the work we have been conducting and the results we have obtained as part of the ORA+ project. We also discussed industry developments and issues (such as the technical and practical issues associated with Hololens). |
Year(s) Of Engagement Activity | 2017 |