Neural pathways underlying human 3D motion perception

Lead Research Organisation: University of Bradford
Department Name: Sch of Life Sciences

Abstract

We use our eyes and brain to move confidently within our surroundings without bumping into things, identifying objects as dangerous or attractive and parsing subtle changes in facial expression. Vision is a hugely complex process that uses much of the brain's resources and involves a constant trade-off between energetic efficiency, speed and accuracy. How is this achieved?

Clues to answer this question come from fundamental biology. Anatomically, it is striking that there are multiple pathways in the visual system and neurons in different visual areas and pathways appear differentially sensitive to certain types of visual information, such as colour or motion.

It is clear that some visual brain areas and pathways have evolved at different times and for different functions. Dedicating different pathways to different functions can be a way of reducing the complexity of the processing problem - allowing the brain to compute independent properties in parallel. Here, we are interested in a specific set of pathways that seem to show strong independence of this type: those involved in the perception of motion in three dimensional space.

Whilst motion is known to be critical for the 'where' functions of the dorsal pathway, very little attention has been placed on how binocular information for motion is processed, nor what pathways carry out that processing. In this project we explore how binocular visual information about motion-in-depth (MID) is processed and carried by several different visual pathways.

Two computational processes have been proposed for using binocular information for MID, and there is evidence for each of them being useful for human vision. Are these signals processed along different fundamental pathways in the brain? Why is it interesting to ask this question? (1) Because neither pathway is fully understood: the sites and natures of the computations involved in processing MID in two ways have not been identified. Even more intriguingly, while one pathway, has been much studied, the other is barely explored, very poorly understood and potentially ancient, in an evolutionary sense. (2) The two pathways might perform different functions, and we propose a series of studies to specifically explore what those functions might be.

Our project has very broad scope, we explore the nature of the putative pathways at the anatomical level, using functional magnetic resonance imaging (fMRI) to localize function, and source-imaged electroencephalography (EEG), which can be used to understand the temporal dynamics of visual processing. We will use psychophysical behavioural studies to study what computational processes take place during MID perception, using both a normal population to explore normal function, and a clinical group of subjects (strabismic amblyopes) that we know have compromised MID processing using one specific pathway. Additionally using Transcranial Magnetic Stimulation (TMS) to degrade information in a particular visual area we will test the causal relevance of MID-responsive regions identified with fMRI and EEG. Finally, we will also employ eye-tracking methods to understand what specific sources of MID information are useful for. Using all these techniques will allow us to get a full picture of the processes underlying MID. To achieve this we require the expertise of three institutions, and we will need to host two RA's, one with visual behavioural and eye tracking skills, the other with imaging credentials.

The work proposed in this project is primarily core visual neuroscience. However, it has implications for human health. One of our techniques will exploit the fact that a person with a squint (strabismic amblyopes) is unable to use a core source of MID information, namely binocular disparity. There are hints that this group may be able to use other sources of MID. Our group will be the first to explore this issue comprehensively.

Technical Summary

Anatomically, it is striking that there are multiple pathways in the visual system. Physiologically, neurons in different visual areas appear specifically sensitive to different visual information, like colour, motion or objects. Why? Dedicating different pathways to different functions can be a way of reducing the complexity of the processing problem. Here, we are interested in a specific set of pathways, those involved in the perception of motion in three dimensions.

Whilst motion is known to be critical for the 'where' functions of the dorsal pathway, very little attention has been placed on how binocular information for motion is processed, nor what pathways carry out that processing. In this project we explore how specifically binocular visual information about motion-in-depth (MID) is potentially processed and carried by several different visual pathways.

Two computational processes have been proposed for using binocular information for MID, and there is evidence for each of them being useful for human vision. One is the rate of change of binocular disparity over time (changing disparity, CD), the other is the binocular combination of monocular motion signals (inter-ocular velocity difference, IOVD). Both are known to be used in human vision. Our aim here is to determine if these signals processed along different fundamental pathways in the brain. We target the magnocellular (magno), parvocellular (parvo) and koniocellular (konio) pathways. The former two are well studied; disparity is carried by both pathways, motion predominantly by the magno. Of particular interest, is the fact that the konio pathway projects to extra-striate cortex without passing through V1, thus providing an alterative pathway to higher visual areas. We will use brain imaging (fMRI, EEG) and visual psychophysics, to explore where, how and why MID information may be separated across different pathways.

Planned Impact

Specific Users / Stakeholders
Our proposal is for core human neuroscience research with no immediate application to UK-plc. However, we have identified two groups of possible indirect stakeholders, and two groups of direct stakeholders:
(1) Technologists and display developers, using stereoscopic displays and stereoscopic content producers.
(2) Medical professions involved in diagnosing and understanding binocular vision deficits.
(3) The general public: both consumers and producers of new stereoscopic content.
(4) Project researchers, who will be trained in interdisciplinary skills and exchange subject-specific knowledge.

Plans for engagement
(1) Technologists developing 3D material
Whilst the basic neuroscientific knowledge we generate will not be of direct interest to this group, they should be interested in the our characterisation of the utility of CD and IOVD information, in other words, how important these two sources of information are for 3D display. Harris or RA1 will attend the international conference "Stereoscopic Displays and Applications", set up by IEEE, towards the end of the project to disseminate our results in the appropriate way for this audience.
(2) Medical professionals
Our work should be of relevance to medical professionals interested in amblyopia and strabismus, conditions that are linked and thought to be due to deficits in the development of binocular vision. One of our experiments will test a population of strabismic observers. For our purposes, we use that population's differently developed vision to test hypotheses, however, our experimental results will have relevance to the understanding of the conditions, and for their potential treatment. Towards the end of the project, we plan to invite our clinical contacts to a workshop session organised under the auspices of the Bradford Institute for Health Research to discuss the clinical implications of our work.
(3) The general public
We intend to make full use of the Press Relations Offices at each of the Institutions involved to maximise our outreach. To this end both RA's will attend the BBSRC Media Training Course.
Wade hosts public engagement websites to teach the public about colour vision, http://www.vischeck.com/ and infant visual development http://www.tinyeyes.com/ . We will build on this experience to develop a website to demonstrate the two forms of binocular motion in depth systems that can be isolated both and allow users to generate stimuli that contain mixtures of each cue for educational purposes.
Binocular vision is also a very popular topic with the general public of all ages, due in part to the recent resurgence of 3D film. This therefore provides an excellent opportunity for public engagement, to involve people in understanding how the brain is needed for sight and how 3D technology works. We have experience and equipment to deliver public displays at Science Fairs on how the psychology and neuroscience of binocular vision link to optics and ophthalmology. We plan to attend opens days and science fairs in Scotland and Yorkshire.
(4) Project researchers
The proposal is an interdisciplinary collaboration. We will appoint experienced postdoctoral researchers in St. Andrews and York. Each individual will experience training and will be exposed to research across disciplines, allowing us to deliver genuine inter-disciplinary scientists to the UK research community, at the end of the project. This aim will be achieved via our regular (12 in total) face to face meetings between all members of the project group.

Publications

10 25 50
publication icon
Benjamin AV (2018) The Effect of Locomotion on Early Visual Contrast Processing in Humans. in The Journal of neuroscience : the official journal of the Society for Neuroscience

publication icon
Bloj M (2016) Bias effects of short- and long-term color memory for unique objects. in Journal of the Optical Society of America. A, Optics, image science, and vision

publication icon
Kaestner M (2019) Asymmetries between achromatic and chromatic extraction of 3D motion signals. in Proceedings of the National Academy of Sciences of the United States of America

publication icon
Weiß D (2015) Color constancy revisited: A better approach in Journal of Vision

 
Description We have made considerable progress on all aims:

A paper on the cortical representation of chromatic MiD signals is nearing completion and will be submitted to the journal 'Neuroimage' in April 2018. The most striking finding is a qualitative difference in the way that MiD signals are transmitted in the different chromatic pathways: broadly, the S-cone isolating pathway is preferentially sensitive to IOVD signals while the achromatic pathway is more sensitive to CD information. Our data are complementary to those from another lab working on MiD representation in macaque using fMRI and we are in discussion with this group about submitting two papers (human an macaque results) in tandem.

In a second paper, to be submitted shortly after the first, we show that attentional modulation of the two different MiD cues also suggests parallel processing pathways that overlap partially but not completely. These experiments demonstrate that attentional modulation of fMRI responses are highly significant in both size and extent and represent a novel way to examine both stimulus- and task-based neuronal responses in humans.

A third fMRI paper examining eye of origin representations across cortex is also in preparation - all data collection is complete and we have almost finished analysis. We find that eye of origin information is present not just in early visual areas (specifically V1) but also, to some degree, in motion selective area hMT. This finding is in agreement with recent single unit work in macaques and indicates that IOVD information in particular could be computed in these higher visual areas.

We have completed data collection on a novel EEG experiment examining the time-course of MiD signal processing. Using state-of-the-art machine learning algorithms we are able to classify different types and directions of MiD cues based solely on the pattern of electrode responses at specific times after stimulus onset. Our results indicate that CD and IOVD signals drive very different cortical networks and that the direction of CD, but not IOVD MiD stimuli can be decoded from neuronal responses around 300ms after stimulus onset.

We have also completed three psychophysical studies: The first, which is under review at IOVS, demonstrates that amblyopic subjects very rarely have preserved IOVD MiD pathways. We report on a single subject (out of 15 tested) whose vision in this domain was preserved, despite having little or no stereoacuity. This result is important because it demonstrates that IOVD and CD signals may have independent neuronal substrates.

In a second paper we map out the range of spatiotemporal parameters that govern IOVD and CD sensitivity. We show that the two different cues operate optimally over different ranges of motion speed and disparity and that together they provide good sensitivity to MiD over a wide range of potential natural stimulus conditions. This paper is in preparation for submission in Summer 2018.

In a third paper (under review, Journal of Vision), we have shown so far IOVD information can likely only be totally isolated using de-correlated rather than anti-correlated motion. The well known increase in duration thresholds with stimulus size for lateral motion is not present for motion in depth. Thus there appears to be a fundamental difference between motion in depth and lateral motion information. this work has been presented at the Applied Vision Association and European Conference on Visual Perception.

Finally, a series of experiments measuring eye movements have just been completed, in collaboration with Tony Norcia's lab (Stanford University), to explore how different sources of visual information about motion in depth drive vergence eye movements. Results show that IOVD information is not a good driver of vergence, whether de-correlated or anti-correlated. This work is currently being written up for publication.
Exploitation Route Findings were taken to national and international conferences over 2016-17 and have been submitted or are being prepared for submission to top journals.
Sectors Education

 
Description The primary goals of the research comprised core human neuroscience research, but the results have scientific impacts beyond the specific outcomes detailed elsewhere. The project itself explores the basic visual pathways that analyse motion, a core feature of the perceptual system for a mobile animal, such as ourselves. We outlined 4 groups of user and stakeholders in our Pathways to Impact document, technologists, medical professional, general public and project researchers. At this, point, we have contributed to impact deliverables for all four of these. (a) Engagement with medical professionals. We have engaged with Dr. Alison Bruce, Head Orthoptist at Bradford Teaching Hospitals NHS Foundation Trust/ NIHR Post Doctoral Research Fellow at the Bradford Institute for Health Research (BIHR). She helped establish a Yorkshire-based (University of York/ University of Bradford) pool of suitable study participants. RA Maloney presented the findings of the work with amblyopes at the British Congress of Optometry and Vision Science 2017. (b) Public engagement. The Harris lab have developed displays for public science festival events, so far taking part in the St. Andrews University Science Open Day, 2015 and St. Andrews Explorathon (European Researchers Night) 2016 and 2017. Harris has given two talks in 2015, to young adults in 6th form, as part of a Sutton Trust Summer School, and for the St. Andrews Open Association lectures. and Harris has talked at Cafe Scientifique (Dunkeld and Pitlochry) in 2016 and has deliver two Quiz a Whiz interviews for the Royal Society of Edinburgh (2016-17). (c) RA Maloney has completed a media training courses (Royal Society) in 2017. The whole group, including 3 PhD students working in similar areas, meet 4 times a year for project meetings. These give the RA's the opportunity to interact with scientists trained in other areas and to learn to collaborate in an interdisciplinary fashion. Finally, (d) Technologists: We have presented our findings at the Electronic Imaging conference in Burlingame, CA in early 2018. This conference is dominated by industry and our session on 'The Engineering Challenges of Virtual Reality' was attended by representatives from major Silicon Valley technology companies including Apple and Google. We have also engaged with representatives from Google's 'Daydream' team and are collaborating on potential VR/Health applications in the future.
First Year Of Impact 2015
Sector Healthcare
Impact Types Economic

 
Description Leverhulme Project Grant
Amount £200,344 (GBP)
Organisation The Leverhulme Trust 
Sector Academic/University
Country United Kingdom
Start 03/2018 
End 03/2021
 
Description ViiHM network: Visual image interpretation in humans and machines 
Organisation University of Birmingham
Department School of Psychology Birmingham
Country United Kingdom 
Sector Academic/University 
PI Contribution EPSRC funded network, I am a member.
Collaborator Contribution A network funded by EPSRC -- Schofield at Birmingham is the PI
Impact None yet
Start Year 2014
 
Description Work with Dr Alison Bruce - Head Orthoptist and NIHR Post-doctoral Research Fellow, Bradford Institute for Health Research and Bradford Teaching Hospitals NHS Foundation Trust 
Organisation Bradford Institute for Health Research (BIHR)
Country United Kingdom 
Sector Academic/University 
PI Contribution Access to expertise in basic vision science
Collaborator Contribution Access to expertise in management of binocular vision disorders
Impact none yet, but papers are under review
Start Year 2010
 
Description Explorathon 2017: Harris lab exhibit at this European Researchers night event, highlighting research activity on vision. 
Form Of Engagement Activity Participation in an open day or visit at my research institution
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact Explorathon 2017: Harris lab exhibit 'Vision: more than meets the eye' at this European Researchers night event, highlighting research activity on vision.
Year(s) Of Engagement Activity 2017
URL http://www.explorathon.co.uk/standrews
 
Description Visual Perception in Humans and Animal Camouflage, Explorathon 18 exhibit 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact Visual Perception in Humans and Animal Camouflage, Interactive presentation at science fair, Explorathon 2018, held at Dundee Science Centre (science museum).
Year(s) Of Engagement Activity 2018