Early advantage of luminance for object representation and its cross-talk with chromatic pathways in human visual scene analysis

Lead Research Organisation: University of Aberdeen
Department Name: Psychology

Abstract

Detection and identification of objects is the most crucial goal of human visual perception. Several parallel channels in the visual system processes incoming information with that purpose, segmenting the images in both eyes through a series of rapid hierarchically organized stages. This rapid hierarchical processing stream involves a cascade of neural activity that encompasses a series of brain areas, from primary sensory regions that analyse separate visual features (low-level vision), through parts that organise the percept into figures and background (mid-level vision), to parts of the brain that store semantic knowledge on familiar objects (high-level vision). A coherent representation of our environment is thus formed in less than 300 milliseconds of processing time within a range of highly varied brain regions. In the study on visual perception, it is crucial to investigate in which way does the brain manage to coordinate the processing of information on simple visual features such as colour and luminance along each of the transformative stages (low, mid, high-level) that lead to the perception of objects. It is well-known from studies on animals that separate visual channels in mammalian brain process achromatic and chromatic information: magnocellular pathway processes luminance information, while parvo- and koniocellular pathways predominantly subserve colour processing. Both are crucial for everyday object vision but their contributions differ, with luminance considered to be more relevant for rapid processing of lines, edges, shape and motion and colour being more relevant for segmentation of visual scenes. But the extent to which various visual pathways function independently or interactively at different stages of visual processing remains unknown even after many years of study, due to the difficulties in 1) producing stimuli that selectively elicit processing along different pathways and 2) analysing the rapidly evolving neural processes that subserve normal human vision. We intend to conduct a study that will overcome these problems through an innovative experimental approach which joins electroencephalography's (EEG's) ability to divulge millisecond differences in rapid neural processes that underlie human visual perception with the tools of colour psychophysics which allow us to separate out different visual processing streams by defining our stimuli in three-dimensional colour space (a luminance dimension and two chromatic dimensions). The stimulus displays will be controlled through a visual stimulus generator that would enable systematic and concurrent control of inputs into chromatic and achromatic mechanisms. The timecourse of cortical activations and their underlying generators will be identified and the interactions between different pathways modelled by using stimuli that elicit excitations in single (magno-, parvo- or konio-) or multiple (two or all three combined) pathways. The findings of this study will provide an important insight into the ways in which the human brain utilises different types of information during parallel visual processing and will thus significantly contribute to current knowledge on the relations between parallel (magnocellular, parvocellular or koniocellular) and hierarchical (low, mid, high-level) processing. We will be able to describe the neural mechanisms that allow preferential inputs of luminance information into object representation processes and thus enable it to drive everyday vision. A further advantage will be provided by the description of koniocellular contributions to vision which have so far not been studied extensively since this pathway has only recently become the subject of systematic study in human participants.

Technical Summary

Luminance information plays a unique role in everyday vision by providing inputs that drive the efficiency and speed of object representation. Chromatic information also contributes to high-level vision, both independently and through interacting with luminance information at higher stages of cortical processing - but the extent to which they function independently or interactively remains unknown. In our study, we will join electroencephalography's (EEG's) ability to divulge millisecond differences in rapid neural processes with the tools of colour psychophysics which allow us to bias processing towards the luminance or chromatic pathways. We will define our stimuli in three-dimensional colour space whose directions correspond to magno-, parvo- and koniocellular pathways. This will enable us to test the early neural markers of processing that is biased towards a single pathway, drives two pathways or drives all three pathways at different stages of the visual hierarchy (low-, mid- or high-level). Early event-related activity elicited by stimuli that are processed predominantly along a single pathway will be measured to obtain its characteristic activation. This will allow us to model the evolution of rapid cortical responses elicited by summation of this information from different geniculate pathways. Source localisation will allow us to to look for differences in the excitation of various extrastriate areas that relate to the degree of activation of the luminance pathway. In study 1 which examines low-level vision, participants will perform orientation discrimination between circular horizontal or vertical Gabors. In study 2 which examines mid-level vision, participants will discriminate displays with fully-random arrays of gabors from displays with a pseudo-random array containing a contour. In study 3 which examines high-level vision, participants will discriminate line-drawings of real-life objects from scrambled images of such objects (so-called non-objects).

Planned Impact

The primary impact of the proposed project will be an increased understanding of processes that underlie visual perception. Perceptual experiences form the basis of human consciousness - for example, the percept of colour red is related to an internal awareness of the sensation of redness, which is a question that is of interest to cognitive scientists, philosophers as well as the public at large. Many popular science books treat this topic. But the study of basic analysers in vision (luminance or colour pathways, for example) has so far often been neglected by scientific writers as unapproachable to the general public due to its focus on the study of elementary neural processes related to simple stimuli (e.g. gratings, gabor patches, lines of different orientation). The proposed study will answer important questions about the role of elementary visual processes for everyday vision. It will demonstrate how certain types of information have preferential role in forming high-level representations of our environment. This is a matter that is of intrinsic interest to all members of our society. The research will also be highly interesting to both vision scientists and clinical neuroscientists, as deficits in the luminance pathway have been identified in a series of conditions such as dyslexia (K. Nandakumar and S.J. Leat, (2008), Clin. Exp. Optom., 91, 333-340), schizophrenia (P.D. Butler, et al., (2007), Brain, 130, 417-430), autism (K. Plaisted and G. Davis, (2005), Cah. Psychol. Cogn.-Curr. Psychol. Cogn., 23, 172-179) or Parkinson's disease (W.G. Sannita, et al., (2009), Vision Res., 49, 726-734). Research by Butler and colleagues shows that subcortical magnocellular deficits drive secondary cortical impairments in schizophrenia. By investigating the function of the visual pathways and their potential interactions in high-level vision, the proposed work will provide a starting point for clinically applied research which could potentially lead to symptom alleviation or prevention techniques. The work will be submitted for publication in high-impact journals. The resluts will also be communicated through media with an interest in scientific breakthroughs. Another major way of publicising the proposed research will occur through networking at scientific meetings, within the UK research community as well as abroad. Some of these meetings are dedicated to the application of scientific work and thus bring together the UK scientific community and the engineering sector: e.g. the Applied Vision Association or the UK colour group. This will provide an opportunity to network with experts who are ideally placed to implement possile insights gained from our work into improvements of human-machine interaction devices. For example, air traffic control instruments could benefit from knowledge about types of visual information that result in highest rates of detection or fastest reaction times. The reputation of the United Kingdom as one of the leading countries for fundamental research into visual perception (in particular, colour vision) and a center for cutting-edge discoveries in brain and cognitive sciences will be upheld by the advances made in the proposed research. The appearance of this work in top journals may attract more world-class researchers to work at institutions within the UK and further advance the scientific community here.
 
Description We have found that when participants are discriminating unfamiliar shapes (non-objects) and familiar shapes (objects) whose contours are defined by various combinations of luminance and colour there are contrast-dependent windows for interaction between the luminance and colour visual mechanisms, which allow discrimination to be facilitated by colour channels. Within these limits, the amount of luminance needed to discriminate shapes and objects can be decreased by addition of colour. However, when the luminance signal is strong enough, it appears to dominate discrimination.

We have also assessed the neural markers of these processes by using electroencephalography (EEG), which indicated that these interactions occur relatively early, around 150-300ms after the image has been observed. Comparing colour only with colour-luminance combinations shows that early brain responses are object-sensitive only when the image contains both colour and luminance signals. We extended these findings by using several colour-luminance combinations at threshold and at suprathreshold contrast levels. The early EEG response occurred earlier and generally had larger amplitude at suprathreshold and was again object-sensitive only for the full combination of colour and luminance signals, perhaps due to the increase in sensitivity to some mid-level property of the object images. We concluded this line of experiments by using EEG to conduct a detailed assessment of cortical summation and attentional modulation of combined chromatic and luminance signals. Again, our results indicated that colour and luminance contrast were not processed independently, both in terms of perception and selective attention.

In our second line of experiments, we focused on colour and luminance in perceptual organisation / mid-level vision. First, we examined the effects that backgrounds have on object classification by adding background elements to the images of objects. In all combinations of colour and luminance signals there was no adverse effect of background, except for a combination of shortwavelength cones (S-cones) with luminance. These adverse effects of shortwavelength-cone signals combined with luminance signals on the extraction of objects from their background indicate that the role of the colour signals in the segregation of objects from the background may not be just facilitatory, but can actually produce negative effects when certain conditions are met. We are following these results up with psychophysical studies aimed at establishing the source of this suppresive interaction, examining if it is due to masking or crowding. We also looked at symmetry perception for patterns defined by colour and luminance, to test for potential influences of early mechanisms on this important determinant of figure/background segregation. Our findings at threshold support models that propose an important contribution of low-level mechanisms to symmetry perception, in line with distinct contrast sensitivity functions for colour and luminance, with colour providing low frequency information useful for symmetry detection and luminance providing high frequency information useful for detection of detail.

In summary, experiments conducted as a part of this project have shown that the role of colour in the processing of contours and shapes cannot be neglected. In terms of complex scenes, chromatic signals are also likely to influence the processing of shapes within backgrounds. Finally, chromatic information added to luminance information significantly affects attentional selection even at early neural sites. Therefore, interactions between colour and luminance extend throughout the visual processing hierarchy.
Exploitation Route We organised a family event called 'Vision Vibes', showcasing interesting visual perception phenomena at the British Science Festival in Aberdeen in September 2012. The audience had a chance to talk to researchers about their work and pick up a handout leaflet outlining, among other things, our BBSRC-funded research into colour and luminance contributions to human vision. The event was registered with the Science Technology Engineering Mathematics (STEM) network, of which Dr Jasna Martinovic is an ambassador.

One exploitation route concerns the role of basic vision research in sustaining the improvement of machine vision systems: by knowing the properties of human vision very well, it is possible to implement these highly effective heuristics in machine vision development. Human vision has evolved to utilise the limited resources at its disposal optimally and knowing the exact way in which chromatic signals sustain luminance signals during the integration of contours, classification of objects, symmetry perception or attentional selection will help to design better machine vision systems.
Sectors Aerospace, Defence and Marine,Digital/Communication/Information Technologies (including Software)

URL https://sites.google.com/site/drjasnamartinovic/home
 
Description We organised a family event called 'Vision Vibes', showcasing interesting visual perception phenomena at the British Science Festival in Aberdeen in September 2012. The audience had a chance to talk to researchers about their work and pick up a handout leaflet outlining, among other things, our BBSRC-funded research into colour and luminance contributions to human vision. The event was registered with the Science Technology Engineering Mathematics (STEM) network, of which Dr Jasna Martinovic is an ambassador. We are currently pursuing exploitation routes in industry. One exploitation route concerns the role of basic vision research in sustaining the improvement of machine vision systems: by knowing the properties of human vision, it is possible to implement these highly effective heuristics in machine vision development. Human vision has evolved to utilise the limited resources at its disposal optimally and knowing the exact way in which chromatic signals interact with luminance signals during the integration of contours and classification of objects will help to design better machine vision systems.
First Year Of Impact 2012
 
Description eastbio research experience placement
Amount £2,300 (GBP)
Organisation Biotechnology and Biological Sciences Research Council (BBSRC) 
Sector Public
Country United Kingdom
Start 05/2017 
End 08/2017
 
Description responsive mode
Amount £343,632 (GBP)
Funding ID BB/R009287/1 
Organisation Biotechnology and Biological Sciences Research Council (BBSRC) 
Sector Public
Country United Kingdom
Start 08/2018 
End 04/2021
 
Description standard
Amount £625,414 (GBP)
Funding ID EP/P007600/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 01/2017 
End 12/2019
 
Description Collaboration with Dr Corinna Haenschel on luminance and chromatic signals in working memory 
Organisation City, University of London
Country United Kingdom 
Sector Academic/University 
PI Contribution This collaboration has developed during the course of the project, currently my former dissertation student Maciej Kosilo is doing a PhD under Dr Haenschel's supervision, investigating basic visual inputs into working memory. The studies are currently performed on normal observers but will be followed up with studies on patients with schizophrenia.
Collaborator Contribution Dr Haenschel's contribution is to extend the findings on colour/luminance interactions in vision into visual working memory tasks, as well as into the area of different populations (schizophrenia patients).
Impact Disciplines involved are vision research, cognitive neuroscience and clinical neuroscience. Maciej Kosilo's PhD thesis project is entitled 'Early visual stimulus encoding and visual working memory' and is meant to produce further outcomes.
Start Year 2012
 
Title Large stimulus set of Gaborised objects and non-objects 
Description 375 Gaborised objects and non-objects have been piloted and used in the experiments we conducted over the two year period. Upon publication of the main study that used these stimuli, the stimulus set will be made publicly available on the project website. 
Type Of Technology New Material/Compound 
Year Produced 2011 
Impact We have made the stimuli available on the website at the end of the project in 2013 and are currently waiting to see if it's been used by other researchers. 
URL https://sites.google.com/site/drjasnamartinovic/home
 
Description Vision Vibes event at the British Science Festival in Aberdeen 2012 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? Yes
Geographic Reach National
Primary Audience Public/other audiences
Results and Impact The one-day 'Vision Vibes' event involved drop-in demonstrations of visual illusions in an Aberdeen city centre shopping mall and drop-in demonstrations in the University Library of various vision science equipment and displays (e.g., live eye tracking), with handouts available providing a lay summary of the PIs research. The event was covered by the local press and was held during the family weekend, at the end of the festival. As a trained member of STEM network for public engagement, Dr Martinovic trained several staff and students to be demonstrators at the event and also ensured that health and safety regulations were observed.

we do not have data on any notable impacts
Year(s) Of Engagement Activity 2012
URL http://www.britishscienceassociation.org/sites/default/files/root/festival/BSFprog12%20lr%202.pdf