Object vision and population-code representation

Lead Research Organisation: MRC Cognition and Brain Sciences Unit

Abstract

Recognising the objects we see around us is effortless. Seeing is much easier for us than high-level cognitive tasks like playing chess. This belies the computational complexity of object recognition. Computers have surpassed us at chess, but object recognition is an open problem of artificial intelligence. In this programme, we study how the brain achieves visual object recognition so effortlessly and efficiently. The visual images are converted into neural signals in the retina and send to the visual cortex. At early stages of processing the visual system recognises edges and other local features. The visual information is then transformed through a sequence of representational stages that lead to recognition. We present real-world images to people while scanning their brain activity. By analysing the brain-activity patterns representing the stimuli at each stage of processing, we investigate how the brain performs the feat of recognition so swiftly and without requiring us to make any conscious effort. We also study how our goals (e.g. looking for a particular kind of object, such as food) affect the process of visual perception and how our brain representations give rise to the unique way each of us perceives the relationships between objects in the world.

Technical Summary

The objective of this programme is to understand the computational and neural mechanisms of visual object perception in health and disease and their variation across individuals.
Object recognition is effortless for humans and animals, and yet it is one of the unsolved problems of artificial intelligence. Our central challenge is to understand the transformation of representations along the ventral visual stream, from the early visual representation to the level at which natural categories and semantic dimensions are represented, and the computational mechanisms that enable the brain to perform object recognition so swiftly and more reliably than current computer vision systems.
We study representations of objects, places, and faces and higher-level semantic content in healthy volunteers and also in selected clinical populations, starting with autism. Our main methodology is high-resolution functional magnetic resonance imaging combined with pattern-information analyses.
 
Description 7th Framework Programme: Starting Grant
Amount € 1,310,639 (EUR)
Funding ID 261352 
Organisation European Research Council (ERC) 
Sector Public
Country European Union (EU)
Start 09/2011 
End 09/2016
 
Description British Academy Postdoctoral Fellowship - Johan Carlin
Amount £169,478 (GBP)
Organisation The British Academy 
Sector Academic/University
Country United Kingdom
Start 12/2012 
End 12/2015
 
Description German National Academic Foundation
Amount £2,000 (GBP)
Organisation German National Academic Foundation 
Sector Public
Country Germany
Start 10/2011 
End 10/2013
 
Description Postdoc grant to Linda Henriksson from Aalto University, Helsinki, Finland
Amount £140,000 (GBP)
Organisation Aalto University 
Sector Academic/University
Country Finland
Start 09/2011 
End 08/2013
 
Description Probing Visual Short-Term Memory representations with Dynamically Adaptive Imaging
Amount £183,742 (GBP)
Funding ID 091540/Z/10/Z 
Organisation Wellcome Trust 
Sector Charity/Non Profit
Country United Kingdom
Start 10/2010 
End 02/2014
 
Description Rubicon
Amount £66,668 (GBP)
Organisation Netherlands Organisation for Scientific Research (NWO) 
Sector Public
Country Netherlands
Start 05/2011 
End 02/2013
 
Description The Brain, a Prediction Machine?
Amount £188,163 (GBP)
Funding ID pf130086 
Organisation The British Academy 
Sector Academic/University
Country United Kingdom
Start 09/2013 
End 09/2016
 
Description Yousef Jameel Scholarship
Amount £83,000 (GBP)
Organisation University of Cambridge 
Sector Academic/University
Country United Kingdom
Start 10/2012 
End 08/2014
 
Title Matlab RSA toolbox 
Description We have introduced a MatLab toolbox for representational similarity analysis. 
Type Of Material Improvements to research infrastructure 
Provided To Others? No  
Impact This toolbox has been completed and is available for use by other researchers (in neural data analysis), a paper has also been produced with regards to the RSA toolbox. 
URL http://www.mrc-cbu.cam.ac.uk/methods-and-resources/toolboxes/license/
 
Description Autism Research Centre and Prof. Simon Baron-Cohen 
Organisation University of Cambridge
Department Autism Research Centre (ARC)
Country United Kingdom 
Sector Academic/University 
PI Contribution We have an ongoing collaboration with the Autism Research Centre and Prof. Simon Baron-Cohen investigating the Representations of objects in people with an Autism Spectrum Condition.
Collaborator Contribution At the MRC-CBU, we are collecting behavioural and neuroimaging data for 20 male ASC participants that we obtained from the ARC volunteer database. Drs. Michael Lombardo and Meng-Chuan Lai are also helping in collecting the autism diagnostic interviews.
Impact So far, we have measured a total of 8 ASC volunteers and 10 male matched controls. Our plan is to recruit a total of 20 male ASC and 20 male matched control participants (we need 12 more ASC and 10 more controls).
Start Year 2011
 
Description Comparison of multivariate classifiers and response normalizations for pattern-information fMRI 
Organisation National Institutes of Health (NIH)
Department National Institute of Mental Health (NIMH)
Country United States 
Sector Public 
PI Contribution N Kriegeskorte: leadership, fMRI data acquisition, initial analysis programming, main writer of the paper, senior author M Misaki: analysis programming, analysis execution, figure preparation, paper writing Y Kim: contribution to initial analysis programming PA Bandettini: consulting
Collaborator Contribution - started project as a postdoc at the NIMH - local collaborators M Misaki, Y Kim, and PA Bandettini contributed to the project
Impact 20580933
Start Year 2007
 
Description Disentangling ongoing cortical dynamics from stimulus-driven fMRI reponse-pattern similarity 
Organisation Aalto University
Department Brain Research Unit and AMI Centre
Country Finland 
Sector Academic/University 
PI Contribution One of our PhD students has been assisting with the modelling aspect of this project, and also contributing to the on-going discussions.
Collaborator Contribution Our partner in Aalto University is leading this project. Our partner in St. Louis has provided the data.
Impact None to date. Manuscript is in preparation at time of submission.
Start Year 2013
 
Description Disentangling ongoing cortical dynamics from stimulus-driven fMRI reponse-pattern similarity 
Organisation Washington University in St Louis
Department Psychology Department
Country United States 
Sector Academic/University 
PI Contribution One of our PhD students has been assisting with the modelling aspect of this project, and also contributing to the on-going discussions.
Collaborator Contribution Our partner in Aalto University is leading this project. Our partner in St. Louis has provided the data.
Impact None to date. Manuscript is in preparation at time of submission.
Start Year 2013
 
Description Distance measures for fMRI pattern analysis 
Organisation Western University
Department Brain and Mind Institute
Country Canada 
Sector Academic/University 
PI Contribution One of my PhD students is the principle researcher on the above project, working within Joern Diedrichsen's lab at UCL.
Collaborator Contribution Dr Diedrichsen supervises the research.
Impact The have been no outcomes to date. This collaboration is multi-diciplinary, involving Neuroscience and Computer Science.
Start Year 2013
 
Description FMRI voxels: compact kernels or complex spatiotemporal filters 
Organisation Medical Research Council (MRC)
Department MRC Cognition and Brain Sciences Unit
Country United Kingdom 
Sector Public 
PI Contribution First author: N Kriegeskorte Coauthors: R Cusack (MRC CBU), PA Bandettini (NIH)
Collaborator Contribution theoretical discussions, computer simulations, paper writing- started working on this project as a postdoc at the NIMH - my postdoc advisor PA Bandettini contributed to the project
Impact 19800408
Start Year 2008
 
Description FMRI voxels: compact kernels or complex spatiotemporal filters 
Organisation National Institutes of Health (NIH)
Department National Institute of Mental Health (NIMH)
Country United States 
Sector Public 
PI Contribution First author: N Kriegeskorte Coauthors: R Cusack (MRC CBU), PA Bandettini (NIH)
Collaborator Contribution theoretical discussions, computer simulations, paper writing- started working on this project as a postdoc at the NIMH - my postdoc advisor PA Bandettini contributed to the project
Impact 19800408
Start Year 2008
 
Description Face-identity change activation outside the face system: "release from adaptation" may not always indicate neuronal selectivity 
Organisation Maastricht University (UM)
Department Faculty of Psychology and Neuroscience
Country Netherlands 
Sector Academic/University 
PI Contribution M Mur: fMRI data acquisition, data analysis, figure preparation, contributions to design, paper writing (first author) N Kriegeskorte: idea, conceptual leadership, study design, fMRI data acquisition, initial analysis, guidance, paper writing (senior author) DA Ruff: contributions to stimulus generation and design J Bodurka: fMRI acquisition design PA Bandettini: consulting, guidance
Collaborator Contribution First author M Mur funded by this organisation- started this projected there as a postdoc - NIMH covered most of the cost of the project
Impact 20051364
 
Description Face-identity change activation outside the face system: "release from adaptation" may not always indicate neuronal selectivity 
Organisation National Institutes of Health (NIH)
Department National Institute of Mental Health (NIMH)
Country United States 
Sector Public 
PI Contribution M Mur: fMRI data acquisition, data analysis, figure preparation, contributions to design, paper writing (first author) N Kriegeskorte: idea, conceptual leadership, study design, fMRI data acquisition, initial analysis, guidance, paper writing (senior author) DA Ruff: contributions to stimulus generation and design J Bodurka: fMRI acquisition design PA Bandettini: consulting, guidance
Collaborator Contribution First author M Mur funded by this organisation- started this projected there as a postdoc - NIMH covered most of the cost of the project
Impact 20051364
 
Description High-level visual object representations are constrained by position 
Organisation National Institutes of Health (NIH)
Department National Institute of Mental Health (NIMH)
Country United States 
Sector Public 
PI Contribution D Kravitz: data acquisition, analysis programming, figure preparation, paper writing N Kriegeskorte: consulting on design, analysis, and interpretation, analysis programming CI Baker: guidance, paper writing
Collaborator Contribution - started collaborating on this project as a postdoc at NIMH - NIMH covered almost all costs of the project
Impact 20351021
Start Year 2007
 
Description Model Testing Project2 - RSA/Voxel Receptive Fields 
Organisation Aalto University
Department Brain Research Unit and AMI Centre
Country Finland 
Sector Academic/University 
PI Contribution A PhD student in my lab is running the project as the first author, doing the majority of the analysis.
Collaborator Contribution Our partner at Washington University acquired all the fMRI data. The other partner helps in discussion.
Impact Linda Henriksson, Seyed-Mahdi Khaligh-Razavi, Nikolaus Kriegeskorte. Population-code representations of natural images across human visual areas. Annual meeting of the Vision Sciences Society, Naples, Florida, USA, 2013.
Start Year 2013
 
Description Model Testing Project2 - RSA/Voxel Receptive Fields 
Organisation Washington University in St Louis
Department Psychology Department
Country United States 
Sector Academic/University 
PI Contribution A PhD student in my lab is running the project as the first author, doing the majority of the analysis.
Collaborator Contribution Our partner at Washington University acquired all the fMRI data. The other partner helps in discussion.
Impact Linda Henriksson, Seyed-Mahdi Khaligh-Razavi, Nikolaus Kriegeskorte. Population-code representations of natural images across human visual areas. Annual meeting of the Vision Sciences Society, Naples, Florida, USA, 2013.
Start Year 2013
 
Description Object vision and psychophysics 
Organisation University of Minnesota
Department Department of Psychology
Country United States 
Sector Academic/University 
PI Contribution Expertise in object vision and pattern-information fMRI
Collaborator Contribution Expertise in pasychophysics and visual adaptation
Impact None to date
Start Year 2013
 
Description Paper about current opinion on circular analysis, a widespread statistical error in systems neuroscience 
Organisation University of California, San Diego (UCSD)
Department Department of Psychology
Country United States 
Sector Academic/University 
PI Contribution N Kriegeskorte: idea, organisation of author group, content E Vul: organisation of author group, content TE Nichols: content MA Lindquist: content RA Poldrack: content
Collaborator Contribution cowrote theoretical paper
Impact 20571517
Start Year 2009
 
Description Pattern geometry modelling 
Organisation University College London
Department Institute of Cognitive Neuroscience
Country United Kingdom 
Sector Academic/University 
PI Contribution This collaboration is in it's infancy and so this has not yet been clearly defined.
Collaborator Contribution This collaboration is in it's infancy and so this has not yet been clearly defined.
Impact None to date but several research papers and thesis chapters should result, along with contriubtion to the RSA toolbox.
Start Year 2012
 
Description Predicting behaviour from representational geometry 
Organisation University of Maryland
Department Department of Psychology
Country United States 
Sector Academic/University 
PI Contribution Expertise in object vision and pattern-information fMRI
Collaborator Contribution Expertise in MEG decoding and psychophysics
Impact 24001004 23915056 23908380
Start Year 2011
 
Description 'Helping Researchers Publish in Sciences, Technology, Engineering and Mathematics' Workshop 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact This event explored the resources to help with the process of publishing research in STEM disciplines - from recording observations to editing to peer review. It also offered the chance to learn about available tools and options in publishing and reviewing, and ask questions of the experts.
Year(s) Of Engagement Activity 2017
URL http://www.data.cam.ac.uk/helping-researchers-publish-sciences-technology-engineering-and-mathematic...
 
Description Bayesian Theory Interest Group 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Type Of Presentation Workshop Facilitator
Geographic Reach Local
Primary Audience Other academic audiences (collaborators, peers etc.)
Results and Impact 60+ people (post-docs, PhD students, programme leaders, resrach assisants) are invovled in a series of talks looking at Bayesian Theory.

Mailing list has new subscribers weekly and we often see a rise in external visitors tot he Unit, to attend the talks.
Year(s) Of Engagement Activity 2013
URL http://imaging.mrc-cbu.cam.ac.uk/imaging/CbuImaging/Bayesian_theory
 
Description CBSU Science festival 
Form Of Engagement Activity Participation in an open day or visit at my research institution
Part Of Official Scheme? Yes
Type Of Presentation Workshop Facilitator
Geographic Reach Local
Primary Audience Public/other audiences
Results and Impact 90+ members of the public attended our Science Night event, visiting demonstrations, talks and posters. Visitors were very engaged and many, many questions were asked.

Excellent feedback from visitors. Many people had to be turned away due to reaching capacity within 20 mins of the event opening.
Year(s) Of Engagement Activity 2012,2013
URL http://www.cambridgesciencefestival.org/Home.aspx
 
Description Cambridge Neuroscience Meeting 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Other audiences
Results and Impact Niko Kriegeskorte delivered talk on 'Imaging the nervous system'.
Year(s) Of Engagement Activity 2015
 
Description Cambridge Neuroscience Workshop 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Other audiences
Results and Impact Niko Kriegeskorte was invited to participate in the interdisciplinary workshop on "Computational vision: from brains to machines".
Year(s) Of Engagement Activity 2015
 
Description CoSMo 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact Niko Kriegeskorte delivered a talk as part of the Fifth Annual Computational Sensory-Motor Neuroscience Summer School.
Year(s) Of Engagement Activity 2015
URL https://www.bcf.uni-freiburg.de/events/conferences-workshops/150628-cosmo
 
Description Cold Spring Harbor Laboratories 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact Niko Kriegeskorte regular presents at this summer workshop.
Year(s) Of Engagement Activity 2013,2014,2015
URL https://meetings.cshl.edu/courses.aspx?course=c-neudata&year=15
 
Description Columbia University 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Niko Kriegeskorte was invited to give a talk on 'Deep neural networks: a new framework for modelling brain information processing', as part of a seminar.
Year(s) Of Engagement Activity 2016
 
Description Cosyne 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Niko Kriegeskorte co-organised the 'Computational mechanisms of object recognition in ventral visual pathway' workshop, at the Computational and Systems Neuroscience meeting (Cosyne).
Year(s) Of Engagement Activity 2015
 
Description Durham 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Postgraduate students
Results and Impact Niko Kriegeskorte was invited to deliver a seminar.
Year(s) Of Engagement Activity 2015
 
Description Edinburgh SPM Course 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Postgraduate students
Results and Impact Delivered talk as part of Edinburgh SPM Course.
Year(s) Of Engagement Activity 2015
URL http://www.sbirc.ed.ac.uk/cyril/SPM-course/SPM_course_home.html
 
Description Graduate Seminar 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Postgraduate students
Results and Impact Niko Kriegeskorte delivered a talk on 'How to write good papers' as part of the graduate seminar series.
Year(s) Of Engagement Activity 2015
 
Description Invited talk in Oxford 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Other audiences
Results and Impact Title: Testing complex brain-computational models to understand how the brain works
Year(s) Of Engagement Activity 2017
 
Description Invited talk in York 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Other audiences
Results and Impact Title: Testing complex brain-computational models to understand how the brain works
Year(s) Of Engagement Activity 2017
 
Description MRC Centenary Open Day 
Form Of Engagement Activity Participation in an open day or visit at my research institution
Part Of Official Scheme? Yes
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact As part of the nation-wide MRC centenary celebrations, the CBSU held an Open Day. Many of our lab members were involved in running demonstrations and presenting posters.

250 visitors to the Unit. Excellent feedback given.
Year(s) Of Engagement Activity 2013
URL http://www.centenary.mrc.ac.uk/
 
Description Max Planck Institute for Human Cognitive and Brain Sciences 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Niko Kriegeskorte was invited to give an an overview of the research program, and it's future direction.
Year(s) Of Engagement Activity 2016
 
Description NeuroHAM 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Niko Kriegeskorte was invited to deliver the talk Deep Neural Net Visual Representation Resembles Inferior Temporal Cortex, as part of this 3-day conference.
Year(s) Of Engagement Activity 2015
URL http://neuroham.bu.edu/
 
Description Presentation at Cosyne International conference 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact Presenting my work under the title: Testing complex brain-computational models to understand how the brain works
Year(s) Of Engagement Activity 2017
URL http://www.cosyne.org/c/index.php?title=Cosyne_17
 
Description Publishing Culture in the Life Sciences 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Type Of Presentation Keynote/Invited Speaker
Geographic Reach Regional
Primary Audience Other academic audiences (collaborators, peers etc.)
Results and Impact 60-90 graduate students, postdocs, and faculty members attended this panel discussion on scientific publishing culture and how we can change it. I was one of five invited speakers. I gave a brief talk and contributed my ideas to the discussion.

I made a number of new contacts. The event was a big success and raised awareness among the younger generation that it's up to us to change the culture of science - including the way we conduct peer review -- for the better.
Year(s) Of Engagement Activity 2013
 
Description Recurring school visits 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Type Of Presentation Keynote/Invited Speaker
Geographic Reach Regional
Primary Audience Schools
Results and Impact Sixth form students from several schools visit the Unit to see what we do here and what a career in science involves.

Schools repeatedly ask to return and feedback is very positive.
Year(s) Of Engagement Activity 2012,2013
 
Description Royal Society Meeting 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Other audiences
Results and Impact Niko Kriegeskorte was invited to give a talk as part of the Royal Society meeting 'Interpreting BOLD: a dialogue between cognitive and cellular neuroscience'.
Year(s) Of Engagement Activity 2015
URL https://royalsociety.org/events/2016/01/interpreting-bold/
 
Description Stanford Neurosciences Institute 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Niko Kriegeskorte was invited to deliver a talk as the 2nd Annual Symposium of the Stanford Neurosciences Institute
Year(s) Of Engagement Activity 2015
 
Description University of Pennsylvania 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Postgraduate students
Results and Impact Niko Kriegskorte was invited to give a talk on Deep neural networks: a new framework for modelling brain information processing, as part of a one-day series.
Year(s) Of Engagement Activity 2016
 
Description University of Western Ontario 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Other audiences
Results and Impact Niko Kriegeskorte was invited to give a talk at Brain and Mind Institute, Department for Computer Science, University of Western Ontario.
Year(s) Of Engagement Activity 2016
 
Description Washington post-publication peer review debate 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Type Of Presentation Keynote/Invited Speaker
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact The final event of the annual meeting of the Professional & Scholarly Publishing (PSP) Division of Association of American Publishers at the Ritz Carlton Hotel in Washington, DC in February 2013 was an Oxford style debate entitled "Resolved: Post-publication peer review will achieve better scientific results than pre-publication peer review, while reducing costs for researchers and funders". I was invited as one of two scientists pitted against two representatives of the industry. I presented a talk arguing for open evaluation (OE) by postpublication peer review and debated these ideas before an audience of scholarly publishing professionals from the industry.

Several people from the industry requested copies of my slides, I was invited to participate in further events on these issues, and I summarised my impression of the state of the debate in this animation video: http://www.youtube.com/watch?v=9XnVadLilPo
Year(s) Of Engagement Activity 2013
URL http://www.youtube.com/watch?v=9XnVadLilPo