Brain processes predicting future perception: cortical feedback and visual predictions

Lead Research Organisation: University of Glasgow
Department Name: School of Psychology

Abstract

Brain processes predicting future perception: cortical feedback and visual predictions We investigate cortical processing strategies using functional brain imaging. Previous evidence has shown that early visual areas receive feedback from higher visual areas. We have shown previously that this feedback activity helps to predict future perception. In three functional magnetic resonance imaging (fMRI) experiments we will test feedback and prediction with natural visual scenes (experiment 1 and 2) or with natural viewing conditions (experiment 3). To extract meaning the brain has to set incoming sensory information into the context of previous and future events. From moment to moment the visual system needs to decide what is present in its environment and how to interact with it. Perception 'postdicts' events if they occur unforeseen, that is perception is a reconstruction of the recent past, if upcoming events are however predictable the visual system incorporates this knowledge in the perceptual processing. Little is known about the brain processes underlying this predictive coding. One of today's biggest scientific challenges is the identification of important cortical processing principles. Tentative models that try to identify the essential principles are relatively rare. Feedback and prediction play an important role in some recent theories about brain processing. The memory-prediction framework of intelligence has been a relatively recent approach to isolate essential processing strategies. In this theory, the main tasks of cortical modules are to extract a plausible categorisation that helps to predict future events. Cortical areas integrate upcoming and neighbouring information and forward the current interpretation to higher areas and at the same time back-project the results to lower tier areas. Lower cortical areas compare the upcoming information with the prediction of higher cortical areas and generate a new pattern or error signal if the prediction was incorrect. The memory prediction framework assumes that the processing principles are quite general and apply to all cortical areas including primary sensory areas. Some evidence in favour of feedback and prediction has used artificial / easy to predict / stimuli. Here we will test in primary visual cortex (area V1) whether complex predictions can be formed with natural visual scenes. Two experiments will use complex visual stimuli taken from 3D-movies to initiate complex expectations. Subsequently the predictions will be tested using a newly developed fMRI-anticipation design. The third experiment will test visual predictions in the context of eye-movements.

Technical Summary

This proposal is about cortical feedback and visual prediction. In recent years we have shown that V1 receives feedback from specialised visual motion areas (Muckli et al. 2005, PLoS) and that this feedback helps to predict future perception (Schwiedrzik et al. 2007, VisRes). With this grant we aim to investigate cortical feedback and visual prediction in natural visual scenes. The central importance of feedback and prediction for cortical processing has been highlighted recently by the memory-prediction framework described in the book 'On Intelligence' (Hawkins & Blakeslee, 2004). We propose three new experiments that will test hypotheses derived from the memory-prediction framework. In the first experiment we will record rapid event related fMRI while natural 3D-images are presented in correct or scrambled temporal sequences. Presentation in the correct temporal sequence will induce complex apparent motion and should trigger predictive coding. Such predictions will be tested with a probe stimulus for which we expect to see BOLD signal attenuation if it correctly fits with the prediction. The second experiment will use multivariate analysis strategies to extract information from non-stimulated subregions of area V1. We will use 3D-pictures of natural scenes and place an occluder on top of the image. Following our predictions about cortical feedback and visual prediction and the memory-prediction framework we propose that this non-stimulated cortical region might carry significant informationby means of feedback or lateral interactions. The third experiment will use our previous long range apparent motion paradigm and combine it with eye-movements. In this experiment we will test whether pre-saccadic perceptual expectations will be fed-back to V1 to represent the new retinal positions of the post-saccadic stimulation scene. With these three experiments we will be able to test whether cortical feedback codes for perceptual expectations in natural scenes.

Publications

10 25 50
publication icon
Alink A (2010) Stimulus predictability reduces responses in primary visual cortex. in The Journal of neuroscience : the official journal of the Society for Neuroscience

publication icon
Maus GW (2010) Does Area V3A Predict Positions of Moving Objects? in Frontiers in psychology

publication icon
Muckli L (2013) Network interactions: non-geniculate input to V1. in Current opinion in neurobiology

publication icon
Muckli L (2015) Contextual Feedback to Superficial Layers of V1. in Current biology : CB

publication icon
Muckli L (2010) What are we missing here? Brain imaging evidence for higher cognitive functions in primary visual cortex V1 in International Journal of Imaging Systems and Technology

publication icon
Petro LS (2013) Decoding face categories in diagnostic subregions of primary visual cortex. in The European journal of neuroscience

 
Description Four years after the final report (2016):

• The BBSRC grant was the seed for a new and very successful research direction. We established three different ways to investigate cortical feedback and pioneered its role for predictive processing. These research strands received new Funding: 5 years ERC consolidator grant (2012-2017), and is now part of the EU flagship Human Brain Project, grant (2016-).

• Our results on Predictive Processing contributed to a paradigm shift in Neuroscience, Philosophy and Cognition. As an example several monographs of philosophers advocate the idea that Predictive processing has revolutionized the way we should think about the brain (Andy Clark, Hakob Hohwy). Our research plays a central role in these books.

• We were invited to many Symposia and workshops, some of which lead to special issues, others to new grant proposals and review articles about our novel findings and our follow up research (ERC grant).

• Symposium on predictive coding and schizophrenia created high impact, new research and new collaborations, with potential change in treatment.

• The field of Neuroscience is changing - textbooks will in years to come will have chapters abut predictive processing and eventually it might be the core narrative.
• Comparable to evolution-theory in biology, Predictive processing will be the core theory explaining the purpose of the brain as a prediction machine


From final report 2012:
1) We discovered predictive coding brain signals in V1. We found that predicted stimuli were easier to detect and evoked less brain activity in V1 (Alink et al. 2010). Our finding were quickly recognized in the field (de-Wit, et al. J Neurosci 2010). We replicated related results in Schizophrenic patients (Sanders et al. 2012) and extended our findings using related motion illusions (Maus et al., 2010) and eye-movements (Vetter et. al. 2012), and using TMS (Vetter et al. 2015).

2) We found evidence for contextual processing in V1. We decoded the content of occluded parts of visual scenes from the brain activity from non-stimulated parts of V1 (Smith and Muckli, 2010, PNAS). Vetter et al. demonstrated multisensory contextual processing in V1 of blindfolded subjects (Vetter 2014).

3) We started a special topic in Frontiers in Psychology on Predictive Coding
Exploitation Route The field of Neuroscience is changing - textbooks will in year to come start to have chapters abut predictive processing and eventually it might be the core narrative.

Comparable to evolution-theory in biology, Predictive processing will be the core theory explaining the purpose of the brain as a prediction machine.

Eventually the effect on artificial intelligence will be large - current models are weak on recurrent processing - we are developing new algorithm in the follow-up grants that will move the field forward and lead eventually to an information technology revolution - making artificial intelligence adaptable to new environments.

we are now collaborating with Sheffield robotics (through HBP) based on our original findings (Smith et al 2010; Vetter et al. 2014).
Sectors Digital/Communication/Information Technologies (including Software)

Education

Healthcare

URL http://muckli.psy.gla.ac.uk/
 
Description Healthcare: a group in Newcastle has used ideas from our cortical feedback studies for the development of treatment of tinnitus. Engineering: a group in Korea is training robots with predictive coding algorithms and tasks motivated by our research. Job creation: the BBSRC grant lead to a follow-up grant for 5 years which brings EU money to Scotland: ERC grant which financed 2 post doc position and 2 PhD students (started 2012). Moreover 3 more post-doc positions are created with EU flagship grant that was received in 2016.(Title: Context-sensitive multisensory object recognition a deep network model constrained by multi-level, multi-species data ), Granting body: The Human Brain Project (HBP) FET Flagship and European Union (call for Expression of Interest) Human Brain Project SP3 Amount: Work package coordinator (L. Muckli) total approx. € 2.000.000 (Brigitte Roeder, Nikolaus Kriegeskorte, Matthew Larkum, Christiaan Levelt), Principle Investigator (L. Muckli) approx. € 658.000 Philosophers have written monographs about predictive coding in which our research is cited (Andy Clark, Jakob Hohwy).
First Year Of Impact 2010
Sector Education,Healthcare
Impact Types Cultural

Economic

 
Description Scotish Parlament info about brain reading workshop report
Geographic Reach National 
Policy Influence Type Membership of a guideline committee
 
Description ERC consolidator grant 'Brain reading of contextual feedback and predictions '
Amount € 1,500,000 (EUR)
Funding ID ERC-StG-2012-311751 
Organisation European Research Council (ERC) 
Sector Public
Country Belgium
Start 11/2012 
End 10/2017
 
Description human brain project flagship SP3
Amount € 2,000,000 (EUR)
Funding ID [HBP] SGA1 
Organisation Human Brain Project 
Sector Academic/University
Country Switzerland
Start 03/2016 
End 03/2018
 
Title Multivariate Toolbox developed for the Smith and Muckli 2010, PNAS paper was reused in several papers 
Description Multivariate analysis toolbox developed in the lab in 2010 reused for several studies and shared between labs. Fraser Smith is now Lecturer in East Anglia 
Type Of Material Improvements to research infrastructure 
Provided To Others? No  
Impact MVPA Classification tool 
 
Description Pilot access for ultra-high field fMRI in Maastricht, for second contraol study at 7T following design of Smith and Muckli (2010, PNAS) BBSRC funded 
Organisation Maastricht University (UM)
Department Maastricht Brain Imaging Center
Country Netherlands 
Sector Academic/University 
PI Contribution Federico DeMartino and Rainer Goebel, and Elia Formisano, supported the pilot experiment. Scanning costs and in-kind sallery contribution helped the pioneering study to be published in 2015 (Muckli et al.). It marks a break-through in layer specific fMRI as described above this study lead to further grands and eventually to further invest ment and our participation in a big university investment in 7 Tesla fMRI (where i am now co-leading the efforts in Glasgow)
Collaborator Contribution My partners allowed the pioneer study - they finainced pilot data helped the recording the sequence optimisation the know how. I developed the paradigm and the analysis - together this was a break through
Impact the publication Smith and Muckli (2010, PNAS) is BBSRC funded the Muckli et al. (2015, Current Biology) is funded by the ERC grant. It is a replication of the Smith and Muckli (2010) at high resolution. Curr Biol. 2015 Oct 19;25(20):2690-5. doi: 10.1016/j.cub.2015.08.057. Epub 2015 Oct 1. The outcome of this collaboration based on Smith and Muckli (2010, PNAS) was a paper Muckli et al (2015) a grant of ERC granted in 2012 - 2017 a EU grant in 2016, with new collaborations (650.000 for Glasgow; 2 Mill for the consortium) and indirect an imaging centre for excellence ICE, Glasgow opened in 2017 (32 Mill.)
Start Year 2010
 
Description functional imaging at world leading Ultra-High-Field (7-Tesla ) Imaging center in Minneapolis, USA 
Organisation University of Minnesota
Department Centre for Magnetic Resonance Research
Country United States 
Sector Academic/University 
PI Contribution Professor Essa Yacoub, and Prof Kamil Urgubil CMRR, Centre Magnetic Resonance Research, University of Minnesota, Minneapolis, https://www.cmrr.umn.edu/ Dr Federico DeMartino, CMRR and Prof. Rainer Goebel University of Maastricht
Collaborator Contribution Based on the Smith and Muckli (2010) PNAS study we started a collaboration measuring fMRI at ultra high field. Federico De Martino worked at the CMRR and recorded the data. later we made a control study in Maastricht. The design of our study was idealy suited to investigate the effect of cortical feedback - now with ultra-high resolution. We were able to separate the different layers of cortex. Using a method we had pioneered for this design in Smith and Muckli (2010), we could show that cortical feedback reaches the superficial layers of early visual cortex, and we became the first team in the world showing that information analysis opens a new dimension for layer specific fMRI. These results were published last year Muckli et al. (2015) Current Biol. and mark a break-through in functional MRI. In 2017, Glasgow is becoming a center for ultra-high field MRI (7 Tesla) and our pioneering study and close collaboration with the High - field centre in Maastricht and Minneapolis allows a huge progress. The new Glasgow ICE center has gained collaborative development grants of 32 Million GBP. The BBSRC grand was an importand seed fund that triggered my involvement in 7T-field and contributes to the success of ultra-high-field scanning today.
Impact The out come of this collaboration based on Smith and Muckli (2010, PNAS) was a paper Muckli et al (2015) a grant of ERC granted in 2012 - 2017 a EU grant in 2016, with new collaborations (650.000 for Glasgow; 2 Mill for the consortium) and indirect an imaging centre for excellence ICE, Glasgow opened in 2017 (32 Mill.) the publication Smith and Muckli (2010, PNAS) is BBSRC funded the Muckli et al. (2015, Current Biology) is funded by the ERC grant. It is a replication of the Smith and Muckli (2010) at high resolution. Curr Biol. 2015 Oct 19;25(20):2690-5. doi: 10.1016/j.cub.2015.08.057. Epub 2015 Oct 1. Contextual Feedback to Superficial Layers of V1. Muckli L1, De Martino F2, Vizioli L3, Petro LS3, Smith FW4, Ugurbil K5, Goebel R6, Yacoub E5. Author information 1 Centre for Cognitive Neuroimaging (CCNi), Institute of Neuroscience and Psychology, College of Medical, Veterinary and Life Sciences, University of Glasgow, 58 Hillhead Street, G12 8QB Scotland, UK. Electronic address: lars.muckli@glasgow.ac.uk. 2 Faculty of Psychology and Neuroscience, Department of Cognitive Neurosciences, Maastricht University, Oxfordlaan 55, 6229 EV Maastricht, the Netherlands; Center for Magnetic Resonance Research (CMRR), Department of Radiology, University of Minnesota, 2021 Sixth Street SE, Minneapolis, MN 55455, USA. 3 Centre for Cognitive Neuroimaging (CCNi), Institute of Neuroscience and Psychology, College of Medical, Veterinary and Life Sciences, University of Glasgow, 58 Hillhead Street, G12 8QB Scotland, UK. 4 School of Psychology, University of East Anglia, Norwich Research Park, Norwich NR4 7TJ, UK. 5 Center for Magnetic Resonance Research (CMRR), Department of Radiology, University of Minnesota, 2021 Sixth Street SE, Minneapolis, MN 55455, USA. 6 Faculty of Psychology and Neuroscience, Department of Cognitive Neurosciences, Maastricht University, Oxfordlaan 55, 6229 EV Maastricht, the Netherlands; Department of Neuroimaging and Neuromodeling, Netherlands Institute for Neuroscience, Meibergdreef 47, 1105 BA Amsterdam, the Netherlands.
Start Year 2010
 
Description 06 Apr 2010 BBC-Scotland News Drive, radio interview 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Media (as a channel to the public)
Results and Impact Radio Interview Following the Smith and Muckli (2010, PNAS) publication and Press release
Year(s) Of Engagement Activity 2010
 
Description 2010 (PR- to Smith and Muckli 2010 PNAS Publication, World wide Media Coverage) 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Media (as a channel to the public)
Results and Impact Press releas was picked up by many national and international new magazines and news paper - BBC, coverage, Spiegel online coverage, Spektrum, US, and on online forum around the world
Year(s) Of Engagement Activity 2010
 
Description 2014 (Radio - Deutschlandfunk Wissenschaft im Brennpunkt: Das Hypothesengenie) 19.1.2014 / 03.07.2013 
Form Of Engagement Activity A broadcast e.g. TV/radio/film/podcast (other than news/press)
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Media (as a channel to the public)
Results and Impact German National Radio - 1 hour documentation about the research field with in-depth interview with (Lars Muckli, and Karl Friston)
Deutschlandfunk Wissenschaft im Brennpunkt: Das Hypothesengenie
Year(s) Of Engagement Activity 2014
URL http://www.deutschlandfunk.de/hirnforschung-das-hypothesengenie.740.de.html?dram%3Aarticle_id=274367
 
Description 2015 public talk on Aesthetics following the research out come started with BBSRC grant 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Public/other audiences
Results and Impact Interested artistic public - German Media Radio and print , Academics from philosophy aesthetics, and Modern Art
Year(s) Of Engagement Activity 2015
 
Description PR to Vetter et al (2014) Current Biology publication 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Media (as a channel to the public)
Results and Impact Media - Coverage arround the world. Auditorial stimulation that activates visual cortex created a large attention in many online forum and in national and international news outlet (Australia-Media, Spiegel-online Germany, BBC, Metro-UK)
Year(s) Of Engagement Activity 2014
 
Description Public talk in German Embassy 2016 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact Academic audience 100 from different fields
Year(s) Of Engagement Activity 2016
 
Description yearly about 20 talks in international institutes 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Postgraduate students
Results and Impact Talks in conferences - key notes, in universities, etc 20 every year on average international USA, EU, Brazil,
Year(s) Of Engagement Activity 2010,2011,2012,2013,2014,2015,2016