Distributed neural processing of self-generated visual input in a vertebrate brain

Lead Research Organisation: University of Sussex
Department Name: Sch of Engineering and Informatics


Most of what we do on a day-to-day basis involves the ongoing and fluid coordination between our senses and our actions. For example, making a cup of tea involves processing a constant stream of visual and tactile (touch) information to continuously correct how we move our muscles in order to avoid spilling milk or breaking a mug. Our brains coordinate this flow of sensory to motor information effortlessly, yet it is an ability that modern engineering still cannot rival; think of the clumsy robots in this year's DARPA Robotics Challenge. The ability to coordinate sensory input and motor actions is also often impaired in diseases like Parkinson's and dyspraxia. Understanding this ability is therefore not only a central goal of modern neuroscience but also one that promises to deliver advances for engineering and facilitate the treatment of disease.

Over the last century, neuroscience research has revealed that areas across the brain are required to process sensory and motor information during active behaviours. These include sensory and motor systems but also areas such as the cerebellum and the basal ganglia. While a partial understanding of the underlying processes in specific circuits has been achieved, a full understanding would ideally require recordings of the neural activity from across the brain in a behaving animal. This type of experiment has been impossible in the past for two reasons: 1. Neural recording techniques require a great degree of stability between recording devices and neural tissue; thus most experiments involve heavily restrained, or anesthetized, animals, which prevents meaningful brain/environment interactions. 2. Typical brain recordings have been limited to either small numbers neurons at cellular resolutions or indirect recordings from large areas of brain tissue at low spatial and/or temporal resolution. To address these challenges will combine advanced techniques in experimental and computational neuroscience. First, a virtual reality for a swimming larval zebrafish; this will allow us to record from a non-moving brain but allow fictive behaviour. Second, light-sheet microscopy, a technique that can simultaneously image 10000's of neurons from across the zebrafish brain. Third, distributed computing techniques, which will enable us to analyse the enormous data sets (upto a terabyte per trial) acquired from these experiments.

We will use these tools to address three fundamental questions about brain function in behaving animals. First, when animals actively engage the world the brain receives two types of sensory input: Sensory input caused by changes in the external world, e.g. the optic flow experienced by a fish as water sweeps past its retina, and sensory input that is a consequence of their own actions, e.g. the optic flow experienced by the fish that results from its own swimming. These two types of inputs convey different types of information but arrive together on the retina. Thus a central question we will ask is what are the brain-wide circuits that allow the fish to distinguish between them. Second, animals readily adapt their behaviour when the sensory inputs caused by their own actions do not meet their expectations. For example, fish modulate the strength of swimming when changes in water viscosity cause a mismatch between the actual and expected consequences of their swimming, i.e., when their swimming does not propel them as far as they expect. We will ask what the distributed neural circuits are that allow fish to detect these mismatch errors. We will combine our results to produce a biologically plausible model of closed-loop control in an actively swimming fish that reproduces experimental observations and could be used to inspire robotic control systems.

This project will develop new techniques to record and analyse large neural datasets and provide unique insight into the distributed and dynamic nature of brain function necessary for successful active behaviour.

Technical Summary

During movement sensory input and motor output are bound in a closed-loop: motor actions shape sensory input and sensory inputs inform future motor commands. Studies over the last century have revealed that the neural circuits processing this closed-loop flow of information are widely distributed, i.e., including sensory and motor systems and intervening areas such as the cerebellum and basal ganglia. In this project, to characterise these brain-wide neural circuits, we will combine: 1. A swimming virtual reality (VR) environment for larval zebrafish. 2. Light-sheet microscopy to image neurons across the brain. 3. Distributed computing techniques to analyse the large data sets produced by this setup with established (PCA, ICA) and leading edge analysis techniques (e.g. Granger causality).

We will address three core questions that emerge from a consideration of closed-loop processing. Q1. What are the distributed neural circuits that allow fish to distinguish between the visual consequences of voluntary action (reafferent input) and visual input originating in the external environment (exafferent input). Q2. What are the distributed neural circuits that allow fish to detect when visual consequences of motor action (reafferent input) do not meet expectations?, indicating a failure to control their surrounding environment. Q3. Lastly how do fish distinguish between mismatch errors caused by external environment (exafferent input; as in Q1) and those caused by a failure to control their surrounding environment (as in Q2). We will construct, and test, a computational model of the observed circuit dynamics and computations.

This investigation will develop new algorithms to identify circuits in large neural datasets, elucidate the biological basis of closed-loop processing to inspire robotic control systems, and identify aspects of brain function that are contingent on closed-loop dynamics and thus may have been overlooked in traditional open-loop approaches.

Planned Impact

In addition to the academic beneficiaries of this work it will impact on a) the public health sector, b) the commercial private sector, c) public communication and d) the wider academic community, as described below.

Public health sector
Zebrafish are becoming an important model system for the study of brain dysfunction and, at the same time, are becoming a leading systems for rapidly screening pharmaceutical compounds which target specific behavioural deficits. Our setup has the potential to enhance screening techniques by identifying higher order (brain-wide) functional (during behaviour) effects of drugs. It will promote a more holistic understanding of sensorimotor coordination, which will allow a more refined characterization of the deficits caused by diseases like dyspraxia, Parkinson's and Huntington's and potentially impact on the development of high end prosthetics.

The private commercial sector
Over the last decade there has been a significant rise in the commercialisation of autonomous robots that work in dynamic real-world environments. Understanding the neural circuits that allow animals survive and adapt during active behaviours is potentially a richer source of ideas than more traditional open-loop approaches for this application. Our research group, the Centre for Computational Neuroscience and Robotics (CCNR), has strong ties to the robotics community and industry and is well placed to exploit this avenue.

Big data approaches are increasingly seen as a central to scientific projects but also have significant socio-economic relevance. The development of a connectivity analysis toolbox for 'big data' is relevant for many domains. For example, these techniques can be used to understand potential relationships between measures of online (social media) discussion and offline events. In this context we have had expressions of interest from CASM Consulting LLP.

Public communication
Both LL and CLB are actively involved in outreach activities. We will present
the core science, and more generally promote STEM subjects, at school 6th forms, contribute to public meetings, such as Cafe Scientifique and the Brighton Science Festival.

Over the last few decades there has been an increasing tendency to take a narrowly neurocentric view of behaviour. For example, in Francis Crick's book 'An astonishing hypothesis' he suggests that "your joys and your sorrows, your memories and your ambitions, your sense of personal identity and free will, are in fact no more than the behaviour of a vast assembly of nerve cells and their associated Molecules". This idea contrasts with a movement in the cognitive sciences which has long argued that understanding behaviours requires an explicit appreciation of brain/body/environment interactions (e.g. see 'Out of our heads: Why you are not just your brain' by Alva Noe). The goals of this project resonates with, and extends, this latter view by suggesting that not only behaviour, but brain dynamics, are contingent on an appreciation of brain-environment interactions. We feel these broad ideas will be of significant interest to the public, and have potential implications for public health, e.g., promoting exercise and rich environments for mental health, and we will disseminate them through public talks, organised discussions as well as popular science articles.

The wider academic community
The advent of VR approaches in neuroscience provides a unique opportunity to revisit our understanding of the biological basis of sensorimotor control. The achieve impact in this area we intend to organise a broad workshop between experimental neuroscientists working with VR approaches in mice and fish and sensorimotor theorists from behavioural neuroscience and robotics. The goal of the workshop will be to allow those working with VR to contextualise their results in terms modern sensorimotor theory and to inspire new algorithm and mechanisms for theorists and engineers.


10 25 50
publication icon
Aguilera M (2022) How particular is the physics of the free energy principle? in Physics of life reviews

publication icon
Baltieri M (2019) PID Control as a Process of Active Inference with Linear Generative Models. in Entropy (Basel, Switzerland)

publication icon
Baltieri M (2019) Generative models as parsimonious descriptions of sensorimotor loops in Behavioral and Brain Sciences

publication icon
D Fernandes AS (2018) Visual Classical Conditioning in Wood Ants. in Journal of visualized experiments : JoVE

publication icon
Millidge B (2021) Whence the Expected Free Energy? in Neural computation

publication icon
Seth AK (2020) Curious Inferences: Reply to Sun and Firestone on the Dark Room Problem. in Trends in cognitive sciences

publication icon
Tschantz A (2020) Learning action-oriented models through active inference. in PLoS computational biology

Description In many animals vision is central to the control of locomotory behaviours. In particular, innate motor responses to optic flow allow animals to quickly respond to unplanned perturbation, such as gusts of air in flying animals or changes of current in fish. The optomotor response (OMR) in larval zebrafish has become a key model system for investigating the neural basis of this response. During this response, the movement of the image of the riverbed across the retina, as the fish is washed downstream, causes the fish to swim forward in a series of swim bouts (periods of swimming followed by cessation). It is typically assumed that this response serves to stabilize the fish's overall position and prevent the fish drifting downstream. But many aspects of the behavior itself, its function and underlying control algorithms are still not well understood. Our results fill this gap and provide a detailed understanding of the behavior and its underlying algorithms. Some findings are unexpected and challenge common assumptions. For example, the OMR, far from successfully holding position, is only partial and varies systematically with feedback gain; the termination of swim bouts may not be explicitly controlled; and the reversed optic flow experienced during swim bouts is apparently not exploited for control despite carrying most of the information about self-movement. These results beg questions about the role of the OMR in fish behaviour and shed new light on its underlying dynamics which will be crucial for future work aimed at identifying the neural basis of this behavior.
Exploitation Route These results are of particular relevance to neuroscientists using zebrafish optomotor response as a model system and will have a significant impact on future neurobiological research. More generally, the study should interest experimental and computational neuroscientists investigating the neural circuits that generate other sensorimotor behaviors, behavioral scientists and ethologists concerned with modeling the mechanisms underlying animal behavior, and engineers interested in biomimetic robotics.
Sectors Other

Description Data-driven Inference of Models from Embodied Neural Systems In Vertebrate Experiments
Amount £149,053 (GBP)
Funding ID 892715 
Organisation Marie Sklodowska-Curie Actions 
Sector Charity/Non Profit
Country Global
Start 03/2020 
End 04/2022
Title µSPIM Toolset 
Description SPIM Toolset is a an all-in-one control solution for a Selective Plane Illumination Microscope (SPIM), build around the open-source Micro-Manager platform that has been adopted in a wide range of imaging solutions. The main aim of the project is to providea open-source flexible and hardware-agnostic software solution that can be adapted to the specific lightsheet imaging applications while retaining a gentle learning curve. µSPIM Toolset: A software platform for selective plane illumination microscopyD Saska, P Pichler, C Qian, CL Buckley, L Lagnado Journal of Neuroscience Methods 347, 108952 
Type Of Material Data handling & control 
Year Produced 2021 
Provided To Others? Yes  
Impact This has already been adopted by at least 2 experimental labs we know of. 
URL https://uspim.org/#/
Description Travel grant with the National Institute for Physiological Sciences, Japan 
Organisation National Institute for Physiological Sciences
Country Japan 
Sector Academic/University 
PI Contribution We have traveled over to Osaka to collaborate and teach a tutorial on the Free Energy Principle
Collaborator Contribution They collaborate with us and hosted the tutorial.
Impact International Tutorial
Start Year 2019
Title Laser Sheet Microcsopy Data Processing Software 
Description We are about to release open source software for the control of a laser sheet microscope and a data processing pipeline. 
Type Of Technology Software 
Year Produced 2019 
Impact This is open software, open hardware framework which will enable other labs to flexibly setup a laser-sheet microscope. 
URL https://www.sciencedirect.com/science/article/pii/S0165027020303757
Description Invited talk at the Timescale of Life Conference Edinburgh 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Postgraduate students
Results and Impact Timescales of Life and Mind is a post-graduate interdisciplinary conference focused on the use of different timescales as a methodological approach to studying the applications of Predictive Processing and the Free-Energy Principle.
Year(s) Of Engagement Activity 2019
URL https://blogs.ed.ac.uk/tlmconference/
Description Organised a Workshop on the Neurobiology of Control 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact This was a workshop held at the conference Cosyne 20 and invited academics across model systems, robotics and cognitive science. It involved 10 invited speakers.

Workshop description
"The ultimate goal of neuroscience is to understand the operation of neural circuits in behaving animals, thus, fundamentally the brain should be understood as a control system facilitating the coordination of the body and environment in the pursuit of adaptive behavior. The flow of information during sensorimotor processing spans the entire brain and the activity of the circuits involved are strongly modulated by the presence of active behaviors. Thus while a partial understanding in specific circuits and restrained preparations has been achieved, a full understanding ideally requires recordings of the neural activity from across the brain in a behaving animal. While this has historically been technically prohibitive, recent technological advancements, in particular, the development of virtual reality and imaging techniques, provide an opportunity to record from large populations of neurons during active behavior. Thus there is a timely need to revisit our understanding of the neurobiological basis of sensorimotor control in light of these new data.

We argue that understanding the brain as a control system necessitates a change in both technical language and the central questions asked about neural function. Specifically, it promotes a move away from the language of information coding to stochastic dynamical systems and promotes questions not primarily about categorization and decision-making but about behavioral regulation. This workshop will bring together communities working on neurobiological control across the full gamete of model organisms (worm, fly, fish, mouse, monkey, human) together with representatives of communities were the ideas of control are much more matured, i.e., sensorimotor cognitive scientists and roboticists. The goal is to develop a shared technical perspective and identify common questions, challenges, and theories across systems. This workshop will appeal to experimentalists who want to contextualize their results in terms of modern sensorimotor theory and inspire new algorithms and mechanisms for theorists and engineers."
Year(s) Of Engagement Activity 2020
URL http://www.cosyne.org/c/index.php?title=Workshops2020_1_10
Description Taught a Tutorial on the Free Energy Principle 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact We were invited to teach a 2 day tutorial on the Free Energy Principle in Osaka Japan at the National Institute for Physiology (NIPs). This involved a detailed intorduction to the formalism and hands-on coding examples.
Year(s) Of Engagement Activity 2019
URL http://www.nips.ac.jp/~myoshi/nins_tutorial2019/indexe.html