Dopamine-induced hippocampal plasticity: A synaptic model of foraging in mice

Memory is important for normal function in society and for our identity as human beings. Many factors influence how well we remember. For example, intuitively, receiving a reward helps you to learn. Receiving a reward after you have learnt can also help you remember. But how is it that later events can help strengthen memories for what happened earlier? This is the conundrum we will address in this proposal; we will study the mechanisms for how reward influences memory of preceding experience.

We can investigate this in mice by studying how they learn the location of a reward and then use their memory to navigate back to that same place at a later time. To this end, we will use computer models to predict how mice respond to rewards, and how they find rewarded locations. Then, we will test these predictions in real mice. We will give the mice a food reward at specific locations in a simple maze and then study how they navigate to these rewarded locations. If our computer prediction is correct, the mice will use a part of the brain known as the hippocampus to mark the rewarded locations on a mental map of the maze and then use this map to navigate back to this place. To test this idea, we will use a new technique known as optogenetics to activate or silence part of the mouse brain using a laser. We will then test whether the hippocampus is necessary for the animals to find the rewards, and whether a reward signal needs to enter the hippocampus to mark the reward locations on the mental map. The results should give us new insights into how memory works and how animals navigate using their memory of rewarded locations.

The overall aim of this proposal is to investigate possible behavioural implications of the retroactive modulation of hippocampal plasticity that we recently discovered. The objective is to incorporate this novel learning rule in a computer model of the hippocampal network, in order to predict the behaviour of mice in a simple navigation task, and then test the predictions during equivalent learning tasks in behaving animals. Specifically, we will design a hippocampal network model consisting of 'place cells' coding for the location of the agent (artificial animal), projecting onto actor neurons determining the speed and direction of the agent. The new plasticity rule will be implemented at the synapses between place cells and actor neurons. Preliminary results suggest two fundamental advantages of the new learning rule over conventional reinforcement learning rules in such a network: 1) the agent learns from the absence of reward to enhance efficient exploration, and 2) the agent quickly 'unlearns' reward locations once reward is exhausted or absent. We will test the predictions in behaving mice using optogenetics to stimulate or silence dopaminergic reward input into the hippocampus during the task, using DAT-cre mice to express the optogenetic molecule selectively in dopaminergic neurons of the ventral tegmental area (VTA). We will also silence the plastic hippocampal CA3-CA1 synapses at different stages of task performance, using Grik4-cre mice cross-bred with Ai35 mice, to enable temporally-restricted optogenetic silencing of CA3 input. Finally, we would directly monitor synaptic weights at CA3-to-CA1 synaptic connections using extracellular multi-site recording of optogenetically-evoked field EPSPs during the task. The results of this study would indicate whether the computationally attractive properties of the novel learning rule discovered in a brain slice preparation operate in the intact mouse brain.

We have identified three areas of possible impact:
1. High-tech industry. The results of our studies are likely to be of significant interest for machine learning. The algorithms used by the brain will be important for biologically inspired reinforcement learning algorithms, and we will collaborate with industry partners at Google DeepMind to explore these algorithms for commercial use. Moreover, algorithms capable of seeking novelty and navigating towards rewarding locations would be useful in robotics software to enable robots and drones to search more efficiently through an environment. We will contact investigators in robotics to explore possible interactions.
2. Education. Better understanding of the mechanisms of reward-based learning could have implications for theories of education, and we will discuss with researchers in the neuroscience of education to explore potential practical applications in education.
3. Brain disorder. Better understanding of reward processing and how it influences memory could have consequences for treatment of normal age-related memory decline as well as memory disorder such as dementia. Moreover, understanding not only how the brain learns, but also unlearns behaviours that are no longer appropriate, may be of interest for several types of disorder, including OCD, drug addiction and PTSD. We will share our insights with clinicians to enable them to utilise new basic understanding for treating patients.

Who will benefit from the research?
In addition to academic beneficiaries, high-tech industry, the education sector, and parts of the pharmaceutical industry working to develop effective drug therapies for neurological diseases will benefit from the proposed work. In addition, school children, teachers and the general public will benefit from an increase in general knowledge about synaptic plasticity. Therefore, there is the potential for beneficial impact on both health and wealth of the UK.

How will they benefit from this research?
Enabling us to adapt to our environment is one of the most fundamental functions of the brain, a process that synaptic plasticity is believed to underpin. Since memory is so integral to all our lives, gaining knowledge about the mechanisms of synaptic plasticity is of interest not only to the academic community but also to the wider public.
High-tech industry: Our work could benefit developers of smart technologies, since the learning rule explored in this project might well inspire new machine learning algorithms, new learning rules for intelligent robotics and new implementations in neuromorphic engineering. We will make use of CC's contacts at Google DeepMind to encourage the use of our model of plasticity in novel machine learning techniques.
Educational impact: Basic understanding of learning and memory processes may lead to improvements in educational methods. We will contact educational neuroscientists to explore. Moreover, we will train a new generation of scientists, some of whom will go on to do research, but others will be equipped with a solid skillset for working in pharmaceutical, biotechnological, or engineering companies using machine learning or robotics, and also banks and insurance companies that use artificial neural network techniques.
Pharmaceutical industry: Research into numerous neurological diseases, including learning disabilities, autism and Alzheimer's disease, has found that changes in synaptic plasticity may contribute to disease symptoms. The social impact and economic costs of these diseases are enormous. Therefore our work might in the longer-term benefit society from better understanding of the mechanisms that underlie such diseases, and could benefit the economy both in terms of costs saved in care for patients suffering from these conditions, and in benefits from drugs developed and sold by UK-based companies. We acknowledge that these indirect benefits may take decades before they are realised.