Credit Assignment and Reinforcement Learning in Spiking Neural Networks on Neuromorphic Hardware

Artificial neural networks have made great advances over the last decade, through the development of reliable training methods and accelerated hardware. This has allowed neural networks to perform a variety of Artificial Intelligence (AI) tasks, such as image recognition, language processing and defeating human experts at games such as chess and go. This has, in turn led, to widespread use of Artificial Neural Networks (ANNs) in AI applications across industry and academia. However, while these systems can be tuned to perform well in specific tasks, their abilities as general intelligent systems remain severely lacking when compared to neural networks in the human brain. Brains are able to process information from a variety of stimuli, continuously integrate new information without forgetting that learned previously, and consume a fraction of the energy of computational systems.
These impressive qualities of the human brain have led to the research field of neuromorphic engineering: exploring computational systems and algorithms taking inspiration from the brain to improve performance in neural networks. A key concept here is mimicking the brain's use of spiking neurons, where communication between neurons is via small pulses of electricity known as spikes. However, a key challenge when using neuromorphic hardware and spiking neural networks is training. The algorithms typically used to train ANNs -- such as back-propagation of errors and their minimization via gradient descent -- are much harder to apply due to the spiking communication.
This research will therefore explore training techniques for brain-like spiking neural networks. It will take advantage of the neuromorphic platform SpiNNaker, a computer optimized for real-time simulation of large-scale networks of brain-like neurons. It will investigate methods for online learning in neural networks to generate intelligent agents which learn through interaction with their environment. Specifically, it will explore reinforcement learning and the credit assignment problem, the process through which a learning system will attribute outcomes, which are often delayed past the initial event. This will be done using information local to neurons, such as eligibility traces and local activity, and through exploration of learning signals capturing system error and potential rewards.
This work stands to improve greatly the capability of neuromorphic architectures to train deep learning algorithms while taking advantage of their low power use and real-time computation abilities. Furthermore, a biologically plausible model for credit assignment could improve our understanding of long-term synaptic plasticity in the brain, which underlies physiological mechanisms such as learning and memory.
Potential benefits that may emerge may be a better understanding of how biological neurons learn and improved Machine Learning systems capable of assimilating new data with less time and effort than 'conventional' ANNs.