Theory of Deep Learning: Dynamics of Continual Learning

Animals, including humans, learn from a continuous stream of information under ever-changing conditions. Crucially, this learning happens with minimal interference of previously learned skills, even though processed by shared sensory streams and encoded in the same nervous system. This ability of continuous knowledge acquisition is a vital part of human and animal life and a hallmark of intelligent behaviour. Despite a large research body on human and animal learning and the recent identification of underlying neural substrates and synaptic mechanisms, a quantitative theory that explains and can make accurate predictions about how humans and animals continuously add new knowledge to their nervous systems without losing access to previously acquired knowledge is missing. In particular, continuous learning remains an unsolved problem for current brain inspired artificial neural networks. The goal of the proposed research project is to derive an analytic solution of the learning dynamics in artificial neural networks in a teacher-student setup for a range of continual learning paradigms in order to generate and test predictions for the dynamics of continuous learning in humans and animals. Such a quantitative theory of continual learning dynamics is essential to understand lifelong learning.