# Phenomenology of Deep Learning

Lead Research Organisation:
University of Oxford

Department Name: Oxford Physics

### Abstract

Recent Machine Learning (ML) breakthroughs in industry and the sciences rely on neural network architectures with multiple layers, a

branch of ML called Deep Learning (DL). Over the last decade, much of this progress was possible thanks to the impressive growth of

the size of datasets and neural networks. In contrast, the understanding of the foundations of DL has not followed this successful

trend. In fact, there is an enormous gap between the practical success of DL and our understanding of why DL works so well. It is

widely acknowledged that to expand the scope of ML applications and obtain reliable artificial intelligence systems, we must achieve

a fundamental understanding of DL.

A physics-based framework can provide a unique perspective to pressing questions in DL and contribute to filling the gap between

practical developments and theoretical foundations. We stress that, while ML applications are broadly used in physics, the flow of

ideas in the opposite direction, i.e., the use of concepts and techniques from theoretical physics to understand modern deep learning,

has only started to be explored.

In this MSCA, we will exploit and capitalize on the striking similarities between deep neural networks and effective field theory in

physics. The goal of "Phenomenology of Deep Learning" (PHENO-DL) is to contribute to the development of an effective theory of

deep learning. We will make use of physics model-building tools to attack foundational questions (e.g., how do deep neural networks

generalize?) and remarkable phenomena in deep learning (namely, double descent and adversarial examples). To this end, we will

adopt a setup inspired by established methods from theoretical physics, which have been recently applied to neural networks. In

particular, to explain the neural network's expressivity and capacity, we will use the interplay between the renormalisation group in

physics and the 'hierarchy of features' in the different layers in a deep neural network.

branch of ML called Deep Learning (DL). Over the last decade, much of this progress was possible thanks to the impressive growth of

the size of datasets and neural networks. In contrast, the understanding of the foundations of DL has not followed this successful

trend. In fact, there is an enormous gap between the practical success of DL and our understanding of why DL works so well. It is

widely acknowledged that to expand the scope of ML applications and obtain reliable artificial intelligence systems, we must achieve

a fundamental understanding of DL.

A physics-based framework can provide a unique perspective to pressing questions in DL and contribute to filling the gap between

practical developments and theoretical foundations. We stress that, while ML applications are broadly used in physics, the flow of

ideas in the opposite direction, i.e., the use of concepts and techniques from theoretical physics to understand modern deep learning,

has only started to be explored.

In this MSCA, we will exploit and capitalize on the striking similarities between deep neural networks and effective field theory in

physics. The goal of "Phenomenology of Deep Learning" (PHENO-DL) is to contribute to the development of an effective theory of

deep learning. We will make use of physics model-building tools to attack foundational questions (e.g., how do deep neural networks

generalize?) and remarkable phenomena in deep learning (namely, double descent and adversarial examples). To this end, we will

adopt a setup inspired by established methods from theoretical physics, which have been recently applied to neural networks. In

particular, to explain the neural network's expressivity and capacity, we will use the interplay between the renormalisation group in

physics and the 'hierarchy of features' in the different layers in a deep neural network.