Dynamically Adjusting Model Capacity During Training

In recent years, the energy consumption required for training and deploying large deep-learning models has continuously increased, raising significant concerns about energy supply and environmental impact. The cost of large neural networks is primarily determined by their size, which depends on architectural choices, such as the number of layers. Setting these parameters is difficult because they must be fixed before training, even though their impact on performance only becomes clear after training. Currently, finding the right model size involves trial and error, often requiring human intervention.

My research focuses on developing training procedures that automatically adjust model capacity during training, particularly in continual learning scenarios. In these settings, data arrives in a stream of batches and models must efficiently adapt to new data distributions without compromising performance on previous tasks or requiring complete retraining. By dynamically growing or shrinking the size of models based on the complexity of incoming data, we can optimally allocate computational resources across different learning phases. This approach significantly reduces energy consumption and training costs while eliminating the need for manual intervention in tuning model architectures

The proposed research has broad relevance across various fields, particularly when retraining or storing all data is impractical or prohibitively expensive. Examples include models that must adapt in real-time to changing environments without ongoing retraining, such as autonomous systems and financial markets, personalised healthcare solutions that handle sensitive data with privacy considerations, and large-scale data processing tasks involving substantial amounts of user data within secure ecosystems.

This research aligns with the goals of the EPSRC Mathematical Sciences area, addressing key challenges in scalable, adaptive machine learning systems.

The primary CDT impact will be training 75 PhD graduates as the next generation of leaders in statistics and statistical machine learning. These graduates will lead in industry, government, health care, and academic research. They will bridge the gap between academia and industry, resulting in significant knowledge transfer to both established and start-up companies. Because this cohort will also learn to mentor other researchers, the CDT will ultimately address a UK-wide skills gap. The students will also be crucial in keeping the UK at the forefront of methodological research in statistics and machine learning.
After graduating, students will act as multipliers, educating others in advanced methodology throughout their career. There are a range of further impacts:
- The CDT has a large number of high calibre external partners in government, health care, industry and science. These partnerships will catalyse immediate knowledge transfer, bringing cutting edge methodology to a large number of areas. Knowledge transfer will also be achieved through internships/placements of our students with users of statistics and machine learning.
- Our Women in Mathematics and Statistics summer programme is aimed at students who could go on to apply for a PhD. This programme will inspire the next generation of statisticians and also provide excellent leadership training for the CDT students.
- The students will develop new methodology and theory in the domains of statistics and statistical machine learning. It will be relevant research, addressing the key questions behind real world problems. The research will be published in the best possible statistics journals and machine learning conferences and will be made available online. To maximize reproducibility and replicability, source code and replication files will be made available as open source software or, when relevant to an industrial collaboration, held as a patent or software copyright.