Towards Efficient and Certifiable Deep Learning

Research Context and Impact
Recent advances in neural networks have revolutionized computer vision, language, and natural sciences, leading to their widespread adoption. However, the increasing prominence and scale of these systems has brought forth two crucial questions that demand attention ahead of practical deployment. Firstly, how data-efficient are we in training these models? With the growing complexity and size of neural networks, the resource and time consumption required for training escalate significantly. Enhancing data efficiency is vital for enabling widespread adoption and practical use, reducing the dependence on large labeled datasets, and accelerating the training process. Secondly, how safe and reliable are these systems? This pertains to their certifiability, encompassing robustness in safety-critical decision-making contexts and correctness/reliability within the operational domain of deployed models. Ensuring the dependability and trustworthiness of neural networks becomes paramount as they find applications in critical domains.

Aims and Objectives
The overarching goal of the project is to introduce data-efficient techniques to train neural networks and scalable methods that allow us to certify them. The proposed problem can naturally be divided into two parts:
- Explore the application of self-supervised learning techniques to multimodal learning paradigms in an effort to improve the data efficiency of learning processes (following [1, 2]).
- Investigate formal verification techniques to obtain and certify networks are (i) robust within a safety context or (ii) reliable within their applicability domain if they are replacing experts. This is achievable through smoothing-like approaches (following [3, 4]) or by designing of highly efficient and parallelizable optimization algorithms to solve convex relations (following [5, 6]).

Novelty of Research Methodology
The methodology for the development of the project will be based on the existing works mentioned in the previous section, as well as some other related literature.

Alignment to EPSRC's Strategies and Research Areas
The project is well aligned with the "Verification and correctness" and "Artificial intelligence technologies" EPSRC research areas.

Companies or Collaborators Involved
The project will be developed in collaboration with FiveAI, as well as with Dr. Pawan M. Kumar at Google DeepMind.

References:
[1] Balestriero, Randall, et al. "A cookbook of self-supervised learning." arXiv preprint arXiv:2304.12210 (2023).
[2] Gui, Jie, et al. "A survey of self-supervised learning from multiple perspectives: Algorithms, theory, applications and future trends." arXiv preprint arXiv:2301.05712 (2023).
[3] Cohen, Jeremy, Elan Rosenfeld, and Zico Kolter. "Certified adversarial robustness via randomized smoothing." international conference on machine learning. PMLR, 2019.
[4] Yang, Greg, et al. "Randomized smoothing of all shapes and sizes." International Conference on Machine Learning. PMLR, 2020.
[5] Zhang, Huan, et al. "Efficient neural network robustness certification with general activation functions." Advances in neural information processing systems 31 (2018).
[6] Wang, Shiqi, et al. "Beta-crown: Efficient bound propagation with per-neuron split constraints for neural network robustness verification." Advances in Neural Information Processing Systems 34 (2021): 29909-29921.

AIMS's impact will be felt across domains of acute need within the UK. We expect AIMS to benefit: UK economic performance, through start-up creation; existing UK firms, both through research and addressing skills needs; UK health, by contributing to cancer research, and quality of life, through the delivery of autonomous vehicles; UK public understanding of and policy related to the transformational societal change engendered by autonomous systems.

Autonomous systems are acknowledged by essentially all stakeholders as important to the future UK economy. PwC claim that there is a £232 billion opportunity offered by AI to the UK economy by 2030 (10% of GDP). AIMS has an excellent track record of leadership in spinout creation, and will continue to foster the commercial projects of its students, through the provision of training in IP, licensing and entrepreneurship. With the help of Oxford Science Innovation (investment fund) and Oxford University Innovation (technology transfer office), student projects will be evaluated for commercial potential.

AIMS will also concretely contribute to UK economic competitiveness by meeting the UK's needs for experts in autonomous systems. To meet this need, AIMS will train cohorts with advanced skills that span the breadth of AI, machine learning, robotics, verification and sensor systems. The relevance of the training to the needs of industry will be ensured by the industrial partnerships at the heart of AIMS. These partnerships will also ensure that AIMS will produce research that directly targets UK industrial needs. Our partners span a wide range of UK sectors, including energy, transport, infrastructure, factory automation, finance, health, space and other extreme environments.

The autonomous systems that AIMS will enable also offer the prospect of epochal change in the UK's quality of life and health. As put by former Digital Secretary Matt Hancock, "whether it's improving travel, making banking easier or helping people live longer, AI is already revolutionising our economy and our society." AIMS will help to realise this potential through its delivery of trained experts and targeted research. In particular, two of the four Grand Challenge missions in the UK Industrial Strategy highlight the positive societal impact underpinned by autonomous systems. The "Artificial Intelligence and data" challenge has as its mission to "Use data, Artificial Intelligence and innovation to transform the prevention, early diagnosis and treatment of chronic diseases by 2030". To this mission, AIMS will contribute the outputs of its research pillar on cancer research. The "Future of mobility" challenge highlights the importance the autonomous vehicles will have in making transport "safer, cleaner and better connected." To this challenge, AIMS offers the world-leading research of its robotic systems research pillar.

AIMS will further promote the positive realisation of autonomous technologies through direct influence on policy. The world-leading academics amongst AIMS's supervisory pool are well-connected to policy formation e.g. Prof Osborne serving as a Commissioner on the Independent Commission on the Future of Work. Further, Dr Dan Mawson, Head of the Economy Unit; Economy and Strategic Analysis Team at BEIS will serve as an advisor to AIMS, ensuring bidirectional influence between policy objectives and AIMS research and training.

Broad understanding of autonomous systems is crucial in making a society robust to the transformations they will engender. AIMS will foster such understanding through its provision of opportunities for AIMS students to directly engage with the public. Given the broad societal importance of getting autonomous systems right, AIMS will deliver core training on the ethical, governance, economic and societal implications of autonomous systems.