Benchmarking for AI for Science at Exascale (BASE)

Advances in Artificial Intelligence (AI) and Machine Learning (ML) have enabled the scientific community to advance the frontiers of knowledge by learning from complex, large-scale experimental datasets. With the scientific community generating huge amounts of data from observatories to large-scale experimental facilities, AI for Science at Exascale is on the horizon.

However, in the absence of systematic approaches to evaluate AI models and AI algorithms at exascale, the AI for Science community, and, in fact, the general AI community, are facing a major barrier ahead.

This proposal aims to setup a working group with an overarching goal of identifying the scope and plans for developing AI benchmarks to enable the development of AI for Science at Exascale, in ExCALIBUR - Phase II.

Although AI Benchmarking is becoming a well-explored topic, a number of issues are still to be addressed, including, but not limited to:

a) There are no efforts aimed at AI benchmarking at exascale, particularly for science;

b) A range of scientific problems involving real-world large-scale scientific datasets, such as those from experimental facilities or observatories, are largely ignored in benchmarking; and

c) It is worth having benchmarks to serve as a catalogue of techniques offering template solutions to different types of scientific problems.


In this proposal, when scoping the development of an AI benchmark suite, we will aim to address these issues. In developing a vision, a scope and a plan for this significant challenge, the working group will not only engage with the community of scientists from a number of disciplines, and industry, but will also engineer a scalable and functional AI benchmark, so as to learn and embed the practical aspects of developing an AI benchmark into the vision, scope, and plan. The exemplary benchmark will focus on removing noise from images, which is a common issue across multiple disciplines including, life sciences, material sciences and astronomy. The specific problems from each of these disciplines are, removing noise from cryogenic electron microscopic (cryo-em) datasets, denoising X-Ray tomographic images, and minimising the noise from weak lensing images, respectively.

The core of our delivery is a scope and plan for the development of a suite of AI benchmarks in Phase II of the ExCALIBUR programme. When delivered, we see potential impacts across a number of different communities. We discuss these below in turn.

1. Scientists

There are two ways in which the outcomes of this proposal would impact the scientific community: one is enabling easier adoption of AI/ML methods. Secondly, enabling development of better, and scalable AI algorithms for a number of scientific problems.

With a catalogue of AI/ML methods available as template techniques for a range of representative problems from different disciplines, we expect that domain scientists can make use of these benchmarks to build better models or to solve similar problems. The benchmarks are aimed at exascale cases, and such, is likely to include models that can cope with very large datasets and to scale better with computational resources, so as to minimise the training time. This would speed up the way that AI or machine learning is adopted to different problems across scientific domains.


2. Hardware Manufacturers

With benchmarks being able to offer insights into the performance of AI software, these can either be attributed to software or hardware issues. Among these, hardware issues can be identified when profiled across multiple architectures. This will enable hardware manufacturers to develop improved hardware systems. It will also enable hardware-software co-design, particularly within the AI space with exascale in consideration.


3. Software / Service Providers

With benchmarks being available to solve certain types of problems in science, software companies or service providers, such as those who provide scientific software will be able to engineer better software than the ones available in benchmark. Furthermore, these benchmarks can also enable improved software for exascale systems, such as software profilers.


4. Large-scale System Users

Communities who procure large-scale systems, such as national laboratories or data centres, can use these benchmarks to decide the relative and absolute performance of different systems. It can also enable procurement of hardware resources for a specific user community, using relevant benchmarks within the suite.


5. The AI Community

The benchmarks that we are aiming to develop are based on real cases (opposed to being synthetic), and can be used across different aspects of AI, such as explainability, transfer learning, and other areas of AI research.