ExCALIBUR HES Benchmarking for Performance Portable Applications
Lead Research Organisation:
UNIVERSITY COLLEGE LONDON
Department Name: Centre for Advanced Research Computing
Abstract
Application benchmarking is a crucial activity in the UK's path to Exascale. It ensures future Exascale
systems are understood by the community so that UK Science Applications, particularly those
developed as part of ExCALIBUR, can take advantage of the scientific opportunities at Exascale. Linear
solvers are a ubiquitous pattern in many ExCALIBUR codes, and this project will develop a suitable
benchmark applicable to many of UK codes. Application benchmarks are part of the enabling software
stack that allows the performance of hardware to be assessed for scientific workloads .
It is vital to rigorously measure the performance of benchmarks in a systematic way to enhance the
transparency and enable reproducibility. International conferences in this space are now mandating
reproducibility information. This project provides the tooling to make collecting and analysing
performance data straightforward. This will allow application owners to develop the skills and
experience to measure the performance of their code across the gamut of the UK's supercomputer
facilities, and improve their code based on benchmarking as they target Exascale.
The increasing diversity of computer architectures now requires applications to strive to be
Performance Portable (PP); that is, achieve a high level of efficiency on a range of different processors
with minimal changes to the source code. Measuring PP using analysis such as those in our recent
studies needs to become as common in HPC as scaling tests or roofline analyses. This is especially
true for ExCALIBUR codes, where the likely Exascale architectures may not be known for several years.
This project will ensure this vital PP analysis can become ubiquitous in HPC software development.
The ReFrame tool has been developed by the CSCS and ETH Zurich to aid in the automated collection
of performance data. ReFrame is an already established approach within ExCALIBUR. This project will
augment the rudimentary performance analysis available in ReFrame to include our own PP
Methodology developed with collaborators at Intel . By incorporating this alongside the Benchmarking
Initiative, we will empower the supercomputing community in the UK to measure the performance
portability of their code across the UK's HPC infrastructure. This aligns with the H&ES mission to ready
UK codes and software for Exascale.
systems are understood by the community so that UK Science Applications, particularly those
developed as part of ExCALIBUR, can take advantage of the scientific opportunities at Exascale. Linear
solvers are a ubiquitous pattern in many ExCALIBUR codes, and this project will develop a suitable
benchmark applicable to many of UK codes. Application benchmarks are part of the enabling software
stack that allows the performance of hardware to be assessed for scientific workloads .
It is vital to rigorously measure the performance of benchmarks in a systematic way to enhance the
transparency and enable reproducibility. International conferences in this space are now mandating
reproducibility information. This project provides the tooling to make collecting and analysing
performance data straightforward. This will allow application owners to develop the skills and
experience to measure the performance of their code across the gamut of the UK's supercomputer
facilities, and improve their code based on benchmarking as they target Exascale.
The increasing diversity of computer architectures now requires applications to strive to be
Performance Portable (PP); that is, achieve a high level of efficiency on a range of different processors
with minimal changes to the source code. Measuring PP using analysis such as those in our recent
studies needs to become as common in HPC as scaling tests or roofline analyses. This is especially
true for ExCALIBUR codes, where the likely Exascale architectures may not be known for several years.
This project will ensure this vital PP analysis can become ubiquitous in HPC software development.
The ReFrame tool has been developed by the CSCS and ETH Zurich to aid in the automated collection
of performance data. ReFrame is an already established approach within ExCALIBUR. This project will
augment the rudimentary performance analysis available in ReFrame to include our own PP
Methodology developed with collaborators at Intel . By incorporating this alongside the Benchmarking
Initiative, we will empower the supercomputing community in the UK to measure the performance
portability of their code across the UK's HPC infrastructure. This aligns with the H&ES mission to ready
UK codes and software for Exascale.
| Description | The diversity in processor technology used by High Performance Computing (HPC) facilities is growing, and so applications must be written in such a way that they can attain high levels of performance across a range of different CPUs, GPUs, and other accelerators. Measuring application performance across this wide range of platforms becomes crucial, but there are significant challenges to do this rigorously, in a time efficient way, whilst assuring results are scientifically meaningful, reproducible, and actionable. We have developed a methodology for measuring and analysing the performance portability of a parallel application, and an open-source software framework which combines and extends adopted technologies to provide a usable benchmarking tool. We have demonstrated the flexibility and effectiveness of the methodology and benchmarking framework by showcasing a variety of benchmarking case studies which utilise a stable of supercomputing resources at a national scale. |
| Exploitation Route | The outcomes of this project are straightforward to put to use by users of high performance computing (HPC), which spans a wide range of industries. The methodology is published, and the software is available open-source with accompanying documentation and tutorials. The tools we have developed can be used to understand and develop the performance and portability of computational applications, and to maintain and monitor HPC systems. |
| Sectors | Other |
| URL | https://github.com/ukri-excalibur/excalibur-tests |
| Title | Benchmarking framework |
| Description | This framework allows HPC centers to deploy benchmarks in a reproducible and portable way. |
| Type Of Material | Improvements to research infrastructure |
| Year Produced | 2023 |
| Provided To Others? | Yes |
| Impact | The framework is being used in DiRAC sites DiAL2 and DiAL3 to detect changes in performance after system upgrades. It is being considered for deployment in the UK Exascale project. |
| URL | https://github.com/ukri-excalibur/excalibur-tests |
| Description | Development of CSD3 environments |
| Organisation | University of Cambridge |
| Country | United Kingdom |
| Sector | Academic/University |
| PI Contribution | Installation of tools on CSD3, porting of benchmarks, training and support |
| Collaborator Contribution | Updated environments for CSD3. FeniCS benchmarks |
| Impact | Support for CSD3 |
| Start Year | 2023 |
| Description | Development of NESO/NEPTUNE benchmarks |
| Organisation | Culham Centre for Fusion Energy |
| Country | United Kingdom |
| Sector | Academic/University |
| PI Contribution | Issues and pull requests to exCALIBUR-NEPTUNE/NESO repository. Development of NESO benchmarks in excalibur-tests repository. |
| Collaborator Contribution | Review of pull requests and support with NESO build system. Input data for benchmarks |
| Impact | Bug fixes to NESO-spack, NESO-particles |
| Start Year | 2023 |
| Description | Porting DiRAC benchmarks |
| Organisation | Science and Technologies Facilities Council (STFC) |
| Department | Distributed Research Utilising Advanced Computing |
| Country | United Kingdom |
| Sector | Academic/University |
| PI Contribution | Training and support |
| Collaborator Contribution | Porting of DiRAC procurement benchmarks sphng, ramses, trove, swift |
| Impact | Regular use of benchmarking framework for systems monitoring by DiRAC |
| Start Year | 2023 |
| Title | BabelStream Spack package |
| Description | BabelStream is a benchmarking software package used to measure main memory bandwidth. In this project, we contributed a Spack package to the main Spack package repository. |
| Type Of Technology | Software |
| Year Produced | 2023 |
| Open Source License? | Yes |
| Impact | By enabling the benchmark in Spack, a commonly used package manager for a number of HPC supercomputing centres, we have made it easier to build and install the BabelStream benchmark. |
| URL | https://github.com/spack/spack/pull/36164 |
| Title | excalibur-tests |
| Description | excalibur-tests is a framework for automated deployment of benchmarks across heterogeneous HPC systems. It provides automation for building and running benchmarks in a reproducible and documented way. The first release is planned for April 2024. |
| Type Of Technology | Software |
| Year Produced | 2024 |
| Open Source License? | Yes |
| Impact | The software has attracted interest from leading scientific computing institutes in the UK, including: Edinburgh Centre for Parallel Computing, University of Cambridge and DiRAC. It has also led to contributions to the underlying open-source projects spack and reframe. |
| URL | https://github.com/ukri-excalibur/excalibur-tests |
| Title | reframe-hpc/reframe: ReFrame 4.1.0 |
| Description | Release Notes Changes since 4.0.5 New features Introduce a
--dry-run option (#2746) Allow
num_tasks to be
None (#2778) Allow custom JSON formatting for log records in the
httpjson handler (#2752) Print the first lines of stderr and stdout in case of sanity failures (#2762) Relax the regex matching for test names and tags (#2736) Enhancements Properly combine
append-type command-line options defined in execution modes (#2788) Bug fixes Fix listing of non-fixture-based dependencies (#2798) Fix use of
sched_options by the backend schedulers (#2796) Other Update GitHub Actions to use setup-python@v4 (#2775) |
| Type Of Technology | Software |
| Year Produced | 2023 |
| Open Source License? | Yes |
| Impact | The project has made contributions to several releases of ReFrame during 2023. |
| URL | https://zenodo.org/record/7686066 |
| Title | spack/multiple releases |
| Description | Spack is a multi-platform package manager that builds and installs multiple versions and configurations of software. It works on Linux, macOS, and many supercomputers. Spack is non-destructive: installing a new version of a package does not break existing installations, so many configurations of the same package can coexist. Spack offers a simple "spec" syntax that allows users to specify versions and configuration options. Package files are written in pure Python, and specs allow package authors to write a single script for many different builds of the same package. With Spack, you can build your software all the ways you want to. |
| Type Of Technology | Software |
| Year Produced | 2023 |
| Open Source License? | Yes |
| Impact | Over the lifetime of this project, many (more than 50) pull requests to spack have been merged. These have been included in relases starting from 0.18.0 to the most recent 0.21.2 |
| URL | https://github.com/spack/spack |
| Description | Cambridge RSE seminar |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Professional Practitioners |
| Results and Impact | This talk in the Cambridge RSE seminar series presented the software tools being developed by the project |
| Year(s) Of Engagement Activity | 2023 |
| URL | https://talks.cam.ac.uk/show/index/69831 |
| Description | HPCDays23 talk |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Presentation by Dr Tom Deakin on this project, including the motivation for it and how to get involved, at the Durham HPC Days Spring 2023 workshop, to an audience of 50+ people, including international visitors from large supercomputing centres internationally. This led to a number of following discussions with ExCALIBUR project partners on how they can contribute to the benchmark suite. |
| Year(s) Of Engagement Activity | 2023 |
| URL | https://tobiasweinzierl.webspace.durham.ac.uk/research/workshops/durham-hpc-days-spring-2023/ |
| Description | Invited talk at ANL/NERSC/ORNL Performance Portability Training Series |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Dr Tom Deakin was invited to speak on performance portability at the ANL/NERSC/ORNL Performance Portability Training Series, where he shared results and outcomes from this ExCALIBUR project. |
| Year(s) Of Engagement Activity | 2024 |
| URL | https://www.olcf.ornl.gov/calendar/performance-portability-for-next-generation-heterogeneous-systems... |
| Description | Invited talk at CECAM Workshop |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Dr Tom Deakin was invited to speak at the Perspectives and challenges of future HPC installations for atomistic and molecular simulations workshop on the topic of benchmarking and performance portability, where he presented results and outcomes from this ExCALIBUR project. As the meeting was recent (February 2024) there are no known impacts at this time. |
| Year(s) Of Engagement Activity | 2024 |
| URL | https://www.cecam.org/workshop-details/perspectives-and-challenges-of-future-hpc-installations-for-a... |
| Description | Invited talk at On the trail to exascale event at Hartree Centre |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Dr Tom Deakin presented on the outcomes of this project at the On the trail to exascale collaborative event between the Hartree Centre and the USA Exascale Computing Project. As a result of this, there was a blog post discussing some of the ideas from the talk published by the Hartree Centre. |
| Year(s) Of Engagement Activity | 2023 |
| URL | https://www.hartree.stfc.ac.uk/news/2023/06/09/on-the-trail-to-exascale/ |
| Description | Keynote talk at IXPUG @ ISC 2023 |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Dr Tom Deakin gave a keynote talk on performance portability in the IXPUG workshop at International Supercomputing 2023 to around 40+ people, sharing details and results from this ExCALIBUR Benchmarking project. Following on from this, Tom has discussions with Industry about further benchmarks that could be explored. |
| Year(s) Of Engagement Activity | 2023 |
| URL | https://www.ixpug.org/events/isc23-ixpug-workshop |
| Description | Knowledge Exchange at Supercomputing 2024 (SC24) conference |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | The project provided programme level support for the ExCALIBUR community to engage with the annual Supercomputing (SC24) conference from 17th to 22nd November 2024 in Atlanta Georgia, in collaboration with UKRI. SC is the HPC community's largest global conference, with around 15,000 attendees. The project supported the ExCALIBUR presence at SC through a variety of mechanisms including travel and subsistence assistance and an exhibition booth. ExCALIBUR participation included three workshops, two papers, two Birds of a Feather sessions and a panel session. ResearchFish reporting on these activities will be recorded individually by the relevant individuals and projects. |
| Year(s) Of Engagement Activity | 2024 |
| URL | https://excalibur.ac.uk/events/excalibur-at-supercomputing-2024/ |
| Description | Talk in ExCALIBUR Bristol workshop |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Professional Practitioners |
| Results and Impact | Talk and poster at ExCALIBUR Bristol workshop. |
| Year(s) Of Engagement Activity | 2023 |
| URL | https://excalibur.ac.uk/events/spf-workshop/ |
| Description | Talk in RSECon 2023 |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Professional Practitioners |
| Results and Impact | A talk in the Research Software Engineering 2023 conference presenting the findings and tools produced by the project |
| Year(s) Of Engagement Activity | 2023 |
| URL | https://virtual.oxfordabstracts.com/#/event/4430/submission/74 |
| Description | Talk in Supercomputing 23 |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | This talk in the first HPCTESTS workshop in Supercomputing presented the paper on the tools and methodologies developed in this project to an expert international audience in the main conference of the field. It sparked active discussion and many new collaboration opportunities. |
| Year(s) Of Engagement Activity | 2023 |
| URL | https://sc23.supercomputing.org/proceedings/workshops/workshop_pages/ws_hpct103.html |
| Description | Workshop in CIUK2023 |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Professional Practitioners |
| Results and Impact | Hands-on workshop in the Computing Insight UK 2023 conference to train attendees to use the tools developed by the project |
| Year(s) Of Engagement Activity | 2023 |
| URL | https://www.scd.stfc.ac.uk/Pages/CIUK2023.aspx |
| Description | Workshop in RSECon 2023 |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Professional Practitioners |
| Results and Impact | Hands-on workshop in RSECon 2023 to train attendees in the use of the tools produced by the project |
| Year(s) Of Engagement Activity | 2023 |
| URL | https://virtual.oxfordabstracts.com/#/event/4430/program?session=66825&s=2760 |