Lattice Field Theory at the Exascale Frontier

Lead Research Organisation: University of Edinburgh
Department Name: Sch of Physics and Astronomy


Lattice Field Theory (LFT) provides the tools to study the fundamental forces of nature using numerical simulations. The traditional realm of application of LFT has been Quantum Chromodynamics (QCD), the theory describing the strong nuclear force within the Standard Model (SM) of particle physics. These calculations now include electromagnetic effects and achieve sub percent accuracy. Other applications span a wide range of topics, from theories beyond the Standard Model, to low-dimensional strongly coupled fermionic models, to new cosmological paradigms. At the core of this scientific endeavour lies the ability to perform sophisticated and demanding numerical simulations. The Exascale era of High Performance Computing therefore looks like a time of great opportunities.

The UK LFT community has been at the forefront of the field for more than three decades and has developed a broad portfolio of research areas, with synergetic connections to High-Performance Computing, leading to significant progress in algorithms and code performance.
Highlights of successes include: influencing the design of new hardware (Blue Gene systems); developing algorithms (Hybrid Monte Carlo) that are used widely by many other communities; maximising the benefits from new technologies (lattice QCD practitioners were amongst the first users of new platforms, including GPUs for scientific computing); applying LFT techniques to new problems in Artificial Intelligence.

The research programme in LFT, and its impact, can be expanded in a transformative way with the advent of pre-Exascale and Exascale systems, but only if key challenges are addressed. As the number of floating point operations per second increases, the communications between computing nodes are lagging behind, and this imbalance will severely affect future LFT simulations across the board.

These challenges are common to all LFT codebases, and more generally to other communities that are large users of HPC resources. The bottlenecks on new architectures need to be carefully identified, and software that minimises the communications must be designed in order to make the best usage of forthcoming large computers. As we are entering an era of heterogeneous architectures, the design of new software must clearly isolate the algorithmic progress from the details of the implementation on disparate hardware, so that our software can be deployed efficiently on forthcoming machines with limited effort.

The goal of the EXA-LAT project is to develop a common set of best practices, KPIs and figures of merit that can be used by the whole LFT community in the near future and will inform the design and procurement of future systems. Besides the participation of the LFT community, numerous vendors and computing centres have joined the project, together with scholars from 'neighbouring' disciplines. Thereby we aim to create a national and international focal point that will foster the activity of scholars, industrial partners and Research Sotfware Engineers (RSEs). This synergetic environment will host training events for academics, RSEs and students, which will contribute to the creation of a skilled work force immersed in a network that comprises the leading vendors in the subject.

EXA-LAT will set the foundations for a long-term effort by the LFT community to fully benefit of Exascale facilities and transfer some of the skills that characterise our scientific work to a wider group of users across disciplines.

Planned Impact

EXA-LAT will develop new methods for exascale computing that will bring benefits with ample reach. Key beneficiaries of our project will be HPC solution providers, and in particular our commercial project partners, i.e. Arm, Intel, Mellanox, NAG, NVidia. These companies are a respectable set of world-leading software and hardware providers that are actively developing Exascale solutions to be deployed in future releases of their top-end portfolio products. Through the partnership, they will get access to the expertise of a world-renowned community that is developing advanced Exascale software solutions of wide applicability. Our industry partners will be able to have direct access to our codes and hence will be able to optimise their future product releases for one of the most demanding fields of applications. This will provide them with a clear advantage over the competition. At the same time, having the opportunity to test their offers on LFT code will allow them to refine the design of future architectures and software packages in collaboration with LFT, following a model that has been successfully used in the past, e.g., in a collaboration between the UK LFT community and IBM, which has led to the successful commercial exploitation of the BlueGene/Q product line.

In addition to producing a unique sample of high performance code for Exascale architectures, through its community work EXA-LAT will establish new programming methods and devise novel algorithms for Exascale-capable systems. It is anticipated that the underlying methodology will diffuse to the wider HPC world, not only in academia, but also in industrial settings. In order to favour this uptake, we will collaborate with three diverse centres of Doctoral training, developing teaching material for quick adoption in their taught courses and upskilling doctoral students through exposure to our Exascale programming methods. This activity will result in a novel cohort of programmers that, educated in a high-level academic setting, will take key jobs in industry, advancing methods and tools in their respective companies, for the benefit of the latter and of the general economy.

At the same time, we will encourage direct uptaking of our programming methods and algorithm by direct interaction with the industry. This will be facilitated by the Impact Lab (Plymouth) and the Beyond the Lab initiative (Edinburgh), which are two programmes led by investigators in this proposal. The aim of the Impact Lab is to bring the area's world-class R&D assets together within a business-focused innovation ecosystem, to stimulate the development of new products and services and to provide the support required to see these through to market success. The Impact Lab is acting as a hub for SMEs in Devon to provide training on the most advanced techniques in HPC, Data Science, Big Data and Mathematical Modelling. The Impact Lab will disseminate the findings of this project to relevant SMEs through direct interaction and organisation of CPD courses. Beyond the Lab is an initiative established in 2019 by the Institute for Particle and Nuclear Physics at the University of Edinburgh in order to foster the links between the researchers in Particle Physics and Industry through individual partnerships. As a part of its activity, Beyond the Lab will engage with commercial companies that can benefit from the Exascale resources developed by EXA-LAT in order to provide them with a path towards exploitation of those resources.

The general public is also among the wide beneficiaries of the EXA-LAT project. As a part of our activity, we will reach to a wide lay audience in order to raise awareness of both the open problems in Particle Physics tackled by the UK LFT community and the challenges and benefits of Exascale computing for the latter and for the wider society. Part of this engagement will be the participation to outreach events (e.g., Swansea Science Festival).


10 25 50
Description Exascale computers will soon be available to the scientific community for research purposes. This new generation of hardware will allow new, ambitious scientific objectives to be achieved provided that the practitioners in the field of High Performance Computing develop the knowledge and the software to use effectively the new architectures. New programming paradigms need to be developed and implemented effectively. The Exalat project has laid the foundations for the design of new algorithms and their benchmarks on existing supercomputers, while simultaneously surveying the scientific opportunities for the Lattice Field Theory community in the UK, and establishing an effective training pattern for all practitioners in the field, academics, students, PDRAs and RSEs. We are currently in the process of finalising a set of partnership activities with companies active in the HPC sector.
Exploitation Route The Exalat project has set up activities in four main areas: benchmarking, algorithm development, training and data curation. Workshops have been organised where the activities were discussed and results presented. The response from the UK community has been positive, with a seemingly high level of engagement. All the outcomes of these activities are public and available to the whole community, in particular the training material, most of which has been developed especially for Exalat, will be collected in our website and will build in time a unique collection of material produced by practitioners in the field. A significant effort has been dedicated to start joint activities with other Excalibur work packages. We plan to continue these activities regularly in the coming years.
Sectors Creative Economy,Digital/Communication/Information Technologies (including Software),Education,Financial Services, and Management Consultancy

Description The main non-academic impact of High Performance Computing projects like this one comes from the training of a new generation of practitioners who are capable of developing software on the latest architectures available on the market. Through our training events we have had the opportunity to train approximately 100 people this year. Lattice Field Theory has a long track record of producing computer-literate graduates who then work in multiple sectors of the economy (digital, finance, education). The work done by the Exalat project has been influential in defining and delivering the benchmark of a new DiRAC HPC system at Edinburgh. The benchmark routines developed in Exalat have been contributed to Excalibur and used in the RSE working group.
First Year Of Impact 2021
Sector Digital/Communication/Information Technologies (including Software)
Impact Types Economic

Title SOMBRERO Benchmarking Suite 
Description SOMBRERO is suite of benchmarks for HPC computing systems. SOMBRERO is derived from HiRep, a research code used in theoretical particle physics for studying strong interactions beyond the Standard Model. The physical properties of the interactions simulated by HiRep give SOMBRERO a unique ability to vary the ratio between computation and communication by simply changing the internal degrees of freedom of the physical model. This ability makes SOMBRERO an ideal tool for exploring performance on supercomputing systems under workloads of varying intensity. 
Type Of Technology Software 
Year Produced 2021 
Open Source License? Yes  
Impact SOMBRERO has become part of the UKRI ExCALIBUR benchmarking suite and has in fact provided a template for developing other benchmarks. SOMBRERO has been featured in the Linux Format magazine, which targets a wide lay audience. SOMBRERO is being adopted by leading HPC companies to assess the performance of their systems. 
Description Exalat kickoff meeting 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact The advent of new Exascale computers, based on heterogeneous architectures, will open new possibilities and bring new challenges.
The purpose of this workshop is to identify the challenges and draw a roadmap for the community to prepare for the challenges ahead.

The main focus will be on:

Scientific objectives and HPC requirements for the various codebases.
Benchmarking and identification of bottlenecks. Hardware choices.
Optimization and code development.
Training opportunities.
Links to other HPC communities.
Year(s) Of Engagement Activity 2020