Turbulence at the exascale: application to wind energy, green aviation, air quality and net-zero combustion

Lead Research Organisation: Imperial College London
Department Name: Aeronautics

Abstract

This proposal brings together communities from the UK Turbulence Consortium (UKTC) and the UK Consortium on Reacting Flows (UKCRF) to ensure a smooth transition to exascale computing, with the aim to develop transformative techniques for future-proofing their production simulation software ecosystems dedicated to the study of turbulent flows. Understanding, predicting and controlling turbulent flows is of central importance and a limiting factor to a vast range of industries. Many of the environmental and energy-related issues we face today cannot possibly be tackled without a better understanding of turbulence.

The UK is preparing for the exascale era through the ExCALIBUR programme to develop exascale-ready algorithms and software. Based on the findings from the Design and Development Working Group (DDWG) on turbulence at the exascale, this project is bringing together communities representing two of the seven UK HEC Consortia, the UKTC and the UKCTRF, to re-engineer or extend the capabilities of four of their production and research flow solvers for exascale computing: XCOMPACT3D, OPENSBLI, UDALES and SENGA+. These open-source, well-established, community flow solvers are based on finite-difference methods on structured meshes and will be developed to meet the challenges associated with exascale computing while taking advantage of the significant opportunities afforded by exascale systems.

A key aim of this project is to leverage the well-established Domain Specific Language (DLS) framework OPS and the 2DECOMP&FFT library to allow XCOMPACT3D, OPENSBLI, UDALES and SENGA+ to run on large-scale heterogeneous computers. OPS was developed in the UK in the last ten years and it targets applications on multi-block structured meshes. It can currently generate code using CUDA, OPENACC/OPENMP5.0, OPENCL, SYCL/ONEAPI, HIP and their combinations with MPI. The OPS DSLs' capabilities will be extended in this project, specifically its code-generation tool-chain for robust, fail-safe parallel code generation. A related strand of work will use the 2DECOMP&FFT a Fortran-based library based on a 2D domain decomposition for spatially implicit numerical algorithms on monobloc structured meshes. The library includes a highly scalable and efficient interface to perform Fast Fourier Transforms (FFTs) and relies on MPI providing a user-friendly programming interface that hides communication details from application developers. 2DECOMP&FFT will be completely redesigned for a use on heterogeneous supercomputers (CPUs and GPUS from different vendors) using a hybrid strategy.

The project will also combine exascale-ready coupling interfaces, UQ capabilities, I/O & visualisation tools to our flow solvers, as well as machine learning based algorithms, to address some of the key challenges and opportunities identified by the DDWG on turbulence at the exascale. This will be done in collaboration with several of the recently funded ExCALIBUR cross-cutting projects.

The project will focus on four high-priority use cases (one for each solver), defined as high quality, high impact research made possible by a step-change in simulation performance. The use cases will focus on wind energy, green aviation, air quality and net-zero combustion. Exascale computing will be a game changer in these areas and will contribute to make the UK a greener nation (The UK commits to net zero carbon emissions by 2050). The use cases will be used to demonstrate the potential of the re-designed flow solvers based on OPS and 2DECOMP&FFT, for a wide range of hardware and parallel paradigms.
 
Description It is expected that the UK will deploy its first exascale system by 2026. In anticipation, the UK research community is preparing for the arrival of the exascale era through the ExCALIBUR programme to develop exascale-ready algorithms and software. With the many opportunities that exascale computing will bring to the turbulence community, there are abundant challenges. While the Navier-Stokes equations are a well-established and -accepted- mathematical model to describe the motions of a turbulent flow, their solutions can be extremely challenging to obtain, due to the chaotic and inherently multi-scale nature of turbulence. The smallest scales impact the largest scales, and
small changes to boundary conditions, initial conditions, or mesh resolution, for example, can have a dramatic impact on the solution. The turbulent scales are typically separated by many orders of magnitude. As a result, simulations of turbulent flows can present a unique set of challenges such as the need for accurate turbulent models, global communications, meshing generation an unfavourable computation to communication ratio and I/O (visualisations) bottlenecks. This project has identified two broad categories of critical challenges and opportunities in the exascale era with items distinctly pertaining to the turbulence community for the first category and to High Performance Computing (HPC) for the second one.

In the first category, the project observed the following:
• Exascale computing will only enable an incremental increase in the Reynolds number accessible through high-fidelity simulations, for which only the most energetic turbulent scales are simulated. However, there will be opportunities for multi-physics high-fidelity simulations (e.g., flame-wall interaction, including sprays with associated chemical kinetics; combustion at elevated pressures).
• In the exascale era, high-fidelity simulations of turbulent flows based on 100-1000 billion mesh points and performed on 10-100 million cores for 100-1,000 hours (depending on the complexity of the flow) will become the norm, not the exception.
• Shifting to high-order methods is identified as a key strategy for leveraging exascale machines, whilst capturing more turbulent scales than low-order methods for a given mesh resolution.
• Obtaining accurate data in the high Reynolds number regimes will be pivotal for the design of turbulence models which are crucially needed for industrial applications (when cost constraints for industrial designs, or short turnaround times, means that simulating all the turbulent scales of the flow is unfeasible).
• There is great potential for Machine Learning (ML) to open new opportunities for scientific discovery on upcoming exascale systems (turbulence models, flow prediction, data optimisation).
• Exascale computing will allow for Uncertainty Quantification (UQ) techniques to be performed in an accurate and timely manner for large-scale simulations of turbulent flows, especially for estimating uncertainty in turbulence models. It is a unique opportunity for increasing confidence in turbulence simulations.

In the second category, largely applicable to scientists and engineers who intend to use exascale systems, the following critical challenges and opportunities have been identified:
• Power-usage restrictions are leading to a decrease in processor clock rates, an increase in core counts, more complex memory hierarchies and less available memory bandwidth per processor core. Consequently, the resulting hardware architectures have relied on massive-parallelism and are set to continue for the exascale era. Different systems continue to emerge with hardware vendors racing to achieve the ExaFlop/s performance through a diverse mix of heterogeneous and homogeneous / many-core systems along with multi-level memory hierarchies and programming paradigms. A separation of concerns approach (Domain Specific Language, DSL) that separates the science source (what is to be computed) from its parallel implementation (how to program the hardware) is emerging as the way forward.
• The limiting of bandwidth due to increased core counts have shifted focus of optimisations to reducing data movement. It is crucial to manage load balances and minimise communications. Thus communication-avoiding techniques such as cache-blocking tiling, one-sided and near-neighbour communications (reducing collectives), have become important. Such complex optimisations again could be applied through a DSL approach on large code-bases automatically.
• An increase in domain sizes will require a significant increase of the time needed to gather converged statistics (for a fixed time step). Unless the spatial resolution per thread (for scalability) decreases even faster as the number of available threads increases, this will lead to a longer time to solution.
• There is a growing gap between compute capacity versus I/O capabilities. The storage hierarchy is becoming increasingly heterogeneous so I/O (3D visualisations and check-pointing procedures) are very likely going to become a major bottleneck for simulations of turbulent flows.
Exploitation Route All our software are open-source and can freely be used by others.
Sectors Aerospace

Defence and Marine

Education

Energy

Environment
 
Title 2DECOMP&FFT 
Description The 2DECOMP&FFT library is a software framework written in modern Fortran to build large scale parallel applications. It is designed for applications using three-dimensional structured meshes with a particular focus on spatially implicit numerical algorithms. However, the library can be easily used with other discretisation schemes based on a structured layout and where pencil decomposition can apply. It is based on a general-purpose 2D pencil decomposition for data distribution and data Input Output (I/O). A 1D slab decomposition is also available as a special case of the 2D pencil decomposition. The library includes a highly scalable and efficient interface to perform three-dimensional Fast Fourier Transforms (FFTs). The library has been designed to be user-friendly, with a clean application programming interface hiding most communication details from application developers, and portable with support for modern CPUs and NVIDIA GPUs (support for AMD and Intel GPUs to follow). 
Type Of Technology Software 
Year Produced 2023 
Open Source License? Yes  
Impact Possibility to use GPU hardware 
URL https://www.theoj.org/joss-papers/joss.05813/10.21105.joss.05813.pdf
 
Title OpenSBLI 
Description OpenSBLI is a Python-based modelling framework that is capable of expanding a set of differential equations written in Einstein notation, and automatically generating C code that performs the finite difference approximation to obtain a solution. This C code is then targetted with the OPS library towards specific hardware backends, such as MPI/OpenMP for execution on CPUs, and CUDA/OpenCL for execution on GPUs. The main focus of OpenSBLI is on the solution of the compressible Navier-Stokes equations with application to shock-boundary layer interactions (SBLI). However, in principle, any set of equations that can be written in Einstein notation may be solved. 
Type Of Technology Software 
Year Produced 2019 
Open Source License? Yes  
Impact See list of publications 
URL https://opensbli.github.io/
 
Title Xcompact3d 
Description Xcompact3d is a Fortran-based framework of high-order finite-difference flow solvers dedicated to the study of turbulent flows. Dedicated to Direct and Large Eddy Simulations (DNS/LES) for which the largest turbulent scales are simulated, it can combine the versatility of industrial codes with the accuracy of spectral codes. Its user-friendliness, simplicity, versatility, accuracy, scalability, portability and efficiency makes it an attractive tool for the Computational Fluid Dynamics community. XCompact3d is currently able to solve the incompressible and low-Mach number variable density Navier-Stokes equations using sixth-order compact finite-difference schemes with a spectral-like accuracy on a monobloc Cartesian mesh. It was initially designed in France in the mid-90's for serial processors and later converted to HPC systems. It can now be used efficiently on hundreds of thousands CPU cores to investigate turbulence and heat transfer problems thanks to the open-source library 2DECOMP&FFT (a Fortran-based 2D pencil decomposition framework to support building large-scale parallel applications on distributed memory systems using MPI; the library has a Fast Fourier Transform module). When dealing with incompressible flows, the fractional step method used to advance the simulation in time requires to solve a Poisson equation. This equation is fully solved in spectral space via the use of relevant 3D Fast Fourier transforms (FFTs), allowing the use of any kind of boundary conditions for the velocity field. Using the concept of the modified wavenumber (to allow for operations in the spectral space to have the same accuracy as if they were performed in the physical space), the divergence free condition is ensured up to machine accuracy. The pressure field is staggered from the velocity field by half a mesh to avoid spurious oscillations created by the implicit finite-difference schemes. The modelling of a fixed or moving solid body inside the computational domain is performed with a customised Immersed Boundary Method. It is based on a direct forcing term in the Navier-Stokes equations to ensure a no-slip boundary condition at the wall of the solid body while imposing non-zero velocities inside the solid body to avoid discontinuities on the velocity field. This customised IBM, fully compatible with the 2D domain decomposition and with a possible mesh refinement at the wall, is based on a 1D expansion of the velocity field from fluid regions into solid regions using Lagrange polynomials or spline reconstructions. In order to reach high velocities in a context of LES, it is possible to customise the coefficients of the second derivative schemes (used for the viscous term) to add extra numerical dissipation in the simulation as a substitute of the missing dissipation from the small turbulent scales that are not resolved. Xcompact3d is currently being used by many research groups worldwide to study gravity currents, wall-bounded turbulence, wake and jet flows, wind farms and active flow control solutions to mitigate turbulence. 
Type Of Technology Software 
Year Produced 2019 
Open Source License? Yes  
Impact see list of publications 
URL http://www.incompact3d.com