Turbulence at the exascale: application to wind energy, green aviation, air quality and net-zero combustion
Lead Research Organisation:
Imperial College London
Department Name: Aeronautics
Abstract
This proposal brings together communities from the UK Turbulence Consortium (UKTC) and the UK Consortium on Reacting Flows (UKCRF) to ensure a smooth transition to exascale computing, with the aim to develop transformative techniques for future-proofing their production simulation software ecosystems dedicated to the study of turbulent flows. Understanding, predicting and controlling turbulent flows is of central importance and a limiting factor to a vast range of industries. Many of the environmental and energy-related issues we face today cannot possibly be tackled without a better understanding of turbulence.
The UK is preparing for the exascale era through the ExCALIBUR programme to develop exascale-ready algorithms and software. Based on the findings from the Design and Development Working Group (DDWG) on turbulence at the exascale, this project is bringing together communities representing two of the seven UK HEC Consortia, the UKTC and the UKCTRF, to re-engineer or extend the capabilities of four of their production and research flow solvers for exascale computing: XCOMPACT3D, OPENSBLI, UDALES and SENGA+. These open-source, well-established, community flow solvers are based on finite-difference methods on structured meshes and will be developed to meet the challenges associated with exascale computing while taking advantage of the significant opportunities afforded by exascale systems.
A key aim of this project is to leverage the well-established Domain Specific Language (DLS) framework OPS and the 2DECOMP&FFT library to allow XCOMPACT3D, OPENSBLI, UDALES and SENGA+ to run on large-scale heterogeneous computers. OPS was developed in the UK in the last ten years and it targets applications on multi-block structured meshes. It can currently generate code using CUDA, OPENACC/OPENMP5.0, OPENCL, SYCL/ONEAPI, HIP and their combinations with MPI. The OPS DSLs' capabilities will be extended in this project, specifically its code-generation tool-chain for robust, fail-safe parallel code generation. A related strand of work will use the 2DECOMP&FFT a Fortran-based library based on a 2D domain decomposition for spatially implicit numerical algorithms on monobloc structured meshes. The library includes a highly scalable and efficient interface to perform Fast Fourier Transforms (FFTs) and relies on MPI providing a user-friendly programming interface that hides communication details from application developers. 2DECOMP&FFT will be completely redesigned for a use on heterogeneous supercomputers (CPUs and GPUS from different vendors) using a hybrid strategy.
The project will also combine exascale-ready coupling interfaces, UQ capabilities, I/O & visualisation tools to our flow solvers, as well as machine learning based algorithms, to address some of the key challenges and opportunities identified by the DDWG on turbulence at the exascale. This will be done in collaboration with several of the recently funded ExCALIBUR cross-cutting projects.
The project will focus on four high-priority use cases (one for each solver), defined as high quality, high impact research made possible by a step-change in simulation performance. The use cases will focus on wind energy, green aviation, air quality and net-zero combustion. Exascale computing will be a game changer in these areas and will contribute to make the UK a greener nation (The UK commits to net zero carbon emissions by 2050). The use cases will be used to demonstrate the potential of the re-designed flow solvers based on OPS and 2DECOMP&FFT, for a wide range of hardware and parallel paradigms.
The UK is preparing for the exascale era through the ExCALIBUR programme to develop exascale-ready algorithms and software. Based on the findings from the Design and Development Working Group (DDWG) on turbulence at the exascale, this project is bringing together communities representing two of the seven UK HEC Consortia, the UKTC and the UKCTRF, to re-engineer or extend the capabilities of four of their production and research flow solvers for exascale computing: XCOMPACT3D, OPENSBLI, UDALES and SENGA+. These open-source, well-established, community flow solvers are based on finite-difference methods on structured meshes and will be developed to meet the challenges associated with exascale computing while taking advantage of the significant opportunities afforded by exascale systems.
A key aim of this project is to leverage the well-established Domain Specific Language (DLS) framework OPS and the 2DECOMP&FFT library to allow XCOMPACT3D, OPENSBLI, UDALES and SENGA+ to run on large-scale heterogeneous computers. OPS was developed in the UK in the last ten years and it targets applications on multi-block structured meshes. It can currently generate code using CUDA, OPENACC/OPENMP5.0, OPENCL, SYCL/ONEAPI, HIP and their combinations with MPI. The OPS DSLs' capabilities will be extended in this project, specifically its code-generation tool-chain for robust, fail-safe parallel code generation. A related strand of work will use the 2DECOMP&FFT a Fortran-based library based on a 2D domain decomposition for spatially implicit numerical algorithms on monobloc structured meshes. The library includes a highly scalable and efficient interface to perform Fast Fourier Transforms (FFTs) and relies on MPI providing a user-friendly programming interface that hides communication details from application developers. 2DECOMP&FFT will be completely redesigned for a use on heterogeneous supercomputers (CPUs and GPUS from different vendors) using a hybrid strategy.
The project will also combine exascale-ready coupling interfaces, UQ capabilities, I/O & visualisation tools to our flow solvers, as well as machine learning based algorithms, to address some of the key challenges and opportunities identified by the DDWG on turbulence at the exascale. This will be done in collaboration with several of the recently funded ExCALIBUR cross-cutting projects.
The project will focus on four high-priority use cases (one for each solver), defined as high quality, high impact research made possible by a step-change in simulation performance. The use cases will focus on wind energy, green aviation, air quality and net-zero combustion. Exascale computing will be a game changer in these areas and will contribute to make the UK a greener nation (The UK commits to net zero carbon emissions by 2050). The use cases will be used to demonstrate the potential of the re-designed flow solvers based on OPS and 2DECOMP&FFT, for a wide range of hardware and parallel paradigms.
Organisations
| Title | OpenSBLI |
| Description | OpenSBLI is a Python-based modelling framework that is capable of expanding a set of differential equations written in Einstein notation, and automatically generating C code that performs the finite difference approximation to obtain a solution. This C code is then targetted with the OPS library towards specific hardware backends, such as MPI/OpenMP for execution on CPUs, and CUDA/OpenCL for execution on GPUs. The main focus of OpenSBLI is on the solution of the compressible Navier-Stokes equations with application to shock-boundary layer interactions (SBLI). However, in principle, any set of equations that can be written in Einstein notation may be solved. |
| Type Of Technology | Software |
| Year Produced | 2019 |
| Open Source License? | Yes |
| Impact | See list of publications |
| URL | https://opensbli.github.io/ |
| Title | Xcompact3d |
| Description | Xcompact3d is a Fortran-based framework of high-order finite-difference flow solvers dedicated to the study of turbulent flows. Dedicated to Direct and Large Eddy Simulations (DNS/LES) for which the largest turbulent scales are simulated, it can combine the versatility of industrial codes with the accuracy of spectral codes. Its user-friendliness, simplicity, versatility, accuracy, scalability, portability and efficiency makes it an attractive tool for the Computational Fluid Dynamics community. XCompact3d is currently able to solve the incompressible and low-Mach number variable density Navier-Stokes equations using sixth-order compact finite-difference schemes with a spectral-like accuracy on a monobloc Cartesian mesh. It was initially designed in France in the mid-90's for serial processors and later converted to HPC systems. It can now be used efficiently on hundreds of thousands CPU cores to investigate turbulence and heat transfer problems thanks to the open-source library 2DECOMP&FFT (a Fortran-based 2D pencil decomposition framework to support building large-scale parallel applications on distributed memory systems using MPI; the library has a Fast Fourier Transform module). When dealing with incompressible flows, the fractional step method used to advance the simulation in time requires to solve a Poisson equation. This equation is fully solved in spectral space via the use of relevant 3D Fast Fourier transforms (FFTs), allowing the use of any kind of boundary conditions for the velocity field. Using the concept of the modified wavenumber (to allow for operations in the spectral space to have the same accuracy as if they were performed in the physical space), the divergence free condition is ensured up to machine accuracy. The pressure field is staggered from the velocity field by half a mesh to avoid spurious oscillations created by the implicit finite-difference schemes. The modelling of a fixed or moving solid body inside the computational domain is performed with a customised Immersed Boundary Method. It is based on a direct forcing term in the Navier-Stokes equations to ensure a no-slip boundary condition at the wall of the solid body while imposing non-zero velocities inside the solid body to avoid discontinuities on the velocity field. This customised IBM, fully compatible with the 2D domain decomposition and with a possible mesh refinement at the wall, is based on a 1D expansion of the velocity field from fluid regions into solid regions using Lagrange polynomials or spline reconstructions. In order to reach high velocities in a context of LES, it is possible to customise the coefficients of the second derivative schemes (used for the viscous term) to add extra numerical dissipation in the simulation as a substitute of the missing dissipation from the small turbulent scales that are not resolved. Xcompact3d is currently being used by many research groups worldwide to study gravity currents, wall-bounded turbulence, wake and jet flows, wind farms and active flow control solutions to mitigate turbulence. |
| Type Of Technology | Software |
| Year Produced | 2019 |
| Open Source License? | Yes |
| Impact | see list of publications |
| URL | http://www.incompact3d.com |