Turbulent Flow Simulations at the Exascale: Application to Wind Energy and Green Aviation
Lead Research Organisation:
Imperial College London
Department Name: Aeronautics
Abstract
Our daily life is surrounded - and even is sustained - by the flow of fluids. Blood moves through thevessels in our bodies, and air flows into our lungs. Fluid flows disperse particulate air pollution in the turbulent urban as well as indoor environments. Fluid flows play a crucial role for our transportation and our industries. Our vehicles move through air and water powered by other fluids that mix in the combustion chambers of engines. Many of the environmental and energy-related issues we face today cannot possibly be tackled without a better understanding of the dynamics of fluids.
From a practical point of view, fluid flows relevant to scientists and engineers are turbulent ones; turbulence is the rule, not the exception. To date, a complete theory of fluid flow phenomena is still missing because of the complexity of the full equations describing the motion of a fluid.Their understanding and control is however crucial to improve technologies especially with minimal ecological impact as well as to anticipate events, in many areas ranging from engineering applications(e.g., industrial process, propulsion and power generation, car and aircraft design) to environmental sciences and technologies (e.g., air quality, weather forecasting, climate predictions, flood disasters monitoring).
Significant progress has been made recently using petascale computing, and computational fluid dynamics is now a critical complement to experiments and theories. Turbulent flow simulations at the exascale will require significant reformulation of existing flow solvers, implementation of new physics, and development of a more nuanced problem formulation. It has however the potential to produce significant advances in our quest towards a greener future, relying in large parts on a better understanding of the overarching subject of turbulence.
To better understand the opportunities and the challenges that will come with exascale computing for turbulent flows, we propose to create a Design and Development Working Group (DDWG) dedicated to turbulent flow simulations at the exascale, a high priority area of research for the UK. The focus will be on wind energy and green aviation applications as exascale computing will be a game changer in these areas and will contribute to make the UK a greener nation. This DDWG is building upon the experience and expertise of the UK Turbulence Consortium (UKTC) members and the recently funded Collaborative Computational Project (CCP) Turbulence. Two state-of-the-art open source flow solvers,OpenSBLI and Incompact3d, are currently being designed in the UK for pre-exascale and exascale systems using a high-level abstraction framework called OPS, an API with associated libraries and pre-processors to generate parallel executables for applications on multi-block structured meshes. The final output of this DDWG will be the delivery of a report with a strategic research agenda that will clearly articulate the research challenges to be overcome, opportunities, key risks and mitigation for turbulence simulations at the exascale. It will set out a detailed approach to enable development of CFD exascale-ready software through appropriate application-oriented, high-level programming abstractions, with proof-of-concept studies to demonstrate the capabilities of OpenSBLI and Incompact3d for exascale computing.
From a practical point of view, fluid flows relevant to scientists and engineers are turbulent ones; turbulence is the rule, not the exception. To date, a complete theory of fluid flow phenomena is still missing because of the complexity of the full equations describing the motion of a fluid.Their understanding and control is however crucial to improve technologies especially with minimal ecological impact as well as to anticipate events, in many areas ranging from engineering applications(e.g., industrial process, propulsion and power generation, car and aircraft design) to environmental sciences and technologies (e.g., air quality, weather forecasting, climate predictions, flood disasters monitoring).
Significant progress has been made recently using petascale computing, and computational fluid dynamics is now a critical complement to experiments and theories. Turbulent flow simulations at the exascale will require significant reformulation of existing flow solvers, implementation of new physics, and development of a more nuanced problem formulation. It has however the potential to produce significant advances in our quest towards a greener future, relying in large parts on a better understanding of the overarching subject of turbulence.
To better understand the opportunities and the challenges that will come with exascale computing for turbulent flows, we propose to create a Design and Development Working Group (DDWG) dedicated to turbulent flow simulations at the exascale, a high priority area of research for the UK. The focus will be on wind energy and green aviation applications as exascale computing will be a game changer in these areas and will contribute to make the UK a greener nation. This DDWG is building upon the experience and expertise of the UK Turbulence Consortium (UKTC) members and the recently funded Collaborative Computational Project (CCP) Turbulence. Two state-of-the-art open source flow solvers,OpenSBLI and Incompact3d, are currently being designed in the UK for pre-exascale and exascale systems using a high-level abstraction framework called OPS, an API with associated libraries and pre-processors to generate parallel executables for applications on multi-block structured meshes. The final output of this DDWG will be the delivery of a report with a strategic research agenda that will clearly articulate the research challenges to be overcome, opportunities, key risks and mitigation for turbulence simulations at the exascale. It will set out a detailed approach to enable development of CFD exascale-ready software through appropriate application-oriented, high-level programming abstractions, with proof-of-concept studies to demonstrate the capabilities of OpenSBLI and Incompact3d for exascale computing.
Planned Impact
Turbulence is relevant to the transportation, energy supply/generation, biomedical and process sectors in the UK and the world. In addition to creating new software and knowledge, the proposed agenda and the resulting scientific outputs will deliver benefits to the economy and allow the UK to realise its societal goals, especially the one of being a greener nation.
Despite being the largest contributors to harmful emissions, the transportation, energy generation/supply and process sectors are experiencing unprecedented growth around the world. For example, it is estimated that more than 29,000 new large civil airliners, 24,000 business jets, 5,800 regional aircraft and 40,000 helicopters will be required worldwide in 2032 to deal with the constant increase of worldwide air traffic. It is predicated that by 2025 there will be more than 16 billion passengers per year worldwide. The UK is directly concerned by this challenge as it is the second biggest national aerospace industry in the world, with a 17% global market share for a turnover of more than £20 billion every year, sustaining more than 200,000 jobs. Aviation will need to find ways to meet this impressive growing demand whilst reducing its environmental impact - specifically the noise levels and carbon emissions. This can only be achieve with a state-of-the-art ecosystem of exascale-ready software for a better understanding of the overarching subject of turbulence. Many of the UK Turbulence Consortium members and CCP Turbulence members, in collaboration with AIRBUS and the US Air Force, are currently working on drag reduction techniques for airplanes, high-speed trains, automotive vehicles and over the hulls of ships and submarines. Even a 1% reduction in drag can save at least 25,000 gallons of fuel per year per aircraft. Worldwide, this reduction could translate to fuel savings of more than $1 billion per year. The resulting reduction in emissions into the air is equally as impressive.
Since the mid-1990s, Computational Fluid Dynamics (CFD) has been integrated into industrial design and engineering processes, playing a decisive role in improving the quality and efficiency of complex products and significantly reducing the time to market. Petascale computing has enabled simulations at a higher level of precision and complexity, significantly impacting new areas of research. CFD is now recognised as a driver of economic growth and societal well-being and is vital for maintaining international competitiveness. The UK has a long history in Europe of developing cutting-edge applications dedicated to CFD. Because of the rapid evolution of the enabling technologies and the expanding range of applications demand, the UK needs to support and encourage the development of exascale-ready flow solvers, which can produce new knowledge, help to design innovative products and reduce cost and time of their implementation in real life applications. A striking example is the recent purchase by Siemens of the CFD software company CD-Adapco for $970M which clearly shows that turbulence simulations that can improve engineering design for a great range of applications are crucially needed by industries. Our industrial partners, heavily involved with several members of the UK Turbulence Consortium members, believe that exascale computing is crucial to gain insights into turbulence physics and would enabled them to better understand limitations of petascale computing and how to devise improvement strategies to take a competitive lead.
Despite being the largest contributors to harmful emissions, the transportation, energy generation/supply and process sectors are experiencing unprecedented growth around the world. For example, it is estimated that more than 29,000 new large civil airliners, 24,000 business jets, 5,800 regional aircraft and 40,000 helicopters will be required worldwide in 2032 to deal with the constant increase of worldwide air traffic. It is predicated that by 2025 there will be more than 16 billion passengers per year worldwide. The UK is directly concerned by this challenge as it is the second biggest national aerospace industry in the world, with a 17% global market share for a turnover of more than £20 billion every year, sustaining more than 200,000 jobs. Aviation will need to find ways to meet this impressive growing demand whilst reducing its environmental impact - specifically the noise levels and carbon emissions. This can only be achieve with a state-of-the-art ecosystem of exascale-ready software for a better understanding of the overarching subject of turbulence. Many of the UK Turbulence Consortium members and CCP Turbulence members, in collaboration with AIRBUS and the US Air Force, are currently working on drag reduction techniques for airplanes, high-speed trains, automotive vehicles and over the hulls of ships and submarines. Even a 1% reduction in drag can save at least 25,000 gallons of fuel per year per aircraft. Worldwide, this reduction could translate to fuel savings of more than $1 billion per year. The resulting reduction in emissions into the air is equally as impressive.
Since the mid-1990s, Computational Fluid Dynamics (CFD) has been integrated into industrial design and engineering processes, playing a decisive role in improving the quality and efficiency of complex products and significantly reducing the time to market. Petascale computing has enabled simulations at a higher level of precision and complexity, significantly impacting new areas of research. CFD is now recognised as a driver of economic growth and societal well-being and is vital for maintaining international competitiveness. The UK has a long history in Europe of developing cutting-edge applications dedicated to CFD. Because of the rapid evolution of the enabling technologies and the expanding range of applications demand, the UK needs to support and encourage the development of exascale-ready flow solvers, which can produce new knowledge, help to design innovative products and reduce cost and time of their implementation in real life applications. A striking example is the recent purchase by Siemens of the CFD software company CD-Adapco for $970M which clearly shows that turbulence simulations that can improve engineering design for a great range of applications are crucially needed by industries. Our industrial partners, heavily involved with several members of the UK Turbulence Consortium members, believe that exascale computing is crucial to gain insights into turbulence physics and would enabled them to better understand limitations of petascale computing and how to devise improvement strategies to take a competitive lead.
Publications
Hodgkin A
(2022)
Numerical investigation of the influence of shear and thermal stratification on the wind turbine tip-vortex stability
in Wind Energy
Bempedelis N
(2022)
Analytical all-induction state model for wind turbine wakes
in Physical Review Fluids
Bempedelis N
(2023)
Turbulent entrainment in finite-length wind farms
in Journal of Fluid Mechanics
Giannenas A
(2022)
A Cartesian Immersed Boundary Method Based on 1D Flow Reconstructions for High-Fidelity Simulations of Incompressible Turbulent Flows Around Moving Objects
in Flow, Turbulence and Combustion
| Description | It is expected that the UK will deploy its first exascale system by 2025. In anticipation, the UK research community is preparing for the arrival of the exascale era through the ExCALIBUR programme to develop exascale-ready algorithms and software. With the many opportunities that exascale computing will bring to the turbulence community, there are abundant challenges. While the Navier-Stokes equations are a well-established and -accepted- mathematical model to describe the motions of a turbulent flow, their solutions can be extremely challenging to obtain, due to the chaotic and inherently multi-scale nature of turbulence. The smallest scales impact the largest scales, and small changes to boundary conditions, initial conditions, or mesh resolution, for example, can have a dramatic impact on the solution. The turbulent scales are typically separated by many orders of magnitude. As a result, simulations of turbulent flows can present a unique set of challenges such as the need for accurate turbulent models, global communications, meshing generation an unfavourable computation to communication ratio and I/O (visualisations) bottlenecks. This project has identified two broad categories of critical challenges and opportunities in the exascale era with items distinctly pertaining to the turbulence community for the first category and to High Performance Computing (HPC) for the second one. In the first category, the project observed the following: • Exascale computing will only enable an incremental increase in the Reynolds number accessible through high-fidelity simulations, for which only the most energetic turbulent scales are simulated. However, there will be opportunities for multi-physics high-fidelity simulations (e.g., flame-wall interaction, including sprays with associated chemical kinetics; combustion at elevated pressures). • In the exascale era, high-fidelity simulations of turbulent flows based on 100-1000 billion mesh points and performed on 10-100 million cores for 100-1,000 hours (depending on the complexity of the flow) will become the norm, not the exception. • Shifting to high-order methods is identified as a key strategy for leveraging exascale machines, whilst capturing more turbulent scales than low-order methods for a given mesh resolution. • Obtaining accurate data in the high Reynolds number regimes will be pivotal for the design of turbulence models which are crucially needed for industrial applications (when cost constraints for industrial designs, or short turnaround times, means that simulating all the turbulent scales of the flow is unfeasible). • There is great potential for Machine Learning (ML) to open new opportunities for scientific discovery on upcoming exascale systems (turbulence models, flow prediction, data optimisation). • Exascale computing will allow for Uncertainty Quantification (UQ) techniques to be performed in an accurate and timely manner for large-scale simulations of turbulent flows, especially for estimating uncertainty in turbulence models. It is a unique opportunity for increasing confidence in turbulence simulations. In the second category, largely applicable to scientists and engineers who intend to use exascale systems, the following critical challenges and opportunities have been identified: • Power-usage restrictions are leading to a decrease in processor clock rates, an increase in core counts, more complex memory hierarchies and less available memory bandwidth per processor core. Consequently, the resulting hardware architectures have relied on massive-parallelism and are set to continue for the exascale era. Different systems continue to emerge with hardware vendors racing to achieve the ExaFlop/s performance through a diverse mix of heterogeneous and homogeneous / many-core systems along with multi-level memory hierarchies and programming paradigms. A separation of concerns approach (Domain Specific Language, DSL) that separates the science source (what is to be computed) from its parallel implementation (how to program the hardware) is emerging as the way forward. • The limiting of bandwidth due to increased core counts have shifted focus of optimisations to reducing data movement. It is crucial to manage load balances and minimise communications. Thus communication-avoiding techniques such as cache-blocking tiling, one-sided and near-neighbour communications (reducing collectives), have become important. Such complex optimisations again could be applied through a DSL approach on large code-bases automatically. • An increase in domain sizes will require a significant increase of the time needed to gather converged statistics (for a fixed time step). Unless the spatial resolution per thread (for scalability) decreases even faster as the number of available threads increases, this will lead to a longer time to solution. • There is a growing gap between compute capacity versus I/O capabilities. The storage hierarchy is becoming increasingly heterogeneous so I/O (3D visualisations and check-pointing procedures) are very likely going to become a major bottleneck for simulations of turbulent flows. |
| Exploitation Route | All our software are open-source and available to the scientific community. |
| Sectors | Aerospace, Defence and Marine,Education,Energy,Transport |
| Description | Turbulence at the exascale: application to wind energy, green aviation, air quality and net-zero combustion |
| Amount | £2,670,328 (GBP) |
| Funding ID | EP/W026686/1 |
| Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
| Sector | Public |
| Country | United Kingdom |
| Start | 12/2021 |
| End | 11/2024 |
| Title | OpenSBLI |
| Description | OpenSBLI is a Python-based modelling framework that is capable of expanding a set of differential equations written in Einstein notation, and automatically generating C code that performs the finite difference approximation to obtain a solution. This C code is then targetted with the OPS library towards specific hardware backends, such as MPI/OpenMP for execution on CPUs, and CUDA/OpenCL for execution on GPUs. The main focus of OpenSBLI is on the solution of the compressible Navier-Stokes equations with application to shock-boundary layer interactions (SBLI). However, in principle, any set of equations that can be written in Einstein notation may be solved. |
| Type Of Technology | Software |
| Year Produced | 2019 |
| Open Source License? | Yes |
| Impact | See list of publications |
| URL | https://opensbli.github.io/ |
| Title | Xcompact3d |
| Description | Xcompact3d is a Fortran-based framework of high-order finite-difference flow solvers dedicated to the study of turbulent flows. Dedicated to Direct and Large Eddy Simulations (DNS/LES) for which the largest turbulent scales are simulated, it can combine the versatility of industrial codes with the accuracy of spectral codes. Its user-friendliness, simplicity, versatility, accuracy, scalability, portability and efficiency makes it an attractive tool for the Computational Fluid Dynamics community. XCompact3d is currently able to solve the incompressible and low-Mach number variable density Navier-Stokes equations using sixth-order compact finite-difference schemes with a spectral-like accuracy on a monobloc Cartesian mesh. It was initially designed in France in the mid-90's for serial processors and later converted to HPC systems. It can now be used efficiently on hundreds of thousands CPU cores to investigate turbulence and heat transfer problems thanks to the open-source library 2DECOMP&FFT (a Fortran-based 2D pencil decomposition framework to support building large-scale parallel applications on distributed memory systems using MPI; the library has a Fast Fourier Transform module). When dealing with incompressible flows, the fractional step method used to advance the simulation in time requires to solve a Poisson equation. This equation is fully solved in spectral space via the use of relevant 3D Fast Fourier transforms (FFTs), allowing the use of any kind of boundary conditions for the velocity field. Using the concept of the modified wavenumber (to allow for operations in the spectral space to have the same accuracy as if they were performed in the physical space), the divergence free condition is ensured up to machine accuracy. The pressure field is staggered from the velocity field by half a mesh to avoid spurious oscillations created by the implicit finite-difference schemes. The modelling of a fixed or moving solid body inside the computational domain is performed with a customised Immersed Boundary Method. It is based on a direct forcing term in the Navier-Stokes equations to ensure a no-slip boundary condition at the wall of the solid body while imposing non-zero velocities inside the solid body to avoid discontinuities on the velocity field. This customised IBM, fully compatible with the 2D domain decomposition and with a possible mesh refinement at the wall, is based on a 1D expansion of the velocity field from fluid regions into solid regions using Lagrange polynomials or spline reconstructions. In order to reach high velocities in a context of LES, it is possible to customise the coefficients of the second derivative schemes (used for the viscous term) to add extra numerical dissipation in the simulation as a substitute of the missing dissipation from the small turbulent scales that are not resolved. Xcompact3d is currently being used by many research groups worldwide to study gravity currents, wall-bounded turbulence, wake and jet flows, wind farms and active flow control solutions to mitigate turbulence. |
| Type Of Technology | Software |
| Year Produced | 2019 |
| Open Source License? | Yes |
| Impact | see list of publications |
| URL | http://www.incompact3d.com |