A scalable dynamical core for Next Generation Weather and Climate Prediction - Phase 2
Lead Research Organisation:
University of Bath
Department Name: Mathematical Sciences
Organisations
Publications
Adams S. V.
(2018)
LFRic: Meeting the challenges of scalability and performance portability in Weather and Climate models
in arXiv e-prints
Bastian P
(2019)
Matrix-free multigrid block-preconditioners for higher order discontinuous Galerkin discretisations
in Journal of Computational Physics
Betteridge J
(2021)
Multigrid preconditioners for the hybridised discontinuous Galerkin discretisation of the shallow water equations
in Journal of Computational Physics
Cullen, Mike, Freitag, Melina A., Kindermann, Stefan, Scheichl, Robert
(2013)
Large Scale Inverse Problems: Computational Methods and Applications in the Earth Sciences
Dedner A
(2015)
Efficient multigrid preconditioners for atmospheric flow simulations at high aspect ratio
in International Journal for Numerical Methods in Fluids
Maynard Christopher
(2020)
Multigrid preconditioners for the mixed finite element dynamical core of the LFRic atmospheric model
in arXiv e-prints
Mitchell L
(2016)
High level implementation of geometric multigrid solvers for finite element problems: Applications in atmospheric modelling
in Journal of Computational Physics
Müller E
(2014)
Matrix-free GPU implementation of a preconditioned conjugate gradient solver for anisotropic elliptic PDEs
in Computing and Visualization in Science
Müller E
(2014)
Massively parallel solvers for elliptic partial differential equations in numerical weather and climate prediction
in Quarterly Journal of the Royal Meteorological Society
Müller E
(2015)
Petascale solvers for anisotropic PDEs in atmospheric modelling on GPU clusters
in Parallel Computing
| Description | In Phase 2 of this programme grant, we have extended our results from Phase 1, solved a physically more relevant problem, implemented and tested our methods within two parallel, discretisation platforms, DUNE and firedrake, as well as stand-alone solvers on novel architectures, such as GPUs = |
| Exploitation Route | We have continued publishing all our results in top journals and presenting at a large range of relevant meetings in meteorology and in HPC. The papers are well cited and we are receiving invitations from other researchers and centres for collaborations, in particular we had discussions about a joint project with the European Centre for Medium Range Weather Forecasting (ECMWF). The implementation of the results at the Met Office is still ongoing and we are still actively involved in it. As documented in an Industrial Impact Case Study submitted for REF2021, the multigrid solver developed for the ENDGame model has been used operationally by the Met Office since 9th Dec 2020 and has made a significant impact. This has also been highlighted in the Met Office research news article https://www.metoffice.gov.uk/research/news/2020/multigrid-solver and in the press-release https://www.bath.ac.uk/announcements/a-sunny-outlook-for-speedier-weather-forecasts/ As pointed out in the REF2021 Impact Case Study, the new multigrid code has been picked up and is also used by one of the Met Office's international partners, the NIWA research institute in New Zealand. |
| Sectors | Environment |
| URL | https://www.metoffice.gov.uk/research/news/2020/multigrid-solver |
| Description | The solution of the pressure correction equation in atmospheric models is one of the computational bottlenecks if implicit time stepping methods are used. Hence any improvements to the solver algorithm and its implementation will allow the Met Office to deliver more accurate forecasts in a shorter time. In this project we have demonstrated the benefits of an improved, tensor product multigrid algorithm and showed that it scales well on modern massively parallel supercomputer hardware, which will be used for running the next generation atmospheric models. The key impact of this project has been to convince the Met Office to use multigrid solver technology across its current and next-generation numerical climate- and weather-forecast models. In addition to increasing the Met Office's reputation as a world- leading weather- and climate prediction centre, those improved forecasting capabilities have an impact on a wide range of industries and all parts of society (e.g. by aiding emergency responders and by informing government policy to prepare for changes in the earth's climate). Impact from this project has been realised on two different timescales: (1) a multigrid solver has been implemented in the current ENDGame dynamical core, and we demonstrated that this reduces the solution time for the pressure equation by a factor of around two. Tests of the solver in operational configurations have shown that the total model runtime is reduced by 10%-15%, which is a significant saving for a mature model of this complexity. Following successful testing and approval, the multigrid solver has been used operationally since Dec 9th 2020. (2) The next generation "LFRic" forecast model is based on mimetic finite element discretisations and will become operational within the next 5-10 years. One of the research outputs of this project was to develop bespoke solver technology for this discretisation. A new multigrid solver is now an integral part of the Fortran 2003 LFRic model; it is expected to be the standard solver in the next-generation numerical forecast code. In collaboration with Met Office scientists, we demonstrated the superior performance and scalability of the multigrid solver technology under realistic conditions. |
| First Year Of Impact | 2020 |
| Sector | Environment |
| Impact Types | Policy & public services |
| Description | EPSRC Impact Acceleration Award (IAA) |
| Amount | £27,329 (GBP) |
| Funding ID | IAA428 Mueller |
| Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
| Sector | Public |
| Country | United Kingdom |
| Start | 05/2021 |
| End | 10/2021 |
| Title | DUNE implementation of tensor-product multigrid solver |
| Description | Implementation of a geometric tensor-product multigrid solver in the DUNE C++ library for grid based applications. The code generalises other implementations of the algorithm since it supports more generals grids and more realistic pressure equations encountered in atmospheric modelling. |
| Type Of Technology | Software |
| Year Produced | 2015 |
| Open Source License? | Yes |
| Impact | The code was used to obtain results in the following paper: Dedner, A., Müller, E. and Scheichl, R., 2016. Efficient multigrid preconditioners for atmospheric flow simulations at high aspect ratio. International Journal for Numerical Methods in Fluids, 80(1), pp.76-102 |
| URL | https://bitbucket.org/em459/tensorproductmultigrid |
| Title | Mimetic finite element multigrid solver |
| Description | The Met Office will use a mimetic finite element discretisation for its next generation forecast model LFRic. To test the performance of bespoke multigrid solvers in this discretisation and to compare to existing AMG solvers, this code solves the pressure equation which arises in the gravity wave propagation scenario in a shallow atmosphere. All code is implemented in the Firedrake/PyOP2 finite element library. |
| Type Of Technology | Software |
| Year Produced | 2016 |
| Open Source License? | Yes |
| Impact | This software will inform the design of solvers for the next generation Met Office forecast model LFric. Currently the data structures and algorithms implemented in the Firedrake version of the code are translated into the Fortran 2003 code base used by the Met Office. The code was also crucial to obtain results in the following paper: Mitchell, L. and Müller, E.H., 2016. "High level implementation of geometric multigrid solvers for finite element problems: applications in atmospheric modelling." Journal of Computational Physics, 327, pp.1-18. |
| URL | https://github.com/firedrakeproject/firedrake-helmholtzsolver |
| Title | Multi-GPU implementation of tensor-product multigrid algorithm |
| Description | This multi-GPU implementation of a tensor-product multigrid solver was used to solve a simplified pressure correction equation and test the performance of the solver on multi-GPU clusters. |
| Type Of Technology | Software |
| Year Produced | 2014 |
| Open Source License? | Yes |
| Impact | Since the Met Office is considering using chip architectures similar to GPUs for their next generation forecast model, the results obtained with this code will have an impact on the ultimate choice of solver algorithm. The code was used to produce results for the following two publications: Müller, E., Guo, X., Scheichl, R. and Shi, S., 2013. "Matrix-free GPU implementation of a preconditioned conjugate gradient solver for anisotropic elliptic PDEs". Computing and Visualization in Science, 16(2), pp.41-58. Müller, E.H., Scheichl, R. and Vainikko, E., 2015. "Petascale solvers for anisotropic PDEs in atmospheric modelling on GPU clusters". Parallel Computing, 50, pp.53-69. |
| URL | https://bitbucket.org/em459/ellipticsolvergpu |
| Title | Multigrid solver in the ENDGame dynamical core |
| Description | ENDGame is the dynamical core in the Unified Model, which is used by the Met Office for climate- and weather- forecasting. As a result of this NERC funded project a bespoke multigrid solver for the pressure correction equation was implemented in the ENDGame code. |
| Type Of Technology | Software |
| Year Produced | 2019 |
| Impact | Compared to alternative solvers, the bespoke multigrid solver developed during this project reduces the runtime of the Unified Model by 10%-15%. This has a significant impact on the speed and quality with which the Met Office can deliver its key forecast products. |
| URL | https://code.metoffice.gov.uk/trac/um/ |
| Title | Multigrid solver in the LFRic numerical forecast model |
| Description | LFRic is the next generation numerical climate- and weather- prediction code developed by the Met Office. As a result of this NERC funded project, a bespoke multigrid solver for the pressure correction equation was implemented in the LFRic code. |
| Type Of Technology | Software |
| Year Produced | 2019 |
| Open Source License? | Yes |
| Impact | Compared to alternative methods, the bespoke multigrid solver is significantly more efficient. It reduces the total runtime of the model and improves parallel scalability. |