Parallel Scalability of Elliptic Solvers in Weather and Climate Prediction

The UK Met Office is one of the world leaders in weather and climate prediction, and the Met Office's global forecast model is used by many other centres worldwide to drive their individual local area models. However, many short scale phenomena as well as important longterm dynamics are still difficult to predict accurately due to the limited spatial resolution of global models and the additional errors introduced by local area models. Novel computing architectures with more than 10^5 cores provide a chance to push these boundaries and to keep the UK Met Office at the forefront of developments.

Decades of experience with numerical weather and climate prediction have produced a good understanding of the core dynamics inherent in atmospheric flow and of their stable and accurate numerical approximations. As outlined in the call, the Met Office's Unified Model uses lattitude-longitude grids and achieves high efficiency on parallel computers with up to 1000 cores. However, (artificial) grid clustering at the poles renders these grids impractical for large-scale computations, and so one of the core tasks in this NERC Programme is the search for suitable alternative grids. Several separate proposals address this issue. However, the equations governing atmospheric flow form a time-dependent system of differential equations which strongly couple the solution everywhere on the globe (the famous "butterfly effect"). Most current atmospheric dynamics models use semi-implicit time discretisation schemes which provide some global coupling of the equations at each time step. This prevents the system from becoming unstable and as a consequence it allows for larger time steps than fully explicit schemes, which include no global coupling. Since the cost of the forecast is proportional to the number of time steps, a scheme that allows for larger time steps (with satisfactory accuracy) seems preferable. But these benefits come at a price, especially in the context of large-scale problems and on massively parallel architectures. An elliptic system for the pressure has to be solved in each time step, leading to a very large, ill-conditioned algebraic system, the solution of which is difficult to parallelise efficiently. There are two main factors that make the scaling of this elliptic solve to large problem sizes and to large processor numbers difficult: algorithmic scalability and parallel scalability. Since the solution operator for the elliptic equation couples the pressures globally, only multilevel iterative solvers which use a hierarchy of discretisations on grids of varying resolution allow optimal, linear growth in cost (algorithmic scalability). But in a massively parallel computing environment, where global communication is costly, it is necessary to implement these solvers well, keeping most of the communication local, to ensure that the computational cost continues to scale optimally to 100K or more processors (parallel scalability).

This proposal addresses this problem and will thus facilitate the best possible decisions on the design of the Met Office's future dynamical core, thus guaranteeing the UK's competitiveness in this key societal/technological challenge. An optimal scalability of semi-implicit schemes has not been achieved in atmospheric flow up to now, but success of the Project Partners, IWR Heidelberg and Lawrence Livermore National Lab, on simpler model elliptic problems shows that it is possible. The PIs experience over the years in obtaining optimal scalability of elliptic solvers on the most current architectures in various application areas, most notably for elliptic problems from atmospheric flow discretised on latitude-longitude grids up to 256 cores, as well as his status as one of the world's leading theoretical analysts of multilevel iterative elliptic solvers and his links to other world leading groups in this field, mean that that he is ideally equipped to achieve this goal.

The proposed research will have enormous impact on all those fields where semi-implicit schemes are needed to treat the diffusive parts of the system and where solving elliptic systems is the most costly computational component of the analysis.

Who? The focus of the research is on applying existing and on developing new highly efficient parallel techniques for solving elliptic systems that arise when using a semi-implicit approach to global modelling of atmospheric flow, ensuring optimal scalability on modern multicore architectures. A number of very important areas would benefit directly from the outcomes:
1. The meteorological and climatological forecasting community, in particular the UK Met Office and other users of the UM and related tools .
2. The oceanographic community who deal with similar models that require scalable elliptic solvers.
3. The water resources management sector, where weather and climate prediction has always played a major role.
4. The military, where accurate weather predictions are crucial to the safety of the troops and to strategic decisions.
5. Other environmental sectors, in particular the subsurface waste management or the carbon capture and storage sectors.
Large elliptic systems are at the heart of many environmental models. They usually constitute the bottleneck in large scale computations, particularly on massively parallel computers, and thus all the areas above would benefit directly from the proposed research.

How? In computational approaches to model physical systems, semi-implicit treatment of diffusive processes is used to ensure the the stability of the numerical scheme. This approach leads to large elliptic systems that often constitute the most expensive part in the simulation and classical solvers have been observed not to scale well beyond 1000s of processors on massively parallel architectures. Future simulation and forecasting tools, such as the Met Office's dynamical core, will have to address this issue in a more satisfactory manner. Similar comments could be made regarding many of the other areas mentioned above. Thus, when our methods are applicable we expect them to reduce computation times significantly and bring many important problems within reach of analysis. The benefit will be in removing the computational bottleneck of parallel elliptic solvers which are at the heart of many parallel forecasting and simulation tools in engineering and environmental applications.

What? The primary beneficiary of this research will be the weather and climate prediction sector. Transfer of the benefits of the research will be achieved by working closely with the Met Office and with other similar institutions such as the ECMWF in Reading. Nils Wedi from the ECMWF will act as one of the advisers of the project. The University of Bath and the PI in particular have an excellent and effective established relationship with the Met Office. The entire project will be carried out in close collaboration with the other project teams on this programme and with the Met Office. The PDRA and the PI will visit the Met Office regularly, thus keeping them informed constantly about the progress. Regular progress reports will also be posted on the programme Wiki and given at regular programme meetings. In Phase 2 of the programme the methods developed and tested in this project will be considered for inclusion in the next generation dynamical core at the Met Office and thus become also available to the weather and climate forecasting community as a whole through the many links and clients the Met Office has worldwide. In addition we will also disseminate our results at appropriate international meetings.

Measures of success. The key measure will be adoption of semi-implicit schemes in the new dynamical core and their implementation in Phase 2 of the programme. But through our links with the project partners we hope to also see a wider use of the methods which we develop.