AtmosFOAM parallel scaling on HECToR and New Test Cases which expose Grid Scale Oscillations

Lead Research Organisation: University of Reading
Department Name: Meteorology


Some of the best weather and climate forecasting models of the past few decades appear not to work well on the computer architectures of the coming decades. Further increases in computer power need to be accompanied by reduced electricity consumption and this, along with hardware cost, means that future generations of high performance computers will consist of a tens or hundreds of thousands of cheaper, less powerful processing cores, such as are used by computer games. Forecasting models need to be designed differently so that the computation can be distributed efficiently over so many less powerful cores. In designing new forecasting models that will perform well on next generation computers it is essential to compare different aspects of the model independently and it is essential to be able to test simplified model versions rigorously before the entire model is assembled. These two aspects will be covered separately in this project: (1) It is proposed to further develop a simplified model of the global atmosphere: AtmosFOAM, to run on 10,000 processing cores of the UK high performance computer, HECToR. AtmosFOAM is written using the OpenFOAM software library which has already been run on HECToR. Other models written using OpenFOAM have been shown to scale well to 1,000 or more cores on HECToR. It is aimed to reach a horizontal spatial resolution of 15km with 50 vertical levels meaning around 150 million computational points with 10 to 15 thousand points on each core. This problem size should be large enough to scale to 10,000 cores. Two of the bottlenecks to parallel scaling (identified by the Met Office) are semi-implicit time stepping which allows longer time steps but requires synchronised, global communication across all processing cores, and the smoothing of the solutions near the north and south poles where the lines of constant longitude converge. Semi-implicit time stepping is important because it means that the time step does not have to be short enough to resolve sound waves. However if explicit time stepping is used instead in the horizontal, less processor communication is needed but smaller time steps are needed. At some point it will become beneficial to go back to the modelling techniques of the 1980s and use time stepping which is explicit in the horizontal. This project will help to find out when this change in numerical algorithm must be made by comparing the accuracy and parallel performance of the two time stepping schemes in a model which is otherwise identical. A solution to the problem of the convergence of the lines of constant longitude towards the poles is to use a hexagonal-icosahedral grid of the sphere instead of a latitude-longitude grid. This tessellation is more like a football but it divides the sphere into many more hexagons. Parallel performance and accuracy will be compared on the two grid types using a model which is otherwise identical in order to give a clean comparison of the two grids. All code developments will be freely available and open source under the Gnu public licence (GPL). (2) An independent part of the research will be to create new test cases for simplified models of the atmosphere which mimic the strong activity at very small scales that occur in the real world which cannot be resolved in a global model. For example thunderstorms and very steep mountains cannot be fully resolved but can dramatically influence the larger scales. In these new test cases, terms will be added to the equations which mimic these influences at just a single point. These forcings will generate grid scale oscillations or other unrealistic behaviour in inadequate models which otherwise might appear to work well for other existing simple test cases. This could prevent inadequate models from being adopted by operational centres.
Description A number of numerical techniques for weather and climate forecsting models have been analysed and tested. This has informed the UK Met Office of how to write their next weather and climate forecasting model suitable for modern massively parallel computer architectures
Exploitation Route They could be used by other operational forecasting centres.
Sectors Aerospace, Defence and Marine,Environment

Description The findings of my research have been used by the UK Met Office to inform the design of the new model.
First Year Of Impact 2003
Sector Aerospace, Defence and Marine
Impact Types Policy & public services

Title AtmosFOAM 
Description A suite of numerical models and methods for atmospheric modelling using emerging numerical techniques 
Type Of Material Computer model/algorithm 
Year Produced 2014 
Provided To Others? Yes  
Impact Papers and collaborative code development with students 
Description Met Office/NERC/STFC Gung Ho project 
Organisation Meteorological Office UK
Country United Kingdom 
Sector Public 
PI Contribution Under this grant, work is ongoing on the Gung Ho project joint between the Met Office, academics from six Universities and STFC to design and build the dynamical core of a new weather and climate forecasting model. In particular, the contributions by Hilary Weller to this project have been joint with Nigel Wood at the Met Office, Sarah-Jane Lock at Leeds (now at ECMWF), John Thuburn at Exeter and Colin Cotter at Imperial College.
Collaborator Contribution A new dynamical core of the global atmosphere is being developed
Impact Lock, S.-J., Wood, N. and Weller, H. (2014) Numerical analyses of Runge-Kutta implicit-explicit schemes for horizontally explicit, vertically implicit solutions of atmospheric models. Quarterly Journal of the Royal Meteorological Society. ISSN 1477-870X doi: 10.1002/qj.2246 Weller, H. and Shahrokhi, A. (2014) Curl free pressure gradients over orography in a solution of the fully compressible Euler equations with implicit treatment of acoustic and gravity waves. Monthly Weather Review. ISSN 1520-0493 doi: 10.1175/MWR-D-14-00054.1 Weller, H. (2013) Non-orthogonal version of the arbitrary polygonal C-grid and a new diamond grid. Geoscientific Model Development, 7. pp. 779-797. ISSN 1991-959X doi: 10.5194/gmd-7-779-2014 Weller, H., Lock, S.-J. and Wood, N. (2013) Runge-Kutta IMEX schemes for the Horizontally Explicit/Vertically Implicit (HEVI) solution of wave equations. Journal of Computational Physics, 252. pp. 365-381. ISSN 0021-9991 doi: 10.1016/ Weller, H. (2012) Controlling the computational modes of the arbitrarily structured C-grid. Monthly Weather Review, 140 (10). pp. 3220-3234. ISSN 1520-0493 doi: 10.1175/MWR-D-11-00221.1 Weller, H., Thuburn, J. and Cotter, C. J. (2012) Computational modes and grid imprinting on five quasi-uniform spherical C-grids. Monthly Weather Review, 140 (8). pp. 2734-2755. ISSN 1520-0493 doi: 10.1175/MWR-D-11-00193.1
Start Year 2011
Description Talk at Parliamentary Links Day 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Policymakers/politicians
Results and Impact I gave a talk on:
"Gung-Ho: Designing a New Weather and Climate Forecasting Model"
I do not know who was in the audience. No time for questions was given afterwards.

not known
Year(s) Of Engagement Activity 2011