Hardware Acceleration of Co-Simulation for the Study of Extreme Weather Events

Lead Research Organisation: University of Glasgow
Department Name: School of Computing Science


This is a proposal for a two-month research visit to the Disaster Prevention Research Institute (DPRI) of the University of Kyoto in Japan, to work with Prof. Tetsuya Takemi of the Atmospheric and Hydrospheric Disasters Division on hardware acceleration of co-simulation of extreme weather events. In particular, our aim is to accelerate co-simulation of the Weather Research and Forecasting (WRF) model and custom simulators such as the Large Eddy Simulator, and to reduce the run times for combined simulations by an order of magnitude. This reduction in run time will allow scientists to perform simulations of extreme weather events at much higher precision.

This visit is a follow-on visit from our previous visit in 2012, which established to collaboration and led to a publication at the HPCS conference.

** Focus of the Project

* Numerical Weather Prediction Models

The particular NWP applications to be used in the proposed work are:
- The Weather Research and Forecasting Model (WRF). This is the leading model in climate research. It is an open-source (http://wrf-model.org/) next-generation mesoscale numerical weather prediction system designed to serve both operational forecasting and atmospheric research needs. However, because of the complexity of its design, the WRF model currently does not make use of hardware acceleration (except for a small number of experimental modules).
- The Large Eddy Simulator is developed at the DPRI specifically to study the effects of severe weather events such as hurricanes on urban areas.

* Co-simulation

The focus of the project is co-simulation, an approach where two simulators run in parallel and the outputs of one simulator serve as inputs for the other. Co-simulation is a very important mechanism to achieve more efficient NWP simulations: many scientists develop custom simulators that however rely on inputs from existing simulators such as WRF. This is in particular the case in the study of severe weather events where the scientists want to change the governing equations: modifying the WRF core is generally not an options because of the complexity of the system. However, the current approach, which involves running WRF and writing the results of the run to a file, then reading the generated data into the custom simulator, is extremely ineffective because of the need to generate huge amounts of data and store them on hard disks. Hard disk access is typically 1000x slower than memory access. Having to read input data from disk at every time step of the simulator results in very slow operation. In co-simulation, the data generated by the first simulator (WRF) is transferred via memory to the second simulator (e.g. LES) at every time step.

* Accelerating the process

The code for the WRF model is very complex. We have shown in that there is scope for accelerating WRF, but it will take several years to have a fully accelerated version of WRF. Consequently, for this research visit, we plan to use WRF in its current form, and run it on a compute cluster using MPI, which is the most efficient way to run WRF.

However, in practice creating a GPU-capable version of a small custom simulator is feasible and an expert can do this in a few weeks (we have for example already created a GPU-capable version of the LES). By running the second simulator on a GPU or other accelerator, we achieve full co-simulation at the speed of the WRF simulation. During the research visit, we want to create the system that will make co-simulation between WRF and a GPU-accelerated simulator possible.

Planned Impact

The main purpose of this research visit is to strengthen a long-term collaboration including research groups in Japan and the UK, with the aim of accelerating the full Weather Research and Forecasting (WRF) model and adding support for FPGA acceleration. We describe here the potential impact of the aims of this long-term collaboration.

* Impact of Improved Weather and Climate Modeling

In the longer term the project is expected to have a considerable impact on climate research and weather forecasting by enabling faster, higher-precision simulations. This in its turn is of great potential benefit as it will help reduce the impact of global warming, and specifically severe weather events such as storms and floods. The World Economic Forum (WEF) estimates the cost of such events at more than $250 billion for the next 10 years. According to the WEF study "Global Risks 2011", no other events have a higher likelihood than storms and cyclones, and no other event has a higher estimated cost ($1000 billion) than climate change. Consequently, reducing the economic impact of such events is an absolute top priority for the world economy.

Better forecasting will also result in better utilisation of renewable resources such as wind and wave energy, which will help the UK to meet its carbon emission reduction targets.

Apart from these benefits resulting from the improved weather and climate models, the technology developed can also benefit other areas of the UK economy, as
the approach used for the weather models can be used for many other types of scientific computing.

* Impact of FPGA Accelerated Computing

FPGA acceleration is potentially very high impact because FPGAs have a much lower power consumption than GPUs or conventional CPUs. They provide the ability to deliver increased computational power without the high cost in terms of power consumption, space and equipment incurred by scaling a conventional supercomputer. As such they enable both low-cost access to HPC resources and scalability into exascale performance. "Exascale computing" is the term used for the efforts to scale the performance of the current supercomputers with a factor of 1000. The main problem to be solved is power consumption, and this is why the use of FPGAs is seen as very attractive.

Finally, FPGA acceleration can also be used for general-purpose computing in data centres, and as shown in our recent work with HP (to be published at the ISPASS conference) this can lead to a factor of 10 reduction in Total Cost of Ownership, mainly through the dramatic reduction of the energy bill of the data centre. The technology we aim to develop to for FPGA acceleration of scientific computing applications will make it much easier to create FPGA-accelerated applications and thus lower the threshold to the adoption of FPGAs in data centres.


10 25 50
Description We have found that it is possible to combine models into larger simulation systems using a novel method that is better suited for future manycore systems.
Exploitation Route This was a 2-month research visit so the bulk of the work still remains to be done.
Sectors Agriculture, Food and Drink,Energy,Environment,Healthcare,Pharmaceuticals and Medical Biotechnology

Description The work has been received with interest in particular by insurance companies interested in flood predictions, as this requires coupling of weather models with models of the soil and river systems.
First Year Of Impact 2014
Sector Financial Services, and Management Consultancy
Impact Types Economic

Description Royal Academy of Engineering Distinguished Visiting Fellowship
Amount £4,000 (GBP)
Funding ID DVF1617\5\92 
Organisation Royal Academy of Engineering 
Sector Learned Society
Country United Kingdom
Start 09/2016 
End 11/2016
Description Collaboration with Aizu University 
Organisation University of Aizu
Country Japan 
Sector Academic/University 
PI Contribution We collaborate with climatologist Prof. Hameed on model coupling for the study of the Indian Ocean Dipole. Our know-how is the acceleration, parallelisation and coupling of climate models.
Collaborator Contribution Prof. Hameed is the subject expert. He selects the models, defines the coupling scenarios, provides input on how to build and run the simulations etc.
Impact There are no outcomes yet. This work was funded by the Royal Academy of Engineering.
Start Year 2016
Description Collaboration with Kyoto University Disaster Prevention Research Institute 
Organisation University of Kyoto
Department Disaster Prevention Research Institute (DPRI)
Country Japan 
Sector Academic/University 
PI Contribution We collaborate with Prof Takemi on acceleration and parallelisation of athmospheric simulations as well as coupling different simulators. DPRI does not have the know-how to do this.
Collaborator Contribution Prof Takemi and his team are the subject experts. The provide the simulation code as and give us training in how to build and use them, and provide inputs on alternative implementations of algorithms etc. Quite simply we can't do this work without them.
Impact This is a multi-disciplinary collaboration and the outputs are publications as well as simulation software e.g. https://github.com/wimvanderbauwhede/MPI-LES and https://github.com/wimvanderbauwhede/LES and https://github.com/wimvanderbauwhede/gmcf
Start Year 2014
Title Glasgow Model Coupling Framework 
Description Model Coupling is increasingly important in fields such as climate science: individual models are already hugely complicated, so a conventional integration by merging the code bases would require a very large effort by a highly specialised team. Instead, exchanging information between running models is a more practical approach. The aim of GMCF is to make Model Coupling easier and more suited to modern heterogeneous manycore architectures. Our approach is to use modern language, compiler and runtime technologies so that in the end the user only has to write a scenario to realise the coupling. This is a long-term goal (many years). 
Type Of Technology Software 
Year Produced 2014 
Open Source License? Yes  
Impact The main impacts are is a collaboration, funded by the Royal Academy of Engineering, with Prof. Hameed of Aizu University on the study of the effect of the Indian Ocean Dipole on the El Niño Southern Oscillation. This work is currently in progress. 
URL https://github.com/wimvanderbauwhede/gmcf