Hardware Acceleration of Co-Simulation for the Study of Extreme Weather Events

This is a proposal for a two-month research visit to the Disaster Prevention Research Institute (DPRI) of the University of Kyoto in Japan, to work with Prof. Tetsuya Takemi of the Atmospheric and Hydrospheric Disasters Division on hardware acceleration of co-simulation of extreme weather events. In particular, our aim is to accelerate co-simulation of the Weather Research and Forecasting (WRF) model and custom simulators such as the Large Eddy Simulator, and to reduce the run times for combined simulations by an order of magnitude. This reduction in run time will allow scientists to perform simulations of extreme weather events at much higher precision.

This visit is a follow-on visit from our previous visit in 2012, which established to collaboration and led to a publication at the HPCS conference.

** Focus of the Project

* Numerical Weather Prediction Models

The particular NWP applications to be used in the proposed work are:
- The Weather Research and Forecasting Model (WRF). This is the leading model in climate research. It is an open-source (http://wrf-model.org/) next-generation mesoscale numerical weather prediction system designed to serve both operational forecasting and atmospheric research needs. However, because of the complexity of its design, the WRF model currently does not make use of hardware acceleration (except for a small number of experimental modules).
- The Large Eddy Simulator is developed at the DPRI specifically to study the effects of severe weather events such as hurricanes on urban areas.

* Co-simulation

The focus of the project is co-simulation, an approach where two simulators run in parallel and the outputs of one simulator serve as inputs for the other. Co-simulation is a very important mechanism to achieve more efficient NWP simulations: many scientists develop custom simulators that however rely on inputs from existing simulators such as WRF. This is in particular the case in the study of severe weather events where the scientists want to change the governing equations: modifying the WRF core is generally not an options because of the complexity of the system. However, the current approach, which involves running WRF and writing the results of the run to a file, then reading the generated data into the custom simulator, is extremely ineffective because of the need to generate huge amounts of data and store them on hard disks. Hard disk access is typically 1000x slower than memory access. Having to read input data from disk at every time step of the simulator results in very slow operation. In co-simulation, the data generated by the first simulator (WRF) is transferred via memory to the second simulator (e.g. LES) at every time step.

* Accelerating the process

The code for the WRF model is very complex. We have shown in that there is scope for accelerating WRF, but it will take several years to have a fully accelerated version of WRF. Consequently, for this research visit, we plan to use WRF in its current form, and run it on a compute cluster using MPI, which is the most efficient way to run WRF.

However, in practice creating a GPU-capable version of a small custom simulator is feasible and an expert can do this in a few weeks (we have for example already created a GPU-capable version of the LES). By running the second simulator on a GPU or other accelerator, we achieve full co-simulation at the speed of the WRF simulation. During the research visit, we want to create the system that will make co-simulation between WRF and a GPU-accelerated simulator possible.

The main purpose of this research visit is to strengthen a long-term collaboration including research groups in Japan and the UK, with the aim of accelerating the full Weather Research and Forecasting (WRF) model and adding support for FPGA acceleration. We describe here the potential impact of the aims of this long-term collaboration.

* Impact of Improved Weather and Climate Modeling

In the longer term the project is expected to have a considerable impact on climate research and weather forecasting by enabling faster, higher-precision simulations. This in its turn is of great potential benefit as it will help reduce the impact of global warming, and specifically severe weather events such as storms and floods. The World Economic Forum (WEF) estimates the cost of such events at more than $250 billion for the next 10 years. According to the WEF study "Global Risks 2011", no other events have a higher likelihood than storms and cyclones, and no other event has a higher estimated cost ($1000 billion) than climate change. Consequently, reducing the economic impact of such events is an absolute top priority for the world economy.

Better forecasting will also result in better utilisation of renewable resources such as wind and wave energy, which will help the UK to meet its carbon emission reduction targets.

Apart from these benefits resulting from the improved weather and climate models, the technology developed can also benefit other areas of the UK economy, as
the approach used for the weather models can be used for many other types of scientific computing.

* Impact of FPGA Accelerated Computing

FPGA acceleration is potentially very high impact because FPGAs have a much lower power consumption than GPUs or conventional CPUs. They provide the ability to deliver increased computational power without the high cost in terms of power consumption, space and equipment incurred by scaling a conventional supercomputer. As such they enable both low-cost access to HPC resources and scalability into exascale performance. "Exascale computing" is the term used for the efforts to scale the performance of the current supercomputers with a factor of 1000. The main problem to be solved is power consumption, and this is why the use of FPGAs is seen as very attractive.

Finally, FPGA acceleration can also be used for general-purpose computing in data centres, and as shown in our recent work with HP (to be published at the ISPASS conference) this can lead to a factor of 10 reduction in Total Cost of Ownership, mainly through the dramatic reduction of the energy bill of the data centre. The technology we aim to develop to for FPGA acceleration of scientific computing applications will make it much easier to create FPGA-accelerated applications and thus lower the threshold to the adoption of FPGAs in data centres.