Next generation Numerical Weather Prediction: 4DVar ensembles and Particle Filters

Data assimilation is a method to combine numerical models with observations. It is used in all environmental sciences and essential to be able to simulate the real world, instead of a pure model world which has little to do with reality. With the increasing resolution of geophysical models both the size and the nonlinearity of these models increase. Also the number of observations increases and the observation operators, which connect the model variables to the observations, become more and more complex and nonlinear, like new satellite observations and radar observations in weather forecasting. Obviously, the data-assimilation methods have to fully allow for these nonlinearities.

Present-day implementations in numerical weather prediction are all based in linearisations. For example, the (Ensemble) Kalman Filter assumes linear updates, and variational methods like 4DVar solve a weakly nonlinear problem through linear iterations. A further problem with variational methods is that error estimates are hard to obtain, and for highly nonlinear problems inaccurate.

A few operational weather prediction centres have started experimenting with ensembles of 4DVar's. This has the potential of solving the nonlinearity problem, and at the same time provides an error estimate. Recently, the European Centre for Medium Range Weather Forecasts (ECMWF) started experimenting with ensembles of 4DVar solutions, generated by perturbing the observations, with very promising results. It is known from Kalman Filter (or rather Smoother) theory that when this ensemble is cycled through several data-assimilation cycles its spread will approximate the error covariance of the system. In that case, the ensemble is a sample from the correct distribution. However, for a strongly nonlinear system the Kalman filter theory does not hold, and it is unclear what the ensemble means, and there is a strong scientific and operational need to understand what these ensembles mean, and how we can improve them.

On the other hand, it is well-known that we can represent the underlying distributions by a set of particles, i.e. a set of model states, in a so-called particle filter. Particle filters are fully nonlinear both in model evolution and analysis step. A fundamental problem, the so-called 'curse of dimensionality' has hampered their use in geoscience applications. Very recently a solution has been found by the PI that has the potential to revolutionize data assimilation in highly nonlinear geophysical systems (Van Leeuwen, 2010a; Van Leeuwen, 2010b). The latter paper describes applications to relatively simple (up to 1000-dimensional) highly nonlinear systems that previously needed hundreds to thousands of model integrations, and now only of the order of 20 model integrations.

This research proposal explores the possibilities of combining 4DVar ensembles with ideas from Particle Filtering for the next generation numerical weather prediction. A simple and exciting idea is to use 4DVar solutions as particles in the Particle Filter, and this is one of the directions we will investigate. But we will also investigate other ways to generate 4DVar ensembles that are meaningful in nonlinear systems. A strong point is that we will have direct access to the operational ECMWF system, allowing us to efficiently operate between relatively simple academic models and the operational world.

The outcome of this research would benefit the whole geophysical community. First and foremost, the ECMWF and other national meteorological centres like the Met Office will benefit from a very easy implementation of the scheme if successful since their software is used extensively. Furthermore, other meteorological centers, coastal management, air pollution management, oil industry, and insurance companies will all benefit from this research on the long run. All these communities use some form of data assimilation and encounter highly nonlinear problems that need sophisticated data-assimilation methods. The benefit will come from the results we achieve with the new techniques developed in this proposal on real large-scale problems. If the methods are successful the communities will start using them.

Active 'workfloor' collaboration between the University of Reading and ECMWF ensures a quick dissemination of the results in the meteorological and oceanographic communities. Risk Management Solutions in London have shown interest in particle filtering for i.e. hurricane and tornado forecasting in the USA. If successful the developed methods can be implemented straight forwardly in the high-resolution (1km) Unified Model of the Met Office to enhance weather forecasting for extreme events, with enormous potential for health and safety.

Through the NCEO, of which the PI is co theme leader of the Data Assimilation theme efficient exchange of results within the NCEO community is guaranteed. Furthermore, the NCEO Knowledge Exchange manager Andy Shaw will organize workshops to promote NCEO achievements outside NCEO, and results from this project will contribute to that activity. All results will be published in scientific journals and discussed in scientific meetings all over the world to ensure maximal exposure to the scientific community.