Evaluating probabilistic predictions in the presence of observation error (Ref: 4165)

Accurate predictions are integral to the success of most of the emerging applications of data science and artificial intelligence. To make successful prediction systems, it is necessary to analyse their predictions in order to monitor performance, guide improvements, demonstrate benefits and inform decisions about their use.

The definitive measures of predictive performance are proper scoring rules, which compute a score for each prediction based on the observed outcome. If outcomes are observed with error, however, then these scores are biased and can favour inferior prediction systems.

Recent research has shown how to construct unbiased proper scoring rules for specific types of scoring rule and observation error. This project will generalise these results to enable unbiased assessments of predictions in a wider range of applications. For example, new scoring rules will be constructed for predictions of multi-dimensional outcomes, time series and spatial fields.

The project will demonstrate the benefits of the new measures by using them to analyse the performance of predictions in fields such as meteorology and economics. Potential benefits include helping to set common standards for evaluating prediction systems, and accelerating improvements in predictions systems by ensuring that better systems are identified and their capabilities are accurately quantified.