Uncertainty Estimation and Assessment in Machine Learning

This project relates to EPSRC key research areas "Statistics and applied probability" and "Artificial intelligence technologies". The goal is to explore rigorous ways of assessing uncertainty estimates, and to improve current tools with respect to these metrics. From credit scoring to collision prevention in self-driving cars, reliable uncertainty estimation is crucially important for ethical and safe deployment of machine learning algorithms.

Design of flexible and scalable uncertainty estimators in machine learning has recently received much attention. Unfortunately, most effort concentrated on improving properties of the estimators which are only indirectly related to calibration of the uncertainty estimates. Moreover, the problem of assessing quality of uncertainty estimates has been largely avoided by the machine learning community, but for a few exceptions like Vovk et al., (2006). Contrarily, statisticians have studied this area for several decades (Neyman, 1937; Szabo et al., 2014); we expect their ideas to be indispensable for further progress. Substantial conceptual and algorithmic advances are needed to develop a practical approach applicable to large modern models and datasets. We plan to explore ways of assessing quality of uncetainty estimates provided by modern approximate inference techniques, and use these to design more reliable and safe algorithms. Early signs that this might be feasible have appeared in recent publications (Chwialkowski et al., 2016; Jing et al., 2016).

V. Vovk, A. Gammerman, G. Shafer (2006) Algorithmic Learning in a Random World. Springer.
J. Neyman (1937) Outline of a Theory of Statistical Estimation Based on the Classical Theory of Probability. Philosophical Transactions of the Royal Society of London A, 236: 333-380.
B. Szabo, A. W. van der Vaart, J. H. Zanten (2014) Frequentist coverage of adaptive nonparametric Bayesian credible sets. The Annals of Statistics 43.4: 1391-1428.
K. Chwialkowski, H. Strathmann, A. Gretton (2016) A kernel test of goodness of fit. In Proceedings of the International Conference on Machine Learning (ICML).
L. Jing, M. G'Sell, A. Rinaldo, R. J. Tibshirani, L. Wasserman (2016) Distribution-Free Predictive Inference For Regression. arXiv preprint arXiv:1604.04173.