Fast Bayesian Deep Learning

Lead Research Organisation: University of Liverpool
Department Name: Electrical Engineering and Electronics

Abstract

Over recent years, Deep Learning has been demonstrated to offer impressive performance across a range of application domains. In part, this is because of its ability to exploit modern many-cored processors (e.g., GPUs).

At its heart, Deep Learning makes use of stochastic gradient ascent, an approach that is known to struggle in the context of both local maxima and (the more numerous) saddle-points, degrading performance possible using a fixed dataset. Techniques have been developed that replace the stochastic gradient ascent with numerical Bayesian methods typified by Markov chain Monte Carlo (MCMC). Unfortunately, while such use of MCMC makes it theoretically possible to always find the global maximum, MCMC is so slow that this is hard to achieve in practice. This has restricted the extent to which MCMC has been used in the context of Deep Learning.

Recent work by the University of Liverpool has developed a variant of the Sequential Monte Carlo (SMC) sampler, as a generic alternative to MCMC. SMC samplers appear to offer the same global optimisation capability as MCMC but with substantially reduced computational cost. This comes about, in part, because SMC samplers are inherently parallel, making it possible to exploit, for example, GPUs.
Deep Learning achieves parallelism by distributing the processing across the data. SMC samplers achieve parallelism by distributing the consideration of the many hypotheses that the data could imply to be true. This studentship will begin by investigating the extent to which a combination of Deep Learning and SMC samplers can be parallelised by simultaneously considering parallelism across the data and the hypotheses. The studentship will then go on to undertake research that is only possible as a result of the existence of fast Bayesian Deep Learning (e.g., related to robust outlier detection).

Publications

10 25 50

Studentship Projects

Project Reference Relationship Related To Start End Student Name
EP/S023445/1 31/03/2019 29/09/2027
2297823 Studentship EP/S023445/1 30/09/2019 29/09/2023 Vincent Beraud
 
Description - New state of the art filter (statistical method often applied to meteorology). It outperforms the current methods in various settings, either with simple or complex models.
- New Artificial Neural Networks, using probabilistic methods to obtain uncertainty estimates. The method is scalable and probabilistically accurate.
Exploitation Route - Improve the sampling method.
- Explore other neural architectures (such as memory).
- Apply the method to lots of industries.
Sectors Aerospace, Defence and Marine,Digital/Communication/Information Technologies (including Software),Financial Services, and Management Consultancy,Security and Diplomacy

 
Description It is used by the Governement for defence and security.
First Year Of Impact 2022
Sector Aerospace, Defence and Marine,Digital/Communication/Information Technologies (including Software),Security and Diplomacy
Impact Types Economic,Policy & public services