Automatic Learning of Latent Force Models

Machine Learning (ML) is a rapidly evolving field of research within Computer Science which employs statistical models and learning algorithms in order to derive insights from data. Perhaps the most prominent application of ML is to prediction problems, where by 'training' models on labelled data (the label may be discrete for classification problems, or continuous for regression problems), a model can be constructed which is capable of predicting the label when presented with new data; this is known as supervised learning. On the other hand, unsupervised machine learning techniques aim to either detect patterns in unlabelled data by clustering data-points into groups, or reduce the dimensionality of data for easier visualisation and computation [1]. Gaussian Processes (GPs) are an extremely useful and robust form of ML model, as unlike many other learning techniques, they allow uncertainties in the inference process to be accurately quantified and can be applied to both supervised and unsupervised learning problems. GPs can be thought of as a probability distribution over different functions which may fit a given set of data-points; a prior distribution is specified and after data is observed, a posterior distribution is computed; the posterior allows for predictions to be made with an associated uncertainty that is not usually obtainable via other ML techniques. Perhaps the most useful characteristic of GP models is that they allow for prior knowledge of the system to be incorporated into the prior distribution by means of the covariance function (commonly known as the
'kernel'), and the choice of this function has a large impact on the performance of the model [2].

Latent Force Models (LFMs) and Deep Gaussian Processes (DGPs) are two more recent developments which both build on the fundamental GP model. The former does so by utilising physically inspired covariance functions in the form of differential equations, in order to create a hybrid
model, combining the flexibility of typical data-driven ML models with the improved extrapolation achieved by mechanistic models underpinned by differential equations [3]. The latter combines layers of GPs to create an architecture similar to that of a Deep Neural Network (DNN), resulting in a flexible, non-parametric class of model which is capable of tackling a host of very complex supervised and unsupervised learning problems [4]. My research proposal is centred around a new type of ML model, the Deep Latent Force Model (DLFM), which effectively takes the form of a DGP, but with the GP at each node of the network replaced by a LFM. Such a model would combine the impressive extrapolation qualities of LFMs and their ability to accurately model physical systems, with the scalability to large datasets and complex systems that DGPs provide. Combining these desirable features into a single model is a research objective well worth pursuing, due to the new applications which such features may facilitate, especially in the fields of robotics, air pollution and neuroscience.