Learning Highly Structured Sparse Latent Variable Models

Technological advances have brought the ability of collecting and analysing patterns in high-dimensional databases. One particular type of analysis concerns problems where the recorded variables indirectly measure hidden factors that explain away observed associations. For instance, the recent National NHS Staff Survey of 2009, taken by over one hundred thousand staff members, contained several questions on job satisfaction. It is only natural that the patterns of observed answers are the result of some common hidden factors that remain unrecorded. In particular, such answers could arguably be grouped by factors such as perceptions of the quality of work practice, support of colleagues and so on, that are only indirectly measured.

In practice, when making sense out of a high-dimensional data source, it is useful to reduce the observations to a small number of common factors. Since records are affected by sources of variability that are unrelated to the actual factors (think of someone having a bad day, or even typing wrong information by mistake), removing such artifacts is also part of the statistical problem. A model that estimates such transformations is said to perform "dimensionality reduction" and "smoothing".

There are a variety of methods to accomplish such tasks. At one end of the spectrum, there are models that assume the data match some very simple patterns such as bell curves and pre-determined factors. Others are very powerful, allowing for flexible patterns and even an infinite number of factors that are inferred from data under some very mild assumptions. The proposed work tries to bridge these extremes: the shortcomings of the very flexible models are subtle but important. In particular, they can be very sensitive to changes in the data - meaning some very different conclusions about the hidden factors might be achieved if a slightly different set of observations is provided. Moreover there are computational concerns: calculating the desired estimates usually requires an iterative process, a process that needs some initial guess about these estimates. So, even for a fixed dataset, results can vary considerably if such an initial guess is not carefully chosen. Our motivation is that if one does have these concerns, one might as well take the trouble of incorporating domain knowledge about the domain. The upshot: we do not aim to be general, and instead target applications where some reasonable domain knowledge exists. In particular, we focus on problems where the hidden targets of interest are pre-specified, but infinitely many others might exist. While we map our data to a fixed space of hidden variables, we provide an approach that is robust to the presence of an unbounded number of other, implicit, common factors. The proposed models are adaptive: they account for possible extra variability between the given hidden factors that would be missed by the simpler models. At the same time, they are designed to be less sensitive to initial conditions while being less sensitive to small changes in the datasets.

Institutions and research groups with high-dimensional data often found themselves with records that are either noisy measurements of some unobserved factors of interest, or heavily confounded by hidden common causes. In this case, estimating a small-dimensional representation of the data is useful for data summarization and visualization; as a smoothing device to estimate relationships between unobservable factors; and as an alternative representation for imputing missing values and providing features that can be used for prediction tasks and clustering.

In particular, applications that fit well the assumptions of our proposal are processes where large surveys are collected - be it in the government (NHS, Home Office and other departments), industry (marketing and employee surveys) and health services (questionnaires applied to patients and staff). Another source of applications come from natural scientists that have theories on how the data were generated (say, postulating hidden functional modules of the cell that cause gene expression levels) and want to understand the consequences of their assumptions. Moreover, in financial domains, a substantial number of variables correspond to entities such as assets (relatively easy to categorize as belonging to particular sectors of the economy) and market indicators (designed to measure theoretical market factors).

Essentially, we plan to change how data analysis practice is done in domains where variables follow a natural and relatively simple partition, but one that is arguably not perfect and can be improved by allowing for residual associations due to infinitely many other factors. In many cases, this natural partition is often implied by the way data collection was designed. For instance, a company interested in market segmentation may find more useful (say, in a predictive sense or by providing insights that translate more directly to policy making) to generate latent embeddings of its customers according to a pre-determined set of factors used in the very design of the questionnaire - as opposed to (say) having an potentially unbounded number of factors being generated by a non-parametric model. However, since the analyst cannot predict the infinitely other implicit factors that explain the associations in the data, the model has to be robust to these other possible factors. The resulting pre-determined latent variables retain their interpretability, but now can fit the data better.

It is certainly the case that several applications do not fit this format, and there is definitely no shortage of very important domains where non-parametric models for latent structure should be the approach of the choice. In any case, it is only sensible that we provide a choice: often, sophisticated statistical methods for dimensionality reduction are ignored by the practitioner because they promise a level of automation and flexibility that simply is not there. The burden of providing information to the model is simply postponed to the stage of trying to make sense of the resulting factors. Where applicable, the philosophy of this proposal is to fit the data well while making the statistical model respect the goals of the analysis, instead of the opposite.

Quantifying the hidden factors that explain observed associations is a hard task, one of the ultimate goals of science and data analysis in general. The proposal tackles the problem using advanced computational tools while listening to practical needs that are not explicitly exploited in the state of the art. The resulting work aims at showing that a new level of results can be achieved if the science behind such applications is exploited in their full potential.