Multi-task and multi-modal deep learning on heterogenous mobile sensing data

The underlying objective of this PhD is the development of novel machine learning models and techniques tailored to high-dimensional and noisy measurements that come from mobile and wearable devices, that can incorporate data of dynamic and static nature, of multiple tasks and of varying level of sparsity, for the ultimate aim of improving health and wellbeing.
The focus will be on neural networks and similar architectures that involve representation learning and non-linear interactions between input data. My initial experiments are based on the Emotionsense dataset, attempting to infer mood using mobile sensors, self-reports, and various metadata. The following three broad topics form the core of my research work for the duration of my PhD: -Combine dynamic-static features with multi-modal models.
Prior work has incorporated multiple features for the modelling of mood but either the original sensor time series were transformed to features through laborious feature engineering or there were no extra static metadata used. My main hypothesis here is that we can learn informative features by using the raw time-series along with static features.
Multi-modal deep learning has been proved successful when there is some relationship between the modalities, e.g. audio-visual speech classification performs better by learning representations for speech audio which are coupled with videos of the lips. Mobile sensors like the accelerometer and the GPS are inherently related since they come from the same device, both tracking movement. Therefore, the aim of this topic would be to develop novel algorithms for better user modeling. A possible improvement on the combination of heterogeneous features presents a broader contribution beyond mobile sensing, being applicable to potentially every data-driven field that deals with variables of different nature.
-Exploit similar tasks with multi-task learning.
The modularity of neural networks enables the learning of similar tasks in parallel (e.g. predict both dimensions of mood -happiness and calmness- in the same model). They do so by either shaping the data as multidimensional tensors so that the final layers output multiple sequences or by building a network with individual output branches or forks that optimize different losses and back-propagate the error to the shared layers. Multi-task learning has been proven successful across diverse applications, from computer vision, language and speech processing to drug discovery. Therefore, the aim of this topic would be to develop novel models that exploit similar tasks to improve mood prediction and create robust, generalizable personalization at scale.
-Deal with data scarcity and model interpretability.
Mobile sensing studies collect fine-grained, multi-modal data from multiple sources which usually are noisy or missing. Increasingly, such studies take place in the wild over long periods of time, where the issue of missing data is aggravated. For example, the Emotionsense study which tracked users' behaviour through their smartphone sensors and asked them to report their mood, was running for almost 3 years. During that time participants were likely to not always carry their phones, and sometimes miss filling out surveys and reports. While more data sources can lead to better results, this comes at a price as each noisy source that is added, the intersection of data-points with clean data of each modality becomes smaller and smaller. This problem of missing data becomes even more distinct as the period of data collection and the number of tracked individuals grow. Thus, the aim of this topic would be to develop novel techniques to overcome the dramatic data sparsity in longitudinal mobile studies and provide interpretable insights on the time-series level.