Exploring the modelling of behaviour and context using deep learning under constrained computing platforms with applications to Digital Health

Lead Research Organisation: University of Oxford
Department Name: Computer Science


This project falls within the ESPRC Artificial Intelligence Technologies research area.

The central question of the study is the effective modelling of consumer health to determine user behaviour and context using data-mining methods.
Today's forms of mobile sensing, ranging from phone apps to wearable devices, typically monitor relatively simple dimensions of behaviour and context; for instance, sleep duration and step counts. However, advances in areas like deep learning are demonstrating computational models are possible for much more complex phenomena (e.g., user emotion, social interactions), at a level of robustness that they can be useful in real-world environments. Simultaneously, advances in the computational power of constrained devices (e.g., low-power GPUs, small-form-factor hardware accelerators) are increasing the sophistication of algorithms that are feasible to execute on these platforms.

Data mining has widespread applications as a useful process for extracting meaningful information from large datasets. In particular, its application in the modelling of health data on mobile devices has generated considerable interest. Such interest is chiefly motivated by breakthroughs in both software and hardware, namely deep learning methods and device computational power.

This research will involve an examination of current models and a subsequent software innovation to produce efficient models suited for constrained computing platforms. Current usage of data mining models often involves a trade-off between performance and efficiency. A prudent research question would be to tackle algorithmic redundancies and innovate for methods with relevance to constrained computing platforms such as wearables.

The main objectives to be achieved through this project include, but are not limited to, the following:
- modelling sensor data from constrained platforms using deep learning principles and
algorithms such that the interpretation of user behaviour and context reaches greater breadth and accuracy;
- developing new system resource-efficient deep learning methods suited to
constrained computing platforms (such as wearable devices and embedded platforms);
- investigating potential efficiency gains in deep learning methods through
software/algorithmic innovation or novel hardware/processor directions.

The novelty of the research lies on the potential solutions that might result from experimenting with varying machine learning architectures.

Finally, this project also aligns to EPSRC's strategy in delivering intelligent technologies and systems. The project also adheres to broader Cross-ICT priorities since it seeks to look at real healthcare data, such that the project is ICT-centric but not necessarily solely related to ICT.


10 25 50
publication icon
Radu V (2018) Multimodal Deep Learning for Activity and Context Recognition in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

publication icon
Tong C (2019) Tracking Fatigue and Health State in Multiple Sclerosis Patients Using Connnected Wellness Devices in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Studentship Projects

Project Reference Relationship Related To Start End Student Name
EP/N509711/1 01/10/2016 30/09/2021
1892895 Studentship EP/N509711/1 01/10/2017 31/03/2021 Eu Gen Tong
Description As a result of work funded through this award, I have produced 3 referred conference papers, 1 referred workshop paper and 1 referred poster (please refer to the list of publications). Radu et al (2017) investigate the use of deep learning to model multimodal data for activity and context recognition; Tseng et al (2018) propose binary filters for convolutional neural networks as a means to reduce memory and computation requirements in deploying deep learning; Tong et al (2019) study the use of machine learning to model patient health states using data from connected wellness devices at home. In Tong et al (2018), we presented a study into the use of machine learning to model the Big-Five Personality of users using a Large-scale Networked Mobile and Appliance Data. In Tong et al (2020), we investigate whether accelerometers, and by extension other inertial sensors, are still appropriate for activity recognition, given the rise of imagers (small embedded image sensors).


Conference Papers
C. Tong, M. Craner, M. Vegreville, N. D. Lane. Tracking Fatigue and Health State in Multiple Sclerosis Patients Using Connnected Wellness Devices. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT), Volume 3 Issue 3. Also in Ubicomp '19 and MobiUK '19.

VWS. Tseng, S. Bhattacharya, J. Fernandez Marques, M. Alizadeh, C. Tong, N. D. Lane. Deterministic Binary Filters for Convolutional Neural Networks. The 27th International Joint Conference on Artificial Intelligence (IJCAI '18).

V. Radu, C. Tong, S. Bhattacharya, N. D. Lane, C. Mascolo, M. K. Marina, F. Kawsar. Multimodal Deep Learning for Activity and Context Recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT), Volume 1 Issue 4. Also in Ubicomp '18 and MobiUK '18.

Workshop Papers
C. Tong, S. A. Tailor, N. D. Lane. Are Accelerometers for Activity Recognition a Dead-end? The 21st International Workshop on Mobile Computing Systems and Applications (Hotmobile '20).

C. Tong, GM. Harrari, A. Chieh, O. Bellahsen, M. Vegreville, E. Roitmann, ND. Lane. Poster: Inference of Big-Five Personality Using Large-scale Networked Mobile and Appliance Data. The 15th ACM International Conference of Mobile Systems, Applications, and Systems (MobiSys '18).
Exploitation Route Our studies on using machine learning on connected wellness devices data offer a practical perspective of the model performance on such domains and might prompt future research into further utilizing data generated from these devices (beyond smartphone and smartwatches) for healthcare research. Our work investigating imager-based mobile systems for human activity recognition proposes a new generation of embedded sensors, which might be taken forward by efforts to manufacture mage sensors which are ever smaller; it might also be put to use by researchers interested in automatic detection of Activities of Daily Living (ADL).
Sectors Digital/Communication/Information Technologies (including Software),Healthcare,Leisure Activities, including Sports, Recreation and Tourism

Description BDI wearable camera dataset 
Organisation University of Oxford
Department Big Data Institute
Country United Kingdom 
Sector Academic/University 
PI Contribution I am contributing towards building an activity recognition model using data collected by participants in a wearable camera study conducted by the BDI (CAPTURE-24). My current contribution includes building a preprocessing pipeline for the raw images.
Collaborator Contribution They have conducted the study in 2015 where they recruited participants to use wearable cameras to record their activities. They have compiled a dataset with wearable camera images as well as their activity annotations.
Impact Null.
Start Year 2019