Multimodal machine learning approaches for measuring mental wellbeing using sensor and online data

Previous studies have demonstrated the advantages of physical activity and sleep on mental health and also provided an association between virtual behaviour, such as social media use and screen time, and mental health problems. We believe that a person's physical or online behaviour alone might not be a reliable indicator of their mental health. Therefore, this study focuses on combining physical and virtual behaviour of individuals to better access mental health. This study involves building machine learning algorithms to predict mental wellbeing based on passively sensed behavioural patterns. We aim to assess which behavioural features provide the most important information for predicting mental wellbeing. Additionally, we investigate if machine learning models that include both physical and virtual behaviour can better predict mental health. This study is conducted by using data collected via a custom-made app. This app is designed to run unobtrusively in the background of an individual's smartphone. It also provides a platform via ecological momentary assessment (EMA) for users to register information about their emotions and other important events.

A significant aspect of this research lies in data collection. Data will be collected using a custom mobile phone app involving both physical (e.g., accelerometer data, GPS, sleep data) and online (e.g., app usage, screen time, social media use) behaviour and self-reported EMA data (e.g., stress, mood, happiness). The second contribution of this research will be to develop novel state-of-the-art multimodal federated machine learning algorithms that can effectively utilise the collected data to predict mental well-being (using the EMA as ground truth). The third contribution will be to address privacy-focused models for users' data privacy and security. The final contribution informs the previous aspects and will involve consultation with all stakeholders to inform data collection and machine learning applications. The key novelty of this research project lies in the development of novel multimodal machine learning and privacy-driven models that can fuse and learn from physical and online behaviour effectively, in order to better predict mental health status.

Impact on Health and Care
The CDT primarily addresses the most pressing needs of nations such as the UK - namely the growth of expenditure on long term health conditions. These conditions (e.g. diabetes, depression, arthritis) cost the NHS over £70Bn a year (~70% of its budget). As our populations continue to age these illnesses threaten the nation's health and its finances.

Digital technologies transforming our world - from transport to relationships, from entertainment to finance - and there is consensus that digital solutions will have a huge role to play in health and care. Through the CDT's emphasis on multidisciplinarity, teamwork, design and responsible innovation, it will produce future leaders positioned to seize that opportunity.

Impact on the Economy
The UK has Europe's 2nd largest medical technology industry and a hugely strong track record in health, technology and societal research. It is very well-placed to develop digital health and care solutions that meet the needs of society through the creation of new businesses.

Achieving economic impact is more than a matter of technology. The CDT has therefore been designed to ensure that its graduates are team players with deep understanding of health and social care systems, good design and the social context within which a new technology is introduced.

Many multinationals have been keen to engage the CDT (e.g. Microsoft, AstraZeneca, Lilly, Biogen, Arm, Huawei ) and part of the Director's role will be to position the UK as a destination for inwards investment in Digital Health. CDT partners collectively employ nearly 1,000,000 people worldwide and are easily in a position to create thousands of jobs in the UK.

The connection to CDT research will strongly benefit UK enterprises such as System C and Babylon, along with smaller companies such as Ayuda Heuristics and Evolyst.

Impact on the Public
When new technologies are proposed to collect and analyse highly personal health data, and are potentially involved in life or death decisions, it is vital that the public are given a voice. The team's experience is that listening to the public makes research better, however involving a full spectrum of the community in research also has benefits to those communities; it can be empowering, it can support the personal development of individuals within communities who may have little awareness of higher education and it can catalyse community groups to come together around key health and care issues.

Policy Makers
From the team's conversations with the senior leadership of the NHS, local leaders of health and social care transformation (see letters from NHS and Bristol City Council) and national reports, it is very apparent that digital solutions are seen as vital to the delivery of health and care. The research of the CDT can inform policy makers about the likely impact of new technology on future services.

Partner organisation Care & Repair will disseminate research findings around independent living and have a track record of translating academic research into changes in practice and policy.

Carers UK represent the role of informal carers, such as family members, in health and social care. They have a strong voice in policy development in the UK and are well-placed to disseminate the CDTs research to policy makers.

STEM Education
It has been shown that outreach for school age children around STEM topics can improve engagement in STEM topics at school. However female entry into STEM at University level remains dramatically lower than males; the reverse being true for health and life sciences. The CDT outreach leverages this fact to focus STEM outreach activities on digital health and care, which can encourage young women into computer science and impact on the next generation of women in higher education.

For academic impact see "Academic Beneficiaries" section.