Development of a Multimodal Lifelogging Platform to Support Self-Reflection & Monitor Inflammation Associated With the Experience of Negative Emotions

The project will develop a mobile lifelogging platform that will deliver measures of cardiovascular disease in an everyday situation, such as driving a car. Driving represents a common daily activity, where experiences and expressions of anger have implications for health and safety. As such, this activity can be associated with high levels of negative emotions that have a cumulative impact on long-term health.

Lifelogging is the continuous act of recording and documenting our lives, from the things we do, to the places we visit and even our feelings. Wearable cameras and body sensors allow us to capture rich information from multiple data sources about ourselves. As sensors become more prevalent, within our environment, the range of available data is increasing. This has enabled lifelogs to become richer with information and their use in various application domains, such as digital health, is increasing.

The project will explore how multiple streams of physiological and contextual data can be processed and integrated in real-time to detect the user's state. Measures such as heart rate, pulse wave velocity (PWV), speed of the vehicle, location, and first-person photographs of the environment will be brought together to identify instances of anger and inflammation. A range of signal processing approaches will be applied to these data items (e.g. inter-beat interval from the heart rate will be subjected to Fast Fourier Transform) and artefacts will be identified and either removed or incorporated in real-time.

Currently, it is straightforward to log overt aspects of behaviour, such as photographs, location and movement. However, this project will combine those markers with covert changes in cardiovascular physiology, which aren't perceived directly by the user. Hence, the project is extending a person's awareness of their bodies, how their behaviour and reactions to situations are directly impacting their bodies and the triggers for such behaviour, e.g. traffic congestion at a junction may raise our heart rate, without the user being consciously aware of this physiological change. Repeating this stressful behaviour daily, over a sustained period, could contribute to the development of cardiovascular disease.

Reviewing moments when arterial inflammation occurs and understanding the context of this behaviour leads to an enhanced perception of how daily events affect health. This can lead to positive changes to the person's lifestyle, such as avoiding the junction in question to help prevent triggers leading to the onset of cardiovascular disease.

The system will provide a new method to monitor and influence behaviour, which enables us to enhance and bring the field of lifelogging into alignment with advances in digital health. This is achieved using markers that are clinically relevant in the context of lifelogging technologies and developing techniques to process multi-modal signals in real-time. To the best of the author's knowledge, the integration of such biomedical markers that measures physiological changes in context to prevent the onset of disease has not been addressed in any other developments. Overall, the project attempts to reduce a significant real-world problem with an advanced mobile lifelogging platform. The platform will be evaluated in a real-world scenario to assess its capabilities outside of an artificial environment. This will enable us to gauge its robustness as a real and practical solution to log and quantify behaviour. In this way, the data collected will be used to identify moments of arterial inflammation and the context of those times to promote self-reflection and the implementation of behavioural changes.

The proposed research addresses complex signal processing, data visualisation and mobile architecture design issues. Short term benefits within the next 3 - 10 years includes direct interest from:

1) Commercial Sensor Developers: The wearable technology market is predicted to grow from over $14 billion (2014) to over $70 billion by 2024. Developers building consumer mobile platforms, sensing devices and digital health platforms would benefit from the system as the prototype will provide knowledge to enable the production of such systems to become more advanced and commercialised. This will be achieved through intellectual property (IP) and licensing strategies, facilitated by LJMU's Knowledge Exchange and Commercialisation Team.

2) Charities, e.g. British Heart Foundation (BHF): The BHF is the UK's biggest independent funder of CVD research, currently investing over £88 million over 1,000 projects. The dataset provides valuable insights into the triggers of aggression, which could be used by the BHF and its collaborators. The research team expect to submit a larger collaborative application involving other external groups to the BHF and other aligned charities to maximise the impact of this project.

3) Medical Professionals and the NHS: Medical professionals could harness the information collected by individual's to remotely monitor their patients. Field studies could be setup with the NHS with the aim of identifying patients at risk of developing CVD who could implement behavioural changes to mitigate this occurrence. This would reduce hospital admissions, which costs approx. £4,614 per CVD event, and increase the patient's quality of life.

Indirect beneficiaries:

4) Energy Sector: Modernising our energy system is likely to save the UK £8 billion, within the next 6 years. The technical skills that will be developed during this project (signal processing, data visualisation and sensor systems) are transferrable and can be used by the team on joint projects between industry and academia, which can contribute in the long term to reducing carbon emissions. For instance, modelling the behaviour of a person is directly mirrored in monitoring the behaviour of homes.

Long term benefits within the next 10 - 50 years include direct interest from:

5) The General Public: Allowing individuals to be more in control of their health enables them to live healthier lives and would have a cascading effect on healthcare services. CVD is a substantial contributor to the escalating cost of healthcare in the UK (£8.6 billion), Europe (46 billion EUR) and America ($108.9 billion) annually. Using such a system and being in control of our health, by reflecting and altering behaviour, could reduce hospital admittance and the cost of ongoing care of this condition.

6) Motor Industry: The case study and dataset can be used to understand the physiological responses of driving situations to assess the physiological impacts of stress on the driver. This would enable the development of more sophisticated car systems that could detect and react to the driver and to mitigate these feelings by, for example turning on soothing music.

7) Local Councils and Department for Transport (DfT): Comparing and aggregating the raw data can be generalised to the wellbeing of cities, towns and countries, if the technology is widely adopted. The health of commuters can be to pinpoint potential traffic danger zones. Local councils and the DfT could use this data to improve the roads so that areas are not a hotspot for aggression and stress.

Indirect beneficiaries:

8) Health insurance companies: Providing a system to increase our quality of life could lead to reduced insurance premiums, i.e. the healthier the user the less they pay. Insurance companies already use body mass index (BMI) to calculate premiums. However, using the platform would provide much richer data about our health, which could be used as a benchmark for calculating premiums.