Development of a Multimodal Lifelogging Platform to Support Self-Reflection & Monitor Inflammation Associated With the Experience of Negative Emotions

Lead Research Organisation: Liverpool John Moores University
Department Name: Computing and Mathematical Sciences

Abstract

The project will develop a mobile lifelogging platform that will deliver measures of cardiovascular disease in an everyday situation, such as driving a car. Driving represents a common daily activity, where experiences and expressions of anger have implications for health and safety. As such, this activity can be associated with high levels of negative emotions that have a cumulative impact on long-term health.

Lifelogging is the continuous act of recording and documenting our lives, from the things we do, to the places we visit and even our feelings. Wearable cameras and body sensors allow us to capture rich information from multiple data sources about ourselves. As sensors become more prevalent, within our environment, the range of available data is increasing. This has enabled lifelogs to become richer with information and their use in various application domains, such as digital health, is increasing.

The project will explore how multiple streams of physiological and contextual data can be processed and integrated in real-time to detect the user's state. Measures such as heart rate, pulse wave velocity (PWV), speed of the vehicle, location, and first-person photographs of the environment will be brought together to identify instances of anger and inflammation. A range of signal processing approaches will be applied to these data items (e.g. inter-beat interval from the heart rate will be subjected to Fast Fourier Transform) and artefacts will be identified and either removed or incorporated in real-time.

Currently, it is straightforward to log overt aspects of behaviour, such as photographs, location and movement. However, this project will combine those markers with covert changes in cardiovascular physiology, which aren't perceived directly by the user. Hence, the project is extending a person's awareness of their bodies, how their behaviour and reactions to situations are directly impacting their bodies and the triggers for such behaviour, e.g. traffic congestion at a junction may raise our heart rate, without the user being consciously aware of this physiological change. Repeating this stressful behaviour daily, over a sustained period, could contribute to the development of cardiovascular disease.

Reviewing moments when arterial inflammation occurs and understanding the context of this behaviour leads to an enhanced perception of how daily events affect health. This can lead to positive changes to the person's lifestyle, such as avoiding the junction in question to help prevent triggers leading to the onset of cardiovascular disease.

The system will provide a new method to monitor and influence behaviour, which enables us to enhance and bring the field of lifelogging into alignment with advances in digital health. This is achieved using markers that are clinically relevant in the context of lifelogging technologies and developing techniques to process multi-modal signals in real-time. To the best of the author's knowledge, the integration of such biomedical markers that measures physiological changes in context to prevent the onset of disease has not been addressed in any other developments. Overall, the project attempts to reduce a significant real-world problem with an advanced mobile lifelogging platform. The platform will be evaluated in a real-world scenario to assess its capabilities outside of an artificial environment. This will enable us to gauge its robustness as a real and practical solution to log and quantify behaviour. In this way, the data collected will be used to identify moments of arterial inflammation and the context of those times to promote self-reflection and the implementation of behavioural changes.

Planned Impact

The proposed research addresses complex signal processing, data visualisation and mobile architecture design issues. Short term benefits within the next 3 - 10 years includes direct interest from:

1) Commercial Sensor Developers: The wearable technology market is predicted to grow from over $14 billion (2014) to over $70 billion by 2024. Developers building consumer mobile platforms, sensing devices and digital health platforms would benefit from the system as the prototype will provide knowledge to enable the production of such systems to become more advanced and commercialised. This will be achieved through intellectual property (IP) and licensing strategies, facilitated by LJMU's Knowledge Exchange and Commercialisation Team.

2) Charities, e.g. British Heart Foundation (BHF): The BHF is the UK's biggest independent funder of CVD research, currently investing over £88 million over 1,000 projects. The dataset provides valuable insights into the triggers of aggression, which could be used by the BHF and its collaborators. The research team expect to submit a larger collaborative application involving other external groups to the BHF and other aligned charities to maximise the impact of this project.

3) Medical Professionals and the NHS: Medical professionals could harness the information collected by individual's to remotely monitor their patients. Field studies could be setup with the NHS with the aim of identifying patients at risk of developing CVD who could implement behavioural changes to mitigate this occurrence. This would reduce hospital admissions, which costs approx. £4,614 per CVD event, and increase the patient's quality of life.

Indirect beneficiaries:

4) Energy Sector: Modernising our energy system is likely to save the UK £8 billion, within the next 6 years. The technical skills that will be developed during this project (signal processing, data visualisation and sensor systems) are transferrable and can be used by the team on joint projects been industry and academia, which can contribute in the long term to reducing carbon emissions. For instance, modelling the behaviour of a person is directly mirrored in monitoring the behaviour of homes.

Long term benefits within the next 10 - 50 years include direct interest from:

5) The General Public: Allowing individuals to be more in control of their health enables them to live healthier lives and would have a cascading effect on healthcare services. CVD is a substantial contributor to the escalating cost of healthcare in the UK (£8.6 billion), Europe (46 billion EUR) and America ($108.9 billion) annually. Using such a system and being in control of our health, by reflecting and altering behaviour, could reduce hospital admittance and the cost of ongoing care of this condition.

6) Motor Industry: The case study and dataset can be used to understand the physiological responses of driving situations to assess the physiological impacts of stress on the driver. This would enable the development of more sophisticated car systems that could detect and react to the driver and to mitigate these feelings by, for example turning on soothing music.

7) Local Councils and Department for Transport (DfT): Comparing and aggregating the raw data can be generalised to the wellbeing of cities, towns and countries, if the technology is widely adopted. The health of commuters can be to pinpoint potential traffic danger zones. Local councils and the DfT could use this data to improve the roads so that areas are not a hotspot for aggression and stress.

Indirect beneficiaries:

8) Health insurance companies: Providing a system to increase our quality of life could lead to reduced insurance premiums, i.e. the healthier the user the less they pay. Insurance companies already use body mass index (BMI) to calculate premiums. However, using the platform would provide much richer data about our health, which could be used as a benchmark for calculating premiums.
 
Description A mobile lifelogging platform has been developed to measure negative emotional states during real-life driving. The platform utilizes a number of wearable/mobile sensors to collect physiological and contextual lifelogging data from participants on their daily driving commute to and from work. This data includes acceleration, electrocardiogram (ECG) and photoplethysmogram (PPG) signals, as well as location and photographs of the road to ascertain the context of the drive. The development of this platform has met the first objective of the research.

A data processing pipeline has been created to pre-process the sensor data to 1) filter noise, 2) calculate speed from the acceleration data, 3) calculate cardiovascular measures, including heart rate, heart rate variability (HRV) and pulse transit time (PTT), 4) identify and correct artefacts, 5) extract features from the data and 6) synchronize the physiological with the contextual data. The development of these algorithms has met the second objective of the research.

An interactive interface has also been created that transforms the collected multivariate data into an interactive visualisation, so participants can easily interpret the context of the drive for changing patterns of psychophysiology. The purpose of this has been to create a self-reflective tool whereby participants can reflect on their data and perhaps incite behavioural changes. For example, certain junctions might incite negative emotions and so by understanding this, coping strategies may be employed at this point, such as deep breathing. The development of this interface has met the third objective of the research.

In order to assess the validity of our approach, two real-world case studies have been undertaken that have focused on using our platform to collect a variety of lifelogging data from participants on their daily driving commute to and from work:

1. The purpose of study one was a data collection exercise. This study consisted of thirteen participants - seven female and six males, with an age range from 25 to 57 (mean = 42, SD = 12). Each participant undertook the study for five working days and recorded data during each of their driving journeys to and from work, i.e. ten drives (five mornings and five evening) were recorded per participant. This study has resulted in the collection of 408,113,095 instances of raw lifelogging data.

2. The purpose of study two was to assess if the interactive visualisation influenced the participants behaviour. This study consisted of eight participants - six females and two males, with an age range from 28 to 57 (mean = 39.50, SD = 11.10). Each participant collected data for a total of four working days. The protocol required participants to collect data for two days, review their data via the visualization, and then collect data for a further two days. This study has resulted in the collection of 159,496,783 instances of raw lifelogging data.

The completion of these studies has exceeded the fourth objective of the research.

Several publications have resulted from the research, including:

Chelsea Dobbins and Stephen Fairclough, "A Mobile Lifelogging Platform to Measure Anxiety and Anger During Real-Life Driving" in the 2017 IEEE International Conference on Pervasive Computing and Communications (PerCom'17), Kona, Big Island, Hawaii, USA, 13th - 17th March, 2017, pp. 327-332 .DOI: 10.1109/PERCOMW.2017.7917583

Chelsea Dobbins and Stephen Fairclough, "Wearable Sensors, Driving and the Visualization of Cardiovascular Stress During Everyday Life," in the 1st Neuroadaptive Technology Conference 2017 (NAT'17), Berlin, Germany, 19th - 21st July, 2017. http://neuroadaptive.org/files/NAT17_Berlin_Conference_Programme.pdf

Chelsea Dobbins and Stephen Fairclough, "Detecting Negative Emotions During Real-Life Driving via Dynamically Labelled Physiological Data" in 2018 IEEE International Conference on Pervasive Computing and Communications (PerCom'18), 2018 (Accepted)

Further papers are currently either under review or being prepared.

The project has also contributed to the paper:
C. Dobbins and S. Fairclough, "Lifelogging Technologies to Detect Negative Emotions Associated with Cardiovascular Disease," in Applied Computing in Medicine and Health, 1st ed., D. Al-Jumeily, A. Hussain, C. Mallucci, and C. Oliver, Eds. Elsevier, 2015, pp. 27-44.

The project has also been presented at the ESF Science Meeting on Evaluating Personal Lifelogs in Glasgow, which was held in conjunction with The 12th NTCIR Conference that was being held in Tokyo, Japan. The PI was personally invited by the organizers to give a talk about the project. The event was an opportunity for international colleagues to present their research, as well as a networking in the broader field of lifelogging and quantified self.
Exploitation Route The findings have been taken forward into the formulation of a larger EPSRC proposal that is building on the techniques obtained through this project. This project includes collaborations with NHS hospitals/trusts and industry.
Sectors Digital/Communication/Information Technologies (including Software),Electronics,Healthcare

 
Description The project's impact has been facilitated by attendance at a number of regional/national events that have been attended by industry and healthcare professionals. The team has been involved in presenting the project at a number of events for Sensor City. This is a new collaboration between the University of Liverpool and Liverpool John Moores University that aims to create a University Enterprise Zone whereby industry and academia can come together to create commercially viable solutions. One such event that the project has been presented at was, Doing Business with Sensor City, which attracted 100+ delegates from around the nation. The presentation of the project facilitated discussions with colleagues from industry, academia and clinicians about its applications within health and has led to the formulation of another funding proposal, which has been documented in the Other Outputs & Knowledge/Future Steps section. Attendance at another Sensor City event, Could you be EPIC? - Sensor City, has also facilitated talks for larger projects between the team and the international electronics, defense and telecommunications company Plessey. The project has also been presented at the event, Together for manufacturing - the official launch of LCR 4.0. LCR 4.0 is part funded by the European Regional Development Fund (ERDF) and is unique to the region. It allows regional businesses to access practical support and connect with other SMEs, as well as to utilize academic expertise in finding solutions to industrial challenges. This event attracted 50+ delegates from around the region and was composed of local SME's, as well as LCR 4.0 partners and the region's leading manufacturing experts. The team was invited to participate in the event and provided a demonstration of the project, which facilitated discussions with colleagues from industry.
Sector Digital/Communication/Information Technologies (including Software),Healthcare
 
Title Lifelogging Dataset (study 1) 
Description Raw acceleration, electrocardiogram (ECG) and photoplethysmogram (PPG) signals, as well as photographs of the road have been collected from thirteen participants. Data has been collected from each participant for five days during their driving journeys to and from work. This study has resulted in the collection of 408,041,797 instances of raw lifelogging data. This data is held by the University and so may not be shared with others. 
Type Of Material Database/Collection of data 
Provided To Others? No  
Impact The results indicated that low mean speed was associated with increased heart rate, which was indicative of anger due to journey impedance. This preliminary analysis of the data has been reported in the paper: Chelsea Dobbins and Stephen Fairclough, "A Mobile Lifelogging Platform to Measure Anxiety and Anger During Real-Life Driving" in the 2017 IEEE International Conference on Pervasive Computing and Communications (PerCom'17), Kona, Big Island, Hawaii, USA, 13th - 17th March, 2017 (Accepted) 
 
Title Lifelogging Dataset (study 2) 
Description Raw electrocardiogram (ECG) and photoplethysmogram (PPG) signals, as well as location, speed and photographs of the road have been collected from eight participants. Data has been collected from each participant for four days during their driving journeys to and from work. This study has resulted in the collection of 159,496,783 instances of raw lifelogging data. This data is held by the University and so may not be shared with others. 
Type Of Material Database/Collection of data 
Provided To Others? No  
Impact The results of this dataset are currently in progress and will be reported once published. 
 
Title Lifelogging algorithm 
Description A data processing algorithm has been created that pre-processes raw lifelogging data. The algorithm filters noise, calculates measurements of cardiovascular activity, identifies and corrects artifacts and extracts features from the data. 
Type Of Material Computer model/algorithm 
Provided To Others? No  
Impact The results of this algorithm are currently in progress and will be reported once published. 
 
Description International Conference on Physiological Computing Systems 2016 
Organisation Institute for Systems and Technologies of Information, Control and Communication
Country Portugal 
Sector Charity/Non Profit 
PI Contribution Professor Fairclough was the program co-chair for this annual conference in 2016.
Collaborator Contribution Professor Fairclough was involved in the recruitment of keynote speakers, liaison with INSTICC who acted as conference organisers and reviewed papers.
Impact Abraham Otero, Alan Pope, Andreas Holzinger, Hugo Plácido da Silva, Stephen Fairclough (Eds.). 2016. Proceedings of the 3rd International Conference on Physiological Computing Systems. ISBN: 978-989-758-197-7, published by SCITEPRESS
Start Year 2016
 
Description Neuroadaptive Technology Conference 
Organisation Technical University Berlin
Country Germany 
Sector Academic/University 
PI Contribution Stephen Fairclough is the co-organiser of the the first conference on Neuroadaptive Technology to be held in Berlin in July 2017. Professor Fairclough organised this event in collaboration with Technical University of Berlin.
Collaborator Contribution Professor Fairclough planned the event with a German colleague, together they recruited the programme committee and publicised the conference call. The conference will be held in July 2017 and is expected to attract approximately 100 delegates.
Impact None to date
Start Year 2016
 
Description ESF Science Meeting on Evaluating Personal Lifelogs (Glasgow) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact This European invite-only event was held in conjunction with The 12th NTCIR Conference, which was held in Tokyo, Japan. The event brought together international colleagues to present their own approaches to searching and retrieving lifelogging data, as well as a networking in the broader field of lifelogging and quantified self. The PI was personally invited by the organizers to present the project.
Year(s) Of Engagement Activity 2016
 
Description LCR 4.0 Launch (Liverpool) 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Industry/Business
Results and Impact This event was to launch the LCR 4.0 Project, which is a knowledge transfer programme that has been developed to help SME's develop smarter products, smarter processes and smarter supply chains to increase productivity. The event was attended by local SME's, as well as LCR 4.0 partners and the region's leading manufacturing experts. The team was invited to participate in the event and provided a demonstration of the project.
Year(s) Of Engagement Activity 2016
URL http://lcr4.uk/2016/11/29/together-manufacturing-official-launch-lcr-4-0/
 
Description Public engagement - Sensor City 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Industry/Business
Results and Impact Over 100 delegates attended the Doing Business with Sensor City: Focus on Healthcare event, which was a health specific event that was aimed at discussing the practical applications of sensor systems and technologies within business and the role of sensor technologies within the healthcare industry. The day brought together people from businesses, academic and clinical staff to discuss, network and share their knowledge surrounding the development and deployment of sensor systems. The team presented the project and this sparked discussions surrounding the work and its applications within health.
Year(s) Of Engagement Activity 2015
URL https://www.flickr.com/photos/openlabsinfo/sets/72157662613244556