Lifelearn: Unbounded activity and context awareness

The Wearable Computing market is expected to explode, as evidenced in 2014 and early 2015 with a plethora of new products primarily in the sports and fitness domain. Business Insider in 2013 estimated that 300 million units would be shipped by 2018. What makes wearables (and similarly mobile phones) unique is their contextual intelligence: they use sensors to infer users' context, such as location, activities or social interactions. This contextual intelligence allows a fitness tracker to detect by itself that the user is running, walking, or doing push-ups.

We are motivated by the vision of pervasive "wearable smart assistants" that provide situated contextual support in daily life. They may act as "memory reminders" for people with dementia, or encourage healthy behaviours through supportive prompts presented at the right time (e.g. to fight obesity, diabetes, cardiovascular diseases).

This project deals with the heart of any such assistive technology: the ability to recognise general human activities and context from sensors. Current methods can only recognise pre-defined or "closed sets" set of activities and context. This is insufficient for the scenarios outlined above. In such applications, the set of relevant activities is not necessarily known at design-time, as different users tend to have different routines, routines may change as users change interests, and activities may be performed differently, for instance after an injury. Therefore the set of relevant activities and contexts is potentially unbounded and is said to be "open-ended

The project investigates the methods required to recognise an "open-ended" set of activities and contexts from existing wearables, such as a smartwatch and a mobile phone, following lifelong learning principles. In other words, the system should discover that a user engages in a new activity, even if it was not initially programmed with the knowledge of that activity.

We develop new open-ended learning techniques that can model changing number of classes at runtime. These methods run on a recognition infrastructure comprising software on the wearable devices and on a server. The infrastructure will be made open-source to benefit other projects. We develop methods that discover reoccurring wearable sensor patterns. Repeating patterns may correspond to new activities or contexts. Therefore they are modelled using open-ended learning techniques. Finally, we develop methods to decide whether a discovered pattern is meaningful and what it represents. This is achieved by involving the user and occasionally requesting to provide information about his/her current activity. We compare different feedback options that minimise the number of interruptions and the complexity of the queries. Overall, the system is evaluated on existing data as well as on a new long-term dataset collected within this project.


Our approach is novel and timely. Performance increases in activity recognition are incremental and the inability to deal with unknown activities is most critical for large-scale deployments in daily life scenarios. This project addresses this fundamental limit. This is timely given raising costs of healthcare and calls to rely on technology to address this issue. The outcomes of this project along understanding daily human behaviour may lead to new smart assistants that could help support independent living or assist users in following healthy behaviour change. The outcomes may also find their use in psychology research and in the area of sustainable innovation, such as the assessment of consumer-product interaction and behaviour change initiatives. As such the project has clear societal benefits.


This project is supported by our partners Unilever and Plessey Semiconductors, respectively interested in consumer behaviour research and new products in the healthcare domain.

This project is motivated by the vision of pervasive "smart assistants" that provide situated contextual support in daily life, such as "memory reminders" for persons with dementia. At the heart of such technology lies the ability to recognise general human activities and context. This is the key challenge that this project addresses. In the longer term, the outcomes may become fundamental to address societal challenges in assisted living, health monitoring, and smart assistance.

*Economic and industrial impact

This project develops technologies along mobile, wearable, Internet of Things and behavioural analytics. There are major UK technology companies in the field (e.g. ARM, Imagination Technologies) as well as US technology companies with UK branches (e.g. Qualcomm). Behavioural analytics is beneficial for targeted advertisement or consumer behaviour analysis. The UK has several major companies active in health, nutrition and care (e.g. GSK, Unilever, Body Shop) where behavioural analysis can be used to understand product usage and design new products.

Our project partners Unilever and Plessey Semiconductors illustrate the interest in the technology we develop, which they aim to use respectively for better consumer behaviour research and to create new microelectronics products.

We will reach out to industries through the UK Design Forum (a UK microelectronics forum where companies and academics meet) and the Knowledge Transfer Network of Innovate UK. The PI has also contacts to leading researchers active in companies (Intel, Google, Qualcomm, etc.).

*Societal and quality of life benefits

In the longer term, understanding human daily routines from sensors enables new interventions and services in the field of healthcare. It can support longer independent living; activity monitoring may be used to detect onset of depression; cues may be provided in relevant contextual situations to support a desired lifestyle change (e.g. to fight obesity, diabetes or cardiovascular disease).
The impacts can be major in terms of quality of life for the concerned users. They will also benefit public bodies involved in healthcare (UK NHS, charities) and can increase effectiveness of public services and support evidence-based policies.

*Wider public

There is a growing interest in the general public in "quantified self", i.e. keeping track of one's own routines. Within this project we will inform the general public as to how these technologies operate, their future potential (e.g. with the outcomes of this project), and their possible downsides (e.g. along privacy). We will reach-out to the wider public through talks at existing meetup events and university open-doors (see pathway to impact for details).

*Skilled workforce
The team (postdoc funded by this project and students indirectly associated to it) will gain useful and transferable skills, especially analytical, presentation and organisational skills, that are applicable to several employment sectors:
- big data analytics
- machine learning
- mobile and wearable sensing and applications
- sensing technologies and internet of things
- client/server architectures
- behaviour analytics

*Impact timescales
The infrastructure we develop in the project will be immediately available after the end of the project and may have immediate impact for companies already active in wearable sensing and our project partners.

Open-ended awareness is a novel paradigm that will take more time to be exploited. Based on previous experience we envision uptake in the scientific community within 3 years from the project start (through the workshops we will organise). Wider industry awareness and uptake may be 3-5 years after the project completion.

*Summary

We disseminate our work through high-impact publications and conferences and will reach out to selected industries directly to achieve impact both in the UK and internationally.