GLANCE: GLAnceable Nuances for Contextual Events

This project will develop and validate exciting novel ways in which people can interact with the world via cognitive wearables -intelligent on-body computing systems that aim to understand the user, the context, and importantly, are prompt-less and useful. Specifically, we will focus on the automatic production and display of what we call glanceable guidance. Eschewing traditional and intricate 3D Augmented Reality approaches that have been difficult to show significant usefulness, glanceable guidance aims to synthesize the nuances of complex tasks in short snippets that are ideal for wearable computing systems and that interfere less with the user and that are easier to learn and use.

There are two key research challenges, the first is to be able to mine information from long, raw and unscripted wearable video taken from real user-object interactions in order to generate the glanceable supports. Another key challenge is how to automatically detect user's moments of uncertainty during which support should be provided without the user's explicit prompt.

The project aims to address the following fundamental problems:
1. Improve the detection of user's attention by robustly determining periods of time that correspond to task-relevant object interactions from a continuous stream of wearable visual and inertial sensors.
2. Provide assistance only when it is needed by building models of the user, context and task from autonomously identified micro-interactions by multiple users, focusing on models that can facilitate guidance.
3. Identify and predict action uncertainty from wearable sensing in particular gaze patterns and head motions.
4. Detect and weigh user expertise for the identification of task nuances towards the optimal creation of real-time tailored guidance.
5. Design and deliver glanceable guidance that acts in a seamless and prompt-less manner during task performance with minimal interruptions, based on autonomously built models.

GLANCE is underpinned by a rich program of experimental work and rigorous validation across a variety of interaction tasks and user groups. Populations to be tested include skilled and general population and for tasks that include: assembly, using novel equipment (e.g. an unknown coffee maker), and repair tasks (e.g. replacing a bicycle gear cable). It also tightly incorporates the development of working demonstrations.
And in collaboration with our partners the project will explore high-value impact cases related to health care towards assisted living and in industrial settings focusing on assembly and maintenance tasks.

Our team is a collaboration between Computer Science, to develop a the novel data mining and computer vision algorithms, and Behavioral Science to understand when and how users need support.

Receiving on-the spot training, being able to automatically document industrial processes and being able to better understand users of advanced assistive technology has important and wide spread positive implications across many industries.

It is often ignored the fact that most things that are made, repaired or maintained involve non scripted processes that are performed by a small number of expert, or at least, skilled individuals. This is of concern in many economies and in the UK, lack of training and the ability to increase productivity compared to other economies is of great concern according to the Bank of England [4].

There are many industries within the manufacturing sector that can benefit from methods that with no interruption or extra burden on their workers, allow for the transfer of know-how. Systems that observe, learn and extract the relevant from the irrelevant will be of great help. Training is a prime example of where this process can play a significant role.

High technology sectors such as aerospace have a number of situations where word of mouth is a commonplace yet clearly brittle knowledge transfer method. Consider the case of a turbine engine manufactured by one company, sold to another and maintained by a third party. It is not uncommon that when an engine fails for the first time six years or so after its construction, the small team of about a dozen individuals that built it wont be available to repair it. It can also be the case that no one has repaired a specific issue with such an engine before and thus the first time this is needed will take a substantial amount of time, skill and exploration and thus, high cost. Passing these first and precious few instances of knowledge for training others that will require it is of great potential benefit. The benefit increases further if it simplifies the transmission of this knowledge through geographical locations and across languages and cultures. We have long term aspirations on the impact of the work we are proposed but our industry partner sees some potential benefits in a number of situations in the short to mid term horizon (5-10 years).

Being able to distill from observations of people doing things what is important and being able to present it in manners that are intuitive and unobtrusive is what our project aims to build the foundations for and this has applications beyond industry.

Many perfectly physically able and keen individuals of many ages sharing knowledge that can support one another in-situ will have huge impact on how empowered people feel. This can be hinted by the number of times people resort to trying to find videos on the web on how to do things, and in many cases only to be faced by a difficult sifting process trying to separate experts from amateurs from the irrelevant. Systems that can do this separation automatically will have great social and even cultural implications as people will feel more confident on trying and doing things themselves.

In the longer term, cognitive wearable systems could be of benefit in supporting memory or other neurological conditions. Reminding patients on how to do things in their daily routines and or for supporting their rehabilitation can have positive effects on self esteem and recovery.

Overall, devising more useful methods that understand people uncertainty better and that build models to gain insight on issues such as task relevance or expertise will have many applications anywhere anyone needs help to do or to document actions.