GLANCE: GLAnceable Nuances for Contextual Events

Lead Research Organisation: University of Bristol
Department Name: Computer Science

Abstract

This project will develop and validate exciting novel ways in which people can interact with the world via cognitive wearables -intelligent on-body computing systems that aim to understand the user, the context, and importantly, are prompt-less and useful. Specifically, we will focus on the automatic production and display of what we call glanceable guidance. Eschewing traditional and intricate 3D Augmented Reality approaches that have been difficult to show significant usefulness, glanceable guidance aims to synthesize the nuances of complex tasks in short snippets that are ideal for wearable computing systems and that interfere less with the user and that are easier to learn and use.

There are two key research challenges, the first is to be able to mine information from long, raw and unscripted wearable video taken from real user-object interactions in order to generate the glanceable supports. Another key challenge is how to automatically detect user's moments of uncertainty during which support should be provided without the user's explicit prompt.

The project aims to address the following fundamental problems:
1. Improve the detection of user's attention by robustly determining periods of time that correspond to task-relevant object interactions from a continuous stream of wearable visual and inertial sensors.
2. Provide assistance only when it is needed by building models of the user, context and task from autonomously identified micro-interactions by multiple users, focusing on models that can facilitate guidance.
3. Identify and predict action uncertainty from wearable sensing in particular gaze patterns and head motions.
4. Detect and weigh user expertise for the identification of task nuances towards the optimal creation of real-time tailored guidance.
5. Design and deliver glanceable guidance that acts in a seamless and prompt-less manner during task performance with minimal interruptions, based on autonomously built models.

GLANCE is underpinned by a rich program of experimental work and rigorous validation across a variety of interaction tasks and user groups. Populations to be tested include skilled and general population and for tasks that include: assembly, using novel equipment (e.g. an unknown coffee maker), and repair tasks (e.g. replacing a bicycle gear cable). It also tightly incorporates the development of working demonstrations.
And in collaboration with our partners the project will explore high-value impact cases related to health care towards assisted living and in industrial settings focusing on assembly and maintenance tasks.

Our team is a collaboration between Computer Science, to develop a the novel data mining and computer vision algorithms, and Behavioral Science to understand when and how users need support.

Planned Impact

Receiving on-the spot training, being able to automatically document industrial processes and being able to better understand users of advanced assistive technology has important and wide spread positive implications across many industries.

It is often ignored the fact that most things that are made, repaired or maintained involve non scripted processes that are performed by a small number of expert, or at least, skilled individuals. This is of concern in many economies and in the UK, lack of training and the ability to increase productivity compared to other economies is of great concern according to the Bank of England [4].

There are many industries within the manufacturing sector that can benefit from methods that with no interruption or extra burden on their workers, allow for the transfer of know-how. Systems that observe, learn and extract the relevant from the irrelevant will be of great help. Training is a prime example of where this process can play a significant role.

High technology sectors such as aerospace have a number of situations where word of mouth is a commonplace yet clearly brittle knowledge transfer method. Consider the case of a turbine engine manufactured by one company, sold to another and maintained by a third party. It is not uncommon that when an engine fails for the first time six years or so after its construction, the small team of about a dozen individuals that built it wont be available to repair it. It can also be the case that no one has repaired a specific issue with such an engine before and thus the first time this is needed will take a substantial amount of time, skill and exploration and thus, high cost. Passing these first and precious few instances of knowledge for training others that will require it is of great potential benefit. The benefit increases further if it simplifies the transmission of this knowledge through geographical locations and across languages and cultures. We have long term aspirations on the impact of the work we are proposed but our industry partner sees some potential benefits in a number of situations in the short to mid term horizon (5-10 years).

Being able to distill from observations of people doing things what is important and being able to present it in manners that are intuitive and unobtrusive is what our project aims to build the foundations for and this has applications beyond industry.

Many perfectly physically able and keen individuals of many ages sharing knowledge that can support one another in-situ will have huge impact on how empowered people feel. This can be hinted by the number of times people resort to trying to find videos on the web on how to do things, and in many cases only to be faced by a difficult sifting process trying to separate experts from amateurs from the irrelevant. Systems that can do this separation automatically will have great social and even cultural implications as people will feel more confident on trying and doing things themselves.

In the longer term, cognitive wearable systems could be of benefit in supporting memory or other neurological conditions. Reminding patients on how to do things in their daily routines and or for supporting their rehabilitation can have positive effects on self esteem and recovery.

Overall, devising more useful methods that understand people uncertainty better and that build models to gain insight on issues such as task relevance or expertise will have many applications anywhere anyone needs help to do or to document actions.
 
Description We have developed new algorithms for the processing and parsing of wearable camera imagery. These have been published in to Computer Vision conferences and we will move next to further impact in this area.
Exploitation Route We envisage that when we have completed our investigations, new knowledge on ways to process wearable camera information as well as datasets will be produced. These we will aim to introduce to industry and societal stakeholders for dissemination and potential application.
Sectors Aerospace, Defence and Marine,Digital/Communication/Information Technologies (including Software),Healthcare

 
Title EPIC Skills dataset 2018 
Description The first dataset that aims to capture varying levels of Skill for people doing daily living tasks. 
Type Of Material Database/Collection of data 
Year Produced 2018 
Provided To Others? Yes  
Impact This will allow us and researchers worldwide to study the problem of visually determining skill. 
 
Title EPIC-Kitchens dataset 
Description The largest dataset in first-person (egocentric) vision; multi-faceted non-scripted recordings in native environments - i.e. the wearers' homes, capturing all daily activities in the kitchen over multiple days. Annotations are collected using a novel `live' audio commentary approach. 32 kitchens - 4 cities Head-mounted camera 55 hours of recording - Full HD, 60fps 11.5M frames Multi-language narrations 39,594 action segments 454,255 object bounding boxes 125 verb classes, 331 noun classes 
Type Of Material Database/Collection of data 
Year Produced 2018 
Provided To Others? Yes  
Impact This dataset is becoming the standard for egocentric research. To date (March 2020), it has been cited 143 times. 
URL https://epic-kitchens.github.io/2020
 
Description Collaboration with Kyoto University 
Organisation University of Kyoto
Country Japan 
Sector Academic/University 
PI Contribution This collaboration with the group of Prof Yuichi Nakamura has lead to exchange visits from Kyoto to Bristol and also to joint publications. The collaboration is still ongoing.
Collaborator Contribution Provided material artefacts for data collection, provided work for papers, and overall research discussions for new directions.
Impact Longfei Chen, Yuichi Nakamura, Kazuaki Kondo, Walterio W. Mayol-Cuevas: Hotspot Modeling of Hand-Machine Interaction Experiences from a Head-Mounted RGB-D Camera. IEICE Transactions 102-D(2): 319-330 (2019) Longfei Chen, Yuichi Nakamura, Kazuaki Kondo, Dima Damen, Walterio W. Mayol-Cuevas: Hotspots Integrating of Expert and Beginner Experiences of Machine Operations through Egocentric Vision. MVA 2019: 1-6 Longfei Chen, Kazuaki Kondo, Yuichi Nakamura, Dima Damen, Walterio W. Mayol-Cuevas: Hotspots detection for machine operation in egocentric vision. MVA 2017: 223-226
Start Year 2017
 
Title Rank aware attention network 
Description Deep computational model for the ranking of skill based on attention. This accompanies the CVPR 2019 paper: Hazel Doughty, Walterio W. Mayol-Cuevas, Dima Damen: The Pros and Cons: Rank-Aware Temporal Attention for Skill Determination in Long Videos. CVPR 2019: 7862-7871. 
Type Of Technology Software 
Year Produced 2019 
Open Source License? Yes  
Impact This software accompanies an academic paper. The impact is still to be measured. 
URL https://github.com/hazeld/rank-aware-attention-network
 
Description EPIC@CVPR2019 workshop organisation 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Organisation of this international workshop where we are leading the egocentric perception community. This event was an official workshop at CVPR 2019.
Year(s) Of Engagement Activity 2019
URL https://www.eyewear-computing.org/EPIC_CVPR19/
 
Description EPIC@ICCV2019 workshop organisation 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Organisation of this international workshop where we are leading the egocentric perception community. This event was an official workshop at ICCV 2019.
Year(s) Of Engagement Activity 2019
URL https://www.eyewear-computing.org/EPIC_ICCV19/
 
Description Egocentric Perception Interaction and Computing Workshop at ICCV 2017 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact We lead the organisation of this workshop that we intend to be a series over the duration of the project and beyond. This new EPIC@X series of workshops aims to bring together the various communities that are relevant to egocentric perception including Computer Vision, Multimedia, HCI and the Visual Sciences and is planned to be held on the major conferences in these fields. EPIC@ICCV will accept Full Papers for novel work, and Extended Abstracts for ongoing or already published work. Both research and application works related to Egocentric Perception, Interaction and Computing are encouraged, including those that can be demonstrated or are in the prototype stages. We co-organized this with colleagues thatw e are reaching out for expanded impact and collaboration from universities in the US, Italy and Germany.
Year(s) Of Engagement Activity 2017
URL http://www.eyewear-computing.org/EPIC_ICCV17/
 
Description Organization of Workshop on egocentric activity EPIC @ ECCV 2018 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact This is another workshop organized on the topic of Egocentric Perception Interaction and Computing. It is the second in the series and there is another one coming up in 2019. These workshops are the only event dedicated to the topic of the grant and attract an international audience at top venues.
Year(s) Of Engagement Activity 2018
URL http://www.eyewear-computing.org/EPIC_ECCV18/