Start Making Sense: Cognitive and affective confidence measures for explanation generation using epistemic planning

Lead Research Organisation: Heriot-Watt University
Department Name: S of Mathematical and Computer Sciences

Abstract

Consider a tourist on a walking tour of a city. After reaching a place where they can see they are almost back to the starting point, their tour guide says "Let's go up that hill", pointing to a large hill. "We can get a good view of the city from there." However, on seeing the tired expression on the tourist's face, the guide adds "Or we can stop in that pub over there and take a break." This scenario has two important features: (1) people like to know what is going on. This is especially true in situations where a decision may not have been anticipated or expected. Here, an explanation may be needed not just to justify the decision but also to establish confidence in that choice-in other words, to trust it; (2) being able to read the situation and adapt to the needs of the moment is important when ranking the possible actions that could be taken. Here, a decision may need to be made dynamically. These two features add up to 'dynamic trust maintenance', and are needed for a broad range of the AI systems that are expected to be deployed in the near future, e.g., automated vehicles, service robots, or interactive voice-based assistants.

This project addresses the need for dynamic trust maintenance (which is not generally available in interactive and autonomous systems) by bringing together experimental research in cognitive science involving cooperative joint action with the practical construction of AI planning tools to apply to the task of explanation generation. This challenge will be addressed through these concrete objectives: (1) to study cooperative joint action in humans to identify the emotional, affective, or cognitive factors that are essential for successful human communicative goals, (2) to enhance epistemic planning techniques with heuristics derived from the cognitive science studies, and (3) to deploy the resulting system to generate human-like explanations and evaluate the effectiveness of the resulting approach with human participants.

Planned Impact

Knowledge. The main scientific advances will be new ways to help people interact with planner-based artificial intelligences, and new ways to measure people's trust in such systems. Contributions will include: identifying emotional, affective, and cognitive factors relating to levels of trust in explanations; epistemic planning techniques enriched with heuristics derived from the cognitive science studies; and an empirical evaluation of an implemented system which generates human-like explanations. These have benefits for fields of enquiry including: AI, natural language processing, cognitive science, affective computing, and human computer interaction. The plan is to publish 9 papers in journals and conferences, and to run an expert workshop to help bridge communities.

Society. Social beneficiaries of the project include policy makers who act as intermediaries between technical experts and the general publics, on topics such as the "Right to Explanation". However, the most important "society" impact will ultimately be on public end users empowered by the development of explainable AI. Through our tour guide domain of application, we aim to help people to understand better the strengths and weaknesses of current and future AI, and to benefit from the next generation of intelligent advisors. The plan is to run a co-design workshop, and a public engagement event.

People & Capabilities. This project will contribute to EPSRC's planned network around explainable AI and human-like computing in the UK. We will seek to share our work with researchers at the interface between AI and cognitive science, wherever they are located, helping build a critical mass of researchers engaged with R&D in this area. The project team will be mentored in sharpening their academic profiles, supported in commercialisation, and receive transferrable skills training. To deliver education and inspire the next generation, the team will work with excellent students, especially from the Masters in Design Informatics. The plan is to supervise at least 6 student projects.

Economy We will gain impact via the release of data, software, APIs, and documentation in the public domain under an appropriate license. We will seek to initiate work with an SME project partner with a novel biometric sensor system. Via Informatics Ventures (IV), there is effective entrepreneurship training, coaching for spinoffs, and business development support. The plan is to expose a Design Informatics student cohort (~30 students) to the project ideas, and use IV to support at least 1 entrepreneurial student.
 
Description Humans and virtual agents elicit different types of responses during interaction. Understanding the nature of these responses is therefore essential for improving the design of virtual agents that can interact more effectively with humans, and with higher levels of trust. This project explored human-agent interaction in the context of explainability during multiple studies involving humans and virtual agents in task-based situations.

Initial studies explored the relationship between biometric signals (for instance, skin response, heart rate, facial expression, and eye movement) and levels of confidence during task-oriented interactions between a human and a virtual agent. The results of these studies indicated their potential applicability to several different interactive situations: wherever it is desirable for a system to respond in real-time to a user's emotional state, the system could be trained to identify signals of confusion or uncertainty
and respond immediately to remedy the situation.

Follow-on studies involved developing a virtual agent that uses AI planning techniques to automatically respond to humans when uncertainty is detected. Originally planned as lab-based studies with human participants, the studies were moved online due to COVID-19. The plan-based agent required the construction and representation of the agent's strategy, detailing how instructions and explanations were selected for the human user. Since the agent's knowledge was often uncertain, as users do not always follow instructions accurately, the strategy had to concisely capture the appropriate agent behaviour for millions of possible situations. A process for plan construction and representation was therefore developed that enabled the large space of possible choices to be significantly reduced to the low thousands.

One important aspect of this work involved identifying a collection of communication actions for the agent to support the user in their task. This included basic and more detailed instructions, as well as explanations, which provided more information when necessary. For several types of explanations it was discovered that it was possible to associate each situation with an appropriate agent behaviour. For explanations where this was insufficient, small extensions to the agent's strategy could often be used, for
instance by dividing specific situations and allowing them to be treated individually.

Another direction of research explored the generation of plans that require fewer explanations, for instance by avoiding ambiguous instructions. The project developed a novel approach, using cost penalties, to explore the trade-off between plan length and the number of explanations. Experiments using this approach found solutions that reduced the number of ambiguous instructions by an average of 24% (for thousands of simulated users).

Due to the online nature of the later studies, which made gathering user responses and controlling user interaction more difficult, a new software tool was developed to interact with the plan-based agent using a web browser. The web application recorded, stored, and collated user data, including mouse movements and interactions, together with video and audio recorded with the users' computing equipment. The final study using this tool began near the end of the project, with results expected to be published beyond the
project's end date.
Exploitation Route The outcomes from this funding are applicable to the design of any AI system that must interact with humans, especially in situations where the human's reactions are taken into consideration in order for the AI to respond appropriately. This work also introduces new research directions in the study of affective measures, especially in an online setting, as a tool for improving interactive behaviour in human-AI systems. As a result, the work and results of this project are potentially useful to researchers, developers, and practitioners that make use of AI tools and techniques, or design artificial/virtual AI agents.
Sectors Other

URL http://start-making-sense.org/
 
Description UKRI COVID-19 Grant Extension
Amount £25,608 (GBP)
Organisation United Kingdom Research and Innovation 
Sector Public
Country United Kingdom
Start 11/2020 
End 05/2021
 
Description Using AI-Enhanced Social Robots to Improve Children's Healthcare Experiences
Amount £507,614 (GBP)
Funding ID ES/T012986/1 
Organisation Economic and Social Research Council 
Sector Public
Country United Kingdom
Start 02/2020 
End 01/2023
 
Title Online user study research tool 
Description The COVID-19 situation made the use of in-person interviewing and testing impossible. Instead, a novel online user study research tool was developed and deployed. Using this tool, the user is able to interact with a virtual agent through the Internet and complete the tasks in the study, using a web browser. The web application then records, stores, and collates user data, such as mouse movements and interactions, as well as video and audio recorded using the users' computing equipment. The research tool is adaptable and extensible, and can be reused for similar user studies in the future. It is fully integrated with an already existing online research questionnaire application. 
Type Of Material Improvements to research infrastructure 
Year Produced 2020 
Provided To Others? Yes  
Impact This research tool was used to facilitate the planned studies on the project (albeit with restrictions on the type of data collected), as a mitigation strategy in response to the COVID-19 situation. 
 
Title HTML5 research tool for online user studies 
Description A novel research tool was developed in HTML5 for conducting online user studies. Fully integrated with an existing research questionnaire application, the software allows the user online access to an artificial intelligence agent, while recording such user interactions and data as mouse movements and interactions (clicks, etc.), and recorded video and audio data of the user. A server-side back-end in PHP stores and collates the encrypted data in safe storage. The software is not currently licenced for release but plans are being made to make the software available after the end of the project. 
Type Of Technology Webtool/Application 
Year Produced 2020 
Impact The software has enabled the user studies on the project to take place in an online setting (albeit with some additional limitations on the kind of data that is collected), as a response to the COVID-19 situation. 
 
Description Inited talk (online) to the UKRI Trustworthy Autonomous Systems Node on Trust - Start Making Sense! Cognitive and Affective Confidence Measures for Explanation Generation using Epistemic Planning 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Other audiences
Results and Impact Invited online talk to the UKRI Trustworthy Autonomous Systems Node on Trust. The talk presented an overview of the Start Making Sense project and its research objectives and outcomes to members of the Node.
Year(s) Of Engagement Activity 2022
 
Description Invited lecture (online) at the ICAPS 2020 Online Summer School on Automated Planning & Scheduling - Planning and Plan Execution for Human-Robot Interaction 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact Invited lecture as part of the 2020 International Conference on Automated Planning and Scheduling Summer School. The event was held online due to COVID-19. The project and its work was mentioned during the talk.
Year(s) Of Engagement Activity 2020
URL https://icaps20subpages.icaps-conference.org/students/summer-school/
 
Description Invited talk (online) for Sapienza University, Rome, Italy - Using Epistemic Planning for Task-Based Human-Robot Interaction 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Postgraduate students
Results and Impact Invited online talk for Sapienza University, Rome, Italy. This event was held online due to COVID-19. Research related to the project was presented as part of this talk.
Year(s) Of Engagement Activity 2020
 
Description Invited talk at Sapienza University, Rome, Italy - Epistemic Planning for Task-Based Action and Human-Robot Interaction 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Other audiences
Results and Impact Invited talk at Sapienza University, Rome, Italy as part of a 3-month Visiting Professorship. Research related to the project was presented as part of this talk.
Year(s) Of Engagement Activity 2019
 
Description Invited talk at the University of Edinburgh, UK - Artificial Intelligence (AI): Should we really be worried about the AI Takeover? 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Postgraduate students
Results and Impact Invited talk as part of the Science & Religion Seminar at New College, University of Edinburgh. The project and its work was mentioned during the talk.
Year(s) Of Engagement Activity 2020
 
Description Opening keynote at SRF 2019 - Artificial Intelligence (AI): Technology, trends, and takeover? 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Opening keynote at the Science and Religion Forum (SRF) 2019 in Durham, UK. The talk was a general talk about artificial intelligence but highlighted work being done on the project.
Year(s) Of Engagement Activity 2019
 
Description Research Workshop: Knowledge Engineering for Planning and Scheduling (KEPS) at ICAPS 2019 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Research workshop organised as part of the 2019 International Conference on Automated Planning and Scheduling (ICAPS), the top automated planning conference, held at UC Berkeley, in Berkeley, California, USA. Approximately 30 researchers (university academics, industrial experts, and postgraduate students) attended the workshop. The automated planning techniques used on the project were a central theme of this workshop. A workshop on a similar theme is planned for the future.
Year(s) Of Engagement Activity 2019
URL https://icaps19.icaps-conference.org/workshops/KEPS/
 
Description Research Workshop: Knowledge Engineering for Planning and Scheduling (KEPS) at ICAPS 2020 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Research workshop organised as part of the 2020 International Conference on Automated Planning and Scheduling (ICAPS), the top automated planning conference, held online this year due to COVID-19. Approximately 65 researchers (university academics, industrial experts, and postgraduate students) attended the workshop. The automated planning techniques used on the project were a central theme of this workshop. A workshop on a similar theme has been accepted for the next ICAPS conference.
Year(s) Of Engagement Activity 2020
URL https://icaps20subpages.icaps-conference.org/workshops/keps/
 
Description Research Workshop: Knowledge Engineering for Planning and Scheduling (KEPS) at ICAPS 2020 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Research workshop proposal accepted as part of the 2020 International Conference on Automated Planning and Scheduling (ICAPS), the top automated planning conference, to be held in Nancy, France. Approximately 30 researchers (university academics, industrial experts, and postgraduate students) are expected to attend the workshop. The planning techniques used on the project are a central theme of this workshop. A workshop on a similar theme is planned for the future.
Year(s) Of Engagement Activity 2020
URL https://icaps20.icaps-conference.org/workshops/keps/
 
Description Research Workshop: UK Planning and Scheduling Special Interest Group (UK PlanSIG) 2020 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Research workshop organised as a regular meeting for the UK automated planning and scheduling research community and international partners, held online this year due to COVID-19. Approximately 30 researchers (university academics and postgraduate students) attended the workshop. The automated planning techniques used on the project were a central theme of this workshop. A workshop on a similar theme is planned for the future.
Year(s) Of Engagement Activity 2020
URL https://plansig2020.wordpress.com/
 
Description Talk (online) to the 2022 Measuring Behavior Conference - Start Making Sense: Predicting Confidence in Virual Human Interactions using Biometric Signals 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Talk by Dr Sara Dalzel-Job in the opening session of the 2022 Measuring Behavior Conference, in the general session on new developments in analysis and statistics.
Year(s) Of Engagement Activity 2022
URL https://archive.measuringbehavior.org/mb2022/analysis.html