Scrutable Autonomous Systems

Lead Research Organisation: University of Aberdeen
Department Name: Computing Science

Abstract

The aim of the project is to make so-called distributed autonomous systems "scrutable" by humans. Autonomous systems can perform tasks without continuous human guidance. This project considers autonomous systems consisting of many components (called agents) which have to make joint plans on how to act together. It is important that humans can understand why the system behaves in particular ways, for example, why the agents have decided upon a certain plan. To address this, the project will develop and evaluate computational techniques for gathering information about the planning process, including how the various agents interacted, and presenting this information to humans in an understandable way.

To allow information to be gathered easily, the agents will use so-called argumentation techniques: agents engage in dialogues whereby they justify their decision-making as they agree on joint plans, and each decision is based on a collection of arguments in favour and/or against it. This approach has the additional benefit to potentially produce plans faster, because it offers agents an insight into each others' motives. However, this approach will gather much information, and the information will be complex. To allow humans to inspect this information easily, the project will use a combination of Natural Language and diagrams to present it in an understandable way; simplifying, summarising and aggregating information as needed.

For example, our system might explain:

"To achieve goal G, a plan comprising the sequence of actions A, B, C has been agreed upon. Agents P1, P2 and P3 were involved in the preparation of the plan. P1 argued that A had to be done first, because of its standard operating procedures; P2 and P3 agreed to this. P1 suggested that A should be followed by action D, because it could do A and D with low costs, however P2 disagreed, pointing out that action D did not address safety concerns, and suggested that A should be followed by B. No one objected to this. Finally, P1 suggested that B should be followed by C, and that it could perform the action with the help of P3."

If the user requests further details about the safety concerns, she would be presented with the following:

"Current guidelines establish that actions B and D both achieve the same goal, but B does so addressing safety concerns X, Y and Z; D does not address any safety concerns"

The design of a scrutable argumentation-based planning system faces substantial technological challenges. One challenge is to endow distributed planning with the ability to judiciously record its process, whilst creating useful plans quickly. The other is to present the record of the distributed planning process optimally to users. Optimality, in this context, means a combination of clarity and information-richness. We will carry out extensive experiments to find out what aspects of a plan (and the motivation behind it) to emphasise, and where details are better left unsaid. Business partners will help us by taking part in these experiments, and by engaging in discussions about the types of insight users demand.

The project will seek to find solutions that apply across a large range of applications, varying from ones that involve highly complex software agents and robots to ones that involve a plurality of simple, sensor-based agents. We hypothesise that our argumentation-based approach is suitable for modelling disparate scenarios, regardless of their complexity, and that Information Presentation techniques, coupled with state-of-the-art requirements gathering and user-based evaluation, has much to offer across this range.

Planned Impact

Businesses using autonomous systems

Autonomous systems are being used in many UK and multi-national businesses, in particular in the areas of Aerospace, Aviation, Defense and Energy Generation and Supply. As recognized in this funding call, it is of vital importance for the successful deployment of autonomous systems that humans are kept "in the loop" with timely and relevant information and can humans can interact with the autonomous systems effectively. The importance of transparency and user involvement in autonomous systems was also highlighted in a recent US government report which stated that human-performance problems with autonomous systems are exacerbated by "inadequate visibility into how automation is working, inadequate feedback about the automation's activities, lack of facilities for operators to communicate with automation, inadequate explanation of its reasoning processes" (O'Hara & Higgins, 2010, p53). To achieve effective scrutability, it is mandatory to know what information presentations would be useful for different business users, including the required levels of aggregation and summarisation. Dissemination of the results to the wider business community is also vital for the exploitation of the research results.


General public

Autonomous systems have a great potential to be used in businesses. However, the general public is likely to be worried by the potential impact of of time- and safety-critical autonomous systems if they malfunction. It is therefore important to raise public understanding of how such systems work, and how they can be made safer by allowing humans to inspect how the system behaves. Autonomous systems are also a very exciting computer science research area, and for the future of the UK economy it is important that young people are enthused about careers related to computer science.


Early-career researchers

It is of strategic importance to the UK to attract and train early-career researchers in its core areas of expertise, such as distributed autonomous systems (DAS) and information presentation especially natural language generation (NLG). The early-career researchers who will work on this grant (3 research fellows and 1 research assistant) will benefit from learning how to apply their research across the traditional DAS and NLG boundaries, and from learning how to perform research that involves end-users throughout the project. They will also benefit from learning how to engage with business communities and the general public. In addition, we will train other early-career researchers in the topics of this research.

Publications

10 25 50
publication icon
Aldewereld H (2016) Group Norms for Multi-Agent Organisations in ACM Transactions on Autonomous and Adaptive Systems

publication icon
Arieli O (2013) A QBF-based formalization of abstract argumentation semantics in Journal of Applied Logic

publication icon
Awad E (2017) Pareto optimality and strategy-proofness in group argument evaluation in Journal of Logic and Computation

publication icon
Caminada M (2016) Preferred semantics as socratic discussion in Journal of Logic and Computation

publication icon
Caminada M (2017) On the Equivalence between Assumption-Based Argumentation and Logic Programming in Journal of Artificial Intelligence Research

publication icon
Caminada M (2015) On the equivalence between logic programming semantics and argumentation semantics in International Journal of Approximate Reasoning

publication icon
Caminada MWA (2013) Explaining the outcome of knowledge-based systems; A discussion-based approach in Proceedings of the Society for the Study of Artificial Intelligence and the Simulation of Behaviour

 
Description We developed different ways to present complex information contained in plans to humans, using a combination of graphical and textual representations. We discovered that some widely held intuitions regarding information presentation were not confirmed by our experiments (e.g., filtering out some of the plan's steps for a particular purpose does not imply an improvement in understanding). The plans are the outcome of autonomous systems' decision processes; we record the rationale for each step of the plan (using formal argumentation) thus enabling the scrutiny of the system's autonomous operation.
Exploitation Route Any system with autonomous behaviour would need means to expose its internal processes, in order to increase transparency and trust. Our research provides insights and tools to automatically generate plan representations (combining graphics and text), also enabling humans to scrutinise the rationale for the plan. Various sectors adopting autonomy in their processes would benefit from these insights and tools.
Sectors Aerospace, Defence and Marine,Construction,Creative Economy,Digital/Communication/Information Technologies (including Software),Energy,Healthcare,Government, Democracy and Justice,Security and Diplomacy,Transport

URL http://scrutable-systems.org/experiments.html
 
Description Two industrial partners, Schlumberger and BAE Systems, have been working with us to develop a demonstrator exploring use-cases. We have had expressions of interest from other industrial entities, such as Sellafield. It is, however, work in progress. Very importantly, as autonomy becomes more common, we have engaged with businesses and the general public (via various engagement activities) to improve their understanding of autonomous systems, and the need to make such systems more transparent (and scrutable) in order to increase trust in them.
First Year Of Impact 2013
Sector Aerospace, Defence and Marine,Creative Economy,Digital/Communication/Information Technologies (including Software),Energy,Financial Services, and Management Consultancy,Government, Democracy and Justice
Impact Types Economic

 
Title Aggregation 
Description This work discusses the idea, implicit in some treatments of aggregation in Natural Language Generation (NLG), that less redundant linguistic structure s must always be preferable to more redundant structures that express the same information. We demonstrate experimentally that this view is mistaken and argue for a non-directional approach to aggregation that is able to add or remove redundancies depending on a range of factors. We argue that aggregation, understood in this way, is not only relevant for NLG, but also to Summarisation, Text Simplification, and Machine Translation, and to human language production decisions as well. We carried out two experiments to address effects of aggregation. The first used very simple aggregation of the form; "Load the van and the truck and the lorry" compared with unaggregated controls like "Load the van. Load the truck. Load the lorry", and found that the aggregated forms took longer to read than the controls. The second experiment replicated this finding even when more naturalistic aggregation was used, e.g., "Load the van, the truck, and the lorry". Data sets and scripts for analysis in R are provided at the links below. 
Type Of Material Database/Collection of data 
Year Produced 2014 
Provided To Others? Yes  
Impact It has resulted in a manuscript under revision that we submitted to the SQUIB editor for Computational Linguistics 
URL http://www.scrutable-systems.org/experiments.html
 
Title Formal Arguments 
Description User data from experiment studying the application of preferences when presented with conflicting arguments (presented in plain English). 
Type Of Material Database/Collection of data 
Provided To Others? No  
Impact Results published in: F. Cerutti, N. Tintarev, N. Oren. Formal Arguments, Preferences, and Natural Language Interfaces to Humans: an Empirical Evaluation. ECAI'14 (acceptance rate fp: 28%). 
URL http://www.scrutable-systems.org/experiments.html
 
Title Lexical Ambiguity 
Description Work in collaboration with Psychologists in Aberdeen. They had run an experiment on lexical ambiguity, and wanted to build some mixed effects statistical models for their results, which I helped with. The R code for these analyses with documentation in pdf is available at this URL 
Type Of Material Database/Collection of data 
Year Produced 2014 
Provided To Others? Yes  
Impact It has resulted in a journal paper under revision aimed at Language and Cognitive Processes originally. 
URL https://bitbucket.org/matt_green/lcp2014ambiguity
 
Description SASSY-BAES 
Organisation BAE Systems
Country United Kingdom 
Sector Private 
PI Contribution Computational modelling of use-case scenarios, and their use in demonstrator (tool) to showcase research.
Collaborator Contribution Development of use-case scenarios and feedback on early versions of demonstrator (tools).
Impact Computational models used in connection with our demonstrator, and explained in papers. The collaboration is multi-disciplinary, involving psychologists, military personnel and training team.
Start Year 2012
 
Description SASSY-Schlumberger 
Organisation Schlumberger Limited
Department Schlumberger Cambridge Research
Country United Kingdom 
Sector Academic/University 
PI Contribution Computational modelling of use-case and its use in a demonstrator (tool) to showcase research.
Collaborator Contribution Development of use-case scenarios, feedback on early versions of tool and mechanisms.
Impact Models of use-case and their exploitation in demonstrator. It is multi-disciplinary, involving geologists, operators, engineers and scientists working in the oil and energy industries.
Start Year 2012
 
Title Argumentation Demonstrator 
Description The tool allows the user to create an argumentation graph from a knowledge base of rules. This can serve as a domain modelling aid or a teaching tool for principles of formal argumentation. 
Type Of Technology Software 
Year Produced 2014 
Open Source License? Yes  
Impact The tool is still in development but it has triggered a wider interest in the argumentation community with a number of researchers enquiring about further developments. 
URL https://bitbucket.org/rkutlak/aspic
 
Title SASSY Demonstrator Suite 
Description This suite of functionalities demonstrate individual strands of research within SASSY and how these come together. These technologies are demonstrated in an area of AI plan presentation and execution. 
Type Of Technology Software 
Year Produced 2013 
Open Source License? Yes  
Impact The demonstrator is developed continuously throughout the project. Several publications were based on this demonstrator and it is a very helpful tool when explaining the research to industrial contacts (e.g., DEMOfest 2013 (Glasgow), DEMOfest 2014 (Aberdeen, Edinburgh) and meetings with Schlumberger). 
URL https://bitbucket.org/rkutlak/sassy
 
Description Are You Talking to Me? 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? Yes
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact Public engagement presentation explaining research challenges of SASSY and benefits.

http://www.techfestsetpoint.org.uk/tis/uploads/files/programmes/Techfest_2013_programme.pdf

Student inquired about which degrees that would enable them to pursue this type of research.
Year(s) Of Engagement Activity 2013
URL http://scrutable-systems.org/resources/TechFest_2013.pdf
 
Description Argument Based Practical Reasoning in Normative Domains 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Professional Practitioners
Results and Impact A technical talk presenting some project outcomes was given to the Imperial College Logic group.

bla
Year(s) Of Engagement Activity 2014
 
Description BBC Scotland Radio Programme 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Public/other audiences
Results and Impact BBC Scotland invited Dr. Vasconcelos to be interviewed as part of a radio programme on Robots. The main insight was how humans and robots could engage in fruitful dialogues. The programme was broadcast on Monday 3 March 2014 13.30-14.00.

National exposure of team at Aberdeen as well as contribution to the research landscape in Scotland.
Year(s) Of Engagement Activity 2014
 
Description DEMOfest 2014 (Aberdeen) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Professional Practitioners
Results and Impact An engaging discussion with members academics working on related projects as well as discussions of the applicability of the research in existing industrial applications.

We were approached by a member of Industry Technology Facilitator (ITF) regarding possible commercialisation of the system once the project advances further.
Year(s) Of Engagement Activity 2013,2014
URL https://sites.google.com/site/sicsademofest2013/
 
Description Science Superheros 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? Yes
Geographic Reach Local
Primary Audience Public/other audiences
Results and Impact Event organised by ScienceGrrls at the Satrosphere Science Centre. This is a part of the National Science and Engineering Week event. The idea is to give 3-7 year old children "science super hero" role models to look up to.

Together with other science superheros, we inspired 132 children to imagine themselves as scientists in the future (70 girls and 62 boys)!
Year(s) Of Engagement Activity 2014
URL http://sciencegrrl.co.uk/superheroes-aberdeen/
 
Description Stand-Up Comedy 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? Yes
Geographic Reach Local
Primary Audience Public/other audiences
Results and Impact Judith Masthoff from the University of Aberdeen performed at the Aberdeen Bright Club. Part of the University of Aberdeen May Festival 2013.

Comedy show exposing research issues in a humorous way.

Many comments after the performance showing that it made the audience think about computer science and autonomous systems in particular.
Year(s) Of Engagement Activity 2013
URL https://www.youtube.com/watch?v=-PkVKGw9zz8
 
Description presentation Satrosphere 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Public/other audiences
Results and Impact The presentation got a predominantly young audience interested in robotics and human-computer interaction

I was asked to give my talk also for a second time, which I did.
Year(s) Of Engagement Activity 2014