Scrutable Autonomous Systems

The aim of the project is to make so-called distributed autonomous systems "scrutable" by humans. Autonomous systems can perform tasks without continuous human guidance. This project considers autonomous systems consisting of many components (called agents) which have to make joint plans on how to act together. It is important that humans can understand why the system behaves in particular ways, for example, why the agents have decided upon a certain plan. To address this, the project will develop and evaluate computational techniques for gathering information about the planning process, including how the various agents interacted, and presenting this information to humans in an understandable way.

To allow information to be gathered easily, the agents will use so-called argumentation techniques: agents engage in dialogues whereby they justify their decision-making as they agree on joint plans, and each decision is based on a collection of arguments in favour and/or against it. This approach has the additional benefit to potentially produce plans faster, because it offers agents an insight into each others' motives. However, this approach will gather much information, and the information will be complex. To allow humans to inspect this information easily, the project will use a combination of Natural Language and diagrams to present it in an understandable way; simplifying, summarising and aggregating information as needed.

For example, our system might explain:

"To achieve goal G, a plan comprising the sequence of actions A, B, C has been agreed upon. Agents P1, P2 and P3 were involved in the preparation of the plan. P1 argued that A had to be done first, because of its standard operating procedures; P2 and P3 agreed to this. P1 suggested that A should be followed by action D, because it could do A and D with low costs, however P2 disagreed, pointing out that action D did not address safety concerns, and suggested that A should be followed by B. No one objected to this. Finally, P1 suggested that B should be followed by C, and that it could perform the action with the help of P3."

If the user requests further details about the safety concerns, she would be presented with the following:

"Current guidelines establish that actions B and D both achieve the same goal, but B does so addressing safety concerns X, Y and Z; D does not address any safety concerns"

The design of a scrutable argumentation-based planning system faces substantial technological challenges. One challenge is to endow distributed planning with the ability to judiciously record its process, whilst creating useful plans quickly. The other is to present the record of the distributed planning process optimally to users. Optimality, in this context, means a combination of clarity and information-richness. We will carry out extensive experiments to find out what aspects of a plan (and the motivation behind it) to emphasise, and where details are better left unsaid. Business partners will help us by taking part in these experiments, and by engaging in discussions about the types of insight users demand.

The project will seek to find solutions that apply across a large range of applications, varying from ones that involve highly complex software agents and robots to ones that involve a plurality of simple, sensor-based agents. We hypothesise that our argumentation-based approach is suitable for modelling disparate scenarios, regardless of their complexity, and that Information Presentation techniques, coupled with state-of-the-art requirements gathering and user-based evaluation, has much to offer across this range.

Businesses using autonomous systems

Autonomous systems are being used in many UK and multi-national businesses, in particular in the areas of Aerospace, Aviation, Defense and Energy Generation and Supply. As recognized in this funding call, it is of vital importance for the successful deployment of autonomous systems that humans are kept "in the loop" with timely and relevant information and can humans can interact with the autonomous systems effectively. The importance of transparency and user involvement in autonomous systems was also highlighted in a recent US government report which stated that human-performance problems with autonomous systems are exacerbated by "inadequate visibility into how automation is working, inadequate feedback about the automation's activities, lack of facilities for operators to communicate with automation, inadequate explanation of its reasoning processes" (O'Hara & Higgins, 2010, p53). To achieve effective scrutability, it is mandatory to know what information presentations would be useful for different business users, including the required levels of aggregation and summarisation. Dissemination of the results to the wider business community is also vital for the exploitation of the research results.


General public

Autonomous systems have a great potential to be used in businesses. However, the general public is likely to be worried by the potential impact of of time- and safety-critical autonomous systems if they malfunction. It is therefore important to raise public understanding of how such systems work, and how they can be made safer by allowing humans to inspect how the system behaves. Autonomous systems are also a very exciting computer science research area, and for the future of the UK economy it is important that young people are enthused about careers related to computer science.


Early-career researchers

It is of strategic importance to the UK to attract and train early-career researchers in its core areas of expertise, such as distributed autonomous systems (DAS) and information presentation especially natural language generation (NLG). The early-career researchers who will work on this grant (3 research fellows and 1 research assistant) will benefit from learning how to apply their research across the traditional DAS and NLG boundaries, and from learning how to perform research that involves end-users throughout the project. They will also benefit from learning how to engage with business communities and the general public. In addition, we will train other early-career researchers in the topics of this research.