Correcting robot behaviour through embodied natural language interaction

Lead Research Organisation: University of Edinburgh
Department Name: Sch of Informatics


My work will be in the arena of semantic parsing: i.e., the task of creating automated systems that map natural language input into an executable output that captures the communicative intention of the user. For example, "Put the red block on the blue block" could map onto some formal representation, say "(put, rb_1, bb_1)", that can be executed by a robot.
One common approach to semantic parsing is to endow the system with complete knowledge of the possible actions and states in the domain: the task is to learn how natural language is used to refer to known domain concepts and actions. Typically, these systems assume that the natural language string is either indicative and describes some aspect of the current visual scene, or it is imperative and it describes an action to be carried out. But human language users express many other kinds of coherent relations to their embodied environment. For example, a user might *correct* what a robot is currently doing, by expressing some rule or heuristic that, when accurately understood, should be used to revise the way the robot performs such actions in future. For example, in response to the robot performing an action where it is putting a green block on a red block,, the user might state "Don't put green blocks on blue blocks. Put red blocks on blue blocks". This should alter future actions where the system is putting a block on a blue block.

The first step will be to create a system that changes its behaviour as described above. However, we will make many assumptions about the sensory input such as vision and understanding of discourse, such that these can be treated as deterministic. Once this works successfully, we will drop assumptions about gold standard parsing and visual processing, and model the noisy aspects of the process probabilistically.


10 25 50

Studentship Projects

Project Reference Relationship Related To Start End Student Name
EP/N509644/1 01/10/2016 30/09/2021
1929791 Studentship EP/N509644/1 01/09/2017 31/05/2021 Mattias Appelgren
Description We have developed an agent which learns from corrections given by a teacher in a simulated tower building task. The agent makes use of the context in which the correction was given to interpret how to update the agent's knowledge (and models) of the world to better fit with the content of the utterances.
Exploitation Route Our shows how additional modes of interaction can be used for the purpose of teaching agents about the world. Our hope is that these ideas will be implemented on real robots so as to teach them about novel tasks without the need of any robotics expertise.
Sectors Digital/Communication/Information Technologies (including Software)