Correcting robot behaviour through embodied natural language interaction

My work will be in the arena of semantic parsing: i.e., the task of creating automated systems that map natural language input into an executable output that captures the communicative intention of the user. For example, "Put the red block on the blue block" could map onto some formal representation, say "(put, rb_1, bb_1)", that can be executed by a robot.
One common approach to semantic parsing is to endow the system with complete knowledge of the possible actions and states in the domain: the task is to learn how natural language is used to refer to known domain concepts and actions. Typically, these systems assume that the natural language string is either indicative and describes some aspect of the current visual scene, or it is imperative and it describes an action to be carried out. But human language users express many other kinds of coherent relations to their embodied environment. For example, a user might *correct* what a robot is currently doing, by expressing some rule or heuristic that, when accurately understood, should be used to revise the way the robot performs such actions in future. For example, in response to the robot performing an action where it is putting a green block on a red block,, the user might state "Don't put green blocks on blue blocks. Put red blocks on blue blocks". This should alter future actions where the system is putting a block on a blue block.

The first step will be to create a system that changes its behaviour as described above. However, we will make many assumptions about the sensory input such as vision and understanding of discourse, such that these can be treated as deterministic. Once this works successfully, we will drop assumptions about gold standard parsing and visual processing, and model the noisy aspects of the process probabilistically.