VALUE: Vision, Action, and Language Unified by Embodiment

The primary aim of this project is to develop a simulation of the processes involved in solving the following problem: how to select, based on the agent's knowledge and representations of the world, one object from several, grasp the object and use it in an appropriate manner. This mundane activity in fact requires the simultaneous solution of several deep problems at various levels. The agent's visual system must represent potential target objects, the target must be selected based on task instructions or the agent's knowledge of the functions of the represented objects, and the hand (in this case) must be moved to the target and shaped so as to grip it in a manner appropriate for its use. We propose to develop a robotic simulation model inspired by recent theories of embodied cognition, in which the vision, action and semantic systems are linked together, in a dynamic and mutually interactive manner, within a connectionist architecture. Human experimental work will constrain the temporal and dynamic properties of the system in an effort to develop a psychologically plausible model of embodied selection for action. As much of the cognitive mechanisms leading to the integration between action and vision for actions such as object assembly tasks are not fully known, new empirical studies in this project will also improve our insight of these embodied cognitive dynamics. New experiments and the use of the embodied cognitive model will also be used to further our understanding of language and cognition integration e.g. by providing further predictions and insights on the dynamics of language and action knowledge in object representation.This is an interdisciplinary project which involves expertise and methodologies from cognitive psychology, motor control, and computational/robotics modelling. The interdisciplinary nature of the project and the design and experimentation of cognitive agents make the project highly relevant to the Cognitive Systems Foresight programme