SBE-UKRI: Integrating vision and action through selection history

Most real-world visual scenes are complex and crowded, where multiple objects compete for attention and goal-directed action. In daily life, for example, a person easily picks up a red apple from a grocery display containing many kinds of fruit. Successful interactions with such complex environments require seamless coordination among multiple mechanisms. In particular, mechanisms of attentional selection that help us make sense of the world work in unity with those that underlie action selection, allowing us to generate adaptive movements. This attention-action synergy is at the root of all complex behaviour.

Object selection is guided not only by the well-established factors of perceptual salience (bottom-up) and current goals (top-down), but also selection history. Yet, how selection history links to visually-guided actions has been understudied in real-world scenarios. To overcome this gap, the goal of this proposal is to determine the interplay between mechanisms controlling attentional selection and action selection, particularly when recent selection history biases subsequent behaviour. The proposed project will focus on three understudied aspects of action selection: variation in action execution, effectors, and biomechanical costs, to determine their novel relations with the wealth of research on selection history of perceptual features.

To ensure successful outcomes, an international collaboration between the two research teams, who will play complementary and synergistic roles, will be formed: Dr Song at Brown University (US-PI) - an expert in attention and motor control - will carry out psychophysical experiments in humans including continuous tracking and force field manipulation of goal-directed actions. To translate empirical evidence to testable models, Dr Heinke at University of Birmingham (UoB) (UK-PI) - a computational modelling expert - will develop and implement biologically plausible control architectures for a robot arm (robotics models). This joint endeavour will advance our understanding of the interdependence between attention and action-driven mechanisms to eventually explain adaptive, real-world selection behaviour.

This proposal intends to foster the integration of research, STEM and a broad education. The US-PI will mentor talented undergraduates including nationally selected, underrepresented minorities for summer placements and long-term research programs in the US-PI's lab via partnership with the Leadership Alliance programs at Brown University. UK-PI will contribute to UoB's Outreach program with scientific talks and robotics demonstrations at local high schools and during campus visits of school classes.

Outcomes can contribute to significant technological innovations, including foundation for novel control architecture for robot arms and teleoperators, developing brain-computer interfaces (BCI) for cognitive neural prosthetics and human-computer-interfaces (HCI) such as the widely used touch-user-interfaces (e.g. smart phones).

Finally outcomes can contribute to the rehabilitation of stroke patients. A typical stroke affects the visual system and the motor system. A better understanding of the relationship between the two systems can contribute to the development of novel treatment programs.