SBE-UKRI: Integrating vision and action through selection history
Lead Research Organisation:
University of Birmingham
Department Name: School of Psychology
Abstract
Most real-world visual scenes are complex and crowded, where multiple objects compete for attention and goal-directed action. In daily life, for example, a person easily picks up a red apple from a grocery display containing many kinds of fruit. Successful interactions with such complex environments require seamless coordination among multiple mechanisms. In particular, mechanisms of attentional selection that help us make sense of the world work in unity with those that underlie action selection, allowing us to generate adaptive movements. This attention-action synergy is at the root of all complex behaviour.
Object selection is guided not only by the well-established factors of perceptual salience (bottom-up) and current goals (top-down), but also selection history. Yet, how selection history links to visually-guided actions has been understudied in real-world scenarios. To overcome this gap, the goal of this proposal is to determine the interplay between mechanisms controlling attentional selection and action selection, particularly when recent selection history biases subsequent behaviour. The proposed project will focus on three understudied aspects of action selection: variation in action execution, effectors, and biomechanical costs, to determine their novel relations with the wealth of research on selection history of perceptual features.
To ensure successful outcomes, an international collaboration between the two research teams, who will play complementary and synergistic roles, will be formed: Dr Song at Brown University (US-PI) - an expert in attention and motor control - will carry out psychophysical experiments in humans including continuous tracking and force field manipulation of goal-directed actions. To translate empirical evidence to testable models, Dr Heinke at University of Birmingham (UoB) (UK-PI) - a computational modelling expert - will develop and implement biologically plausible control architectures for a robot arm (robotics models). This joint endeavour will advance our understanding of the interdependence between attention and action-driven mechanisms to eventually explain adaptive, real-world selection behaviour.
Object selection is guided not only by the well-established factors of perceptual salience (bottom-up) and current goals (top-down), but also selection history. Yet, how selection history links to visually-guided actions has been understudied in real-world scenarios. To overcome this gap, the goal of this proposal is to determine the interplay between mechanisms controlling attentional selection and action selection, particularly when recent selection history biases subsequent behaviour. The proposed project will focus on three understudied aspects of action selection: variation in action execution, effectors, and biomechanical costs, to determine their novel relations with the wealth of research on selection history of perceptual features.
To ensure successful outcomes, an international collaboration between the two research teams, who will play complementary and synergistic roles, will be formed: Dr Song at Brown University (US-PI) - an expert in attention and motor control - will carry out psychophysical experiments in humans including continuous tracking and force field manipulation of goal-directed actions. To translate empirical evidence to testable models, Dr Heinke at University of Birmingham (UoB) (UK-PI) - a computational modelling expert - will develop and implement biologically plausible control architectures for a robot arm (robotics models). This joint endeavour will advance our understanding of the interdependence between attention and action-driven mechanisms to eventually explain adaptive, real-world selection behaviour.
Planned Impact
This proposal intends to foster the integration of research, STEM and a broad education. The US-PI will mentor talented undergraduates including nationally selected, underrepresented minorities for summer placements and long-term research programs in the US-PI's lab via partnership with the Leadership Alliance programs at Brown University. UK-PI will contribute to UoB's Outreach program with scientific talks and robotics demonstrations at local high schools and during campus visits of school classes.
Outcomes can contribute to significant technological innovations, including foundation for novel control architecture for robot arms and teleoperators, developing brain-computer interfaces (BCI) for cognitive neural prosthetics and human-computer-interfaces (HCI) such as the widely used touch-user-interfaces (e.g. smart phones).
Finally outcomes can contribute to the rehabilitation of stroke patients. A typical stroke affects the visual system and the motor system. A better understanding of the relationship between the two systems can contribute to the development of novel treatment programs.
Outcomes can contribute to significant technological innovations, including foundation for novel control architecture for robot arms and teleoperators, developing brain-computer interfaces (BCI) for cognitive neural prosthetics and human-computer-interfaces (HCI) such as the widely used touch-user-interfaces (e.g. smart phones).
Finally outcomes can contribute to the rehabilitation of stroke patients. A typical stroke affects the visual system and the motor system. A better understanding of the relationship between the two systems can contribute to the development of novel treatment programs.
Organisations
Publications
Makwana M
(2021)
Dissociating mechanism underlying selection history bias for goal-directed reaching movements
in Journal of Vision
Heinke D
(2021)
Modelling trajectories from choice reaching experiments through submovement decomposition
in Journal of Vision
Heinke D
(2019)
Noise and motion: A new visual search paradigm with multiple random dot kinematograms (RDKs)
in Journal of Vision
Deakin J
(2020)
Evidence for the involvement of perceptual grouping in flanker effects through random dot kinematograms (RDKs).
in Journal of Vision
Heinke D
(2020)
A novel diffusion-based model of choice reaching experiments
in Journal of Vision
Zhang F
(2021)
Canonical specular and velvety material modes form a basic feature in visual search
in Journal of Vision
Zhang Fan
(2021)
Testing human's ability to search for materials in a visual scene using canonical material modes
in PERCEPTION
Makwana M
(2023)
Continuous action with a neurobiologically inspired computational approach reveals the dynamics of selection history.
in PLoS computational biology
Heinke D
(2021)
A failure to learn object shape geometry: Implications for convolutional neural networks as plausible models of biological vision.
in Vision research
Charles Leek E
(2022)
Deep neural networks and image classification in biological vision.
in Vision research
Description | An important focus in the funded work is the method to understand human behaviour with the help of mathematical/computational models. During the initial phase of the grant it became clear that recently developed artificial intelligence methods are increasingly seen as an alternative to the computational method proposed in our work. In response to this development I and two colleagues lead a special issue on this topic. The papers in this special issue as well as our own contributions in this special issue highlighted current shortcomings of this new approach. In particular this approach cannot contribute to understanding of human behaviour as there are no links conceptual approaches. |
Exploitation Route | The plan is to write a follow-on proposal and the results will be explored by robotic engineers. |
Sectors | Aerospace Defence and Marine Digital/Communication/Information Technologies (including Software) Manufacturing including Industrial Biotechology Transport |
URL | https://doi.org/10.1016/j.visres.2022.108069 |