The action-based brain: a provocation to philosophy, robotics and the cognitive sciences
Lead Research Organisation:
University of Reading
Department Name: Philosophy
Abstract
We often assume that our eyes deliver a detailed image of the world around us, but this is a mistake. Instead, as soon as we want to know the detail in a part of the scene, our eyes flick towards it and the detail is revealed. In this example, perception depends on the world, the motor system and an internal representation that enables the world to act as a 'store' of information that is always retrievable when required. Philosophical theories of 'embodied cognition' emphasise this link between the mind and the motor system. What this analysis lacks, however, is a firm neural and computational grounding specifying the proposed implementation. By looking in detail at a specific version of embodiment that has arisen within the neuroscience community and by exposing it to the critical tools of philosophers, we aim to assess the scope and limits of one particular proposal, and also to draw broader lessons for how neuroscience, robotics and philosophy ought to interact, with respect to these general questions about the nature of cognition.
The network will be inherently multidisciplinary, based on the involvement of mathematicians, engineers, roboticists, psychologists, neuroscientists and philosophers, each with different, complementary inputs. The project is based in Philosophy in Reading, led by a philosopher and already includes key international experts in philosophy of mind and embodied cognition. International experts in neuroscience and robotics have also agreed to take part in the discussion and we will extend the group progressively.
The interactions between members of the network will take place through (i) discussion on a wiki page, with new sets of pages being introduced as the debate develops; (ii) joint video conference meetings focused on specific, contentious areas and (iii) a two day conference towards the end of the 12 month period involving international experts in neuroscience, philosophy of mind and cognitive robotics.
We will disseminate the results of the network primarily through a Brain and Behavioral Sciences article. This is a forum with high impact in both the philosophical and neuroscience communities. We will also publish more specialist articles in journals for neuroscience, engineering and philosophy.
By the end of the year, we will develop a more public-facing portal to the wiki and encourage engagement with a broader audience outside academia. At this point we will use facilities such as ResearchGate to reach a wider group of researchers and social media such as Twitter to advertise the discussion within and beyond the academic community.
The network will be inherently multidisciplinary, based on the involvement of mathematicians, engineers, roboticists, psychologists, neuroscientists and philosophers, each with different, complementary inputs. The project is based in Philosophy in Reading, led by a philosopher and already includes key international experts in philosophy of mind and embodied cognition. International experts in neuroscience and robotics have also agreed to take part in the discussion and we will extend the group progressively.
The interactions between members of the network will take place through (i) discussion on a wiki page, with new sets of pages being introduced as the debate develops; (ii) joint video conference meetings focused on specific, contentious areas and (iii) a two day conference towards the end of the 12 month period involving international experts in neuroscience, philosophy of mind and cognitive robotics.
We will disseminate the results of the network primarily through a Brain and Behavioral Sciences article. This is a forum with high impact in both the philosophical and neuroscience communities. We will also publish more specialist articles in journals for neuroscience, engineering and philosophy.
By the end of the year, we will develop a more public-facing portal to the wiki and encourage engagement with a broader audience outside academia. At this point we will use facilities such as ResearchGate to reach a wider group of researchers and social media such as Twitter to advertise the discussion within and beyond the academic community.
Planned Impact
The most immediate planned impacts of the work will be in applied fields such as robotics and the guidance of autonomous vehicles. Currently, the predominant type of representation is based on a 3D, world-centred frame. There are good reasons to believe that this explicit reconstruction is rather different from the way that humans or other animals represent their environment and the robotics community has a strong interest in developing workable alternatives. For example, we collaborate with Paul Newman who heads the Mobile Robotics Group in Oxford and there is a keen interest within the MRG for developing novel, view-based approaches to navigation and mapping, of the kind our network will investigate. The pathways to impact document expands on the pathways by which the network can influence the development of this type of representation.
In computer vision, there may also be impact in the medium term. Here, we collaborate with researchers at Microsoft Research who are interested in object representation and scene representation using image-based techniques that avoid 3D reconstruction. The development of this type of approach is likely to be informed by biological models and human experiments of the sort that are central to the topic discussed in the network. The proposal about brain-function which our network will discuss is partly inspired by problems and developments in computer vision, and this is a topic where theoretical and practical developments provide crucial feedback for one another. Again, the pathways to impact document provides more detail on the way that these links with industry will be developed.
Of course, the long term consequences of discovering a coherent description of brain function including perception, action and thought will be far reaching, with a profound impact on many areas of health, education and industry. The research network will also contribute to this wider long term goal.
In computer vision, there may also be impact in the medium term. Here, we collaborate with researchers at Microsoft Research who are interested in object representation and scene representation using image-based techniques that avoid 3D reconstruction. The development of this type of approach is likely to be informed by biological models and human experiments of the sort that are central to the topic discussed in the network. The proposal about brain-function which our network will discuss is partly inspired by problems and developments in computer vision, and this is a topic where theoretical and practical developments provide crucial feedback for one another. Again, the pathways to impact document provides more detail on the way that these links with industry will be developed.
Of course, the long term consequences of discovering a coherent description of brain function including perception, action and thought will be far reaching, with a profound impact on many areas of health, education and industry. The research network will also contribute to this wider long term goal.
Publications
Glennerster A
(2017)
Perception and action without 3D coordinate frames
Glennerster A
(2023)
Understanding 3D vision as a policy network.
in Philosophical transactions of the Royal Society of London. Series B, Biological sciences
Glennerster A
(2016)
A moving observer in a three-dimensional world.
in Philosophical transactions of the Royal Society of London. Series B, Biological sciences
Glennerster Andrew
(2018)
A single coordinate framework for optic flow and binocular disparity
in arXiv e-prints
Gootjes-Dreesbach L
(2017)
Comparison of view-based and reconstruction-based models of human navigational strategy.
in Journal of vision
Muryy A
(2018)
Pointing Errors in Non-Metric Virtual Environments
Muryy A
(2021)
Route selection in non-Euclidean virtual environments.
in PloS one
Muryy A
(2017)
Navigation and pointing errors in non-metric environments.
in Journal of Vision
Muryy, A
Route selection in non-Euclidean virtual environments
in forthcoming in PLoS ONE
| Description | By growing and collaborating with our Research Network of philosophers, neuroscientists and experts in robotics and computer vision, we have exposed action-based theories of perception and cognition to multidisciplinary analysis. Our key findings focus on the hypothesis that an action-based process, rather than a 3D model or reconstruction of a scene, can explain an observer's visual perception of the 3D layout of the scene and her consequent actions. Key findings include: 1. (Philosophy) There is a definitional problem about distinguishing between action-based and model-based perception, given that both are ways of perceiving in three dimensions. We can solve this problem, distinguishing between the two forms of perception, if we (i) focus on perception of 'vista space' (the region of space that's visible as you move your head and eyes) and (ii) understand action-based perception as a process of comparing current contexts with stored contexts (a process of the kind traditionally invoked to explain motor control). At the level of vista-space perception, such an action-based theory makes substantively different predictions from a model-based theory. For example, the action-based theory predicts that observers' spatial perception may admit of no consistent mapping onto a coordinate frame defined by three axes (a prediction that's confirmed by previous work in Glennerster's lab). Importantly, this kind of action-based theory avoids charges levelled at previous action-based theories in the philosophical literature, such as the charge of 'behaviourism'. 2. (Neuroscience) This action-based theory has a key advantage over model-based theories: we know how the former, but not the latter, can be implemented by neurons, as reflected in previous work on motor control in the cerebellum. However, our research suggests that it is an open question whether the cerebellum itself has the storage capacity required for action-based perception. Further brain areas are probably crucial. 3. (Robotics and Computer Vision) One challenge to our action-based theory turns on the question how stored contexts are learned. Recent work on artificial visual processing by deep learning systems provides a way of understanding how this could be achieved. Some of the latest systems (developed at Carnegie Mellon University) have much in common with our proposed mechanism. For example, they learn stored contexts which combine sensory and motivational information and they do not represent a scene in terms of a 3D coordinate model. An action-based theory of the brain and this work in computer vision have much to learn from one another. Although these findings are listed here according to discipline, each of them has clear multidisciplinary implications and motivations. This meets our objectives of testing the action-based theory through multidisciplinary analysis, and of setting an agenda for future work across disciplines. |
| Exploitation Route | Academic philosophers will benefit from our detailed description of how action-based perception might be realized in the brain. Philosophical discussion has tended to focus on action-based theories which replace perceptual representation with action, criticizing action-based theories for being unacceptably behaviourist. We show how a physiologically realistic action-based theory need not replace perceptual representation with action, but can instead involve the kind of representation traditionally accepted as a means of motor control. This marks an important possibility which should inform subsequent arguments about the nature of perception and action. The close connections we have discovered between our hypothesis and artificial deep learning systems mean that work in neuroscience and computer vision / robotics can learn a great deal from one another. In particular, we have identified a specific niche in which neuroscientific theories can learn from deep learning systems: explaining how the brain learns stored contexts for vision and action without relying on 3D coordinate-frame reconstructions. Similarly, as this work progresses, computer vision and robotics will be in a position to draw on neuroscientists' models of the brain. Our target article, wiki and future collaborations provide a substantial platform for our work to have these impacts. |
| Sectors | Digital/Communication/Information Technologies (including Software),Electronics,Manufacturing, including Industrial Biotechology,Transport |
| URL | https://jamesstazicker.com/research/the-action-based-brain/ |
| Description | Through meetings of the Research Network and other invited visits to collaborators, we have developed connections with: U.S. Navy; Andrew Fitzgibbon at Microsoft; Lindsey Pickup at Optellum Ltd; Abhinav Gupta's robotics and computer vision group at Carnegie Mellon University; Tony Prescott's robotics group at the University of Sheffield; Paul Newman of Oxford Robotics and the commercial spin-off working on driverless cars, Oxbotica; a company making computer-aided glasses to assist the visually impaired through Professor Philip Torr, OxSight; The Arts and Humanities Research Council 'Science in Culture' theme, including collaboration with the UK Space Agency. These collaborations constitute developing pathways to impact in the commercial as well as the academic sphere. In particular, we have identified close connections between human perception and the deep learning systems exploited for artificial vision in commercial navigation systems. This provides a platform for commercial developers and academics to learn from one another's work about artificial cognition and action-based cognition in the human brain. In addition to our publicly available wiki and our planned target article with peer commentary by roboticists, the most concrete pathway to impact so far is Glennerster's collaboration with the U.S. Navy (which has attracted substantial funding from the U.S. Navy, via the Multidisciplinary University Research Initiatives Program). |
| First Year Of Impact | 2016 |
| Sector | Aerospace, Defence and Marine |
| Description | Collaboration with Prof Song-Chun Zhu, UCLA |
| Organisation | University of California, Los Angeles (UCLA) |
| Country | United States |
| Sector | Academic/University |
| PI Contribution | I spent 1 month in UCLA (May-June 2018) working on a joint paper on compositionality in deep neural networks, and on a MURI (US Defense) grant application. |
| Collaborator Contribution | I worked particular with Mark Edmonds in Prof Zhu's group. We have a draft paper comparing model-based and model-free approaches to representation in neural nets and the implications for neuroscience. |
| Impact | Paper in preparation. Multidisciplinary (Neuroscience and Machine Learning). |
| Start Year | 2018 |