Using Split-Brain autoencoders to learn an unsupervised representation of visual saliency in scenes
Lead Research Organisation:
University of Edinburgh
Department Name: Sch of Informatics
Abstract
This research focuses on unsupervised methods of predicting the visual saliency of regions of scenes. We use a split brain auto-encoder model in which two auto-encoders - each representing a 'hemisphere' - are trained to predict the output of the other. This is an extension of the predictive processing model of cognition. According to our model, each hemisphere is engaged in attempting to predict the output of the other and in order to do so, must learn useful representations of the input.
If successful, this research will provide a useful unsupervised method of predicting visual saliency in scenes, as well as improve our understanding of how the brain is able to extract information out of the jumble of input it receives, as well as the importance of hemisphericity as an organising principle of the brain.
Project milestones are as follows:
1.) Conduct a literature review and identify relevant work and similar approaches.
2.) Create a working prototype system
3.) Compare the performance of the system on eye-tracking data with maps of visual saliency across scenes, for example the MIT dataset.
4.) Iterate and improve the model and report on findings.
If successful, this research will provide a useful unsupervised method of predicting visual saliency in scenes, as well as improve our understanding of how the brain is able to extract information out of the jumble of input it receives, as well as the importance of hemisphericity as an organising principle of the brain.
Project milestones are as follows:
1.) Conduct a literature review and identify relevant work and similar approaches.
2.) Create a working prototype system
3.) Compare the performance of the system on eye-tracking data with maps of visual saliency across scenes, for example the MIT dataset.
4.) Iterate and improve the model and report on findings.
People |
ORCID iD |
| Richard Shillcock (Primary Supervisor) | |
| Beren Millidge (Student) |
Studentship Projects
| Project Reference | Relationship | Related To | Start | End | Student Name |
|---|---|---|---|---|---|
| EP/N509644/1 | 01/10/2016 | 30/09/2021 | |||
| 1929930 | Studentship | EP/N509644/1 | 01/09/2017 | 28/02/2021 | Beren Millidge |
| Description | I investigated the properties of split-brain autoencoders for modelling visual saliency. In a 'split-brain' autoencoder the image is split into multiple channels and then networks are trained to predict some channels from others. It was hoped that this would force the network to learn more complex visual representations that could be used for saliency modelling. This work resulted in a preprint (https://psyarxiv.com/csmeb/), however results did not surpass other state of the art methods in machine learning. I then shifted our approach to investigating recent predictive processing and active inference theories in computational neuroscience. These theories propose that the brain primarily perceives and acts through a process of variational inference -- a particular kind of approximation to optimal bayesian inference. While these theories have been worked on and extended for almost a decade and purport to provide computational algorithms (down to the neural level) which can implement this variational inference. Despite this, previous computational implementations of active inference and predictive processing had been limited to tiny toy tasks, and relationships to other frameworks remained deeply unclear. I have tested and empirically tested the scalability of the proposed methods and found that active inference, at least, through the use of deep neural networks as nonlinear function approximators the performance of active inference agents can approach that of the state of the art in deep reinforcement learning (https://arxiv.org/abs/1907.03876), and have implemented continuous time predictive processing algorithms which also achieve reasonable performance on reinforcement learning benchmark tasks (https://psyarxiv.com/kf6wc/, https://psyarxiv.com/4hb58/). I have shown that active inference has deep connections with the control as inference framework in reinforcement learning, although the exact relation remains to be elucidated. Moreover, in work currently in preparation with the University of Sussex, I have worked on extending parts of the theory of active inference by mathematically investigating the core objective -- the expected free energy -- and proposing alternative objectives. |
| Exploitation Route | We believe that our Our work has shown that active inference is also closely related to the control-as-inference framework in reinforcement learning. Understanding deeply how these two frameworks relate could extend and deepen knowledge about how to phrase control problems as inference in graphical models, and could lead to more robust and principled ways to treat uncertainty for control problems for industrial or commercial applications. |
| Sectors | Other |
| Description | Collaborated with Andrea Ravignani on a paper |
| Organisation | Vrije Universiteit Brussel |
| Country | Belgium |
| Sector | Academic/University |
| PI Contribution | My supervisor, Andrea Ravignani, and I collaborated on this work on modelling the creation of acoustic attractors by vocal imitation learning in bats. |
| Collaborator Contribution | I implemented and experimented with the computational models while my supervisor and Andrea conceptualised the study and reviewed the manuscript and made connections to the biological literature. |
| Impact | Paper (under review) on the creation of acoustic attractors via vocal imitation learning in bats. The collaboration is multidisciplinary -- my supervisor (Richard Shillcock) and I are in informatics and did the conceptualisation and computational modelling. Andrea is a biologist and took care to ensure the biological plausibility of the simulations and reviewed the manuscript. |
| Start Year | 2018 |
| Description | Visiting fellowship at University of Sussex |
| Organisation | University of Sussex |
| Department | School of Engineering and Informatics Sussex |
| Country | United Kingdom |
| Sector | Academic/University |
| PI Contribution | Collaborators at the University of Sussex are Christopher Buckley (supervisor) and Alexander Tschantz (PhD student). We are collaborating deeply on papers involving the integration of active inference (a framework for probabilistic decision making from computational neuroscience) and deep reinforcement learning. |
| Collaborator Contribution | Deeply collaborated with University of Sussex academics and PhD students on a range of papers currently in preparation. |
| Impact | Multiple papers are currently under preparation due to this collaboration. |
| Start Year | 2019 |
| Title | Infer-actively |
| Description | The software provides a library implementing active inference - a theory in computational neuroscience, and provides ability to quickly replicate the results of the many papers in the area, and provides state of the art inference algorithms for control problems. |
| Type Of Technology | Software |
| Year Produced | 2020 |
| Open Source License? | Yes |
| Impact | A research lab in italy (headed by Giovanni Pezzulo) has begun using the software for research in active inference |
| URL | https://github.com/alec-tschantz/infer-actively |