Learning the Memorable Information from Images

Memories are very important for people and what we can memorise from images defines a specific information which is important for humans. During a memorisation experiment, people were shown images from a database and then they were asked to indicate what do they remember from those images, if anything. The detailed explanation of the memorisation experiment and the resulting dataset VISCHEMA are provided at: https://www.cs.york.ac.uk/. The resulting dataset defines the areas of images that cause that image to remembered or falsely remembered. These areas together are known as the 'visual memory schema' of the image, which is related to the idea of a psychological schema - that is, certain learned arrangements in images affect memorability.

This PhD project will use a combination of Convolutional Neural Networks, Variational Autoencoders, and Generative Adversarial Networks, alongside deep learning to further explore the idea of a 'visual schema'. The purpose of utilising these machine learning techniques is to expose the potential hidden structure beneath these visual schemas and lead to improved generation of visual schemas for images that do not have a human defined baseline, which currently is restricted to generating drastically reduced resolution visual schema maps, using fully convolutional networks. Learning to generate visual schemas with a generative model is important; the model should learn to understand the underlying features that align with human-held schemas that cause an image to be remembered or falsely remembered. This has clear commercial and educational applications, such as creating advertisements that match a particularly memorable schema.

The initial research will involve generating more baseline visual schemas from psychological experiments in order to improve the effectiveness of machine learning models using this data. This will follow the previous VISCHEMA methodology. This data will be used to train a variational autoencoder that has already been pre-trained on scene datasets (transfer learning, intended to mitigate the possibility of overfitting on the relatively small VISCHEMA dataset). This generative model will be constrained to learn to generate visual schema maps using the VISCHEMA data. Various different encoding/decoding architectures will be explored. The output of this model will be compared to baseline obtained visual memory schemas to determine how well the model has learned the features that describe memorable regions. The learned latent space may lead to better understanding of a visual schema beyond the memorability maps obtained by the VISCHEMA experiment.

In later stages, the machine learning models which generate visual schemas will be expanded to generate memorable images instead, using more complex architectures such as generative adversarial networks. Sufficiently complex latent encodings may capture the features which correspond with memorable regions, and hence can be used to generate new, realistic images for a given class which contain these regions. Additional psychological experiments will be ran to determine whether generated visual schemas agree with human baselines, and whether generated memorable images truly are more memorable as seen by a human.