Seeing the Bigger Picture: Visual Imagination and the Social Brain
Lead Research Organisation:
University College London
Department Name: Institute of Neurology
Abstract
I studied multi-modal aspects of visual imagination in relation to visual art and complex images, defining 'visual imagination' broadly as a dynamic of complex psychological processes that integrate visual information with prior experiences and knowledge to construct internal models of oneself, others and the outside world. This reflects the ultimate aim of my work to develop engaging cultural and clinical resources that strengthen social brain networks, tailored to personal interests, age and cognitive health. I pursued two interrelated research programmes based primarily at the Wellcome Collection, as part of my interdisciplinary residency with Created Out of Mind.
Research Summary
I used complementary neuroscientific and visual research methods to probe relationships between visual imagination and the social brain in neurologically healthy adults and people living with various forms of dementia.The Social Brain Atlas and connectome (Alcalá López et al. , Cerebral Cortex 2017) was recently computed from 3972 functional neuroimaging studies in 22712 healthy adults: to contextualise my research in the social brain, I first translated the social brain connectome to functional infographics (relational spatial representations) of the four hierarchical processing levels of the Social Brain Atlas, and generated visual imagination brain profiles in healthy adults and profiles of canonical dementia syndromes. I used these to generate hypotheses and guide analysis of my neuroscientific experiments.
I recruited three participant cohorts: 17 neurologically healthy adults aged 20-30 years; 20 neurologically healthy adults aged 50+ years; and 11 senior adults living with various forms of dementia.I designed five neuroscientific experiments, in which I used advanced technologies to capture physiological responses and established as well as novel visual research methods to study neuropsychological responses to visual art, complex imagery and colour experiences. I employed an arts-based facilitated conversation methodology, Visual Thinking Strategies (VTS); and I developed novel quantitative methods to analyse recorded eye tracking data, electrodermal activity and speech samples. I used both parametric and non-parametric statistical methods to compare participant cohorts.
In parallel with the neuroscientific research, I developed a series of art experiments at UCL Institute of Making, and my art studio in Limehouse, East London. My artistic research complemented my neuroscientific work by emphasising individual experience over generic perceptual mechanisms: by creating space for personal interactions with art, the research becomes contextualised in the social world. I created two photographic experiments to study how interactions between light, colour and form affect early visual perception; a series of interactive optical instruments to alter wearers' visual perception ('Perspectacles'); and an interactive virtual reality installation based on my neuroscientific experiments. Feedback from my artistic research informed the art exhibition I created to complement my PhD thesis.
Future Directions
Following on from this study, with my collaborators I plan to develop fully interactive, animated digital platforms for the Social Brain Atlas and VTS.
I will seek to replicate and extend my findings in larger cohorts of senior adults living with well-defined dementia diagnoses, including functional neuroimaging techniques to ground behavioural and physiological findings in the brain. Beyond academia, I will use the findings of this study to create programmes for art museums and other cultural institutions as well as commercial applications with my company The Thinking Eye, that aim to strengthen social engagement, careful observation, divergent thinking and communication skills, through visual art interactions. Informed by my PhD findings, the design of these programmes will be tailored to promoting healthy cognitive ageing and wellbeing.
Research Summary
I used complementary neuroscientific and visual research methods to probe relationships between visual imagination and the social brain in neurologically healthy adults and people living with various forms of dementia.The Social Brain Atlas and connectome (Alcalá López et al. , Cerebral Cortex 2017) was recently computed from 3972 functional neuroimaging studies in 22712 healthy adults: to contextualise my research in the social brain, I first translated the social brain connectome to functional infographics (relational spatial representations) of the four hierarchical processing levels of the Social Brain Atlas, and generated visual imagination brain profiles in healthy adults and profiles of canonical dementia syndromes. I used these to generate hypotheses and guide analysis of my neuroscientific experiments.
I recruited three participant cohorts: 17 neurologically healthy adults aged 20-30 years; 20 neurologically healthy adults aged 50+ years; and 11 senior adults living with various forms of dementia.I designed five neuroscientific experiments, in which I used advanced technologies to capture physiological responses and established as well as novel visual research methods to study neuropsychological responses to visual art, complex imagery and colour experiences. I employed an arts-based facilitated conversation methodology, Visual Thinking Strategies (VTS); and I developed novel quantitative methods to analyse recorded eye tracking data, electrodermal activity and speech samples. I used both parametric and non-parametric statistical methods to compare participant cohorts.
In parallel with the neuroscientific research, I developed a series of art experiments at UCL Institute of Making, and my art studio in Limehouse, East London. My artistic research complemented my neuroscientific work by emphasising individual experience over generic perceptual mechanisms: by creating space for personal interactions with art, the research becomes contextualised in the social world. I created two photographic experiments to study how interactions between light, colour and form affect early visual perception; a series of interactive optical instruments to alter wearers' visual perception ('Perspectacles'); and an interactive virtual reality installation based on my neuroscientific experiments. Feedback from my artistic research informed the art exhibition I created to complement my PhD thesis.
Future Directions
Following on from this study, with my collaborators I plan to develop fully interactive, animated digital platforms for the Social Brain Atlas and VTS.
I will seek to replicate and extend my findings in larger cohorts of senior adults living with well-defined dementia diagnoses, including functional neuroimaging techniques to ground behavioural and physiological findings in the brain. Beyond academia, I will use the findings of this study to create programmes for art museums and other cultural institutions as well as commercial applications with my company The Thinking Eye, that aim to strengthen social engagement, careful observation, divergent thinking and communication skills, through visual art interactions. Informed by my PhD findings, the design of these programmes will be tailored to promoting healthy cognitive ageing and wellbeing.
People |
ORCID iD |
| Jason Warren (Primary Supervisor) | |
| Janneke Van Leeuwen (Student) |
Studentship Projects
| Project Reference | Relationship | Related To | Start | End | Student Name |
|---|---|---|---|---|---|
| EP/N509577/1 | 01/10/2016 | 24/03/2022 | |||
| 1814921 | Studentship | EP/N509577/1 | 01/10/2016 | 30/11/2019 | Janneke Van Leeuwen |
| Title | Artistic research |
| Description | My artistic research in the form of optical instruments and visual art objects and installations aimed to complement the neuroscientific projects by exploring the research questions and hypotheses in different dimensions than verbal language. The artistic research resulted in a public exhibition of optical instruments, visual artworks and installations that expanded on the two neuroscientific research projects, complementing the written thesis with the embodied language of visual art. Visitors could freely explore the perceptual effects of the optical instruments and were invited to reflect on the visual artworks with the Visual Thinking Strategies method. |
| Type Of Art | Artistic/Creative Exhibition |
| Year Produced | 2019 |
| Impact | In the embodied artistic research component of my PhD, I expanded on the theoretical framework and findings of my neuroscientific research projects and invited visitors to engage in experiential explorations of the research questions. Photographic works I developed a novel theoretical model for the visual exploration of artworks and complex images, based on the social brain connectome (Alcala-Lopez et al., 2017). My Social Salience Model of Visual Exploration predicted accurately that animate elements (faces, hand actions, other body parts) in visual art and complex images were on average looked at more than inanimate elements. I created six photographic works, which aimed to further investigated the saliency (importance) of different categories of inanimate elements in artworks and complex images. The photographic models were constructed with different combinations of elements from the five perceptual categories in the inanimate domain: • Text elements • Human made objects • Built environment elements • Natural elements • Number elements. Visitors could explore how they personally perceived and interpreted these different elements in each of the six photographic works, guided by the open questions of the Visual Thinking Strategies (VTS) method: • What is going on in this image? • What do you see that makes you think that • What more can you find? Installation The colour panels from which the framed photographic works were suspended, built on the findings of my neuroscientific colour experiments. The 3 wall panels on one side in saturated yellow, red and blue (International Klein Blue) were the most positively rated colours in the neuroscientific research experiments, whereas the 3 wall panels on the opposite side in dark yellow, dark brown (Pantone 448C) and black were the most negatively rated colours. My research findings had also shown that spatial context had an effect on how people experienced the colours, and that senior adults responded more strongly to spatial cues that young adults, which is why the panels were suspended from the wall in different angles to allow viewers to explore how these spatial variations might influence their experiences. Optical instruments To enable visitors to personally explore my research question: 'how does visual perception relate to visual imagination?', I designed optical instruments which I called 'Perspectacles'. The Foveal Perspectacle has a solid frame which only lets light through a 1 mm hole for each eye. The resulting narrow viewpoint corresponds with the human foveal visual field, which only takes up the central 5% of the total visual field, but is responsible for detailed colour and high acuity vision. The Inlay Perspectacle has two open frames, which viewers can fill with many different colour and grating combinations of transparent acrylic glass inlays, allowing them to freely experiment how these might change their perception and interpretation of the colour panels and the photographic art experiments. |
| URL | https://www.thinkingeye.org/artistic |
| Description | In this study I set out to study multi-modal aspects of visual imagination in healthy ageing and dementia, focusing on visual art and complex images in the context of the social brain. Below I have summarised the key research findings of the Thinking Eyes and Colour Spaces projects. Key findings Thinking Eyes experiments: -The visual exploration patterns of visual artworks and complex images showed temporal changes that could be aligned with different processing phases in the human brain. -During the initial perceptual processing phase (0 - 250 ms), the gaze of neurologically healthy adults stayed mostly fixated on the image centres, where participants had been prompted to look beforehand. -During the initial perceptual processing phase (0 - 250 ms), the gaze of senior adults living Posterior Cortical Atrophy stayed significantly less fixated on the image centres, compared to neurologically healthy senior adults. This effect was also observed to a lesser extend in senior adults living with Typical Alzheimer's Disease. During the gist inference phase (250 - 750 ms), top-down effects of social salience significantly influenced the visual exploration of figurative visual artworks and complex images, but not the distorted versions of the images. -During the construct inference phase (>750 ms), in addition to social salience, auditory processing and different viewing strategies also had significant top-down effects on the visual exploration of figurative visual artworks and complex images. -A novel social salience model of visual exploration, based on the principles of foveal vision and the functional profiles of the Social Brain Atlas, predicted the visual exploration patterns of figurative visual artworks and complex images significantly better than an alternative model that only took perceptual factors into account. -Looking at visual artworks and complex images with the Visual Thinking Strategies (VTS) method made all participants - independent of age and neurological health - explore the images in general significantly more compared to when they listened to contextual information, and made them focus especially on animate elements (human and animal faces, hand actions and other body elements). -Looking at visual artworks and complex images while listening to contextual information from a museum catalogue made all participants - independent of age and neurological health - explore the images in general significantly less compared to the VTS method, but made them focus slightly more on inanimate elements (human made objects, built environment, natural, text and number elements). -In young adults, looking at visual artworks and complex images with the Visual Thinking Strategies (VTS) method from their personal perspective significantly increased their resonance with the images, while listening to contextual information decreased their resonance with the images significantly. -In senior adults, looking at visual artworks and complex images with different viewing strategies did not significantly affect their resonance with the images. -In senior adults living with dementia, looking at visual artworks and complex images while listening to contextual information decreased their resonance with the images significantly. -Auditory processing had specific top-down effects on the visual exploration patterns of visual artworks and complex images in senior adults living with various forms of dementia during the silent viewing phase that followed an audio recording. -A novel quantitative analysis method for VTS speech samples showed that individual speech profiles were stable in healthy ageing, but showed both general and specific effects of dementia. The most likely speech parameters to be affected were the content words and semantic words ratios. -Electrodermal activity, indicative of autonomic arousal, was considerably lower during the Thinking Eyes experiments in senior adults living with dementia, compared to the standard neuropsychometric test section of the research. Key findings Colour Spaces experiments: -The visual exploration patterns of monochromatic colour space images showed temporal changes that could be aligned with different processing phases in the human brain. -During the initial perceptual processing phase (0 - 250 ms), the gaze of neurologically healthy adults stayed mostly fixated on the image centres, where participants had been prompted to look beforehand. -During the initial perceptual processing phase (0 - 250 ms), the gaze of senior adults living Posterior Cortical Atrophy stayed significantly less fixated on the image centres, compared to neurologically healthy senior adults. This effect was also observed to a lesser extend in senior adults living with Typical Alzheimer's Disease or Behavioural Frontotemporal Dementia. -During the gist inference phase (250 - 750 ms), depicted spatial context significantly influenced the visual exploration of colour space images. During the construct inference phase (>750 ms), the effects of depicted spatial context significantly on the visual exploration of colour space images became more pronounced. -A novel social salience model of visual exploration, based on the principles of foveal vision and the functional profiles of the Social Brain Atlas, predicted the visual exploration patterns of colour space images no better than an alternative model that only took perceptual factors into account. -The visual exploration patterns of colour room images showed significant effects of healthy ageing and dementia, which might be related to a reduced contrast sensitivity in healthy ageing and both physiological and attentional impairments in various forms of dementia. -Young adults showed significantly stronger pupil contractions to light and saturated colours, compared to senior adults. The results showed no significant differences in the pupil responses to dark colours. -The equally strong pupil response to light and saturated colours in young adults, despite a higher luminance of the light colours could be explained by the Helmholtz-Kohlrausch effect which describes that strongly saturated colours have a higher perceived brightness than colours with the same luminance but a lower saturation. -The finding that the Helmholtz-Kohlrausch effect was only observed in the pupil contractions of young adults, suggests a reduced sensitivity to colour saturation in senior adults, independent of neurological health. -The affective responses to monochromatic colour spaces were broadly aligned with the brightness of the colours, with brighter colours on average evoking more positive feelings in participants regardless of age and neurological health. -The affective responses of neurologically healthy young and senior adults to the colour space images replicated a 2012 survey under 1000 smokers commissioned by the government of Australia that found that Pantone 448C was generally experienced as the most unpleasant colour. -In contrast, the Pantone 448C colour field print was rated significantly more positively by 36% of the senior adults living with dementia, compared to neurologically healthy adults. -The affective responses of young adults to monochromatic colour space images were only significantly affected by material presentation, whereby they preferred the print versions over the digital colour images. -The affective responses to monochromatic colour space images of senior adults, independent of neurological health, were only significantly affected by depicted spatial context, whereby the colour rooms were rated more negatively. -The majority of all participants, independent of age and neurological health, preferred the colour prints over the digital colour presentations. The neurologically healthy senior adults had more often no preference than the young adults and the senior adults with dementia more often preferred the digital colours than the neurologically healthy senior adults. -Young adults and various senior adults living with dementia had a significantly higher electrodermal activity in the left wrist during the Digital Colour Spaces experiment, compared to the neurologically healthy senior adults. This could perhaps be indicative of a more positive emotional arousal in young adults and senior adults living with dementia in response to digital colours, but it could also reflect a stronger autonomic response to the sensory colour stimulations. |
| Exploitation Route | With this study, I hope to make valuable contributions to the academic study of neuroscience and visual art and beyond. My visualisations of the Social Brain Atlas will hopefully be of value to researchers and clinicians in the field of neuroscience, but will also enable a wider public to gain an understanding of the brain dynamics that are involved in visual imagination and other complex social behaviours in healthy ageing and in different forms of dementia. The outcomes of this study will be submitted to peer-reviewed scientific journals and I will seek out diverse public platforms to disseminate the neuroscientific and artistic research to broad audiences to whom the findings will be of relevance. The artistic research of this study, in the form of drawings, optical instruments and visual art objects and installations was presented alongside this thesis. The aim of the artistic research was to complement the rational and reductionistic nature of neuroscientific research by an embodied approach thataddressed the research questions and hypotheses in different dimensions than verbal language. By exploring the multi- layered interactions between the researcher and the subject, the research becomes contextualised in the social world, which is where I hope it will have its greatest impact. With the art programmes that I will develop from this study I will strive to make a positive impact on people's lives by offering visual art interactions that build on the principles of the social brain dynamics to strengthen a sense of agency, social engagement, observation and communication skills. |
| Sectors | Communities and Social Services/Policy,Creative Economy,Education,Healthcare,Culture, Heritage, Museums and Collections |
| URL | https://www.thinkingeye.org/seeing-the-bigger-picture |
| Description | With the findings of my research I am developing Visual Thinking Strategies (VTS) and the Social Brain workshops and training courses which I'm delivering at higher education and cultural institutions with my social enterprise The Thinking Eye, which I founded in 2015. VTS is tailored towards fostering collaborative, inclusive, community-building dialogue, through arts-based facilitated conversations. My research findings enable me to tailor the VTS services of The Thinking Eye to people's personal interests, age and neurological health. Cultural institutions I'm working with in the UK include The Photographers' Gallery and Dulwich Picture Gallery in London, RIBA East Midlands and the Real Creative Futures Programme at New Art Exchange in Nottingham. I'm also working with cultural and academic organisations across North-America and the Netherlands. |
| First Year Of Impact | 2019 |
| Sector | Communities and Social Services/Policy,Creative Economy,Education,Healthcare,Culture, Heritage, Museums and Collections |
| Impact Types | Cultural,Societal |
| Description | Dementia and the Arts: Sharing Practice, Developing Understanding and Enhancing Lives |
| Geographic Reach | Multiple continents/international |
| Policy Influence Type | Influenced training of practitioners or researchers |
| Impact | This course has been accredited by the Royal Society for Public Health (RSPH), offering learners Continuing Professional Development (CPD) points, and has already attracted over 2200 participants during its first run in 2018. The second run started on the 21st of January 2019. |
| URL | https://www.futurelearn.com/courses/dementia-arts |
| Title | A predictive foveal fixation model for visual exploration |
| Description | I devised a hierarchical foveal interest area method to predict on which image features people will fixate on for the longest duation in colour spaces, visual artworks and complex images. This hierarchical foveal interest area model of visual exploration could accurately predict on which image features neurologically healthy people would foveate on for the longest duration. I was also able to identify different fixation patterns in different visual processing stages in people with various forms of dementia. |
| Type Of Material | Physiological assessment or outcome measure |
| Year Produced | 2017 |
| Provided To Others? | No |
| Impact | My ambition is to further develop this method into a monitoring instrument to investigate on personal and cohort levels where people fixate on the longest in colour spaces, visual artworks and complex images. I also will work towards developing this method into a sensitive and humanistic diagnostic tool for clinicians to assess people with who have a potential dementia. |
| Title | Mood Shade Scale |
| Description | To assess current mood, a possible covariate of the subjective measurements in the experiments, I developed a new visual rating scale: the Mood Shade Scale. This 5-point visual rating scale offers a more abstract non-verbal alternative to the prevailing emoticon scales used to measure affective states in people with dementia syndromes. |
| Type Of Material | Physiological assessment or outcome measure |
| Year Produced | 2017 |
| Provided To Others? | No |
| Impact | The Mood Shade Scale was experienced as an intuitive and easy to use visual tool to rate current mood with by both the neurologically healthy as well as the research participants with a dementia diagnosis. The Mood Shade Score scores are now being used in the ongoing research analysis. |
| Title | Resonance Radius Scale |
| Description | The Resonance Radius Scale is a 5-point visual rating scale which I designed to create an abstract, non-verbal tool to measure personal resonance with presented stimuli. |
| Type Of Material | Physiological assessment or outcome measure |
| Year Produced | 2017 |
| Provided To Others? | No |
| Impact | The visual symbols of the Resonance Radius Scale were experienced as an intuitive and easy to use visual tool to rate personal resonance with by both the neurologically healthy as well as the research participants with a dementia diagnosis. The order of the accompanying numbers to indicate the resonance response sometimes created confusion in the participants however and this aspect will be revised before further use of the scale. The Resonance Radius scores are now being used in the ongoing research analysis. |
| Title | The Affect Amplitude Scale |
| Description | The Affect Amplitude Scale is a 5-point visual rating scale which I designed to create an abstract, non-verbal tool to measure people's affective responses to presented stimuli. |
| Type Of Material | Physiological assessment or outcome measure |
| Year Produced | 2017 |
| Provided To Others? | No |
| Impact | The visual symbols of the Affect Amplitude Scale were experienced as an intuitive and easy to use visual tool to rate affective response with by both the neurologically healthy as well as the research participants with a dementia diagnosis. The order of the accompanying numbers to indicate the affective response sometimes created confusion in the participants however and this aspect will be revised before further use of the scale. The Affect Amplitude scores are now being used in the ongoing research analysis. |
| Title | Visual Speech Analysis Method |
| Description | I have devised a quantitative analysis method for speech responses to visual artworks and complex images. With this method I have been able to derive a logical rules system that was able to identify research participants with a dementia diagnosis with 90% reliability based only on their age and mean values on the quantitative speech parameters I developed. |
| Type Of Material | Physiological assessment or outcome measure |
| Year Produced | 2019 |
| Provided To Others? | No |
| Impact | My ambition is to further develop this visual speech analysis method into a sensitive and humanistic screening tool for dementia symptoms in the general public and furthermore develop an arts-based speech training programme that addressses the specific changes in language that occur in various dementia variants. |
| Title | Thinking Eyes Database |
| Description | This database contains multimodal measurements of engagement with colour spaces, complex images and 2-D visual artworks collected in the following 3 research cohorts: -Neurologically healthy young adults (N=17, F=10, M=7) -Neurologically healthy senior adults (N=2-, F=10, M=10) -Senior adults living with various forms of dementia (N=13, F=3, M=10) The collected qualitative and quantitative data includes: -Visual exploration -Auditory perception -Speech production -Electrodermal activity -Heart rate -Internal state evaluation |
| Type Of Material | Database/Collection of data |
| Year Produced | 2018 |
| Provided To Others? | No |
| Impact | This database is currently being used in the analysis of my research outcomes and will form the basis of future scientific publications, therapeutic interventions and educational resources. |
| Description | Gerrit Rietveld Academie |
| Organisation | Gerrit Rietveld Academy |
| Country | Netherlands |
| Sector | Academic/University |
| PI Contribution | Over the past two years I have been invited as a guest lecturer twice, sharing my expertise on the relationship between visual imagination and the social brain with the art and design students at the Gerrit Rietveld Academy. |
| Collaborator Contribution | The Gerrit Rietveld Academy is sponsoring Professor Jeroen Boomgaard to supervise the artistic research of my studentship. |
| Impact | The outcomes of the artistic research will be described in the PhD thesis and will also result in a public art exhibition, planned to take place in September/October 2019. |
| Start Year | 2016 |
| Description | Created Out of Mind: Shaping Public Perception of Dementia through the Arts and Sciences |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Public/other audiences |
| Results and Impact | This participatory event explored how the arts+the sciences can impact research and dementia care. The audience were invited to 'test out' 4 art experiences that were being developed by collaborators at the Wellcome Collection's Hub residency in dementia. People with and without dementia were welcome to attend, around 50 people from the general public took part. |
| Year(s) Of Engagement Activity | 2017 |
| URL | http://www.creativityandwellbeing.org.uk/week/events/created-out-mind-shaping-public-perception-deme... |
| Description | Pint of Science/Creative Reactions |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Public/other audiences |
| Results and Impact | I created an interactive virtual reality installation to introduce my colour research at this public event attended by 200 people, which aimed to challenge the traditional definitions and common misconceptions, while providing the most current research advances and understanding of dementia through a multimedia approach. The installation was a great success, with long queues of people wanting to take part forming. Afterwards, many people came up to me to express their interest and ask me more about my research. |
| Year(s) Of Engagement Activity | 2017 |
| URL | https://pintofscience.co.uk/event/dementia |
| Description | Seminar at the UCL Institute of Making |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Postgraduate students |
| Results and Impact | Every month the UCL Institute of Making invites a researcher to talk about how they navigate the Makespace to develop their academic work. Members are invited to come along to make new connections, and generate ideas for how the tools in the Makespace could be used in their own research. To an audience of about 12 research students I presented the Thresholds/Virtual Colour Rooms Installation I made at the Institute of Making, which was a public adaptation of one of my PhD research projects at the UCL Institute of Neurology and the Wellcome Collection. |
| Year(s) Of Engagement Activity | 2018 |
| URL | https://www.instituteofmaking.org.uk/events/series/research-events |