Space, Place, Sound, and Memory: Immersive Experiences of the Past
Lead Research Organisation:
University of Edinburgh
Department Name: Edinburgh College of Art
Abstract
Listening to music is an experiential activity that connects listeners to their surroundings and to those around them. In part, the recent growth of the live performance industry is a direct consequence of this need to connect and share musical experiences in a communal space.
But the transient nature of live performance presents real challenges. The physical properties and locations of spaces impose constraints on the nature of events and the geographical reach of performances, and, while recordings can capture the sound of performance, they stop short of allowing the listener to feel a sense of presence and participation. This is a challenge that is only amplified when one considers early music performances: even a curated performance in a modern venue loses much of the detail that characterises historic performance.
Immersive technologies offer huge potential for modern audiences to experience these transient qualities, and for allowing performers to recreate historic performances as they would originally have been experienced. This project brings together cross-disciplinary expertise from a range of academic, industry and cultural partners to explore the point where performance practice, gaming and VR, technology, and culture and heritage meet.
We will use 3D imaging, binaural and surround sound, and room-impulse responses to create a software application that allows users to experience the performance of early music in an accurately-modelled historic space. The development of this application will draw on the development experience of our technical partner, Biome Collective, who have significant experience of using mobile technologies to create immersive augmented reality experiences.
We will work with two contrasting spaces and related repertories: St Cecilia's Hall and Rosslyn Chapel. Both sites benefit from pre-existing architectural research, and there are extensive records of historical concerts, which will enable us to recreate particular musical events using instruments from the Russell Collection, and a performance of a sung liturgical service by the renowned Binchois Consort in combination with our software.
From an audience perspective we will explore how immersive media technologies might bring us closer to the original experience of early music, while from the perspective of performers and musicologists, it will allow us, for the first time, to explore systematically concepts of space and place within the context of historic performance.
Through the process of creating this new technology, we will explore and create an outline taxonomy of the key psycho-physical cues that promote the sense of presence and immersion within a shared simulated performance space, describe how they combine to create convincing spaces, and investigate the methods that allow us to measure their efficacy.
By hosting the Binchois Consort's performance in virtual space, we will also have the opportunity to explore a number of questions relating to performance that have the potential to serve as the basis for a much deeper follow-on investigation: To what extent does performing in a virtual space impact upon performance practice? What are the challenges and opportunities involved in bringing together musicians and audiences who are geographically remote to a co-located virtual space? How might immersive media technologies change how we curate physical and digital performance spaces? How might they be used to develop existing audiences and reach new ones, particularly those who are hard-to-reach?
This final question is of direct commercial relevance, particularly to our partner organisation, the Royal Scottish National Orchestra, who are committed to using technology as a tool to develop audiences and extend their existing outreach work, suggesting a possible avenue for commercialisation and further development. The RSNO have kindly agreed to assist us in working directly with their audiences for user testing.
But the transient nature of live performance presents real challenges. The physical properties and locations of spaces impose constraints on the nature of events and the geographical reach of performances, and, while recordings can capture the sound of performance, they stop short of allowing the listener to feel a sense of presence and participation. This is a challenge that is only amplified when one considers early music performances: even a curated performance in a modern venue loses much of the detail that characterises historic performance.
Immersive technologies offer huge potential for modern audiences to experience these transient qualities, and for allowing performers to recreate historic performances as they would originally have been experienced. This project brings together cross-disciplinary expertise from a range of academic, industry and cultural partners to explore the point where performance practice, gaming and VR, technology, and culture and heritage meet.
We will use 3D imaging, binaural and surround sound, and room-impulse responses to create a software application that allows users to experience the performance of early music in an accurately-modelled historic space. The development of this application will draw on the development experience of our technical partner, Biome Collective, who have significant experience of using mobile technologies to create immersive augmented reality experiences.
We will work with two contrasting spaces and related repertories: St Cecilia's Hall and Rosslyn Chapel. Both sites benefit from pre-existing architectural research, and there are extensive records of historical concerts, which will enable us to recreate particular musical events using instruments from the Russell Collection, and a performance of a sung liturgical service by the renowned Binchois Consort in combination with our software.
From an audience perspective we will explore how immersive media technologies might bring us closer to the original experience of early music, while from the perspective of performers and musicologists, it will allow us, for the first time, to explore systematically concepts of space and place within the context of historic performance.
Through the process of creating this new technology, we will explore and create an outline taxonomy of the key psycho-physical cues that promote the sense of presence and immersion within a shared simulated performance space, describe how they combine to create convincing spaces, and investigate the methods that allow us to measure their efficacy.
By hosting the Binchois Consort's performance in virtual space, we will also have the opportunity to explore a number of questions relating to performance that have the potential to serve as the basis for a much deeper follow-on investigation: To what extent does performing in a virtual space impact upon performance practice? What are the challenges and opportunities involved in bringing together musicians and audiences who are geographically remote to a co-located virtual space? How might immersive media technologies change how we curate physical and digital performance spaces? How might they be used to develop existing audiences and reach new ones, particularly those who are hard-to-reach?
This final question is of direct commercial relevance, particularly to our partner organisation, the Royal Scottish National Orchestra, who are committed to using technology as a tool to develop audiences and extend their existing outreach work, suggesting a possible avenue for commercialisation and further development. The RSNO have kindly agreed to assist us in working directly with their audiences for user testing.
Planned Impact
Our key academic beneficiaries are academics working in and around Early Music; Sound/Interaction Design, and Physical/VR/AR Computing. By documenting process and workflow, and generating new acoustic models and sound recordings, we will provide case-study material and data sets for all three groups. In particular, the exploration of early music performance and its relationship to space will provide significant new insights into the understanding of performance practice and generate new research questions that will be the subject of a follow-on funding application.
There are three main non-academic beneficiaries:
1. The national technology sector
While this project is focused on developing software that co-locates geographically-remote musicians and audiences within virtual spaces (either historically-recreated or fictional) to perform and spectate - which has commercial potential - the research will provide insights into the role of auditory feedback in supporting immersion/presence within virtual spaces, which has direct applications in computer gaming and virtual reality. VR is an emerging technology and developers are currently concerned with systems of control, visualization, and movement, particularly around motion sickness. Sound has been somewhat neglected, yet it offers enormous potential to situate users within spaces.
In Biome, we have a partner that has experience of multi-modal software development and understands the commercialization process and routes to market from conceptual prototypes, while in The Binchois Consort, the RSNO, and heritage spaces like St Cecilia's Hall, we have access to established networks of potential users.
2. Early music performers and audiences
The physical dimensions and locations of many historic Scottish spaces impose limits on their use in performance, and the country's unique geography and population distribution presents challenges in engaging geographically-dispersed and other hard-to-reach communities in site-specific performances. This project would equip performers with a new set of tools and methods, and provide audiences with new ways of engaging with and experiencing site-specific performances.
In addition, by providing musicians with unconstrained access to virtual acoustic spaces, performers will be able to rehearse in a site-specific way, which is simply not possible otherwise. This embedding of space in performance practice should lead to new insights and perspectives on established repertoire, the nature of historic performance practice, and perhaps new or rediscovered performance technique in response.
3. National cultural and heritage organizations
The collaborative music-making technologies that this project aims to develop represent an opportunity for cultural organizations, like the RSNO, to greatly expand the reach of their existing engagement work and to grow new audiences, particularly hard-to-reach audiences. By building anonymous user analytics directly into the software, we will have a means of capturing and visualizing its impact.
For heritage agencies, such as Historic Scotland, the technology represents an opportunity to explore, non-invasively, the architectural acoustics of their historic sites, and to situate experiential visitor attractions on site. The technologies developed by this project have the potential to provide greater and easier public access to the outcomes of such site-specific research to the substantial audiences who visit heritage sites each year.
The technology also provides a means of democratizing such research, and provides a vehicle for communities to use the technology to explore, non-invasively, the performance characteristics of historical spaces/events that are not of national importance, but which are, nevertheless, significant to local communities.
There are three main non-academic beneficiaries:
1. The national technology sector
While this project is focused on developing software that co-locates geographically-remote musicians and audiences within virtual spaces (either historically-recreated or fictional) to perform and spectate - which has commercial potential - the research will provide insights into the role of auditory feedback in supporting immersion/presence within virtual spaces, which has direct applications in computer gaming and virtual reality. VR is an emerging technology and developers are currently concerned with systems of control, visualization, and movement, particularly around motion sickness. Sound has been somewhat neglected, yet it offers enormous potential to situate users within spaces.
In Biome, we have a partner that has experience of multi-modal software development and understands the commercialization process and routes to market from conceptual prototypes, while in The Binchois Consort, the RSNO, and heritage spaces like St Cecilia's Hall, we have access to established networks of potential users.
2. Early music performers and audiences
The physical dimensions and locations of many historic Scottish spaces impose limits on their use in performance, and the country's unique geography and population distribution presents challenges in engaging geographically-dispersed and other hard-to-reach communities in site-specific performances. This project would equip performers with a new set of tools and methods, and provide audiences with new ways of engaging with and experiencing site-specific performances.
In addition, by providing musicians with unconstrained access to virtual acoustic spaces, performers will be able to rehearse in a site-specific way, which is simply not possible otherwise. This embedding of space in performance practice should lead to new insights and perspectives on established repertoire, the nature of historic performance practice, and perhaps new or rediscovered performance technique in response.
3. National cultural and heritage organizations
The collaborative music-making technologies that this project aims to develop represent an opportunity for cultural organizations, like the RSNO, to greatly expand the reach of their existing engagement work and to grow new audiences, particularly hard-to-reach audiences. By building anonymous user analytics directly into the software, we will have a means of capturing and visualizing its impact.
For heritage agencies, such as Historic Scotland, the technology represents an opportunity to explore, non-invasively, the architectural acoustics of their historic sites, and to situate experiential visitor attractions on site. The technologies developed by this project have the potential to provide greater and easier public access to the outcomes of such site-specific research to the substantial audiences who visit heritage sites each year.
The technology also provides a means of democratizing such research, and provides a vehicle for communities to use the technology to explore, non-invasively, the performance characteristics of historical spaces/events that are not of national importance, but which are, nevertheless, significant to local communities.
Organisations
- University of Edinburgh (Lead Research Organisation)
- Engineering and Physical Sciences Research Council (Co-funder)
- Historic Environment Scotland (Collaboration)
- Biome Collective (Project Partner)
- St Cecilia's Hall (Project Partner)
- The Royal Scottish National Orchestra (Project Partner)
- The Binchois Consort (Project Partner)
Publications
Selfridge, R
(2019)
Creating Historic Spaces in Virtual Reality Using off-the-shelf Audio Plugins
McAlpine K.
(2021)
3D Audio
Cook, J.
(2022)
Point Cloud to Sound Cloud: Digital Innovation and Historic Sound at Linlithgow Palace
in Magazen
Cook, J.
(2023)
Hearing Historic Scotland: Reflections on Recording in Virtually Reconstructed Acoustics
in Journal of the Alamire Foundation
Title | Music for the King of Scots: Inside the Pleasure Palace of James IV |
Description | A commercial CD released by Hyperion Records, performed by the Binchois Consort. |
Type Of Art | Artefact (including digital) |
Year Produced | 2021 |
Impact | Recording of impact on both the recording industry and audience is still ongoing. |
URL | https://www.hyperion-records.co.uk/dc.asp?dc=D_CDA68333 |
Title | Pilot VR applications |
Description | A pilot VR application creating a reconstruction, in acoustic and image, of St Cecilia's Hall and the chapel of Linlithgow Palace, with appropriately chosen historically informed music. |
Type Of Art | Film/Video/Animation |
Year Produced | 2018 |
Impact | Several other research projects attempting similar reconstructions have contacted us wanting our input. Historic Scotland have sought our input on sound for audio guides for Glasgow Cathedral and Edinburgh Castle. The PI has been asked to be a part of a proposal to produce a similar reconstruction of a medieval hunting lodge in England. |
Description | We have demonstrated that it is possible, through detailed research and through engagement with other historians, archaeologists, musicologists, acousticians, performers, games developers, and curators, to produce VR applications that are able to allow users to interact with a visual and acoustic reconstruction of lost performance spaces, and to experience carefully selected historically informed performance within these. In doing so, we have tested the limits of the current state of the art, id |
Exploitation Route | Our work on acoustic modelling flags up necessary areas for further research in order to make usable real-time renderings of acoustic spaces that can react to user movement, something that would be extremely useful for game engines. Our approach to recording in anechoic conditions can be used to inform other attempts to record in similar conditions. The detailed exploration of the lost spaces of Linlithgow Palace Chapel and St Cecilia's Hall can feed in to future research on the spaces, as well |
Sectors | Creative Economy Digital/Communication/Information Technologies (including Software) Education Leisure Activities including Sports Recreation and Tourism Culture Heritage Museums and Collections |
URL | https://www.ed.ac.uk/edinburgh-college-art/reid-school-music/immersive-history |
Description | To-date, our impact has related primarily to enhancement of quality of life through engagement events which increased the public's awareness of the sounds of history. Our impact and engagement follow-on relates specifically to producing direct impact on the global economic performance of the record label Hyperion and of Historic Environment Scotland, as well as producing a step-change in the understanding of the general public towards music and sound of the past. |
First Year Of Impact | 2018 |
Sector | Culture, Heritage, Museums and Collections |
Impact Types | Cultural |
Description | Follow-on Funding for Impact and Engagement |
Amount | £39,924 (GBP) |
Funding ID | AH/S010653/1 |
Organisation | Arts & Humanities Research Council (AHRC) |
Sector | Public |
Country | United Kingdom |
Start | 02/2019 |
End | 11/2019 |
Description | Historic Environment Scotland |
Organisation | Historic Environment Scotland |
Country | United Kingdom |
Sector | Public |
PI Contribution | The PI has, following the successful partnership as part of both funded projects, continued to work with HES on music for audioguides at a number of sites (Edinburgh Castle, Melrose Abbey, Linlithgow Palace). |
Collaborator Contribution | HES have enabled research by the PI to reach a large number of visitors to their properties. |
Impact | Audioguide for Edinburgh Castle Audioguide for Melrose Abbey Audioguide for Linlithgow Palace |
Start Year | 2019 |
Description | 'Linlithgow Palace Reborn', The Early Music Show |
Form Of Engagement Activity | A broadcast e.g. TV/radio/film/podcast (other than news/press) |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Media (as a channel to the public) |
Results and Impact | BBC Radio 3's The Early Music Show ran a programme about the Binchois Consort CD, which was an output from both AHRC projects. The programme featured music from the CD as well as lengthy discussion between myself and Hannah French about my research for the project. I subsequently fielded a number of emails about the project and sales of the CD increased. |
Year(s) Of Engagement Activity | 2021 |
URL | https://www.bbc.co.uk/programmes/m000xshs |
Description | Coverage in over 100 national and international print media outlets |
Form Of Engagement Activity | A press release, press conference or response to a media enquiry/interview |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Media (as a channel to the public) |
Results and Impact | The release of the Binchois Consort CD output, and the research by which this was underpinned, was covered in articles in over 100 national and international newspapers and magazines, including The Guardian and Telegraph. I was interviewed for several of these. |
Year(s) Of Engagement Activity | 2021 |
URL | https://www.theguardian.com/uk-news/2021/apr/05/gaming-technology-recreates-c16th-music-in-a-scottis... |
Description | Game Music Bit-by-Bit - AES Annual Lecture |
Form Of Engagement Activity | A broadcast e.g. TV/radio/film/podcast (other than news/press) |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Schools |
Results and Impact | The project Co-I, Dr Kenny McAlpine, gave the AES Scotland Annual lecture, filmed by BBC Scotland with the assistance of local college students, and broadcast online. The project PI was invited as a special guest. He gave a 10-minute introduction to the project, including a live demonstration of the prototype. The broadcast was used as the basis of several college lectures on audio engineering. Several members of the audience approached the project team afterwards indicating a new interest in VR audio. |
Year(s) Of Engagement Activity | 2018 |
URL | https://www.youtube.com/watch?v=bUFnxIW5Nso |
Description | Interview with 'Good Morning Scotland', BBC Radio Scotland |
Form Of Engagement Activity | A press release, press conference or response to a media enquiry/interview |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Media (as a channel to the public) |
Results and Impact | An interview with the Good Morning Scotland, BBC Radio Scotland, about the production of the CD output from the two AHRC projects. BBC Radio Scotland's 'Sunday Morning' subsequently booked me for a longer interview piece. |
Year(s) Of Engagement Activity | 2021 |
Description | Interview with 'Sunday Morning, BBC Radio Scotland |
Form Of Engagement Activity | A press release, press conference or response to a media enquiry/interview |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Media (as a channel to the public) |
Results and Impact | An interview with the Sunday Morning, BBC Radio Scotland, about the production of the CD output from the two AHRC projects. BBC Radio 3 subsequently got in touch to design an entire episode of The Early Music Show around our research. |
Year(s) Of Engagement Activity | 2021 |
Description | Interview with BBC World Service |
Form Of Engagement Activity | A press release, press conference or response to a media enquiry/interview |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Media (as a channel to the public) |
Results and Impact | An interview with the BBC World Service about the production of the CD output from the two AHRC projects. There were subsequent email requests for further information. |
Year(s) Of Engagement Activity | 2021 |
Description | South by Southwest 2019 |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | Attendance at South by Southwest as an invited member of the AHRC Immersive Mission including a demonstration to attendees of our pilot project. |
Year(s) Of Engagement Activity | 2019 |
Description | Talk at Linlithgow Museum |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Other audiences |
Results and Impact | I gave a talk to Linlithgow Museum about the project, followed by question and discussion afterwards. Several attendees planned to visit the VR installation output and to purchase the CD output. |
Year(s) Of Engagement Activity | 2021 |
Description | The Binchois Consort: Hearing Historic Scotland |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Public/other audiences |
Results and Impact | A concert by our project partners 'The Binchois Consort' presenting music from our project, around which the project team wove a lengthy lecture about and the history and science behind it. It culminated in a demonstration of the work-in-progress prototype. A delegation from Innovate UK attended, alongside the general public. We had several inquiries from attending tech companies about involvement in follow-on projects. |
Year(s) Of Engagement Activity | 2018 |
Description | Workshop on Space, Place, Sound, and Memory for the MSc in Collections and Curating practices |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Postgraduate students |
Results and Impact | A workshop with current MSc in Collections and Curating Practices students, demonstrating how we used VR in our own curating of space. It sparked several follow-up emails from students. It is hoped that they go on to careers in curation and make use of sound in VR within these. |
Year(s) Of Engagement Activity | 2018 |
Description | World-leading Research: A Celebration of Impact |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | Regional |
Primary Audience | Industry/Business |
Results and Impact | Demonstration of the pilot at an event celebrating impact at which members of staff from the university, and local and national industry partners and other collaborators were present. Many people tried out the software. After experiencing it here, a colleague from Theology wants the project to demonstrate to colleagues and students how to use technology in project such as this to research historical questions. |
Year(s) Of Engagement Activity | 2019 |