Space, Place, Sound, and Memory: Immersive Experiences of the Past

Lead Research Organisation: University of Edinburgh
Department Name: Edinburgh College of Art

Abstract

Listening to music is an experiential activity that connects listeners to their surroundings and to those around them. In part, the recent growth of the live performance industry is a direct consequence of this need to connect and share musical experiences in a communal space.

But the transient nature of live performance presents real challenges. The physical properties and locations of spaces impose constraints on the nature of events and the geographical reach of performances, and, while recordings can capture the sound of performance, they stop short of allowing the listener to feel a sense of presence and participation. This is a challenge that is only amplified when one considers early music performances: even a curated performance in a modern venue loses much of the detail that characterises historic performance.

Immersive technologies offer huge potential for modern audiences to experience these transient qualities, and for allowing performers to recreate historic performances as they would originally have been experienced. This project brings together cross-disciplinary expertise from a range of academic, industry and cultural partners to explore the point where performance practice, gaming and VR, technology, and culture and heritage meet.

We will use 3D imaging, binaural and surround sound, and room-impulse responses to create a software application that allows users to experience the performance of early music in an accurately-modelled historic space. The development of this application will draw on the development experience of our technical partner, Biome Collective, who have significant experience of using mobile technologies to create immersive augmented reality experiences.

We will work with two contrasting spaces and related repertories: St Cecilia's Hall and Rosslyn Chapel. Both sites benefit from pre-existing architectural research, and there are extensive records of historical concerts, which will enable us to recreate particular musical events using instruments from the Russell Collection, and a performance of a sung liturgical service by the renowned Binchois Consort in combination with our software.

From an audience perspective we will explore how immersive media technologies might bring us closer to the original experience of early music, while from the perspective of performers and musicologists, it will allow us, for the first time, to explore systematically concepts of space and place within the context of historic performance.

Through the process of creating this new technology, we will explore and create an outline taxonomy of the key psycho-physical cues that promote the sense of presence and immersion within a shared simulated performance space, describe how they combine to create convincing spaces, and investigate the methods that allow us to measure their efficacy.

By hosting the Binchois Consort's performance in virtual space, we will also have the opportunity to explore a number of questions relating to performance that have the potential to serve as the basis for a much deeper follow-on investigation: To what extent does performing in a virtual space impact upon performance practice? What are the challenges and opportunities involved in bringing together musicians and audiences who are geographically remote to a co-located virtual space? How might immersive media technologies change how we curate physical and digital performance spaces? How might they be used to develop existing audiences and reach new ones, particularly those who are hard-to-reach?

This final question is of direct commercial relevance, particularly to our partner organisation, the Royal Scottish National Orchestra, who are committed to using technology as a tool to develop audiences and extend their existing outreach work, suggesting a possible avenue for commercialisation and further development. The RSNO have kindly agreed to assist us in working directly with their audiences for user testing.

Planned Impact

Our key academic beneficiaries are academics working in and around Early Music; Sound/Interaction Design, and Physical/VR/AR Computing. By documenting process and workflow, and generating new acoustic models and sound recordings, we will provide case-study material and data sets for all three groups. In particular, the exploration of early music performance and its relationship to space will provide significant new insights into the understanding of performance practice and generate new research questions that will be the subject of a follow-on funding application.

There are three main non-academic beneficiaries:
1. The national technology sector
While this project is focused on developing software that co-locates geographically-remote musicians and audiences within virtual spaces (either historically-recreated or fictional) to perform and spectate - which has commercial potential - the research will provide insights into the role of auditory feedback in supporting immersion/presence within virtual spaces, which has direct applications in computer gaming and virtual reality. VR is an emerging technology and developers are currently concerned with systems of control, visualization, and movement, particularly around motion sickness. Sound has been somewhat neglected, yet it offers enormous potential to situate users within spaces.
In Biome, we have a partner that has experience of multi-modal software development and understands the commercialization process and routes to market from conceptual prototypes, while in The Binchois Consort, the RSNO, and heritage spaces like St Cecilia's Hall, we have access to established networks of potential users.

2. Early music performers and audiences
The physical dimensions and locations of many historic Scottish spaces impose limits on their use in performance, and the country's unique geography and population distribution presents challenges in engaging geographically-dispersed and other hard-to-reach communities in site-specific performances. This project would equip performers with a new set of tools and methods, and provide audiences with new ways of engaging with and experiencing site-specific performances.
In addition, by providing musicians with unconstrained access to virtual acoustic spaces, performers will be able to rehearse in a site-specific way, which is simply not possible otherwise. This embedding of space in performance practice should lead to new insights and perspectives on established repertoire, the nature of historic performance practice, and perhaps new or rediscovered performance technique in response.

3. National cultural and heritage organizations
The collaborative music-making technologies that this project aims to develop represent an opportunity for cultural organizations, like the RSNO, to greatly expand the reach of their existing engagement work and to grow new audiences, particularly hard-to-reach audiences. By building anonymous user analytics directly into the software, we will have a means of capturing and visualizing its impact.
For heritage agencies, such as Historic Scotland, the technology represents an opportunity to explore, non-invasively, the architectural acoustics of their historic sites, and to situate experiential visitor attractions on site. The technologies developed by this project have the potential to provide greater and easier public access to the outcomes of such site-specific research to the substantial audiences who visit heritage sites each year.
The technology also provides a means of democratizing such research, and provides a vehicle for communities to use the technology to explore, non-invasively, the performance characteristics of historical spaces/events that are not of national importance, but which are, nevertheless, significant to local communities.

Publications

10 25 50
 
Title Pilot VR applications 
Description A pilot VR application creating a reconstruction, in acoustic and image, of St Cecilia's Hall and the chapel of Linlithgow Palace, with appropriately chosen historically informed music. 
Type Of Art Film/Video/Animation 
Year Produced 2018 
Impact Several other research projects attempting similar reconstructions have contacted us wanting our input. Historic Scotland have sought our input on sound for audio guides for Glasgow Cathedral and Edinburgh Castle. The PI has been asked to be a part of a proposal to produce a similar reconstruction of a medieval hunting lodge in England. 
 
Description We have demonstrated that it is possible, through detailed research and through engagement with other historians, archaeologists, musicologists, acousticians, performers, games developers, and curators, to produce VR applications that are able to allow users to interact with a visual and acoustic reconstruction of lost performance spaces, and to experience carefully selected historically informed performance within these. In doing so, we have tested the limits of the current state of the art, identifying areas for improvement in future projects. We have demonstrated, through a research paper, how close we can come to recreating the acoustic of a space accurately through our reconstruction process, outlining which of the current technologies come closest. Though we have demonstrated that, mathematically, even our best reproduction is not exact, we have also shown that it is indistinguishable for the non-expert user. Improving the accuracy of this reconstruction, within a framework that still allows for users to move within the space is a key aim for our future research - something that could have applicable use in the gaming industry.

We have also learnt a lot about how to record in anechoic conditions, as is required for our VR workflow. We have experimented with different approaches to this and come up with a blended approach which balances the scientific and artistic needs of the recording process. This approach will inform our CD recording with The Binchois Consort, published by Hyperion, which will implement our approach on a much larger, commercial level as part of our follow-on project.

Our research has also given us new insight into both of the spaces which we reconstructed. It has demonstrated that the initial architect of St Cecilia's gave much thought to the acoustic properties of the hall, and that its earliest users made further changes in order to improve it as a performance space. It has also raised detailed questions about the layout of the devotional space within Linlithgow Palace Chapel, something which might impact on our understanding of how the space functioned.

As an unintended consequence of research, I have spent a long period of time researching two Mass cycles which we chose as appropriate for performance within our reconstructed spaces. I have come to the conclusion that both may well be, in contradiction to scholarly consensus, Scottish works. This would push back the earliest date of surviving Scottish Mass cycles by some 50 years (an article on this is forthcoming).
Exploitation Route Our work on acoustic modelling flags up necessary areas for further research in order to make usable real-time renderings of acoustic spaces that can react to user movement, something that would be extremely useful for game engines. Our approach to recording in anechoic conditions can be used to inform other attempts to record in similar conditions. The detailed exploration of the lost spaces of Linlithgow Palace Chapel and St Cecilia's Hall can feed in to future research on the spaces, as well as informing public-facing interpretations of both spaces.
Sectors Creative Economy,Digital/Communication/Information Technologies (including Software),Education,Leisure Activities, including Sports, Recreation and Tourism,Culture, Heritage, Museums and Collections

URL https://www.ed.ac.uk/edinburgh-college-art/reid-school-music/immersive-history
 
Description To-date, our impact has related primarily to enhancement of quality of life through engagement events which increased the public's awareness of the sounds of history. Our impact and engagement follow-on relates specifically to producing direct impact on the global economic performance of the record label Hyperion and of Historic Environment Scotland, as well as producing a step-change in the understanding of the general public towards music and sound of the past.
First Year Of Impact 2018
Sector Culture, Heritage, Museums and Collections
Impact Types Cultural

 
Description Follow-on Funding for Impact and Engagement
Amount £39,924 (GBP)
Funding ID AH/S010653/1 
Organisation Arts & Humanities Research Council (AHRC) 
Sector Public
Country United Kingdom
Start 02/2019 
End 11/2019
 
Description Game Music Bit-by-Bit - AES Annual Lecture 
Form Of Engagement Activity A broadcast e.g. TV/radio/film/podcast (other than news/press)
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Schools
Results and Impact The project Co-I, Dr Kenny McAlpine, gave the AES Scotland Annual lecture, filmed by BBC Scotland with the assistance of local college students, and broadcast online. The project PI was invited as a special guest. He gave a 10-minute introduction to the project, including a live demonstration of the prototype. The broadcast was used as the basis of several college lectures on audio engineering. Several members of the audience approached the project team afterwards indicating a new interest in VR audio.
Year(s) Of Engagement Activity 2018
URL https://www.youtube.com/watch?v=bUFnxIW5Nso
 
Description South by Southwest 2019 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact Attendance at South by Southwest as an invited member of the AHRC Immersive Mission including a demonstration to attendees of our pilot project.
Year(s) Of Engagement Activity 2019
 
Description The Binchois Consort: Hearing Historic Scotland 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Public/other audiences
Results and Impact A concert by our project partners 'The Binchois Consort' presenting music from our project, around which the project team wove a lengthy lecture about and the history and science behind it. It culminated in a demonstration of the work-in-progress prototype. A delegation from Innovate UK attended, alongside the general public. We had several inquiries from attending tech companies about involvement in follow-on projects.
Year(s) Of Engagement Activity 2018
 
Description Workshop on Space, Place, Sound, and Memory for the MSc in Collections and Curating practices 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Postgraduate students
Results and Impact A workshop with current MSc in Collections and Curating Practices students, demonstrating how we used VR in our own curating of space. It sparked several follow-up emails from students. It is hoped that they go on to careers in curation and make use of sound in VR within these.
Year(s) Of Engagement Activity 2018
 
Description World-leading Research: A Celebration of Impact 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Industry/Business
Results and Impact Demonstration of the pilot at an event celebrating impact at which members of staff from the university, and local and national industry partners and other collaborators were present. Many people tried out the software. After experiencing it here, a colleague from Theology wants the project to demonstrate to colleagues and students how to use technology in project such as this to research historical questions.
Year(s) Of Engagement Activity 2019