Space, Place, Sound, and Memory: Immersive Experiences of the Past

Listening to music is an experiential activity that connects listeners to their surroundings and to those around them. In part, the recent growth of the live performance industry is a direct consequence of this need to connect and share musical experiences in a communal space.

But the transient nature of live performance presents real challenges. The physical properties and locations of spaces impose constraints on the nature of events and the geographical reach of performances, and, while recordings can capture the sound of performance, they stop short of allowing the listener to feel a sense of presence and participation. This is a challenge that is only amplified when one considers early music performances: even a curated performance in a modern venue loses much of the detail that characterises historic performance.

Immersive technologies offer huge potential for modern audiences to experience these transient qualities, and for allowing performers to recreate historic performances as they would originally have been experienced. This project brings together cross-disciplinary expertise from a range of academic, industry and cultural partners to explore the point where performance practice, gaming and VR, technology, and culture and heritage meet.

We will use 3D imaging, binaural and surround sound, and room-impulse responses to create a software application that allows users to experience the performance of early music in an accurately-modelled historic space. The development of this application will draw on the development experience of our technical partner, Biome Collective, who have significant experience of using mobile technologies to create immersive augmented reality experiences.

We will work with two contrasting spaces and related repertories: St Cecilia's Hall and Rosslyn Chapel. Both sites benefit from pre-existing architectural research, and there are extensive records of historical concerts, which will enable us to recreate particular musical events using instruments from the Russell Collection, and a performance of a sung liturgical service by the renowned Binchois Consort in combination with our software.

From an audience perspective we will explore how immersive media technologies might bring us closer to the original experience of early music, while from the perspective of performers and musicologists, it will allow us, for the first time, to explore systematically concepts of space and place within the context of historic performance.

Through the process of creating this new technology, we will explore and create an outline taxonomy of the key psycho-physical cues that promote the sense of presence and immersion within a shared simulated performance space, describe how they combine to create convincing spaces, and investigate the methods that allow us to measure their efficacy.

By hosting the Binchois Consort's performance in virtual space, we will also have the opportunity to explore a number of questions relating to performance that have the potential to serve as the basis for a much deeper follow-on investigation: To what extent does performing in a virtual space impact upon performance practice? What are the challenges and opportunities involved in bringing together musicians and audiences who are geographically remote to a co-located virtual space? How might immersive media technologies change how we curate physical and digital performance spaces? How might they be used to develop existing audiences and reach new ones, particularly those who are hard-to-reach?

This final question is of direct commercial relevance, particularly to our partner organisation, the Royal Scottish National Orchestra, who are committed to using technology as a tool to develop audiences and extend their existing outreach work, suggesting a possible avenue for commercialisation and further development. The RSNO have kindly agreed to assist us in working directly with their audiences for user testing.

Our key academic beneficiaries are academics working in and around Early Music; Sound/Interaction Design, and Physical/VR/AR Computing. By documenting process and workflow, and generating new acoustic models and sound recordings, we will provide case-study material and data sets for all three groups. In particular, the exploration of early music performance and its relationship to space will provide significant new insights into the understanding of performance practice and generate new research questions that will be the subject of a follow-on funding application.

There are three main non-academic beneficiaries:
1. The national technology sector
While this project is focused on developing software that co-locates geographically-remote musicians and audiences within virtual spaces (either historically-recreated or fictional) to perform and spectate - which has commercial potential - the research will provide insights into the role of auditory feedback in supporting immersion/presence within virtual spaces, which has direct applications in computer gaming and virtual reality. VR is an emerging technology and developers are currently concerned with systems of control, visualization, and movement, particularly around motion sickness. Sound has been somewhat neglected, yet it offers enormous potential to situate users within spaces.
In Biome, we have a partner that has experience of multi-modal software development and understands the commercialization process and routes to market from conceptual prototypes, while in The Binchois Consort, the RSNO, and heritage spaces like St Cecilia's Hall, we have access to established networks of potential users.

2. Early music performers and audiences
The physical dimensions and locations of many historic Scottish spaces impose limits on their use in performance, and the country's unique geography and population distribution presents challenges in engaging geographically-dispersed and other hard-to-reach communities in site-specific performances. This project would equip performers with a new set of tools and methods, and provide audiences with new ways of engaging with and experiencing site-specific performances.
In addition, by providing musicians with unconstrained access to virtual acoustic spaces, performers will be able to rehearse in a site-specific way, which is simply not possible otherwise. This embedding of space in performance practice should lead to new insights and perspectives on established repertoire, the nature of historic performance practice, and perhaps new or rediscovered performance technique in response.

3. National cultural and heritage organizations
The collaborative music-making technologies that this project aims to develop represent an opportunity for cultural organizations, like the RSNO, to greatly expand the reach of their existing engagement work and to grow new audiences, particularly hard-to-reach audiences. By building anonymous user analytics directly into the software, we will have a means of capturing and visualizing its impact.
For heritage agencies, such as Historic Scotland, the technology represents an opportunity to explore, non-invasively, the architectural acoustics of their historic sites, and to situate experiential visitor attractions on site. The technologies developed by this project have the potential to provide greater and easier public access to the outcomes of such site-specific research to the substantial audiences who visit heritage sites each year.
The technology also provides a means of democratizing such research, and provides a vehicle for communities to use the technology to explore, non-invasively, the performance characteristics of historical spaces/events that are not of national importance, but which are, nevertheless, significant to local communities.