Hearing History: Bringing to life the sounds of the past through Virtual Reality

Lead Research Organisation: University of Edinburgh
Department Name: Edinburgh College of Art

Abstract

Imagine experiencing a musical moment from the past; a rich, sensory experience that combines the flickering light from candles, the exquisite performance of centuries-old music, and the warm acoustics of a performance space that now no longer exists. That is what our initial project, Space, Place, Sound, and Memory, set out to do.

Working closely with game developers, musicologists, architectural historians, and acousticians, we set out to recreate the sounds and sensations of early music performance in virtual reality. We began by painstakingly piecing together two concerts of music from fragmentary sources and records, and then recorded historically-informed performances in an anechoic chamber, a space that is specifically designed to have no natural acoustic. Once complete, we constructed two virtual auditoria, digitally rebuilding St. Cecilia's Hall and the Chapel at Linlithgow Palace from detailed laser scans taken on site. Then, we brought the recordings and virtual spaces together using acoustic ray-tracing to recreate the sensations of hearing a historic performance.

The key outcome of that work was the production of virtual models and software which allows users to don VR headsets and step into the past. For our follow-on project, we would like to capitalise on our partnership with Historic Environment Scotland (HES) and St. Cecilia's Hall to create three dedicated public access points, situated at St. Cecilia's Hall, Linlithgow Palace, and the Engine Shed, HES's digital innovation centre. By partnering with HES, we get the greatest possible reach for our VR software: in 2017/18 Linlithgow Palace attracted 87,254 visitors from across the world. We plan to work closely with the organisation to measure impact in terms of visitor engagement using HES's established KPIs.

In addition to exploring the application of our acoustic modelling software in the heritage sector, we would like to use the follow-on project to explore the viability of our models and experience of anechoic recording in a commercial music production environment. Working with the Binchois Consort and the classical label Hyperion, we will arrange, record, and produce music linked, through detailed research, to Linlithgow Palalce, including lesser-known pieces from Scotland's famous Carvor Choirbook. Building on our VR production workflow, we will work with Hyperion's recording engineers to develop a commercial music production pipeline based around anechoic recordings, using the virtual acoustic models as a tool during production and post-production to situate those anechoic recordings within a historical acoustic space. This will be the first instance of a classical album recorded and produced entirely within VR.

We will develop a companion VR app that allows end-users to download additional musical content and VR models to remix and audition the recordings in different historic spaces and at different points in time. In terms of impact, we will capture data about Hyperion's ability to reach and engage its audiences in new ways and to grow new audiences, and the commercial viability of the app as a new revenue stream through creating and packaging additional paid-for content.

By the end of this phase of the project, we expect the following outputs:

1. Three public-facing VR installations located at high footfall cultural sites across the Scottish Central Belt;
2. A new commercial album, to be released on the Hyperion label, featuring the Binchois Consort performing the music of the Carvor Choirbook;
3. New commercial music workflows and production pipelines based around production and post-production using VR tools;
4. A user-friendly mobile app that allows end-users to remix audio in different spaces and at different points in time;
5. Datasets relating to the impact of the underlying research work on visitor engagement to cultural sites, and on the commercial operation of a record label specialising in early music production.

Planned Impact

There are five key beneficiaries to this project:

1. Heritage organisations (St. Cecilia's Hall and Historic Environment Scotland (HES))
One of the key drivers of HES is in increasing the impact and value of the historic environment as a part of Scotland's economy, primarily through heritage tourism, but also through education. This is concretised in HES's organisational KPIs, which include:
- Increase the overall contribution of heritage tourism to the Scottish economy
- Increase knowledge and understanding of the historic environment
- Enhance engagement with the historic environment
- Provide excellent service to our visitors and service users
By creating a dedicated public installation that allows both the public and HES's specialist digital conservators to access these historic environments and their acoustics in new and engaging ways, our project will enable HES measurably to deliver on these KPIs. We will combine HES's own data gathering with user analytics and on-site interviews to capture data on visitor numbers and engagement, and we will work with specialists at HES to capture impact on knowledge and understanding.

2. Visitors to historic sites
VR and the attendant gamification of space and performance offers new modes of access and engagement for the public, who will further benefit from a better and more nuanced understanding of historic cultures and places. By locating our VR installations in high-footfall visitor attractions with international reach, we will ensure that we capture the broadest and most representative visitor engagement data possible.

3. Independent recording engineers and classical labels, like Hyperion
Traditionally, early music has been produced on-location in modern performance venues or churches. This is problematic because such spaces are not optimised for recording and production and are susceptible to high levels of external ambient noise, and such venues are often subject to restricted access during the day. This can lead to lengthy recording sessions and out-of-hours fees, increasing the cost of production. By carrying out a process analysis and practitioner interviews with Hyperion's sound engineers, we aim to measure the impact of our virtual production process on Hyperion's commercial workflow. We anticipate that it will lead to time and cost savings in production, and greater flexibility during post-production. In addition, we aim to measure the commercial impact of our companion app and downloadable content by carrying out a comparative analysis of revenues and profits of this release against previous CD releases by the Binchois Consort.

4. Performing musicians, like the Binchois Consort
Performing live ensemble vocal music in an anechoic chamber requires a very specific set of performance skills, since the recording environment exposes issues around timing, diction, intonation and tuning. We will train the Consort to adapt their technique specifically for performance and recording in anechoic spaces, and will follow up to investigate the impact, over time, of this training and experience on their performance practice in other contexts. Using comparative analysis of recordings and interviews with singers, we anticipate that the experience will result in a measurable improvement in the ensemble's technique.

5. Audiences of early music
All of the outcomes of this project offer new opportunities and new mechanisms for early music audiences to experience and explore music, and for this audience to be widened. Previously, these audiences have only been able to access early music through CD releases or live concerts. Our VR applications, particularly the companion app and downloadable content, will allow audiences to explore space as a dimension of performance in a way never before possible. We will use user analytics and audience interviews to measure the impact on audiences and how they interact with and understand the music they love.

Publications

10 25 50
 
Title Hearing Historic Scotland CD 
Description A commercial CD released by Hyperion Records, performed by the Binchois Consort. 
Type Of Art Artefact (including digital) 
Year Produced 2020 
Impact Recording of impact on both the recording industry and audience is still ongoing. 
 
Title Linlithgow VR application 
Description A virtual reality experience of a performance in a reconstruction of the Chapel of Linlithgow Palace 
Type Of Art Film/Video/Animation 
Year Produced 2020 
Impact Impact on both Historic Environment Scotland (the owners of the site in which the VR experience is deployed), and the public users of the experience is still being measured. 
 
Title St Cecilia's Hall VR application 
Description A virtual reality experience of a reconstruction of performances within St Cecilia's Hall. 
Type Of Art Film/Video/Animation 
Year Produced 2020 
Impact Impact on both the museum and its visitors is still being measured. 
 
Description This project sought to take the research outcomes of the previous award (Space, Place, Sound, and Memory) and to develop them for public use in Linlithgow Palace and St Cecilia's Hall, and to develop a full commercial CD recording, recorded and produced in Virtual Reality. In the process of developing the VR experience for use in museum and heritage space, in-depth research and user-testing was required in order to understand the changes that needed to be made to the experience in order to allow it to function in the designated public spaces. Changes included switching to wireless headsets, removing the need for external sensors, developing software to allow the experience to start automatically as users put it on their heads, developing gaze control (to remove the need for learning of controls), and building introductory videos. Recording and analysis of the impact on users of Linlithgow Palace and St Cecilia's hall is still underway.

New processes for recording in VR have been developed, including guidelines for extending recording sessions, making use of artificial acoustics in the recording, finding appropriate tempi, and developing existing production methods. Following this research, the PI has been acting as an advisor on other international research looking at recording in anechoic conditions.The CD has now been recorded and will be released in the next few months, after which recording and analysis of impact on the recording industry, and on the early music audience will be undertaken.
Exploitation Route The PI has already been advising on other projects that wish to record in anechoic conditions. This research can be used more broadly in the recording industry for similar projects to recreate historical performances. It can also be used to inform other VR experiences in cultural, heritage, and museum sites. The experiences (and CD) are also useful in Education, both formal and informal.
Sectors Creative Economy,Digital/Communication/Information Technologies (including Software),Education,Culture, Heritage, Museums and Collections

 
Description Non-academic impact on this award is just starting. Visitors to Linlithgow Palace and St Cecilia's Hall are just beginning to be able to use the VR experiences. We are measuring the extent to which this is changing the way that they conceive of the spaces they are visiting, the music they are hearing, and the history they are experiencing. The new VR CD will be released shortly, after which we will start to measure how its production has impacted on the recording industry.
First Year Of Impact 2020
Sector Creative Economy,Digital/Communication/Information Technologies (including Software),Education,Culture, Heritage, Museums and Collections
Impact Types Cultural,Societal,Economic

 
Description Historic Environment Scotland 
Organisation Historic Environment Scotland
Country United Kingdom 
Sector Public 
PI Contribution The PI has, following the successful partnership as part of both funded projects, continued to work with HES on music for audioguides at a number of sites (Edinburgh Castle, Melrose Abbey, Linlithgow Palace).
Collaborator Contribution HES have enabled research by the PI to reach a large number of visitors to their properties.
Impact Audioguide for Edinburgh Castle Audioguide for Melrose Abbey Audioguide for Linlithgow Palace
Start Year 2019
 
Description Public Talk at Bach Chorale Summer School, St Andrews 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact I gave a talk on the VR project to attendees of the Bach Summer School vocal course in St Andrews. It sparked a lot of discussion, including email follow-ups from international attendees several months later. One attendee has since gone on to undertake postgraduate study in a related discipline due to interest in the talk.
Year(s) Of Engagement Activity 2019