ENSEMBLE Performing Together Apart: Enhancing Immersive Multi-Location Co-Performance in Real Time

Lead Research Organisation: Edinburgh Napier University
Department Name: Arts and Creative Industries

Abstract

Summary
The feeling of being immersed within a live performance by an ensemble of world class musicians can be a deeply engaging and highly valued cultural experience for audiences, as well as for the performers themselves. Operas, orchestral performances and musical theatre are popular and attract large followings, but they are costly and highly resource intensive to produce, particularly when the most talented performers are often located in different locations across the globe. Advanced audio-visual streaming technology can potentially provide solutions and new creative opportunities, and Edinburgh Napier University is at the forefront of this experimental work. However, the ENSEMBLE project will provide a complementary, human-centred focus on relatively under explored experiential aspects.

The ENSEMBLE project will look at how musicians can perform live together, seamlessly as a group even though they are separated by distance, mediated through cutting-edge connection technologies. ENSEMBLE will examine the subjective, experiential and contextual factors that support high quality performance experiences, primarily from the perspective of the performers themselves, using qualitative and analytical methods.
With the advent of super low latency network connections, the possibility has emerged for musicians, singers and actors to perform together in multiple remote locations in real time. The latency aspect has been a particular challenge for musicians where even a lag of 30 milliseconds may have an impact resulting in a significant deterioration in performance. With the resolution of some of these latency technical hurdles, additional more experiential challenges have emerged regarding the nuance of representation of remote performance partners of both the co-performer(s) and their audience.

In its simplest terms, the question is how best should remote participants in a performance, be they co-performers (e.g. conductors and orchestras) or audience members, be represented to one another (visually, aurally and kinetically) to result in the most engaging, joyful and immersive experience for all involved?

The ENSEMBLE project seeks to answer this question by exploring the potential of emerging technologies such as multi-person immersive dome environments (IDEs) to enable, enhance, extend and immerse performers and/or audiences in real-time during multi-location and distributed musical and theatrical performances. This in turn will provide insight into the parameters of influence of remote co-performance on both the performer and audience experience, in turn enabling the establishment of UX (user experience) and technological frameworks informing best practice when designing, developing and deploying such technologies into temporary and permanent physical performance spaces and scenarios.

Planned Impact

The research to-date shows that real-time performance is unquestionably possible over academic research networks and that, equally significantly, there is an appetite for this amongst the artists and organisations who would ultimately use it for economic or societal benefit.

While current knowledge might be satisfactory for masterclasses between academic institutions, ENSEMBLE's engagement with representatives from various areas of the creative industries will enable us to roadmap the technical and performance/place issues that need to be addressed if we are work towards commercial exploitation of real-time remote performance.

The identified users and beneficiaries of real-time remote performance can be grouped into two categories: artists/cultural organisations and technology innovation companies.This project will draw upon its existing industry networking links with these groups to form the core membership of the proposed ENSEMBLE workshops with additional membership through invitation during the project lifetime.

Who might benefit?
Edinburgh Napier University has a strong research relationship with the JISC Audio/Video Streaming team and was the remote partner in the UK's first demonstration of the LoLa audio/video streaming technology in 2012. Edinburgh Napier then became one of the founder members of the JISC Arts and Humanities Special Interest Group established that year. The membership of that group now reaches beyond education into industry and includes representatives from UK theatre and music groups. The list below identifies artists and cultural organisations that have either been involved with Edinburgh Napier's research or consultancy or have indicated their willingness to collaborate:

Royal Scottish National Orchestra, BBC Scottish Symphony Orchestra, Scottish Opera, The Rogue Orchestra. Q-Strings, Hebrides Ensemble with Sir James MacMillan, Steve Cradock (Paul Weller), Connect Theatre, Bristol Old Vic, Abbey Road Studios. Creative Scotland, Creative Carbon. Edinburgh International Festivals.

How might they benefit?
Short-term impact - The Rogue Orchestra approached Edinburgh Napier with a view to creating a competitive UK remote recording orchestra for music and film projects by international clients. Following positive market research amongst potential clients in the UK and USA feasibility testing has been undertaken with producer/guitarist Steve Cradock for projects by PP Arnold and Paul Weller. The ENSEMBLE project's collaboration with the technology innovation companies described below will help the orchestra and other UK artists to compete remotely in a global marketplace.

Mid to long-term impact - Companies such as the BBC, Scottish Opera and Abbey Road Studios have expressed interest in Edinburgh Napier's research and their participation in this project will allow us to roadmap the steps required before they can evaluate and commercially exploit remote real-time immersive performance.

The ENSEMBLE project will ensure impact through industry engagement workshops held at the mid and end points of the project lifespan. These multidisciplinary events will engage with artists, live and broadcast events practitioners and technology innovation companies to disseminate results of the project work and enable discussion and feedback.

Publications

10 25 50
 
Title 'Global BRASS' 2019 Durham Brass Festival world first joint concert over distance 
Description Global BRASS, Friday July 19: Two bands 560 miles apart were joined together by technology for a real-time joint concert using LOLA technology and techniques developed during the AHRC ENSEMBLE project. North of England Champions NASUWT Riverside Band played to an audience in Durham's Gala theatre along with Danish Brass Band Champions Concord Brass Band playing from Copenhagen. Durham Brass Festival Music Director Paul Gudgin approached Edinburgh Napier University's long-distance, real-time performance research team to examine the feasibility of linking two brass bands in the UK and Denmark. 
Type Of Art Performance (Music, Dance, Drama, etc) 
Year Produced 2019 
Impact This concert marks a very significant adoption of academic research by the music industry: It demonstrates uptake of long-distance performance by a festival and ticket buying audience. It demonstrates that long-distance, real-time performance is not limited to National Research and Education Networks and can be made possible in venues such as Durham's Gala Theatre. 
 
Title Co-located Armistice 2018 Centenary Concert - 45 simultaneous locations worldwide - #IPLAY4PEACE 
Description The realisation of the real-time remote participation element was a direct result of the first ENSEMBLE project workshop. Armistice day 2019 was selected by the industry participants as the primary example where the team's immersive research could have an impact. By bringing our research to this concert we were able to able to invite participation around the world resulting in a total of 45 sites. 
Type Of Art Performance (Music, Dance, Drama, etc) 
Year Produced 2018 
Impact The TV and press coverage for this performance has raised global awareness of the potential for real-time performance/collaboration in the arts. We are now receiving requests to participate in a 2019 #IPLAY4PEACE global concert from artists and technology companies. The Right Hon Frank Field tabled a House of Commons Motion to congratulate: https://edm.parliament.uk/early-day-motion/52368/taking-first-world-war-commemoration-into-the-future 
URL https://www.napier.ac.uk/about-us/events/play-for-peace-a-concert-for-cooperation
 
Description The AHRC-funded ENSEMBLE project has demonstrated that an immersive dome environment has the potential to significantly enhance the experience of a remote musician during remote, real-time co-performance across long distances. The immersive environment promotes a sense of 'togetherness' and a shared experience of 'space'.

The project team believe ENSEMBLE to be the first use of eye-tracking gasses in real-time performance research. This use demonstrates that the qualitative approach taken by researchers to-date can now be extended by quantitive data around the user experience.

The project's workshops with representatives from creative industries organisations have increased awareness of what is becoming possible in co-located performance. As a result the workshops have stimulated discussion around potential use of this creative technology by artists, promoters and venues. The key uses emerging from these discussions with representatives were the use of the technology to widen participation, extend their market reach and to reduce their carbon footprint.
Exploitation Route Real-time performance over long distance requires low latency and minimising the latency due to image processing was a challenge that the project partner Soluis Sublime had not previously had to address and they feel their work can be developed further.

The technology is now ready to make the transition from the research environment to the commercial arts sector. Two of the creative industries organisations present in the ENSEMBLE workshops have approached with plans to take the research further with live audiences.
Sectors Creative Economy,Digital/Communication/Information Technologies (including Software),Education

 
Description Resulting directly from their participation in the first ENSEMBLE workshop, three creative industries organisations are now building on our findings: Gigabit City infrastructure providers City Fibre are actively exploring how their fibre network might extend the reach of real-time co-located performance to include arts venues and schools within the Gigabit Cities. Similarly, the Edinburgh Jazz and Blues Festival has submitted a funding proposal to use our research to link with audiences and performers in Norway and Belgium. UK Edtech providers Jisc are championing our research at Digifest 19 under the theme "Education in a hyper-connected future" Durham Brass Festival used our research to create a world first 'Global Brass' performance in their 2019 programme The BBC World Service Radio programme 'Digital Planet' used our research to produce a long-distance performance in their 18th Birthday special edition episode.
First Year Of Impact 2019
Sector Creative Economy,Education,Leisure Activities, including Sports, Recreation and Tourism
Impact Types Cultural,Economic

 
Description Collaboration with Jisc to produce a short film and press for Digifest 2019 entitled "ENSEMBLE: innovate and inspire" 
Organisation Jisc
Country United Kingdom 
Sector Public 
PI Contribution The research team were interviewed about the project and how it might contribute to "education in a hyper-connected future"
Collaborator Contribution Following on from their participation in the two ENSEMBLE project workshops, Jisc commissioned and funded a short film about the project with an emphasis on the implications for future education.
Impact Film output. Interdisciplinary: music, immersive User Experience (UX), High-performance computer networks
Start Year 2018
 
Description Armistice Centenary Concert press engagement 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact The research into immersive networked performance undertaken by the ENSEMBLE team fed directly into the 2018 interconnected Armistice Concerts that took place in 45 locations around the world. The anchor concert took place in Edinburgh Napier University's Craiglockhart Campus, the former war hospital where war poets Owen and Sassoon recuperated.

The research was featured in interviews by the BBC, STV and press.
Year(s) Of Engagement Activity 2018
URL https://www.napier.ac.uk/about-us/events/play-for-peace-a-concert-for-cooperation
 
Description Ideation Workshop for creative industries organisations 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Industry/Business
Results and Impact This workshop took place in Edinburgh with representatives from creative industries organisations ranging from orchestras, government, technology providers and arts venues.
The ENSEMBLE research team presented their work and explored with the delegates how this research might impact on future policy and practice in the arts.
Year(s) Of Engagement Activity 2018
 
Description Invited speaker: BBC Academy "Sounds Amazing" 2018 staff conference 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Industry/Business
Results and Impact The ENSEMBLE team was invited by the BBC to present their real-time remote performance research to staff and invited professionals at the 2018 BBC Academy "Sounds Amazing" conference. Our presentation was heard by 300 BBC research, production and operations staff in the BBC Broadcasting House Radio Theatre as well as being viewed by additional staff throughout the BBC via an internal live stream.
Year(s) Of Engagement Activity 2018
 
Description Short film and press about the ENSEMBLE project shown at Jisc's Digifest 19 edtech conference 
Form Of Engagement Activity A broadcast e.g. TV/radio/film/podcast (other than news/press)
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Public/other audiences
Results and Impact Following their participation in two of the ENSEMBLE workshops, Jisc commissioned a short film interviewing us about the project. This film was shown as part of the Digifest '19 edtech event to approximately 2000 HE and FE senior managers and policy leaders.
Year(s) Of Engagement Activity 2019
URL https://www.jisc.ac.uk/digifest
 
Description Workshop with Soluis immersive dome demonstration 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Industry/Business
Results and Impact This workshop took place in Glasgow and was attended by creative industries educators, professionals and technology providers from the UK, Italy and the Czech Republic.
The attendees experienced an immersive demonstration of the team's research in the Soluis/Sublime VR Dome. This demonstration was followed by group discussions around the implications for immersive networked performance and for technology developers.
Year(s) Of Engagement Activity 2018