WEb Audio Virtual Environment Rendering (WEAVER): Online Virtual Acoustics for Sonic Art, Digital Heritage, and Broadcast

Lead Research Organisation: University of York
Department Name: Electronics

Abstract

Auralisation, the audio equivalent of visualization, enables us to audition virtual acoustic environments that once existed, will exist or that are purely fictional. Auralisation is based on obtaining the acoustic room impulse response (RIR) of a space, through measurement if the building already exists, or otherwise via simulation using computational modelling. The result is a 3D acoustic rendering of any sound we wish to hear within the auralised environment. This research plays a key role in architecture, environmental noise assessment, recreating past environments in digital heritage, as well as enabling creativity in music production, broadcast and computer gaming.

The WEb Audio Virtual Environment Rendering (WEAVER) project will revisit the concept of spatial audio composition and rendering in relation to the last ten years of development in auralisation and virtual acoustics research. Since this research started in 2004, the specialised audio technology required to explore spatial composition using virtual acoustics, auralisation, acoustic measurement and modelling is now commonplace in most creative computer music software applications. Acoustic RIR measurements from various sites, buildings and venues, many of significant heritage value or interest have also been gathered and disseminated over this ten-year period via the online Open Acoustic Impulse Response Library (OpenAIR: www.openairlib.net). Over this time OpenAIR has become a key resource for electronic musicians, sound designers, software and computer game developers.

Much of the functionality required to explore and audition spatial audio content and virtual acoustic environments can now, in fact, be offered via a standard web browser and mobile device - additional acoustic RIR data is still needed to auralise these virtual spaces, but the ability to use or listen to these results is now carried by millions of potential users worldwide.

WEAVER will use the Web Audio API to deliver interactive virtual acoustic environments via a web browser, and to further realise some of the creative research aims first articulated over ten years ago. In combination with the research and resources related to OpenAIR, that was also an outcome from this initial work, WEAVER will develop impact and engagement in online sonic art, digital heritage and broadcast.

WEAVER will create novel online sonic art from OpenAIR data. We will scope the Web Audio API to detail its capabilities for developing creative virtual acoustics applications informed by existing examples and current practice, and provide a route to impact through the development of new user contacts for OpenAIR content.

We will work with York Theatre Royal, a regional producing theatre based on a historic site that has hosted a working theatre since 1744, to develop a web resource to engage new audiences with the redevelopment and heritage of the theatre's auditorium space. This presents a unique opportunity to apply our virtual acoustics research in a culturally and historically significant venue during its refurbishment. We will help York Theatre Royal to tell their story and engage new audiences using our expertise in acoustic heritage, auralisation and sound design, brought together through the development of novel, interactive online content.

With BBC Audio Research we will investigate how virtual acoustic rendering informs the experience of listening to audio broadcast material. WEAVER will expand on existing BBC spatial audio rendering for broadcast to include differently rendered virtual acoustic and playback listening environments, and will make use of BBC facilities and test platforms for reaching audiences in order to gain feedback on the effectiveness and future viability of this presentation technology.

At all stages data and code developed will be disseminated by open-source/open-access means, encouraging maximum potential impact for WEAVER and the resources it creates.

Planned Impact

The WEAVER follow-on project will build on research in spatial audio composition and rendering by using the Web Audio API, together with OpenAIR data, to deliver interactive virtual acoustic content for the web, so developing new pathways to impact and engagement in online sonic art, digital heritage and broadcast. The main WEAVER beneficiaries are the partners who have shaped its content and direction.

York Theatre Royal (YTR) will use audio/image data, as gathered from the current theatre space throughout the existing refurbishment and beyond, to animate the history of the building and the site. The digital heritage community, both researchers and those managing museums, galleries and heritage sites, will also benefit from the learning developed from the design of the online materials for engagement that will be produced from this partnership.

WEAVER will enable BBC Audio Research to enhance their existing spatial audio rendering applications to include virtual acoustic and multiple playback options. Through our partnership we will further promote this work to the relatively new Web Audio community, who have recently hosted their first international conference with the BBC already active as contributors.

Additionally, audiences for both YTR and BBC will benefit from the enhanced opportunities that will be offered through WEAVER to digitally engage with the creative content that these organisations curate and deliver.

The wider creative music technology community will also benefit from outputs produced in terms of the novel sonic art created, code developed and RIR data that will be curated and made available online via OpenAIR.

OpenAIR has already created impact in computer music, digital heritage and computer gaming. WEAVER will also deliver impact in the area of computer music, extending to the web developer community through the novel use of OpenAIR data and the Web Audio API. Both data and code developed will be made open-source/open-access, encouraging maximum potential impact for WEAVER and the resources it creates. As the Web Audio community is relatively new, there is a good opportunity to make an early contribution to the field with lasting impact.

WEAVER will enhance the offering and potential audiences for our two high profile partners through the novel online content created. YTR and the BBC are part of the UK's Creative Industries, a socially important, export driven industry, and one of the UK's leading industrial sectors. Digital technology increasingly helps to drive the creative industries, and interactive digital media is becoming more important in conveying narrative and factual entertainment to audiences. Collaborating with our partners at this early stage in Web Audio's development maximizes the potential impact for our work given the national reputation of YTR and the global reach of the BBC. York has recently been designated as UNESCO City of Media Arts, providing an umbrella under which the collaboration with YTR has additional opportunity to make a global impact, with related benefit to the local, regional and UK economy.

Impact developed with YTR will become more beneficial as the theatre reopens in late 2015/early 2016, and starts on the next phase of its own development. WEAVER is also planned to start then, and benefits should be evident within the timescale of the project into the medium term. Integrating with the BBC's online technology will enable WEAVER outputs to deliver impact into the future based on their wider use and rollout. The Audio Research Partnership encourages further collaboration beyond the project timescale and joint dissemination to audiences in academia and industry is anticipated.

The project RA will develop professional skills in audio, web authoring and coding through engagement with the requirements of our two partners, coupled with a valuable industry placement with the BBC that will further opportunities for them across employment sectors.

Publications

10 25 50
 
Title OpenAIR at AHRC Common Ground 
Description Early results from the "WEb Audio Virtual Environment Rendering (WEAVER): Online Virtual Acoustics for Sonic Art, Digital Heritage, and Broadcast" project were presented as an immersive audio-visual soundwork at the AHRC Common Ground event in 21 June 2016. 
Type Of Art Artistic/Creative Exhibition 
Year Produced 2016 
Impact This work was presented to highlight the software toolbox and acoustic database that we had developed as part of this project. 
URL http://www.ahrc.ac.uk/documents/agendas/common-ground-programme/
 
Description The WEb Audio Virtual Environment Rendering (WEAVER) project revisited the concept of spatial audio composition and sound rendering in relation to the last ten years of development in virtual acoustics and auralisation research. It considered the enhanced functionality now offered in creative computer music software applications, and the acoustic measurements that have been gathered and disseminated via the OpenAIR project and website (www.openairlib.net). In particular this project investigated the Web Audio API as a means to deliver such interactive virtual environments using a web browser. The project achieved results across the three following areas:

(1) OpenAIR room impulse response acoustic measurements were combined with web audio rendering technology to deliver online demonstrators of novel sonic art.
This was presented at the Audio Engineering Society 142nd Convention in Berlin in May 2017. This work helps to demonstrate the output from (3) below. In addition, the creative application of this work was demonstrated in an artistic installation at the 2016 AHRC Common Ground event.

(2) The development of an online resource to engage new audiences with the redevelopment and heritage of York Theatre Royal's auditorium space.
This involved the capture of 360-degree image and audio data from different seat locations across the York Theatre Royal auditorium for a single on-stage performance. This audio-visual data was then combined into a test web-page viewer to enable a user to view the performance from each seat location. York Theatre Royal are particularly interested in how this might develop into Virtual Box Office technology, to encourage potential audience members to buy tickets for a show or upgrade their seat, or as a test for VR-based streaming of a performance.

(3) The development and testing of virtual acoustic rendering options for BBC online streaming and audition of audio broadcast material.
We have worked with our project partners at BBC Audio R&D to develop a Web Audio framework that enables OpenAIR data to be delivered to third party applications in a standardised way. This is part of an open-source library that was published and documented in a paper for the Audio Engineering Society 142nd Convention in Berlin in May 2017, using novel online sonic art as a means of demonstrating its functionality (see also (1) above).
Exploitation Route The Web Audio Library we have developed is open-source and will be used by BBC R&D, and through its open-source license, potentially a significant part of the wider audio community.

We are looking to develop our virtual box office project further through the DC-Labs project (EP/M023265/1) to see how it might be leveraged as a useful tool for performing arts venues. The learning we have developed in terms of delivering 360-degree visuals and sound for web or VR is being used by DC-Labs in the DINAR project that resulted in the award winning VikingVR exhibition for York Museums Trust in 2017. This research has helped to found a new AHRC project exploring the acoustics of the historic House of Commons Chamber (AH/P012094/1) as well as contributed towards two AHRC Immersive Experiences projects recently awarded to the University of York.

In 2018, this work has been leveraged for further use through our collaboration with the BBC. This has resulted in an EPSRC/BBC ICASE PhD Studentship, looking at automatic soundscape generation for immersive environments, and the BBC being a key partner on the AHRC Creative Media Labs project (AH/S002839/1). We have also published a paper around the VikingVR work, towards which this project made a contribution.
Sectors Creative Economy,Digital/Communication/Information Technologies (including Software),Culture, Heritage, Museums and Collections

URL https://github.com/UoYWEAVER/OpenAirLib
 
Description The WEb Audio Virtual Environment Rendering (WEAVER) project revisited the concept of spatial audio composition and sound rendering in relation to the last ten years of development in virtual acoustics and auralisation research. It considered the enhanced functionality now offered in creative computer music software applications, and the acoustic measurements that have been gathered and disseminated via the OpenAIR project and website (www.openairlib.net). In particular this project investigated the Web Audio API as a means to deliver such interactive virtual environments using a web browser. The project was funded as a Follow-On project for Impact and Engagement, so was entirely focused on realising impact from our findings via our project partners York Theatre Royal and BBC (Audio) R&D. We captured 360-degree image and audio data from different seat locations across the York Theatre Royal auditorium for a single on-stage performance. This audio-visual data was then combined into a test web-page viewer to enable a user to view the performance from each seat location. We have used this as an example of Virtual Box Office technology for York Theatre Royal, to encourage potential audience members to buy tickets for a show or upgrade their seat, or as a test for VR-based streaming of a performance. Although this has not translated into a product or service to date, it has enabled us to develop skills and experience in authoring VR-content, and developed capacity to deliver audio content in VR contexts. This has had important consequences in enabling us to contribute to other projects and look to other funding opportunities. We have worked with our project partners at BBC Audio R&D to develop a Web Audio framework that enables OpenAIR data to be delivered to third party applications in a standardised way. This is part of an open-source library that has been published and documented in a paper for the Audio Engineering Society 142nd Convention in Berlin in May 2017, using novel online sonic art as a means of demonstrating its functionality. This Web Audio library is open-source and will be used by BBC R&D, and through its open-source license, potentially a significant part of the wider audio community. We have also presented examples of the more creative application of this work at the AHRC Common Ground event in York, 2016. This project also facilitated three-way collaboration between the University of York, BBC R&D and York Theatre Royal - the BBC are researching the use of 'Venue Explorer Technology' making use of hi-definition broadcast to enable viewers to focus on one particular element of a wider scene (e.g. a stadium sports event, concert, or music festival). The WEAVER project enabled all three parties to develop into this area through the use of York Theatre Royal performances as small Venue Explorer Case studies. To date, filming has been completed for two performances and we are looking for means of developing this opportunity further. We will develop our virtual box office and venue explorer concepts further through the DC-Labs project (EP/M023265/1) to see how they might be leveraged as a useful tool for performing arts venues and for broadcast. The learning we have developed in terms of delivering 360-degree visuals and sound for web or VR is being used by DC-Labs in as part of the DINAR project and the award winning VikingVR exhibition for York Museums Trust in 2017, and has also helped to found a new AHRC project exploring the acoustics of the historic House of Commons Chamber (AH/P012094/1), as well as contribute to two successful AHRC Immersive Experiences projects in the recent 2017 call. In 2018, this work has had further ongoing impact. We have published a paper around the VikingVR work, towards which this project made a contribution, and our relationship to the BBC has continued to develop. This has resulted in an EPSRC/BBC ICASE PhD Studentship, looking at automatic soundscape generation for immersive environments, and the BBC being a key partner on the AHRC Creative Media Labs project (AH/S002839/1).
First Year Of Impact 2016
Sector Creative Economy,Digital/Communication/Information Technologies (including Software),Culture, Heritage, Museums and Collections
Impact Types Cultural,Economic

 
Description AHRC Next Generation of Immersive Experiences (York Gaol)
Amount £59,682 (GBP)
Funding ID AH/R008701/1 
Organisation Arts & Humanities Research Council (AHRC) 
Sector Public
Country United Kingdom
Start 01/2018 
End 09/2018
 
Description AI-driven Soundscapes Design for Immersive Environments EPSRC/BBC ICASE PhD Studentship
Amount £84,900 (GBP)
Funding ID EP/S513945/1 
Organisation British Broadcasting Corporation (BBC) 
Sector Public
Country United Kingdom
Start 10/2018 
End 09/2022
 
Description Creative Media Labs: Innovations in Screen Storytelling in the Age of Interactivity and Immersion
Amount £5,544,079 (GBP)
Funding ID AH/S002839/1 
Organisation Arts & Humanities Research Council (AHRC) 
Sector Public
Country United Kingdom
Start 10/2018 
End 03/2023
 
Description Follow-on Funding for Impact and Engagement: Creative Economy Highlight Notice
Amount £202,006 (GBP)
Funding ID AH/P012094/1 
Organisation Arts & Humanities Research Council (AHRC) 
Sector Public
Country United Kingdom
Start 02/2017 
End 07/2018
 
Description Listening to the Commons: The Acoustics of Debate and the Experience of Women in Parliament c. 1834
Amount £8,844 (GBP)
Organisation Arts & Humanities Research Council (AHRC) 
Sector Public
Country United Kingdom
Start 01/2016 
End 07/2016
 
Description Realising the Impact of VR for Heritage and Culture
Amount £15,965 (GBP)
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Academic/University
Country United Kingdom
Start 12/2016 
End 03/2017
 
Description VR Capture and Rendering of School-run Traffic Congestion Problems
Amount £2,000 (GBP)
Organisation City of York Council 
Sector Public
Country United Kingdom
Start 02/2017 
End 10/2017
 
Description Virtual Reality in Theatre
Amount £7,425 (GBP)
Organisation Arts & Humanities Research Council (AHRC) 
Sector Public
Country United Kingdom
Start 12/2016 
End 07/2017
 
Description AECOM 
Organisation AECOM Technology Corporation
Department Acoustics
Country United States 
Sector Private 
PI Contribution We have supported a Royal Society Industry Research Fellow from AECOM for a period of 24 months by hosting the in the AudioLab in the Department of Electronic Engineering.
Collaborator Contribution The Fellow has contributed knowledge and experience to our wider group and participated in relevant activities of mutual interest, including publications. They have also contributed to DC-Labs (EP/M023265/1) projects and the Fellow also worked to develop links between The University of York, AECOM and other third party companies, including Virgin East Coast and the Noise Abatement Society with a view to accessing our research and technology.
Impact Two publications: Southern, A., and Murphy, D.T., "Comparison of road tyre noise auralisation methods", Proc. of Internoise 2016, pp. 3483-3489, Hamburg, Germany, Aug. 21-24, 2016. Southern, A., and Murphy, D.T., "A Framework for Road Traffic Noise Auralisation", Proc. of EuroNoise 2015, Maastricht, The Netherlands, 31 May - 3 Jun., 2015. With one more in progress. Technology demonstrations, including audio for virtual reality as used in the WEAVER (AH/N00356X/1) and DC-Labs (EP/M023265/1) projects.
Start Year 2014
 
Description Arup - Environmental Monitoring 
Organisation Arup Group
Country United Kingdom 
Sector Private 
PI Contribution Our team worked on project (NE/N018745/1) to capture 360degree image and audio data for acoustic environmental monitoring, including sound level monitoring using a variety of measurement methods (professional sound level meter, app/iPhone, distributed real-time sensors). We have shared our data with them, post field-work.
Collaborator Contribution Expertise and advice in environmental monitoring and access to prior work completed by our partners. They enabled us to gather traffic flow data and loaned sound level meters for data gathering.
Impact We are working on developing outputs from this project, and looking to publication and further collaboration.
Start Year 2016
 
Description BBC Audio R&D 
Organisation British Broadcasting Corporation (BBC)
Department BBC Research & Development
Country United Kingdom 
Sector Public 
PI Contribution The researcher working on the WEAVER project was seconded to the BBC in Salford/London for three months when developing skills in relation to the underlying web audio technology we were developing. We have helped members of their own team to work with York Theatre Royal in filming, recording and data capture to help with the BBC Venue Explorer project as a test case study. We have made software publicly available that both parties have made contribution to. This has now resulted in a new EPSRC/BBC ICASE Studentship, starting October 2018.
Collaborator Contribution BBC hosted and supervised our researcher for 3-months between Salford and London. They have provided guidance and development in use of the web audio framework for development work, and contributed towards the publicly available software we have developed as well as co-authored a conference paper with us. We are now co-supervising a PhD student together.
Impact The discussions around this project resulted in the following successful funded application: WEb Audio Virtual Environment Rendering (WEAVER): Online Virtual Acoustics for Sonic Art, Digital Heritage, and Broadcast, AHRC, AH/N00356X/1 As well as a contribution towards: The Digital Creativity Hub, EPSRC, EP/M023265/1 One conference paper is forthcoming: Brown KI, Matthew Paradis M, Murphy DT, "OpenAirLib: A JavaScript library for the Acoustics of Spaces", AES 142nd Convention, Berlin, May, 2017. We have secured further funding as noted above through a EPSRC/BBC ICASE PhD studentship. BBC are also now a partner on our new AHRC Creative Media Labs project (Starting October 2018), reference: AH/S002839/1
Start Year 2014
 
Description York Theatre Royal 
Organisation Theatre Royal York
Country United Kingdom 
Sector Academic/University 
PI Contribution Members of our Research time have worked with York Theatre Royal over a number of instances in (1) the acoustic capture and documentation of the auditorium; (2) the acoustic capture and documentation of the auditorium with a full audience in place; (3) digital capture of live performances from multiple perspectives; (4) the discussion on how digital technology might be developed for an integrated as part of a live performance.
Collaborator Contribution York Theatre Royal have offered access to their venue for measurement and capture work; arranged for specific performances to enable filming, recording and capture; worked with live performances and audiences to enable in-performance capture work. Considerable staff time has been involved in the development of these activities. They have also been leading on discussions on how digital technology might be better integrated into their performance work.
Impact To date, this partnership has helped to inform and resulted in two funded grant applications: (1) The Digital Creativity Hub, EPSRC, EP/M023265/1 (2) WEb Audio Virtual Environment Rendering (WEAVER): Online Virtual Acoustics for Sonic Art, Digital Heritage, and Broadcast, AHRC, AH/N00356X/1 Publications and digital assets are still being developed. This is a multi-disciplinary project, involving digital creativity, electronic engineering, acoustics, music and audio technology, as well as creative arts practice.
Start Year 2014
 
Title OpenAirLib 
Description This is a Javascript library, written using the Web Audio Framework to enable OpenAir audio data from our online website (www.openairlib.net) to be used more freely in third party applications. The application receives commands from the client-side classes, performs queries on the OpenAIR internal database, formats the data and returns it to the client classes. Ultimately this library enables OpenAIR acoustic data to be easily integrated into other online applications that use the Web Audio Framework. 
Type Of Technology Webtool/Application 
Year Produced 2017 
Impact This work was prepared in collaboration with colleagues at BBC R&D, with a view to being used in some of their future online broadcast work. 
URL https://github.com/UoYWEAVER/OpenAirLib
 
Description 1.618 Immersive Sound Podcast 
Form Of Engagement Activity A broadcast e.g. TV/radio/film/podcast (other than news/press)
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Media (as a channel to the public)
Results and Impact I was interviewed about my research in immersive sound for the 1.618 Digital podcast series.
Year(s) Of Engagement Activity 2018
URL https://1618digital.com/immersive-audio-podcast/
 
Description AES/IoA Midlands Section: Sound Design in Our Sound Environment: Soundscape Design, Auralisation and Evaluation in Environmental Acoustics 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Industry/Business
Results and Impact This presentation was invitation to speak to a joint meeting of the Midland section of the Audio Engineering Society and Institute of Acoustics, for their Christmas lecture, held at University of Wolverhampton, Walsall Campus, on 6-Dec-2018. It was an opportunity to present a series of case studies from our group in the area of environmental acoustics and soundscape to an audience of mostly industry practitioners, but also including researchers and students.
Year(s) Of Engagement Activity 2018
 
Description Festival of Ideas: Digital Futures - Telling Stories with Sound and Creative Digital Technology, 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Public/other audiences
Results and Impact An invitation by York Theatre Royal to talk about our collaboration work, focusing on digital creativity. This was organised as part of York's Festival Ideas, an annual free festival of science and cultural events across the city. Presentation title, "Digital Futures: Telling Stories with Sound and Creative Digital Technology, York Theatre Royal, York, 8 Jun. 2016."
Year(s) Of Engagement Activity 2016
 
Description Sound Design in Our Sound Environment: Soundscape Design, Auralisation and Evaluation in Environmental Acoustics 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact This presentation was invited as part of a seminar series around a PhD defence at the Department of Media Technology, KTH, Stockholm, Sweden. It was an opportunity to present a series of case studies from our group in the area of environmental acoustics and soundscape to an audience of researchers, peers and industry, and to promote our recent work in this area at a leading international research group.
Year(s) Of Engagement Activity 2018
 
Description Welcome to the Real World - Immersive Audio for Augmented and Virtual Reality 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Industry/Business
Results and Impact The focus of the Welcome to the Real World - Immersive Audio for Augmented and Virtual Reality event in May 2017 was to highlight some of the UK's most exciting current work in immersive audio research, development and content design. It brought together over 100 leading researchers and practitioners from academic, industry and creative sectors and provided an overview of the immersive audio creative process, considered application workflows and the potential for new developments. It also created a platform for demonstrating some of the most exciting immersive audio experiences created so far by leading teams around the UK. We were asked to organise this workshop by ImmerseUK - Immerse UK is supported by Innovate UK and managed by the Knowledge Transfer Network (KTN) as a Special Interest Group (SIG).
Year(s) Of Engagement Activity 2017
URL http://www.immerseuk.org/new-blog/2017/7/19/the-importance-of-audio-in-augmented-and-virtual-reality