Spatial Audio in Virtual and Augmented Reality Technologies
Lead Research Organisation:
Bangor University
Department Name: Sch of Music
Abstract
Virtual and Augment Reality (VAR) is a rapidly developing area of creative technology. As VAR technologies begin to mature and become more widely adopted, it is clear that the quality and detail of the immersive experience will be of increasing importance. Central to this will be high quality immersive sound. There is a saying in VR: "the ears point the eyes". When we hear a sound that is out of our field of view, we instinctively look round to find its source. Sound is a deeply embedded aspect of any VAR experience and is an important instigator for interaction in immersive environments, yet many VAR companies have been predominantly focused on graphics and image.
The aim of this partnership is to develop new technologies for working creatively with high quality VAR sound in the most detailed ways, and to explore new creative applications for VAR sound. These could include soundscape recording, 'sonic tourism', new forms of creativity, interactive distance-learning, remote performing-art collaboration, and so on. In the future, VAR will find creative uses that are as yet unimagined. This partnership will help UK and Chinese creative industries on that journey of imagination.
The aim of this partnership is to develop new technologies for working creatively with high quality VAR sound in the most detailed ways, and to explore new creative applications for VAR sound. These could include soundscape recording, 'sonic tourism', new forms of creativity, interactive distance-learning, remote performing-art collaboration, and so on. In the future, VAR will find creative uses that are as yet unimagined. This partnership will help UK and Chinese creative industries on that journey of imagination.
Planned Impact
The project will be of benefit to specific SME's in the UK and China, who will gain access to research expertise that can inform product development, particularly that focused on the creative application of spatial audio in VAR environments. They will be able to develop their international competitiveness by drawing on trans-national knowledge resources that they would not otherwise be able to access. They will also benefit from the development of new applications for VAR technology, which will make their products and services more useful and attractive to potential end users.
Two of the SME's are based in Wales, and this could help Wales gain a substantive edge within the highly competitive VAR field and help develop further industry growth and employment within the sector.
Another focus for the research and its application the potential to create high quality VAR soundscape environments. This could help Wales develop its international marketability as a tourist destination, through the development of virtual 'sonic-tourism' experiences.
The development of VAR-audio for distance learning in sonic art and creative audio applications could have long-term benefit to the creative industries sector, which is currently experiencing a skills shortage. It would enable new partnership educational programmes to be developed which would not be constrained by national or geographical boundaries.
Our work with theatre partners developing VAR for remote performing-art collaboration could benefit performing arts organisations in China and the UK by enabling international collaborative work.
The bringing together of partners with such a unique range of research knowledge and practical expertise, across national boundaries, has the potential to give Wales a significant edge in the creative application of new technologies for VAR audio.
Two of the SME's are based in Wales, and this could help Wales gain a substantive edge within the highly competitive VAR field and help develop further industry growth and employment within the sector.
Another focus for the research and its application the potential to create high quality VAR soundscape environments. This could help Wales develop its international marketability as a tourist destination, through the development of virtual 'sonic-tourism' experiences.
The development of VAR-audio for distance learning in sonic art and creative audio applications could have long-term benefit to the creative industries sector, which is currently experiencing a skills shortage. It would enable new partnership educational programmes to be developed which would not be constrained by national or geographical boundaries.
Our work with theatre partners developing VAR for remote performing-art collaboration could benefit performing arts organisations in China and the UK by enabling international collaborative work.
The bringing together of partners with such a unique range of research knowledge and practical expertise, across national boundaries, has the potential to give Wales a significant edge in the creative application of new technologies for VAR audio.
Organisations
- Bangor University (Lead Research Organisation)
- Metakey Acoustics (Collaboration)
- Music King (Collaboration)
- Leeds Beckett University (Collaboration, Project Partner)
- Shanghai Municipal Engineering Design Institute (Collaboration)
- Shanghai Normal University (Collaboration, Project Partner)
- M7Virtual (Collaboration)
- M7Media Ltd (Project Partner)
- Music King (Shanghai) Cultural Dev. Co. (Project Partner)
| Title | Andrew Lewis: 'Canzon in Double Echo', for ensemble and electronics |
| Description | At the turn of the 17th century, composers in Venice were perfecting many years of experiments in spatial music. Their sonic laboratory was St Mark's Basilica, and the most famous and accomplished of them was Giovanni Gabrieli. Canzon in Double Echo (a title borrowed from Gabrieli) uses electronics to pursue similar aims. As with Gabrieli's work, the echoes in Canzon are not mere repetitions, but transformations - we hear each musical object from different multiple vantage points in time and space. In doing so our perception of them changes, sometimes dramatically. In many ways, this musical and sonic phenomenon echoes our human experience of time. Calling past events to mind is not simply to repeat them, but to perceive them through the powerfully transformed echoes of memory - from another place, another time. The musical echoes in Canzon take many forms: answering or echoing phrases, sonic echoes in the imaginary space created by loudspeakers, the echoes of fragments of Gabrieli's original, echoes of musical memories from elsewhere, echoes of echoes, double echoes In the final section of the piece, all the music that has gone before is echoed again, but in a changed state. It is as if the sound of praise that echoed through St Mark's Basilica now echoes beyond its walls, down through the centuries, and then beyond even the confines of space and time, into eternity. |
| Type Of Art | Composition/Score |
| Year Produced | 2020 |
| Impact | This works develops the idea of multiple spatial realities using the model of the 16/17th century technique of cori spezzati (broken choirs) and extends this idea into the domain of live instrumental music through computer processing. It has been performed in public concerts by the Uproar Ensemble, in Cardiff, Aberystwyth, Bristol, Montpellier (France) and Caernarfon, and broadcast on BBC Radio 3's 'New Music Show' |
| URL | https://www.andrewlewis.org.uk/works/canzon.php |
| Title | Andrew Lewis: 'Cori Spezzati', fixed-medium acousmatic music in 8 channels |
| Description | At the turn of the 17th century, composers in Venice were perfecting many years of experiments in spatial music. Their sonic laboratory was St Mark's Basilica, and the most famous and accomplished of them was Giovanni Gabrieli. Cori Spezzati is one of a series of works inspired by Gabrieli's spatial music, the first being Canzon in Double Echo (2020). The title (literally 'broken choirs') refers not only to the distribution of musical material across multiple spaces, but also to the 'brokenness' of some of the sound materials, whose contorted glitches and twisted, explosive discontinuities are the result of the failure of digital software. Since the piece was composed as the war in Ukraine was taking hold, these broken and shattered sounds became associated in my mind and with the daily news reports of broken bodies, broken lives, a broken international order. A question formed in my mind: is it possible for broken choirs to sing together? Cori Spezzati was composed in 2022 in the Electroacoustic Music Studios of Bangor University (Wales, UK) and premiered on July 1 2022 in the Elgar Concert Hall, Birmingham, as part of BEAST FEaST. |
| Type Of Art | Composition/Score |
| Year Produced | 2022 |
| Impact | This piece developed the idea of multiple spatial realities using the model of the 16/17th century technique of cori spezzati (broken choirs). It has been presented at public performances in Birmingham, Bangor and Leeds. |
| URL | https://www.andrewlewis.org.uk/works/cori_spezzati.php |
| Title | Andrew Lewis: 'Three Storms', fixed-medium acousmatic music in 8 channels |
| Description | February 2022 saw a remarkable sequence of three named storms in the UK - Dudley, Eunice and Franklyn - which followed each other within the course of a single week. It was another of the increasingly frequent record-breaking weather events that point towards the dangers of climate change. The musical foundation of Three Storms uses data from buoys located around the south west of England and North Wales, which sample the frequencies of ocean waves at different UK locations, every half hour. These wave motions have then been shifted up into the range of human hearing, making the 'vibrations' of UK coastal waters audible, and compressing a month of activity into seven minutes. Wave data was supplied by the Channel Coastal Observatory (https://coastalmonitoring.org/). I am grateful to Dr David Christie, Research Fellow in Ocean Renewable Energy Modelling in Bangor University's School of Ocean Sciences, for his invaluable help in obtaining and understanding the data. |
| Type Of Art | Artwork |
| Year Produced | 2022 |
| Impact | Three Storms has been performed in public concerts in Bangor and Leeds |
| Title | Andrew Lewis: 'Two Lakes', fixed-medium acousmatic music in 8 channels |
| Description | Lake Nezahualcóyotl is a reservoir in Chiapas created as part of a hydroelectric power scheme. The dam was completed in 1966, and the area around the original lake flooded. Just a year earlier a similar flooding drowned the village of Capel Celyn in Wales, controversially displacing its Welsh-speaking residents. At both lakes, droughts cause the periodic re-emergence of the drowned buildings, with increasing regularity: a 16th century church at Lake Nezahualcóyotl and the ruins of the Welsh village of Capel Celyn. In Wales, these appearances re-awaken painful memories of past injustices, but they are also a very present reminder of the crisis of climate change, in Mexico, Wales and across the globe. 'Two Lakes' was composed in the Electroacoustic Music Studios of Bangor University, Wales. Materials in the piece have been shaped using water level and flow data from both lakes, provided by the Copernicus Climate Change Service and the National River Flow Archive (UK). I am indebted to Dr Iestyn Woolway, of Bangor University's School of Ocean Sciences, for his assistance in accessing and understanding the data. |
| Type Of Art | Composition/Score |
| Year Produced | 2023 |
| Impact | 'Two Lakes' was premiered in Morelia, Mexico, at Visiones Sonores 19 hosted by the Mexican Centre for Music and Sonic Art (CMMAS), and sponsored by the British Council. |
| Title | Nikos Stavropoulos: 'Claustro' (fixed-medium acousmatic music in 5.1 channels) |
| Description | Claustro is a sonic art work, composed in 5.1 cinema surround format. Composer: Nikos Stavropoulos Claustro: derived from the Latin, "claustrum," meaning "shut-in" or "enclosure." Claustro is the third composition in a series of works which explore aural micro-space. A sounding place of improved intelligibility through greater aural intimacy. The work is an invitation to come in and listen out for the thin line between philia and phobia that such places evoke. The discontinuous and non-homogenous nature of acoustic space inspires the arrangement of sound materials here. Recordings of original sound sources were conducted using a micro multichannel array designed and built in collaboration with Huw Mcgregor. |
| Type Of Art | Composition/Score |
| Year Produced | 2019 |
| Impact | Award of Distinction (First place), Fixed medium, MA/IN 2020, Matera (Italy) Mention, Ars Electronica Forum Wallis 2020, Leuk (Switzerland) 2nd prize ex æquo, Fixed medium, 13th Destellos Competition 2019, Mar del Plata (Argentina) |
| URL | https://soundcloud.com/nikos-stavropoulos/claustro-stereo-reduction |
| Title | Shen Lin: 'DiMingQueYin' (fixed-medium acousmatic music in 16 channels) |
| Description | 'DiMingQueYin' is a sonic art work, composed in 16-channel immersive spatial sound format. It is based on the sound of birdsong. Composer: Shen Lin |
| Type Of Art | Composition/Score |
| Year Produced | 2019 |
| Impact | Performance at the 20th MUSICACOUSTICA-BEIJING, 25 October 2019 |
| Title | Shen Lin: 'DiMingQueYin' (fixed-medium acousmatic music, stereo version) |
| Description | 'DiMingQueYin' is a sonic art work, originally composed in 16-channel immersive spatial sound format. This is the stereo version. It is based on the sound of birdsong. Composer: Shen Lin |
| Type Of Art | Composition/Score |
| Year Produced | 2020 |
| Impact | Performance at II Encuentro Internacional Red Ecología Acústica México", December 12 2020 |
| Description | The partnership has been developed through two forums - one in the UK and one in Shanghai - enabling face-to-face meetings between project participants and other interested parties. A mixture of formal and informal sessions allowed us to develop our knowledge of each other's activities and expertise in spatial VAR audio, to understand more fully how the different activities in which we are engaged might fit together, and to develop effective personal connections and working relationships. The UK forum (May) focused on presentations by project participants from Shanghai and the UK. Group discussions developed ideas arising from presentations, and established some key themes for the project. The Shanghai visit (July) comprised a series of meetings which enabled us to explore new partnerships to extend our network: • Meeting with a leading Shanghai design institute. This meeting discussed several public-space landscaping projects, and the potential for creating interactive VAR audio experiences using purpose-installed permanent infrastructure. • Tour of an audio technology manufacturer and meeting with executives to discuss development of microphones for VAR spatial audio capture. • Meeting with the Executive Dean of a Shanghai university to discuss developing educational collaborations. This would especially focus on the use of VAR audio for the teaching of creative spatial audio. • Meeting with a Professor at a Shanghai university, to discuss areas of common ground in the use of AI and machine learning in music and audio creation. We also held a meeting in the UK between participants at in the UK, in order to assess the potential of a new VAR system for working creatively with spatial audio. Development of the 5.1 and ambisonics microphones with a Shanghai company is under discussion. Further discussions are also ongoing on the development of the VAR spatial manipulation system for creative projects. Prior to our partnership meeting in Shanghai we had already begun discussions with another AHRC partnership team on developing a more substantial collaborative partnership. To further this we organised a meeting between the two groups in Shanghai. This meeting allowed us to get to know our respective partners and explore common areas of interest and expertise. |
| Exploitation Route | The partnerships developed are ongoing and evolving, and contribute to the future development of China-UK Creative Industry collaboration more generally. |
| Sectors | Creative Economy,Digital/Communication/Information Technologies (including Software),Electronics,Culture, Heritage, Museums and Collections |
| Description | In the course of our forums and international meetings, several fruitful cooperative partnerships were developed. Our meeting with a Chinese Landscape Design Institute led to cooperation on the design of sound installations for a new green space development in Shanghai, for which AHRC funding was sought. This partnership involved landscape design by the Chinese partner, with sound design by UK partners. In addition, the incorporation of audio infrastructure into the landscape design was informed by the specifications for the sound design elements from the UK side. The development was a 4m Yuan project aimed at improving the quality of life and wellbeing of residents in the urban Jing'an District of Shanghai, and the sound design was a key element of this. Another important partnership was that between UK academic partners and a Shanghai SME, aimed at developing the technical aspects of the sound infrastructure for the park development. This infrastructure was to be incorporated in the building phase of the project. Unfortunately follow-on funding for the project was not secured to continue and develop this work A distinct element of our project is the development of a compact and affordable 5.1 microphone design, in a collaboration between UK academic partners and a UK SME. The project enabled links to be established with a Chinese manufacturer aimed at possible commercial production of the microphone. Further to these developments, we also began collaboration on a system using VR as a tool for spatial sound design. This currently involves UK partners only, but we are currently seeking funding to enable us to connect this with our Chinese activities. Funding has not yet been secured. A further grant application for work in spatial audio is shortly to be submitted to AHRC. |
| First Year Of Impact | 2019 |
| Sector | Communities and Social Services/Policy,Creative Economy,Digital/Communication/Information Technologies (including Software),Electronics,Environment,Manufacturing, including Industrial Biotechology |
| Impact Types | Cultural,Societal,Economic,Policy & public services |
| Description | Leeds Beckett University |
| Organisation | Leeds Beckett University |
| Department | Faculty of Arts, Environment & Technology |
| Country | United Kingdom |
| Sector | Academic/University |
| PI Contribution | The team allowed Leeds Beckett researchers to connect with a wide range of academic partners in the UK and China. |
| Collaborator Contribution | Leeds Beckett undertook the underpinning research that informed the design of a microphone prototype. Leeds Beckett also developed a spatial VR system that forms the basis of current development plans for the project. |
| Impact | None |
| Start Year | 2019 |
| Description | M7 Virtual |
| Organisation | M7Virtual |
| Country | United Kingdom |
| Sector | Private |
| PI Contribution | The team introduced M7 Virtual to the principles behind spatial audio in an acousmatic/sonic art context |
| Collaborator Contribution | M7 Vitural introduced the team to the concepts and practices of 3D audio in a commercial video production context |
| Impact | None |
| Start Year | 2019 |
| Description | Metakey Acoustics Ltd |
| Organisation | Metakey Acoustics |
| Country | United Kingdom |
| Sector | Private |
| PI Contribution | The team provided some of the academic expertise that ed into the microphone design proposed by Metakey |
| Collaborator Contribution | Metakey formulated microphone designs, and interactive improvements to the designs, in response to academic research and the requirements of creative practitioners. |
| Impact | Prototype microphone design |
| Start Year | 2019 |
| Description | Music King Shanghai |
| Organisation | Music King |
| Country | China |
| Sector | Private |
| PI Contribution | Our team has introduced Music King to the practices and requirements of spatial sound design in an acoustic music/sonic art context |
| Collaborator Contribution | Music King has contributed expertise in the design, specification and sourcing of infrastructure for our landscape design project, and also introduced us to a potential collaborator on microphone development. |
| Impact | This partnership has resulted in expanding our network of industry connections in Shanghai, and in taking forward design and specification of audio infrastructure for our urban green space project |
| Start Year | 2019 |
| Description | Shanghai Normal University Music School |
| Organisation | Shanghai Normal University |
| Country | China |
| Sector | Academic/University |
| PI Contribution | We have participated in a network based around multi-lateral meeting and discussions, both in the UK and in Shanghai. The brought together a range of academic and industry partners. |
| Collaborator Contribution | Shanghai Normal University was one of the academic partners in the network described above. |
| Impact | No specific outcomes at present. We are developing proposals for grants to support future developments. |
| Start Year | 2019 |
| Description | Virtual Reality Music and Sound Design for New Urban Public Green Spaces |
| Organisation | Shanghai Municipal Engineering Design Institute |
| Country | China |
| Sector | Public |
| PI Contribution | I and the creative team members has been tasked with designing the soundscape composition for the Jing'an District Jiaozhou Road Street Park development |
| Collaborator Contribution | SMEDI has designed the Jing'an District Jiaozhou Road Street Park development |
| Impact | There are no outputs. We were not successful in winning funds to support the collaboration, and Covid brought it effectively to an end. |
| Start Year | 2019 |
| Title | 5.1 Microphone Prototype |
| Description | A 3D printed 5.1 surround sound microphone prototype, with cardioid capsules |
| Type Of Technology | Physical Model/Kit |
| Year Produced | 2019 |
| Impact | The microphone was demonstrated to partners in Shanghai, and calibration measurements made |
| Title | Calibration software for 5.1 microphone |
| Description | Custom-made calibration software for the 5.1 cardioid microphone developed as part of the project. |
| Type Of Technology | Software |
| Year Produced | 2019 |
| Impact | The software is integral to the use of the microphone. |
| Description | UK-based Forum |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Industry/Business |
| Results and Impact | An international forums enabled face-to-face meetings between project participants and other interested parties from China and the UK. A mixture of formal and informal sessions allowed us to develop our knowledge of each other's activities and expertise in spatial VAR audio, to understand more fully how the different activities in which we are engaged might fit together, and to develop effective personal connections and working relationships. The UK forum (May) focused on presentations by project participants Ming JinDong (Music King, Shanghai), Shen Lin (Shanghai Normal University), Mark Wynne (M7 Virtual), Andrew Lewis (Bangor University), Huw McGregor (Bangor University and Metakey Acoustics) and Nikos Stavropoulos (Leeds Becket University). Group discussions developed ideas arising from presentations, and established some key themes for the project. |
| Year(s) Of Engagement Activity | 2019 |