Transforming Transformation: 3D Models for Interactive Sound Design

Lead Research Organisation: Birmingham City University
Department Name: ADM Birmingham Conservatoire

Abstract

'Transforming Transformation: 3D Models for Interactive Sound Design' will open up a new field of music technology research by exploring human-centred approaches to creative sound design. The project will involve the development of a proof-of-concept system, which enables sound to be manipulated through hand movements as if it were an invisible 3D object. The interaction will be enhanced by real-time visualisation within a virtual 3D space. The research surfaces the notion of a 'natural' interface for sound design taking account of already learned behaviour and innate skills such as motor memory, gesture and spatial awareness. The aim is to enable musicians and sound designers to 'think in sound' when working with technology, catalysing a shift from technology-centric to human-centric models. As a pilot project, we will concentrate on one specific audio processing technique: sound spatialisation, with a view to later amplifying our research findings in a larger project, addressing a wider range of sound design practices. In simple terms: our system will explore the idea that we can make users feel as if they are directly positioning sound in space by 'picking it up and moving it' with their hands, with the aim of serving as an exemplar for future, more detailed studies.

Sound design entails the manipulation of sound for dramatic, realistic, musical, or emotional effect. Typically, audio is processed using techniques such as filtering, time-stretching, pitch shifting, granulation, attenuation and panning. These techniques are exposed to the sound designer through discrete controls such as 'sliders' and 'dials' each of which controls a single parameter in the underlying signal processing system. By contrast, human concepts of sound are characterised by polymorphous, perceptual, metaphorical, symbolic and onomatopoeic associations. Many of these associations are cross-sensory, for example: 'warm', 'bright', 'distant', 'metallic'. Furthermore, they suggest physicality and invite the possibility of direct quasi-tactile manipulation. This is especially true of spatialisation, where there exists a sense of sound localisation within an acoustic space, further evoking the idea of tangibility: 'moving' or 'placing' a sound, or even 'pushing' a sound to provide inertia in a given direction.

Our research therefore challenges existing sound design models and explores the notion of a 'natural' and direct corporeal link between imagined sound and acoustic results. In order to achieve this, we will draw upon the immense benefits offered by the PI's research lab within a leading music Conservatoire. Being based within a Conservatoire environment, provides the project with immediate and unfettered access to high-quality musicians from a range of backgrounds. This serves to ground the research firmly in artistic practice, allowing artistic ideas to both test extant models and serve as the basis for new ones. Formal user experience reviews will be undertaken at milestone stages in the project allowing us to surface users' tacit views and opinions on their experience when using the developed system. The research method will also entail the commissioning of a new musical work utilising the system, to be premiered in a public workshop and concert in the final month of the project. This will serve as both an opportunity for testing in a 'real world' creative scenario and a means of disseminating the projects outputs in a public forum. There will also be a mid-project 'experimental jam' using the in-development system, which will be broadcast live on the internet.

Music technology research has a strong track record for impacts in the sound design industry, particularly in game audio through techniques such as procedural audio generation. Our exploration of novel human-centred approaches to sound design will open up a new field of research and through a firm grounding in creative practice, cross-disciplinary impacts will be all the richer.

Planned Impact

Practice-based music technology research has a strong track record for achieving impacts within the wider creative industries, particularly gaming and software development. This has been evidenced through recent advances in gaming such as 'procedural audio', which originated in experimental music and sound art. It is therefore essential that the digital arts continue to provide leadership in this area, with innovation unrestricted by commercial constraints. However it also important to acknowledge the bi-directionality of this transfer: the digital arts and creative industries now work in powerful synergy, each informing the work of the other. We outline below ways in which we build on this synergy and ensure the cultural and economic impacts of our project.

In order to ensure the robustness and relevance of our work to the commercial sound design sector, we have engaged with Two Big Ears, a company specialising in immersive and interactive audio, with a wide range of sound design and software development experience. This will ensure impacts are all the richer with cross-fertilisation between techniques from creative sound design in areas of artistic practice such as live electronic music, and aspects of the creative industries such as sound design for game audio and sound for media.

Music technology research also has a long history of impact within the wider creative and cultural sector. This can be seen through the recent "Max for Live" software, which is widely used in electronic dance music and incorporates Max, an application that originated as a research project at IRCAM. Our research seeks to follow a similar model by providing a novel approach to interactivity, which may be adapted into a wide range of sound processing scenarios, particularly as advanced motion capture technology becomes more ubiquitous. This will be achieved by fully documenting our software-based research outputs and making them available under open source license through a widely used code collaboration portal.

Our project will also bring direct cultural impacts within the performing arts. 'Sound design' is often thought of as an offline process, but 'live sound design' entails the manipulation of sound in live performance. The immediate benefits of this will be amongst the creative practitioners directly involved in the project as artistic 'experimenters' with the system. In order to ensure these impacts reach a wider audience, the project will involve an official musical commission with a public world premiere at Birmingham Conservatoire in the final month of the project. It will also include a live online 'jam' mid-project where the research will be explained and demonstrated to a global online audience.

Finally, we will strongly promote our project, particularly the emerging themes of human-centred interaction and natural user interfaces for sound design through a project blog and social media channels. The existing networks and and social media accounts of the project partners collectively have a large number of followers, and this will be utilised as the project progresses in order to achieve a two-way dialog with interested communities.

Publications

10 25 50
 
Description We conducted an in-depth user experience and usability study with 15 participants, comparing our 3D system with an extant and widely used 2D system with equivalent functionality.

Based on usability metrics, interview data and UEQ questionnaire data, our findings showed that users were more easily able to learn how to use our 3D system, and found it more 'novel' and more 'appealing'. The data also showed that the extant 2D system operated more reliably with respect to user input and that a number of deficiencies in the implementation of our system (for example the type of sensors used) meant task completion times were slower than with the 2D system.
Exploitation Route Our research forms part of a wider ecosystem of research and (industry) development exploring 3D approaches to sound processing and music production.

Of particular benefit to other researchers will be the results of our extensive usability / user experience test comparing our 3D system to an equivalent widely used 2D system. The results of this study will be published in a leading scholarly journal and will serve as a resource against which other researchers can articulate or explain the successes or failings of their own developments.

The project team also seeks to take forward the findings of the project in a larger follow-up bid.
Sectors Creative Economy,Digital/Communication/Information Technologies (including Software),Education,Culture, Heritage, Museums and Collections

URL http://integra.io/ahrcdigitrans2015/
 
Description Glasgow School of Art, Digital Design Studios 
Organisation Glasgow School of Art
Department Digital Design Studio
Country United Kingdom of Great Britain & Northern Ireland (UK) 
Sector Academic/University 
PI Contribution We worked with this partner on our 2015-15 AHRC Digital Transformations Small Grant project. The contribution of our research team was to lead and manage the project and to undertake system design and testing for software developed by GSA.
Collaborator Contribution GSA contributed software development expertise and resources to the project as well as insight and expertise on 3D virtual environments.
Impact A 3D software environment for touch free spatial audio positioning A presentation / live demo at Music Tech Fest 2015 A proceedings-published paper at the Electronic Visualisation in the Arts conference 2016 A symposium and workshop as part of the Frontiers festival in Birmingham 2016
Start Year 2015
 
Description Two Big Ears 
Organisation Two Big Ears
Country United Kingdom of Great Britain & Northern Ireland (UK) 
Sector Private 
PI Contribution Testing and use of the 3Dception audio framework
Collaborator Contribution Advice and consultancy regarding the installation and integration of 3Dception into our project
Impact A 3D software environment for touch free spatial audio positioning
Start Year 2015
 
Title 3D Sound 
Description The software is a prototype for an immersive 3D application for touch-free spatial audio positioning built using the Unity 3D development environment with the 3Dception and UniOSC plugins. The software has also been described in other contexts using the working title "Integra Forms" 
Type Of Technology Software 
Year Produced 2015 
Open Source License? Yes  
Impact The software is at a prototype / proof-of-concept stage and has not yet been used in practice. However, it has resulted in a number of publications (one peer-reviewed conference paper already accepted and one submitted) including a journal article under development It was also the inspiration for a new musical work, which will be performed in public in 2016. 
URL https://github.com/BirminghamConservatoire/AHRCDigiTrans2015
 
Description Music Tech Fest presentation (Ljubljana) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact I presented the background to our AHRC Digital Transformations project and gave a live demo of the system. Whilst I gave the demo, a photograph of it was tweeted by @MusicTechFest on Twitter, an account that has over 3500 followers.

Further online engagement was achieved through the MusicTechFest and Integra Lab FaceBook pages.
Year(s) Of Engagement Activity 2015
URL http://musictechfest.net/project/jamie-bullock-integra-forms/
 
Description Project website / blog 
Form Of Engagement Activity Engagement focused website, blog or social media channel
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact The activity comprises the development of an online blog for promoting the project. This includes video demonstrations of the project software, hosted on the Vimeo sharing site as the project progresses as well as a number of posts about the research process.

The work is also cross-promoted via the Integra Lab twitter feed and Facebook page, which have over 250 followers
Year(s) Of Engagement Activity 2015
URL http://integra.io/ahrcdigitrans2015/