Transforming Transformation: 3D Models for Interactive Sound Design

'Transforming Transformation: 3D Models for Interactive Sound Design' will open up a new field of music technology research by exploring human-centred approaches to creative sound design. The project will involve the development of a proof-of-concept system, which enables sound to be manipulated through hand movements as if it were an invisible 3D object. The interaction will be enhanced by real-time visualisation within a virtual 3D space. The research surfaces the notion of a 'natural' interface for sound design taking account of already learned behaviour and innate skills such as motor memory, gesture and spatial awareness. The aim is to enable musicians and sound designers to 'think in sound' when working with technology, catalysing a shift from technology-centric to human-centric models. As a pilot project, we will concentrate on one specific audio processing technique: sound spatialisation, with a view to later amplifying our research findings in a larger project, addressing a wider range of sound design practices. In simple terms: our system will explore the idea that we can make users feel as if they are directly positioning sound in space by 'picking it up and moving it' with their hands, with the aim of serving as an exemplar for future, more detailed studies.

Sound design entails the manipulation of sound for dramatic, realistic, musical, or emotional effect. Typically, audio is processed using techniques such as filtering, time-stretching, pitch shifting, granulation, attenuation and panning. These techniques are exposed to the sound designer through discrete controls such as 'sliders' and 'dials' each of which controls a single parameter in the underlying signal processing system. By contrast, human concepts of sound are characterised by polymorphous, perceptual, metaphorical, symbolic and onomatopoeic associations. Many of these associations are cross-sensory, for example: 'warm', 'bright', 'distant', 'metallic'. Furthermore, they suggest physicality and invite the possibility of direct quasi-tactile manipulation. This is especially true of spatialisation, where there exists a sense of sound localisation within an acoustic space, further evoking the idea of tangibility: 'moving' or 'placing' a sound, or even 'pushing' a sound to provide inertia in a given direction.

Our research therefore challenges existing sound design models and explores the notion of a 'natural' and direct corporeal link between imagined sound and acoustic results. In order to achieve this, we will draw upon the immense benefits offered by the PI's research lab within a leading music Conservatoire. Being based within a Conservatoire environment, provides the project with immediate and unfettered access to high-quality musicians from a range of backgrounds. This serves to ground the research firmly in artistic practice, allowing artistic ideas to both test extant models and serve as the basis for new ones. Formal user experience reviews will be undertaken at milestone stages in the project allowing us to surface users' tacit views and opinions on their experience when using the developed system. The research method will also entail the commissioning of a new musical work utilising the system, to be premiered in a public workshop and concert in the final month of the project. This will serve as both an opportunity for testing in a 'real world' creative scenario and a means of disseminating the projects outputs in a public forum. There will also be a mid-project 'experimental jam' using the in-development system, which will be broadcast live on the internet.

Music technology research has a strong track record for impacts in the sound design industry, particularly in game audio through techniques such as procedural audio generation. Our exploration of novel human-centred approaches to sound design will open up a new field of research and through a firm grounding in creative practice, cross-disciplinary impacts will be all the richer.

Practice-based music technology research has a strong track record for achieving impacts within the wider creative industries, particularly gaming and software development. This has been evidenced through recent advances in gaming such as 'procedural audio', which originated in experimental music and sound art. It is therefore essential that the digital arts continue to provide leadership in this area, with innovation unrestricted by commercial constraints. However it also important to acknowledge the bi-directionality of this transfer: the digital arts and creative industries now work in powerful synergy, each informing the work of the other. We outline below ways in which we build on this synergy and ensure the cultural and economic impacts of our project.

In order to ensure the robustness and relevance of our work to the commercial sound design sector, we have engaged with Two Big Ears, a company specialising in immersive and interactive audio, with a wide range of sound design and software development experience. This will ensure impacts are all the richer with cross-fertilisation between techniques from creative sound design in areas of artistic practice such as live electronic music, and aspects of the creative industries such as sound design for game audio and sound for media.

Music technology research also has a long history of impact within the wider creative and cultural sector. This can be seen through the recent "Max for Live" software, which is widely used in electronic dance music and incorporates Max, an application that originated as a research project at IRCAM. Our research seeks to follow a similar model by providing a novel approach to interactivity, which may be adapted into a wide range of sound processing scenarios, particularly as advanced motion capture technology becomes more ubiquitous. This will be achieved by fully documenting our software-based research outputs and making them available under open source license through a widely used code collaboration portal.

Our project will also bring direct cultural impacts within the performing arts. 'Sound design' is often thought of as an offline process, but 'live sound design' entails the manipulation of sound in live performance. The immediate benefits of this will be amongst the creative practitioners directly involved in the project as artistic 'experimenters' with the system. In order to ensure these impacts reach a wider audience, the project will involve an official musical commission with a public world premiere at Birmingham Conservatoire in the final month of the project. It will also include a live online 'jam' mid-project where the research will be explained and demonstrated to a global online audience.

Finally, we will strongly promote our project, particularly the emerging themes of human-centred interaction and natural user interfaces for sound design through a project blog and social media channels. The existing networks and and social media accounts of the project partners collectively have a large number of followers, and this will be utilised as the project progresses in order to achieve a two-way dialog with interested communities.