MAES - Towards Structured Interactive Immersive Musical Experiences

Current research demonstrates that the most successful digital musical instruments are intuitive, exploiting spontaneous gesture-sound associations resulting from life-long experience. Instruments do not have to implement traditional music gestures. In fact, by incorporating expressive gestures from other domains, they can enable large numbers of potential creators/performers to make music; for example, videogame players who may not have formal musical training but already possess sophisticated skills in the use of digital game controllers. Therefore, development of appropriate digital music devices can unleash latent creative potential regardless of musical background.

This project aims to enable music performance by using natural hand actions (e.g. throwing objects, sowing seeds, etc.) - furthering the possibilities afforded by game controllers. It will implement a self contained 'Manual Actions Expressive System' (MAES) consisting of a digital glove controlled by specialised software for the creation of musical gestures. While these gestures result from tracking and analysing hand position, rotation and finger bend, the technology will allow performers to concentrate on natural actions from daily use of the hands (e.g. the physical movement associated with 'throwing' and 'sowing'). For this reason, MAES will not require formal musical training, producing sophisticated sound that is a believable result of the performer's natural actions and providing intimate control of the sound. Thus, it will allow individuals who would not have had the opportunity otherwise, to engage actively in music making. At the same time, it will enable performers to achieve virtuosity by providing gestures that can be adapted to individual requirements: gestures will be developed, tested, validated and demonstrated through the creation of a musical work for live performance.

Research and development has been carried out using hand tracking technology with various degrees of success, but none of this has so far yielded a sufficient repertoire of intuitive gestures that produce sophisticated musical materials. Thus, this project is central to current concerns in gesture and expressivity; being crucial in the development of effective digital interfaces, the creation of new media repertoires and consolidation of new artistic forms.

MAES will complete the first stage of an overarching strategy for the realisation of 'Structured interactive immersive Musical experiences' (SiiMe), in which users advance at their own pace in a virtual environment stimulating all the senses (hearing, sight, smell, etc.) and choose their own trajectory through a musical work, but have to act within its interactive rules and constraints towards a final goal - the realisation of the music. This first stage will identify roadmaps for further research and development: for instance, performance instructions may be subsequently implemented in the form of a videogame, acting as a score with game motivations such as rewards for accurate performances, etc.

MAES will create cultural benefits by:

- Unleashing creative potential regardless of musical background.
- Allowing development of creative skills towards virtuosity.
- Allowing more individuals to embrace active forms of creative engagement (instead of passive listening/viewing).
- Enabling a wide spectrum of aesthetic conceptions and approaches, thus challenging the 'popular'/'art' schism.
- Developing current and new art forms.

Ultimately, SiiMe will also create mid-long term industrial, social, scientific and educational benefits:

- Developing the entertainment industries (e.g. music, games, film).
- Enabling new forms of communal social activity in both real and virtual spaces.
- Providing researchers in other fields with content and technologies for simulation environments for other disciplines, vocational training and new learning environments (e.g. 'participation' in historical events).

This is the first stage of a long term research strategy which will produce pivotal cultural and social shifts, creating significant benefits in these areas and providing substantial mid-long term industrial, social, scientific and educational benefits.

A significant part of this impact concerns two issues. Firstly, it will contribute to the shift, heralded by recent technological development, from the relative isolation of private listening/viewing of readymade artefacts, to active forms of engagement; fostering creative social interaction that enhances quality of life and social wellbeing. Its effects will emulate those incipient in later generations of computer games, in which new paradigms encouraging communal activity at home[1] or in virtual venues[2] (often accessed from home), exemplify a process whereby human beings meet together in order to enjoy a common activity. Technology at the centre of this research will make this possible for music performance in a modern incarnation of the parlour; be it at home or as a 'cyber-soiree'. In this context videogames offer an ideal platform for the realisation of music in the 'new parlour'. Therefore, the current proposal will enable further developments (both public and commercial, see 'Pathways to Impact'), that will benefit the increasingly large numbers of game players and cyber socialites.

Secondly, it will have a significant cultural impact by enhancing creative output and contributing to the post-modern blurring of 'popular'/'art' boundaries, challenging the 'traditional' schism: this project will benefit creators and performers in the modern parlour, who embrace music resulting from a wide spectrum of aesthetic conceptions and approaches. Furthermore, since current technology is used to reduce virtuosity requirements (subsuming complexity while allowing control through simple actions) and performance is based on paradigms that are idiomatic to a wide range of contemporary potential performers, music-making becomes accessible to a whole videogame generation that prefers interactivity and, although lacking formal musical training, already possesses appropriate strategic and motoric performance skills[3]. It is therefore capable of unleashing latent creative potential regardless of musical background, enabling development of creative skills towards virtuosity.

As the degree of sophistication of sonic output and its organisation is subsumed within the technology, this will generate demand for further sophistication in order to achieve more refined musical expression and so on[4], in a feedback cycle that has the potential to develop new artistic expression; eroding further the 'popular'/'art' differentiation until this is made irrelevant.

The outputs are prototypes for further development in partnership with the commercial sector. Therefore, this project will provide innovative content and support for the creative and cultural industries, and media (e.g. music, games, film); fostering UK and global economic performance.

Further long term developments will benefit education through new generations of virtual learning environments (e.g. 'participation' in historical events, 'interaction' with physics' laws), as well as therapeutic practice for the general public through environments for treatment of disabilities, learning disorders, rehabilitation, etc.

---
[1] E.g.: Nintendo Wii Sports (2006) games.
[2] E.g.: Call of Duty 4: Modern Warfare (2007) and Call of Duty: Modern Warfare 2 (2009).
[3] This pool of potential performers is large beyond precedent, as evidenced by videogames sales figures.Tthe BBC (2008) reported that 'spending on games will rise by 42% to £4.64bn in 2008, with sales on music and video at £4.46bn'.
[4] This happens with graphics' sophistication, where demand for fidelity independent of content (i.e. a scene may not be possible but i