MIMIC: Musically Intelligent Machines Interacting Creatively

Lead Research Organisation: Goldsmiths College
Department Name: Computing Department

Abstract

This project is a direct response to significant changes taking place in the domain of computing and the arts. Recent developments in Artificial Intelligence and Machine Learning are leading to a revolution in how music and art is being created by researchers (Broad and Grierson, 2016). However, this technology has not yet been integrated into software aimed at creatives. Due to the complexities of machine learning, and the lack of usable tools, such approaches are only usable by experts. In order to address this, we will create new, user-friendly technologies that enable the lay user - composers as well as amateur musicians - to understand and apply these new computational techniques in their own creative work.

The potential for machine learning to support creative activity is increasing at a significant rate, both in terms of creative understanding and potential applications. Emerging work in the field of music and sound generation extends from musical robots to generative apps, and from advanced machine listening to devices that can compose in any given style. By leveraging the internet as a live software ecosystem, the proposed project examines how such technology can best reach artists, and live up to its potential to fundamentally change creative practice in the field. Rather than focussing on the computer as an original creator, we will create platforms where the newest techniques can be used by artists as part of their day-to-day creative practices.

Current research in artificial intelligence, and in particular machine learning, have led to an incredible leap forward in the performance of AI systems in areas such as speech and image recognition (Cortana, Siri etc.). Google and others have demonstrated how these approaches can be used for creative purposes, including the generation of speech and music (DeepMinds's WaveNet and Google's Magenta), images (Deep Dream) and game intelligence (DeepMind's AlphaGo). The investigators in this project have been using Deep Learning, Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Long Short-Term Memory Networks (LSTMs), and other approaches to develop intelligent systems that can be used by artists to create sound and music. We are already among the first in the world to create reusable software that can 'listen' to large amounts of sound recordings, and use these as examples to create entirely new recordings at the level of audio. Our systems produce outcomes that out-perform many other previously funded research outputs in these areas.

In this three-year project, we will develop and disseminate creative systems that can be used by musicians and artists in the creation of entirely new music and sound. We will show how such approaches can affect the future of other forms of media, such as film and the visual arts. We will do so by developing a creative platform, using the most accessible public forum available: the World Wide Web. We will achieve this through development of a high level live coding language for novice users, with simplified metaphors for the understanding of complex techniques including deep learning. We will also release the machine learning libraries we create for more advanced users who want to use machine learning technology as part of their creative tools.

The project will involve end-users throughout, incorporating graduate students, professional artists, and participants in online learning environments. We will disseminate our work early, gaining the essential feedback required to deliver a solid final product and outcome. The efficacy of such techniques has been demonstrated with systems such as Sonic Pi and Ixi Lang, within a research domain already supported by the AHRC through the Live Coding Network (AH/L007266/1), and by EC in the H2020 project, RAPID-MIX. Finally, this research will strongly contribute to dialogues surrounding the future of music and the arts, consolidating the UK's leadership in these fields.

Planned Impact

We will directly engage stakeholders in the process of music making with creative tools, exploring the role that AI will play in the future of the creative industries. We will bring complex AI and machine learning technologies to the general user of creative software; we will democratise technologies that are still emerging in academia and corporate R&D labs.

These groups will benefit from new software, course materials, events, artistic outputs and industry collaborations:

a) creative practitioners and their audiences; specifically musicians, composers and their audiences;
b) the hacker/maker community;
c) industry professionals; including through existing industry partnerships with record labels (XL, Universal), music technology companies (Akai, Roli, Ableton, Reactable, Cycling74, Abbey Road Red) and our project partner, Google Magenta;
e) learners; including those from secondary and higher education, home learners, and academics and professionals
f) the general public.

A key aim of our project is to create a simplified live coding language for coding in the browser where novices can learn about AI and machine learning through a clear and simple, yet powerful, live coding programming language written on top of JavaScript , which is well supported and popular. This simplified live coding language will be designed specifically for musicians and artists and will allow them to pursue new routes for creating music.

We will facilitate a range of high-quality use cases with creative professionals. This will bridge gaps between research and industry, accelerating the impact of artificial intelligence by deploying it in real-world and professional music-making and listening contexts.

Our events series will bring musicians, composers and audiences together, providing an important platform for the continued dissemination our work and the work of those practitioners whom we support through the creation of new tools.

In addition to concerts, we will run symposia to expand and further develop critical thought in these fields, inviting participation from a range of stakeholders. We will also disseminate and support artistic output through the creation of our platform, making it simple not just to create work, but also to share it amongst friends and colleagues, from both outside and inside our connected communities.

Our background technologies will be open source and available to academics and SMEs alike, allowing them to use contemporary AI in ways that are currently very challenging for novices.

We will generate significant, engaging and unique course materials, associated with our existing MOOC provision, and targeted at a range of different learners, from secondary education, through HE, to home learners, academics and professionals. This will help people to acquire skills in machine learning at any stage.

Our track record indicates we are capable of meeting the significant interest from the general public around these issues. Recent public engagement activities from team members have included:

- applying Deep Learning to the creation of artworks we are currently exhibiting at the Whitney Museum of American Art, with an accompanying paper at SIGGRAPH
- significant press around the use of AI and Machine Learning for music
- generative AI music software created used by Sigur Ros for the release of their most recent single (Route One), and a generative remix of the track broadcast for 24 hours on Icelandic National TV and watched by millions of people online
- contribution to the first computer generated West End musical
- high profile experiments on live national radio, as well as experience developing large scale, online collaboration platforms
- machine learning software for composers and musicians, downloaded over 5,000 times and the world's first MOOC on machine learning for creative practice;
- design of various popular live coding systems (ixiQuarks, ixi lang, the Threnoscope).

Publications

10 25 50
 
Description We have developed a mechanism and platform for generating new forms of music and sound using artificial neural networks are trained only on specific mathematical transformations and not musical representations. This is similar to other work in the field but differs due to a specific pre-processing step which we have found greatly speeds up the training process. This pre-processing step requires a decoding process upon inference. However, the reconstructed signal is of higher quality than that which has been currently made public by other researchers in the field using similar approaches. We have as yet not published these findings but an exhibition with material using our method opens in 2019 and will tour for five years.
Exploitation Route An entirely new method of digital audio synthesis.
Sectors Creative Economy,Digital/Communication/Information Technologies (including Software),Education,Leisure Activities, including Sports, Recreation and Tourism,Culture, Heritage, Museums and Collections

 
Description UAL's Creative Computing Institute has announced a landmark AI sound project in collaboration with Massive Attack, to be unveiled as part of the unprecedented Barbican exhibition AI: More Than Human. [16 May-26 Aug 2019]. This features technology developed as part of the AHRC funded project, MIMIC. Marking the 20th anniversary of seminal 1998 album Mezzanine, Massive Attack will encode the album in strands of synthetic DNA in a spraypaint can - a nod towards founding member and visual artist Robert del Naja's roots as the pioneer of the Bristol Graffiti scene. Each spray can contains around one million copies of Mezzanine-encoded ink. The project highlights the need to find alternative storage solutions in a data-driven world, with DNA as a real possibility to store large quantities of data in the future. Massive Attack's Robert Del Naja is also working with Professor Mick Grierson at the Creative Computing Institute, students from UAL and Goldsmith's College and Andrew Melchior of the Third Space Agency to create a unique piece of art that highlights the remarkable possibilities when music and technology collide. It places Mezzanine at the centre of a new AI sound composition - a co-production between Massive Attack and machine. The album will be fed into a neural network and visitors will be able to affect the resulting sound by their presence and movements, with the output returned in high definition. Mick Grierson, Research Lead at the Creative Computing Institute, UAL: AI is taking on the role of remix producer. We're not replacing creative labour, but using technology to offer an extension of what's currently possible, creating new ways of generating forms of sound. Our research is focused on how to make this kind of system professional, user-friendly and accessible. This will be game-changing for artists; but in a year's time, we expect that anybody will be able to use this kind of technology to create their own interesting sounds, simply and in high definition. Neil McConnon, Head of Barbican International Enterprises: Artificial Intelligence is a key marker of the zeitgeist and we are thrilled to be exploring the subject, both as a motive for scientific progress and a stimulus for creativity. We hope that innovation in science will inspire and encourage discourse around this phenomenon and give a fresh perspective on the world in which we live. This exhibition looks at the journey to date and the potential to collaborate as we evolve together. We hope it will be an enlightening and dynamic experience, relevant to anyone invested in the future. Opening in May 2019, AI: More Than Human is an unprecedented survey of creative and scientific developments in Artificial Intelligence, exploring the evolution of the relationship between humans and technology. Part of Life Rewired, the Barbican's 2019 season exploring what it means to be human when technology is changing everything, AI: More Than Human tells the rapidly developing story of AI, from its extraordinary ancient roots in Japanese Shintoism and Ada Lovelace and Charles Babbage's early experiments in computing, to AI's major developmental leaps from the 1940s to the present day to show how an age-old dream of creating intelligence has already become today's reality. With digital media, immersive art installations and a chance for visitors to interact directly with exhibits to experience AI's capabilities first-hand, this festival-style exhibition takes place all over the Centre to examine the subject from multiple, global perspectives and give visitors the tools to decide for themselves how to navigate our evolving world. It will ask the big questions: What does it mean to be human? What is consciousness? Will machines ever outsmart a human? And how can humans and machines work collaboratively? Following its London run the exhibition will open at the Groniger Forum, Netherlands, before an international tour.
First Year Of Impact 2019
Sector Creative Economy,Digital/Communication/Information Technologies (including Software),Leisure Activities, including Sports, Recreation and Tourism,Culture, Heritage, Museums and Collections
Impact Types Cultural,Societal,Economic