MIMIC: Musically Intelligent Machines Interacting Creatively

Lead Research Organisation: Goldsmiths University of London
Department Name: Computing Department

Abstract

This project is a direct response to significant changes taking place in the domain of computing and the arts. Recent developments in Artificial Intelligence and Machine Learning are leading to a revolution in how music and art is being created by researchers (Broad and Grierson, 2016). However, this technology has not yet been integrated into software aimed at creatives. Due to the complexities of machine learning, and the lack of usable tools, such approaches are only usable by experts. In order to address this, we will create new, user-friendly technologies that enable the lay user - composers as well as amateur musicians - to understand and apply these new computational techniques in their own creative work.

The potential for machine learning to support creative activity is increasing at a significant rate, both in terms of creative understanding and potential applications. Emerging work in the field of music and sound generation extends from musical robots to generative apps, and from advanced machine listening to devices that can compose in any given style. By leveraging the internet as a live software ecosystem, the proposed project examines how such technology can best reach artists, and live up to its potential to fundamentally change creative practice in the field. Rather than focussing on the computer as an original creator, we will create platforms where the newest techniques can be used by artists as part of their day-to-day creative practices.

Current research in artificial intelligence, and in particular machine learning, have led to an incredible leap forward in the performance of AI systems in areas such as speech and image recognition (Cortana, Siri etc.). Google and others have demonstrated how these approaches can be used for creative purposes, including the generation of speech and music (DeepMinds's WaveNet and Google's Magenta), images (Deep Dream) and game intelligence (DeepMind's AlphaGo). The investigators in this project have been using Deep Learning, Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Long Short-Term Memory Networks (LSTMs), and other approaches to develop intelligent systems that can be used by artists to create sound and music. We are already among the first in the world to create reusable software that can 'listen' to large amounts of sound recordings, and use these as examples to create entirely new recordings at the level of audio. Our systems produce outcomes that out-perform many other previously funded research outputs in these areas.

In this three-year project, we will develop and disseminate creative systems that can be used by musicians and artists in the creation of entirely new music and sound. We will show how such approaches can affect the future of other forms of media, such as film and the visual arts. We will do so by developing a creative platform, using the most accessible public forum available: the World Wide Web. We will achieve this through development of a high level live coding language for novice users, with simplified metaphors for the understanding of complex techniques including deep learning. We will also release the machine learning libraries we create for more advanced users who want to use machine learning technology as part of their creative tools.

The project will involve end-users throughout, incorporating graduate students, professional artists, and participants in online learning environments. We will disseminate our work early, gaining the essential feedback required to deliver a solid final product and outcome. The efficacy of such techniques has been demonstrated with systems such as Sonic Pi and Ixi Lang, within a research domain already supported by the AHRC through the Live Coding Network (AH/L007266/1), and by EC in the H2020 project, RAPID-MIX. Finally, this research will strongly contribute to dialogues surrounding the future of music and the arts, consolidating the UK's leadership in these fields.

Planned Impact

We will directly engage stakeholders in the process of music making with creative tools, exploring the role that AI will play in the future of the creative industries. We will bring complex AI and machine learning technologies to the general user of creative software; we will democratise technologies that are still emerging in academia and corporate R&D labs.

These groups will benefit from new software, course materials, events, artistic outputs and industry collaborations:

a) creative practitioners and their audiences; specifically musicians, composers and their audiences;
b) the hacker/maker community;
c) industry professionals; including through existing industry partnerships with record labels (XL, Universal), music technology companies (Akai, Roli, Ableton, Reactable, Cycling74, Abbey Road Red) and our project partner, Google Magenta;
e) learners; including those from secondary and higher education, home learners, and academics and professionals
f) the general public.

A key aim of our project is to create a simplified live coding language for coding in the browser where novices can learn about AI and machine learning through a clear and simple, yet powerful, live coding programming language written on top of JavaScript , which is well supported and popular. This simplified live coding language will be designed specifically for musicians and artists and will allow them to pursue new routes for creating music.

We will facilitate a range of high-quality use cases with creative professionals. This will bridge gaps between research and industry, accelerating the impact of artificial intelligence by deploying it in real-world and professional music-making and listening contexts.

Our events series will bring musicians, composers and audiences together, providing an important platform for the continued dissemination our work and the work of those practitioners whom we support through the creation of new tools.

In addition to concerts, we will run symposia to expand and further develop critical thought in these fields, inviting participation from a range of stakeholders. We will also disseminate and support artistic output through the creation of our platform, making it simple not just to create work, but also to share it amongst friends and colleagues, from both outside and inside our connected communities.

Our background technologies will be open source and available to academics and SMEs alike, allowing them to use contemporary AI in ways that are currently very challenging for novices.

We will generate significant, engaging and unique course materials, associated with our existing MOOC provision, and targeted at a range of different learners, from secondary education, through HE, to home learners, academics and professionals. This will help people to acquire skills in machine learning at any stage.

Our track record indicates we are capable of meeting the significant interest from the general public around these issues. Recent public engagement activities from team members have included:

- applying Deep Learning to the creation of artworks we are currently exhibiting at the Whitney Museum of American Art, with an accompanying paper at SIGGRAPH
- significant press around the use of AI and Machine Learning for music
- generative AI music software created used by Sigur Ros for the release of their most recent single (Route One), and a generative remix of the track broadcast for 24 hours on Icelandic National TV and watched by millions of people online
- contribution to the first computer generated West End musical
- high profile experiments on live national radio, as well as experience developing large scale, online collaboration platforms
- machine learning software for composers and musicians, downloaded over 5,000 times and the world's first MOOC on machine learning for creative practice;
- design of various popular live coding systems (ixiQuarks, ixi lang, the Threnoscope).
 
Description The MIMIC platform launched as part of a large scale MOOC initiative in collaboration with FutureLearn, as planned. The MIMIC project platform was used to provide machine learning, music, arts and accessibility training to over 60,000 FutureLearn learners between May and November 2020. This impact led to the MOOC programmes being included in UK Gov's skills toolkit, and by extension, the MIMIC platform itself is now a part of this national training scheme introducing machine learning to a diverse range of people through music and the arts.

We have developed a mechanism and platform for generating new forms of music and sound using artificial neural networks are trained only on specific mathematical transformations and not musical representations. This is similar to other work in the field but differs due to a specific pre-processing step which we have found greatly speeds up the training process. This pre-processing step requires a decoding process upon inference. However, the reconstructed signal is of higher quality than that which has been currently made public by other researchers in the field using similar approaches. We have now published these findings in the following publication: Grierson M, Yee-King M, McCallum L, Kiefer C, Zbyszynski M. Contemporary Machine Learning for Audio and Music Generation on the Web: Current Challenges and Potential Solutions (2019). Further, the work produced as part of this process has led to a 5 year touring exhibition in collaboration with the band Massive Attack, who used our software and expertise to produce a new remix of their well-known album 'Mezzanine', based entirely on the technology developed as part of the MIMIC project. The 5 year touring exhibition was produced by the Barbican UK and Tokyo Science Museum.

Further to this, we have had a number of new papers accepted for publication. These include "Examining student coding behaviours in creative computing lessons using abstract syntax trees and vocabulary analysis", which has been accepted for presentation at the ACM Annual Conference on Innovation and Technology in Computer Science Education. This paper analyses coding behaviours of 1000's of students learning to program, as captured by our MIMIC website prototype. The data reveals how certain types of demonstration programs can better help students learn to program, in particular demonstrating methods for capturing student learning outcomes via data analysis, which is a fundamental approach that the MIMIC project takes.

MIMIC project also received a best paper award at the Web Audio Conference 2019 for work contributing to the development of live audio on the web. The paper, "Bernardo, Francisco, Kiefer, Chris and Magnusson, Thor An AudioWorklet-based signal engine for a live coding language ecosystem. Web Audio Conference, Norwegian University of Science and Technology (NTNU), Trondheim, Norway, 4-6 December 2019. Published in: Proceedings of Web Audio Conference (WAC-2019).", was contributed by the Sussex MIMIC team and has led to the Mozilla Firefox browser accelerating their inclusion of audio workout functionality in order to support our codebases. Importantly the above achievements relate strongly to our prior outputs, including our maximilian library (originally authored by PI Grierson as part on an earlier AHRC funded project), which has now been updated to version 2 as a result of MIMIC. The WAC paper is principally describing the updates to the maximilian library itself, which now has over 1000 stars on GitHub.

The software platforms and infrastructure we have created have been instrumental in the award of new grants and partnerships. Researchers from MIMIC are heavily involved in a new AHRC TANC project (3million), and have received small awards from Google Creative Labs (20,000), and the Alan Turing Institute (25000). Further, we have been engaged by SMEs to undertake contract research in relation to AI and ML deployment in industry (25k).

In addition, the background research developed whilst generating the main MIMIC outcomes has led to new journal articles and conference papers in areas of AI and ML for media analysis and generation, health and wellbeing.
Exploitation Route Due to the massive success of our MOOCs using MIMIC, we have been asked to create a series of specialised programmes, including a new MSc Data Science and the Creative Industries programme, which has already recruited its first cohort. Furthermore, the programme of work has led to a collaboration with WPP, the largest ad agency in the world.
Sectors Creative Economy

Digital/Communication/Information Technologies (including Software)

Education

Leisure Activities

including Sports

Recreation and Tourism

Culture

Heritage

Museums and Collections

URL https://www.mimicproject.com
 
Description UAL's Creative Computing Institute has announced a landmark AI sound project in collaboration with Massive Attack, to be unveiled as part of the unprecedented Barbican exhibition AI: More Than Human. [16 May-26 Aug 2019]. This features technology developed as part of the AHRC funded project, MIMIC. Marking the 20th anniversary of seminal 1998 album Mezzanine, Massive Attack will encode the album in strands of synthetic DNA in a spraypaint can - a nod towards founding member and visual artist Robert del Naja's roots as the pioneer of the Bristol Graffiti scene. Each spray can contains around one million copies of Mezzanine-encoded ink. The project highlights the need to find alternative storage solutions in a data-driven world, with DNA as a real possibility to store large quantities of data in the future. Massive Attack's Robert Del Naja is also working with Professor Mick Grierson at the Creative Computing Institute, students from UAL and Goldsmith's College and Andrew Melchior of the Third Space Agency to create a unique piece of art that highlights the remarkable possibilities when music and technology collide. It places Mezzanine at the centre of a new AI sound composition - a co-production between Massive Attack and machine. The album will be fed into a neural network and visitors will be able to affect the resulting sound by their presence and movements, with the output returned in high definition. Mick Grierson, Research Lead at the Creative Computing Institute, UAL: AI is taking on the role of remix producer. We're not replacing creative labour, but using technology to offer an extension of what's currently possible, creating new ways of generating forms of sound. Our research is focused on how to make this kind of system professional, user-friendly and accessible. This will be game-changing for artists; but in a year's time, we expect that anybody will be able to use this kind of technology to create their own interesting sounds, simply and in high definition. Neil McConnon, Head of Barbican International Enterprises: Artificial Intelligence is a key marker of the zeitgeist and we are thrilled to be exploring the subject, both as a motive for scientific progress and a stimulus for creativity. We hope that innovation in science will inspire and encourage discourse around this phenomenon and give a fresh perspective on the world in which we live. This exhibition looks at the journey to date and the potential to collaborate as we evolve together. We hope it will be an enlightening and dynamic experience, relevant to anyone invested in the future. Opening in May 2019, AI: More Than Human is an unprecedented survey of creative and scientific developments in Artificial Intelligence, exploring the evolution of the relationship between humans and technology. Part of Life Rewired, the Barbican's 2019 season exploring what it means to be human when technology is changing everything, AI: More Than Human tells the rapidly developing story of AI, from its extraordinary ancient roots in Japanese Shintoism and Ada Lovelace and Charles Babbage's early experiments in computing, to AI's major developmental leaps from the 1940s to the present day to show how an age-old dream of creating intelligence has already become today's reality. With digital media, immersive art installations and a chance for visitors to interact directly with exhibits to experience AI's capabilities first-hand, this festival-style exhibition takes place all over the Centre to examine the subject from multiple, global perspectives and give visitors the tools to decide for themselves how to navigate our evolving world. It will ask the big questions: What does it mean to be human? What is consciousness? Will machines ever outsmart a human? And how can humans and machines work collaboratively? Following its London run the exhibition will open at the Groniger Forum, Netherlands, before an international tour.
First Year Of Impact 2019
Sector Creative Economy,Digital/Communication/Information Technologies (including Software),Leisure Activities, including Sports, Recreation and Tourism,Culture, Heritage, Museums and Collections
Impact Types Cultural

Societal

Economic

 
Description Transforming our world with AI
Geographic Reach National 
Policy Influence Type Citation in other policy documents
URL https://www.ukri.org/about-us/what-we-do/ai-review-transforming-our-world-with-ai/
 
Description Transforming Collections: Reimagining Art, Nation and Heritage
Amount £2,947,162 (GBP)
Funding ID AH/W003341/1 
Organisation Arts & Humanities Research Council (AHRC) 
Sector Public
Country United Kingdom
Start 11/2021 
End 11/2024
 
Title MIMIC project website 
Description MIMIC is a web platform for the artistic exploration of musical machine learning and machine listening. We have designed this collaborative platform as an interactive online coding environment, engineered to bring new technologies in AI and signal processing to artists, composers, musicians and performers all over the world. The MIMIC platform has a built-in audio engine, machine learning and machine listening tools that makes it easy for creative coders to get started using these techniques in their own artistic projects. The platform also includes various examples of how to integrate external machine learning systems for sound, music and art making. These examples can be forked and further developed by the users of the platform. Over the next three years, we aim to integrate brand new and developing creative systems into this platform so that they can be more easily used by musicians and artists in the creation of entirely new music, sound, and media, enabling people to understand and apply new computational techniques such as Machine Learning in their own creative work. MIMIC or "Musically Intelligent Machines Interacting Creatively" is a three year AHRC-funded project, run by teams at Goldsmiths College, Durham University and the University of Sussex. 
Type Of Material Improvements to research infrastructure 
Year Produced 2019 
Provided To Others? Yes  
Impact This website allows anyone to understand and create Machine Learning and AI for use in Music and the Arts through collaborative coding. It also allows people to use other techniques for data-oriented research and knowledge exchange across a range of health, wellbeing and general technology-related sectors. The website was used to support Massive Open Online Courses as part of the UK government's digital skills toolkit, which attracted over 60,000 learners during the first few months of the pandemic. It continues to be widely used in universities and beyond, and has led to new research and KE partnerships. 
URL http://mimicproject.com
 
Description Massive Attack Collaboration for 20th anniversary of their album, Mezzanine 
Organisation University of the Arts London
Country United Kingdom 
Sector Academic/University 
PI Contribution We developed a means to remix their album, Mezzanine, using a unique artificial intelligence algorithm which we developed.
Collaborator Contribution They provided access to an archive of musical STEMS for the purposes of remixing their album.
Impact The remix has been turned into an installation as part of the Barbican's AI: More than Human exhibition, which will tour for 5 years. The work Artificial Intelligence, music technology, Exhibition Design, Culture and Heritage, Digital Signal Processing, Human Computer Interaction.
Start Year 2018
 
Description Nesta 
Organisation Nesta
Country United Kingdom 
Sector Charity/Non Profit 
PI Contribution Professor Grierson worked with Nesta, providing empirical evidence, argument and economic proposal for a new centre for AI in the creative industries. Nesta compiled a report demonstrating the need for such a centre, in part based on this evidence, drawing specifically on work completed as part of the MIMIC project and the research leading up to it.
Collaborator Contribution Nesta co-authored a letter to Chancellor Rishi Sunak promoting the idea of a national centre for AI in the Creative Industries, and received a personal reply saying that the proposal was under consideration.
Impact https://www.pec.ac.uk/assets/publications/PEC-and-Nesta-research-report-The-art-in-the-artificial.pdf https://www.nesta.org.uk/blog/art-artificial/
Start Year 2020
 
Description WPP Plc 
Organisation WPP
Country United Kingdom 
Sector Private 
PI Contribution WPP plc is a British multinational communications, advertising, public relations, technology, and commerce holding company headquartered in London, England. It is considered the world's largest advertising company, as of 2019. They used the outcomes of our project, including work by students working on the MIMIC project to publicise their future approach to creative tool development. Further to this, we are working with them to create new research partnerships in Creative AI.
Collaborator Contribution WPP worked with us to design new training and competency development including a new MSc in Data Science and AI for the Creative Industries
Impact WPP worked with us to design new training and competency development including a new MSc in Data Science and AI for the Creative Industries
Start Year 2019
 
Title Maximilian 2 
Description What's Maximilian? Maximilian is a cross-platform and multi-target audio synthesis and signal processing library. It was written in C++ and provides bindings to Javascript. It's compatible with native implementations for MacOS, Windows, Linux and iOS systems, as well as client-side browser-based applications. The main features are: sample playback, recording and looping support for WAV and OGG files. a selection of oscillators and filters enveloping multichannel mixing for 1, 2, 4 and 8 channel setups controller mapping functions effects including delay, distortion, chorus, flanging granular synthesis, including time and pitch stretching atom synthesis real-time music information retrieval functions: spectrum analysis, spectral features, octave analysis, Bark scale analysis, and MFCCs example projects for Windows and MacOS, susing command line and OpenFrameworks environments example projects for Firefox and Chromium-based browsers using the Web Audio API ScriptProcessorNode (deprecated!) example projects for Chromium-based browsers using the Web Audio API AudioWorklet (e.g. Chrome, Brave, Edge, Opera, Vivaldi) 
Type Of Technology Software 
Year Produced 2019 
Open Source License? Yes  
Impact This is the second revision of the software package, Maximilian, originally released in 2009. This new version works seamlessly on the web, including with threaded signals (Web Audio worklets), and was the subject of a paper that won best paper award at the 2019 Web Audio Conference. 
URL https://github.com/micknoise/Maximilian
 
Title learner.js, interactive machine learning library for the web 
Description A C++ rapid prototyping environment for machine learning, transpiled in javascript and united with a graphical interface that incorporates data collection, model selection, training and inference. 
Type Of Technology Software 
Year Produced 2019 
Open Source License? Yes  
Impact Made implementation of machine learning systems far easier for those with less training, and was deployed to thousands of learners via our massive open online courses based on the MIMIC platform. 
URL http://louismccallum.com/portfolio/learner-js
 
Title mimicproject.com 
Description MIMIC is a web platform for the artistic exploration of musical machine learning and machine listening. We have designed this collaborative platform as an interactive online coding environment, engineered to bring new technologies in AI and signal processing to artists, composers, musicians and performers all over the world. The MIMIC platform has a built-in audio engine, machine learning and machine listening tools that makes it easy for creative coders to get started using these techniques in their own artistic projects. The platform also includes various examples of how to integrate external machine learning systems for sound, music and art making. These examples can be forked and further developed by the users of the platform. 
Type Of Technology Webtool/Application 
Year Produced 2019 
Open Source License? Yes  
Impact Supported 60,000 subscribed FutureLearn users between May and November 2020. Actively used to deliver new MSc programmes on AI and Data Science. 
URL http://mimicproject.com
 
Description FutureLearn Massive Open Online Courses 
Form Of Engagement Activity Engagement focused website, blog or social media channel
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Industry/Business
Results and Impact In partnership with FutureLearn, we created a series of MOOCs which formed part of the UK government's digital skills toolkit. These courses attracted over 60,000 learners in the first 6 months of the pandemic, and were delivered using the MIMIC platform. They have led to a new series of MOOCs on Machine Learning which will be live at the end of 2022
Year(s) Of Engagement Activity 2019,2020,2021,2022
URL https://www.futurelearn.com/partners/ual-creative-computing-institute?all_courses=1
 
Description Network Music Festival 2020 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact The event was publicised as follows : "Exploring innovative digital music, art and research which investigates the impact of networking technology on musical creation and performance practice, Network Music Festival presents cutting edge musical performances, workshops and discussions.
The Network Music Festival took place 2012-2014 at the heart of Birmingham's (UK) creative community. Responding to the move to online music making during the 2020 global pandemic and with increasing concern in artistic communities about the climate emergency, the festival is returning for a global fourth edition, which will take place entirely online 15-18th July 2020.
The main theme for 2020 is communities near and far. As many people find themselves socially distant, the importance of community is not diminished, but rather transformed. People rely on sound to bridge physical gaps, from singing or applauding out their windows, to connecting with others online. Our connections, then, are either intensely local, or virtually borderless. We stay in touch with friends and neighbours, but also find that collaborating across the world is not harder than collaborating across town. In this year's festival, we want to celebrate and strengthen the musical communities people have built with networking tools, as well as exploring the aesthetics, performance practice and technologies around topics such as web-streaming, multi-location performance, collaborative music making environments, accessible and sustainable performance practice and more.
We're happy to be teaming up with the Arts and Humanities Research Council (UK) funded MIMIC project, who are developing a web platform for the artistic exploration of musical machine intelligence, for a special concert of new works developed with MIMIC tools."
Year(s) Of Engagement Activity 2020
URL https://networkmusicfestival.org