Innovating Sema: Community-building of Live Coding Language Design and Performance with Machine Learning

Lead Research Organisation: University of Sussex
Department Name: Sch of Media, Arts and Humanities

Abstract

The Innovating Sema project will secure the long-term existence and use of our innovative live coding platform for machine learning. After an intense development process during the MIMIC grant (AH/R002657/1) we have developed a potential paradigm-shifting system that empowers users of programming languages for artistic expression to design their own languages for interfacing with machine learning. This has not been done before and through our user studies and workshops we are confident that our system works. Our project has the potential to demystify machine learning through hands-on use, thus generating understanding and trust in modern AI. It enables non-expert users to engage with creative AI. Now is the time to secure the longevity and wide use of the Sema system promoting understanding of computing through the two pillars of language design and machine learning.

The MIMIC project is a direct response to significant changes taking place in the domain of computing and the arts. Recent developments in Artificial Intelligence and Machine Learning lead to a revolution in how music and art is being created. However, these are early days concerning the integration of this new technology into software aimed at creatives. Due to the complexities of machine learning, and the lack of usable tools, such approaches have only been usable by experts. In order to study how we might bring these new fruitful technologies of new art making to the people, we created new, user-friendly technologies. They enable laypeople, both composers as well as amateur musicians, to understand and apply these new computational techniques in their own creative work.

The unexpected potential for impact we discovered during the development of Sema is twofold:

1) We were going to create a high level live coding language for machine learning. However, responding to survey feedback, we decided to generalise and scale our approach by creating a system for *other people to design their own* live coding languages, using our language grammar compilation technology. This makes artists equally the creators of their own tools as well as the creators of artistic work with those tools. By designing and installing our system online and building a community that contributes to its development, we are effectively collaborating with users in filling a musical toolbox with tools and instruments.

2) During the development of Sema, we become attuned to how distant machine learning is to the general public. People do not easily understand new AI and this technology is often seen as a negative force in society (from automation anxiety, filter bubbles, bias, and algorithms pushing people to the political extremes). Our project enables people to explore machine learning, and thereby gain an understanding of how machine learning and artificial neural networks are designed and trained, through hands-on experience. Music is an ideal platform to give people an understanding of machine learning as it is a collaborative, fun and risk-free environment for experimentation (unlike self driving cars or stock market algorithms).

By creating an online resource with our software (including language generator, live coding environment, machine learning architectures and models, and user contributed mini-languages) together with solid documentation and tutorials, we will secure the longevity and impact of our Sema project. Through an extensive user-engagement plan, we will create the conditions for an international community of users to grow independently of our own practice and research. This community will be initially formed by our own workshops, mailing lists, media appearances, social media communication, and dissemination through our own networks of colleagues who teach computer music internationally.
 
Description Our key goal, to introduce machine learning to creative coders, and create an environment where they can easily apply machine learning in live coding performances has been reached. The second goal, to encourage people to create their own live coding languages using BNF syntax, has also been achieved, but this is work in progress at the moment.

Implementing machine learning is generally a laborious task. It requires knowledge of the code libraries used and a thorough reading of the documentation. Users find it hard to get started and get a hands-on feeling for how to train a network. Our system makes this simple, intuitive and hands-on.

This award has enabled us to get our Sema technology into the hands of users, to work on the usability of our online system, and to create tutorials and workspaces where users can create user accounts where they can store their code and share with other users. This is required for us to build a lively and thriving user community. The seeds of this community is now emerging and we are pleased with how the work has progressed, although the pandemic has made it harder to run face-to-face workshops with users. In turn we have run online workshops and these have enabled people from across the world to join and learn with us.
Exploitation Route People will create their own languages and pieces using Sema and they can share the unique URL to their work.
Sectors Creative Economy

Education

Electronics

URL http://www.sema.codes
 
Description Artists have started using Sema in their work. The software has influenced work in the field. We have found that users have learned about machine learning for the first time and had a strong encounter with the technology through actual coding. Some of the new findings have been written about in the first book on artistic live coding, published by MIT Press: Live Coding: A User's Manual
Sector Creative Economy,Education,Electronics
Impact Types Cultural

Societal

 
Description Intelligent Instruments: Understanding 21st-Century AI Through Creative Music Technologies
Amount € 199,960 (EUR)
Funding ID 101001848 
Organisation European Research Council (ERC) 
Sector Public
Country Belgium
Start 08/2021 
End 09/2026
 
Description Collaboration with Intelligent Instruments Lab 
Organisation Iceland Academy of the Arts
Country Iceland 
Sector Academic/University 
PI Contribution We have collaborated with researchers at the Intelligent Instruments Lab in developing and running the artist programme.
Collaborator Contribution We have co-created tutorials and vision for the future of the platform.
Impact Tutorials on the website and upcoming artistic content
Start Year 2021
 
Title Sema 
Description Sema lets you compose and perform music in real time using simple live coding languages. It also enables you to customise these languages, create new ones, and infuse your live code with bespoke neural networks, using interactive workflows and small training data sets. Sema is a live coding environment for sound and music similar to SuperCollider, TidalCycles and Gibber. Uniquely, it provides many domain-specific languages and an integrated experience to language design and machine learning in your Web browser. This is achieved through the integration of: 
Type Of Technology Webtool/Application 
Year Produced 2021 
Open Source License? Yes  
Impact We have run various workshops and tutorials during this project. At the moment we have an artist programme where artists are developing work using the platform. 
URL https://sema.codes
 
Title Sema Source Code 
Description Sema is a playground where you can rapidly prototype live coding mini-languages for signal synthesis, machine learning and machine listening. Sema aims to provide an online integrated environment for designing both abstract high-level languages and more powerful low-level languages. Sema implements a set of core design principles: a) Integrated signal engine - In terms language and signal engine integration, there is no conceptual split. Everything is a signal. However, for the sake of modularity, reusability, and a sound architecture, sema's signal engine is implemented by the sema-engine library. b) Single sample signal processing - Per-sample sound processing for supporting techniques that use feedback loops, such as physical modelling, reverberation and IIR filtering. c) Sample rate transduction - It is simpler to do signal processing with one principal sample rate, the audio rate. Different sample rate requirements of dependent objects can be resolved by upsampling and downsampling, using a transducer. The transducer concept enables us to accommodate a variety of processes with varying sample rates (video, spectral rate, sensors, ML model inference) within a single engine. d) Minimal abstractions - There are no high-level abstractions such as buses, synths, nodes, servers, or any language scaffolding in our signal engine. Such abstractions sit within the end-user language design space. 
Type Of Technology Webtool/Application 
Year Produced 2021 
Open Source License? Yes  
Impact Users have been using this and it has impacted on other software development in the fields of live coding and creative coding.