RUDIMENTS: Reflective Understanding of Digital Instruments as Musical Entanglements

Lead Research Organisation: Imperial College London
Department Name: Design Engineering (Dyson School)

Abstract

Many new digital musical instruments (DMIs) are created every year, each of which encodes cultural values both obvious and subtle, reflecting the aesthetic priorities of its designer, its constituent technologies, and those technologies' domains of origin. Recent developments in high-performance, low-cost embedded computing promise new frontiers in machine intelligence to be integrated into instruments. The musical opportunities are vast, but so are the risks: who will set the agenda for deploying this technology, and whose interests and aesthetics will be represented? RUDIMENTS takes a value-sensitive approach to DMI design, using embedded computing (miniaturised computers integrated into physical objects) to support rich sensory experience rather than analytical symbolic intelligence. This interdisciplinary project seeks to reconcile technical and ecological perspectives of DMI research, the former focused on engineering techniques or performer-instrument interaction, the latter on the interplay of environmental factors which shape musical creation. The technical perspective can prioritise analytical concepts over the inexpressibility of human experience, while the ecological perspective offers few practical suggestions for technologists. RUDIMENTS revisits the foundational assumptions of DMI research, stripping away layers of historical technical shortcuts and music theory to return to the fundamentals -- the rudiments -- of musical phenomena. The project proposes that instruments should not be viewed as self-contained technological objects at all, but as entanglements: complex and irreducible webs of relationships between humans and things. Musical entanglement design will provide a far-reaching new technical-artistic basis on which to explore the transformational potential of embedded computing and artificial intelligence, building instruments bottom-up from pre-reflective sensory experience and considering the cultural implications of every design decision.
 
Title Arca performs with the magnetic resonator piano 
Description Arca, a leading electronic musician and multidisciplinary artist, presented a series of concerts featuring the magnetic resonator piano (MRP), an augmented instrument created in my research. Four performances took place at the Armory in New York City in October 2023, and two performances at the Pinault Collection in Paris in March 2024. Collectively the audience for these shows was in the thousands, accompanied by several high-profile media pieces (New York Times, Rolling Stone, etc.) and considerable attention on social media. 
Type Of Art Performance (Music, Dance, Drama, etc) 
Year Produced 2024 
Impact The shows drew considerable attention to the magnetic resonator piano at the research in the Augmented Instruments Laboratory. The shows in Paris, where the magnetic resonator piano was a focal point of the performance, sold out within minutes. 
URL https://www.pinaultcollection.com/en/boursedecommerce/arca-presents-light-comes-name-voice
 
Title Music from the Augmented Instruments Laboratory 2023 
Description The Augmented Instruments Laboratory presented a concert featuring new digital musical instrument research by lab members and featured guests, including Iran Sanadzadeh of Monash University (Australia) and Jeff Snyder of Princeton University. The concert took place at Queen Mary University of London on 17 November 2023. 
Type Of Art Performance (Music, Dance, Drama, etc) 
Year Produced 2023 
Impact The concert was well received and served to deepen collaborations with the two invited international artist-researchers. 
URL http://instrumentslab.org/news/2023/11/13/ail-concert.html
 
Description Cyborg Soloists 
Organisation Royal Holloway, University of London
Country United Kingdom 
Sector Academic/University 
PI Contribution I have provided new musical instruments and related technologies for use in the Cyborg Soloists project, a UKRI Future Leaders Fellowship led by Dr Zubin Kanga at Royal Holloway University of London. I have built and delivered the instruments and supported their use in new pieces.
Collaborator Contribution Composers and performers on the Cyborg Soloists project, including Dr Kanga, have worked with instruments from my lab to create and perform new pieces. From this process I have gathered useful feedback that can improve future instrument research.
Impact Several pieces have been written involving my instruments, including TouchKeys and a new optical keyboard scanner, and these have been performed at different venues and festivals around the UK and Europe.
Start Year 2022
 
Description Royal College of Music - Centre for Performance Science 
Organisation Royal College of Music (RCM)
Country United Kingdom 
Sector Academic/University 
PI Contribution Starting in early 2023, I have forged a collaboration with several researchers and musicians in the Royal College of Music, including members of their Centre for Performance Science, an interdisciplinary research centre which also includes other academics at Imperial. I have made my augmented instrument research, including the magnetic resonator piano, available for use by RCM academics and students, and we have begun exploring future opportunities around studying the interaction between performer and instrument.
Collaborator Contribution RCM has provided space for setting up the magnetic resonator piano. Academics and students have begun exploring the instrument, and we are making plans for future public events.
Impact The collaboration to date is primarily concerned with exploring opportunities and preparing for public events which take place elsewhere. We are currently planning a joint event at the Great Exhibition Road Festival to take place in June 2024, where RCM composers and improvisers will work with new augmented instruments from my research lab.
Start Year 2023
 
Description University of Bologna 
Organisation University of Bologna
Country Italy 
Sector Academic/University 
PI Contribution I am working with Dr Michele Ducceschi at the University of Bologna, Department of Industrial Engineering, on new musical instrument design. My contributions involve hardware sensor and actuator systems and interaction strategies.
Collaborator Contribution The contributions of Dr Ducceschi's lab focus on physical modelling of acoustic musical instruments, i.e. numerically simulating the sound of acoustic systems. They have developed custom versions of their audio plugin software which we are exploring in joint research.
Impact Papers from the collaboration are currently under review and in preparation.
Start Year 2023
 
Title pybela 
Description pybela, developed by QMUL PhD researcher Teresa Pelinski with support from the Augmented Instruments Laboratory and an industrial internship at Bela (http://bela.io), is a toolchain and workflow for training AI models to deploy on the Bela embedded hardware platform. pybela consists of tools for capturing data on a Bela embedded platform (a 1GHz ARM-Linux computer running a real-time operating system), transferring the data to python running on a host computer for training models in pytorch, and deploying lightweight models back on the Bela board. 
Type Of Technology Software 
Year Produced 2023 
Open Source License? Yes  
Impact pybela was used in a workshop at the Timbre Tools Hackathon at QMUL in February 2024. Future research will explore other applications of pybela for instrument design and similar applications. 
URL https://github.com/belaplatform/pybela
 
Description Augmented Instruments Lab at Science Museum Lates 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact On 19 October 2023, the Augmented Instruments Laboratory demonstrated several new musical instruments at the Science Museum "Music Lates" event. That evening, the museum was open to the public with exhibitors from around the UK showing different projects related to music technology.
Year(s) Of Engagement Activity 2023
 
Description Augmented Instruments Laboratory - Open Talks 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact The Augmented Instruments Laboratory has organised a series of public research seminars which are presented as hybrid in-person/live-streamed events and recorded for later watching. Thus far the lab has held four such events:
- 6 October 2023 (Eliot Bates)
- 3 November 2023 (Anna-Kaisa Kalia)
- 17 November 2023 (Iran Sanadzadeh)
- 9 February 2024 (Alex Harker)
Year(s) Of Engagement Activity 2023,2024
URL https://www.youtube.com/@AugmentedInstrumentsLab/streams
 
Description Bela workshop at Timbre Tools hackathon 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Postgraduate students
Results and Impact On 16 February 2024, I presented a workshop on using the Bela embedded platform (bela.io, a spinout of my lab) to create musical instruments. The workshop demonstrated a novel AI workflow created by PhD student Teresa Pelinski in which sensor data could be captured by the Bela board, processed on a computer to train a model, and then the model deployed again on Bela. The workshop served as an introductory session for the Timbre Tools hackathon held at QMUL the following week.
Year(s) Of Engagement Activity 2024
 
Description Guthman Musical Instrument Competition 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact I was invited to be one of three judges in the annual Guthman Musical Instrument Competition held at Georgia Tech (Atlanta, USA) on 8-9 March 2024. The Guthman Competition has been running for over a decade. It brings in instrument designers from around the world, with 10 finalists chosen this year from over 60 entries. The judges choose the winning instruments and present panel discussions and media interviews. Overall the event reached hundreds of people in person and hundreds more on a live stream.
Year(s) Of Engagement Activity 2024
URL https://guthman.gatech.edu
 
Description NIME Summer School - Mexico City 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Public/other audiences
Results and Impact In June 2023, members of the Augmented Instruments Laboratory held a 3-day-long public engagement workshop involving building new digital instruments with Bela, an open-source embedded platform for audio and sensor processing created in the lab. The Spanish-language workshop was held in Mexico City as a summer school preceding the International Conference on New Interfaces for Musical Expression. While the conference targets an international academic audience, the summer school was aimed at members of the local community many of whom were not affiliated with academic institutions. Participants made their own instruments, learning aspects of hardware and programming in the process, and presented their results in a showcase before the conference.
Year(s) Of Engagement Activity 2023
URL https://www.nime2023.org/program/summer-school
 
Description Porto Electronic Music Symposium 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Postgraduate students
Results and Impact On 15-16 December 2023, I presented a keynote talk and instrument-building workshop at the Porto Electronic Music Symposium, a conference organised with national arts funding from Portugal. The workshop showed people how to build simple digital instruments with touch sensing and sound synthesis using the Bela platform (http://bela.io), a spinout project of my research lab.
Year(s) Of Engagement Activity 2023
 
Description Synth Design Hackathon 2023 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Postgraduate students
Results and Impact In February 2023, I presented a 4-day instrument-building workshop as part of the international Synth Design Hackathon. Around 15 students from QMUL participated, grouped into 4 teams building different instrument prototypes. QMUL was one of over a dozen institutions which participated, with several dozen instrument projects overall. One of the QMUL teams was chosen as a winner of the hackathon and presented their instrument at Superbooth 2023 in Berlin.
Year(s) Of Engagement Activity 2023
URL https://www.synthux.academy/events/hackathon-2023
 
Description Talk at WiSSAP 2023 - Kanpur, India 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Postgraduate students
Results and Impact On 20 December 2023 I presented an online talk at WiSSAP 2023, the Winter School of Speech and Audio Processing held in Kanpur, India. The workshop acts as a training and networking event for music and audio researchers in India, especially students. The talk was well received and led to follow-up inquiries on my work.
Year(s) Of Engagement Activity 2023
URL https://wissap23.madhavlab.com