Hackable Instruments: Musical Interface Design for Appropriation, Modification and Creative Destruction
Lead Research Organisation:
Queen Mary University of London
Department Name: Sch of Electronic Eng & Computer Science
Abstract
Digital musical instrument design presents several unique challenges in comparison to the design of most human-computer interfaces. For musical instruments, as for any tool for creative expression, the standards of success in a design are difficult to quantify, since standard HCI (human-computer interaction) metrics of accuracy and speed do not necessarily correlate with what musicians find expressive. More problematic still is the fluid and unpredictable relationship between designer and user. From the saxophone to the electric guitar to the turntable, musicians have been appropriating and even subverting instrument designs to produce music in ways the designer never anticipated. Some of the past century's most transformative changes in musical performance practice have arguably come from performers radically reinterpreting an instrument's capabilities and limitations. Digital musical instrument researchers therefore face a quandary: how can an interface be designed which embraces the user's wilful subversion of its original goals?
This project will produce a design methodology and a hardware-software platform for the creation of 'hackable instruments': instruments which embrace the performer's reinterpretation and modification. This kind of end-user (mis)appropriation is currently more prevalent among acoustic and analogue electronic instruments than digital musical instruments. The typical computer music interface is designed such that arbitrary modification by non-experts is more likely to produce catastrophic failure than interesting results. Moreover, in comparison to acoustic instruments, whose operation is easy to understand, a lack of transparency in how a digital instrument works can discourage the performer from fully exploring its capabilities.
User studies with musicians and instrument designers form the foundation of the project. The goal of these studies is to understand the exploration process musicians employ when adapting an instrument to their needs and to establish what features of an instrument design facilitate creative repurposing. In parallel, fundamental research on audio processing on embedded devices will lead to a hardware instrument platform that is robust to modification by the end user. Using this platform, several instrument designs will be created to test specific hypotheses about the factors promoting 'hackability' in an instrument. These designs will be evaluated in extended user trials which culminate in a public concert performance.
Many of the transformative changes in musical performance practice originated with the creativity of a few individuals, but their impact depended on dissemination throughout a larger community. Community-building is critically important in digital musical instrument research, and the results of this project will be publicly distributed as an open platform for instrument creation. Several initial instrument designs will be created to encourage exploration and modification, and all hardware and software designs will be released under open-source licenses. Thorough documentation will be provided to enable researchers and self-trained hackers alike to make use of the platform well beyond the duration of the project.
This project will produce a design methodology and a hardware-software platform for the creation of 'hackable instruments': instruments which embrace the performer's reinterpretation and modification. This kind of end-user (mis)appropriation is currently more prevalent among acoustic and analogue electronic instruments than digital musical instruments. The typical computer music interface is designed such that arbitrary modification by non-experts is more likely to produce catastrophic failure than interesting results. Moreover, in comparison to acoustic instruments, whose operation is easy to understand, a lack of transparency in how a digital instrument works can discourage the performer from fully exploring its capabilities.
User studies with musicians and instrument designers form the foundation of the project. The goal of these studies is to understand the exploration process musicians employ when adapting an instrument to their needs and to establish what features of an instrument design facilitate creative repurposing. In parallel, fundamental research on audio processing on embedded devices will lead to a hardware instrument platform that is robust to modification by the end user. Using this platform, several instrument designs will be created to test specific hypotheses about the factors promoting 'hackability' in an instrument. These designs will be evaluated in extended user trials which culminate in a public concert performance.
Many of the transformative changes in musical performance practice originated with the creativity of a few individuals, but their impact depended on dissemination throughout a larger community. Community-building is critically important in digital musical instrument research, and the results of this project will be publicly distributed as an open platform for instrument creation. Several initial instrument designs will be created to encourage exploration and modification, and all hardware and software designs will be released under open-source licenses. Thorough documentation will be provided to enable researchers and self-trained hackers alike to make use of the platform well beyond the duration of the project.
Planned Impact
By exploring the creative contributions of musical performers to the design process and forging links among communities, this project holds benefits to several groups beyond academic researchers, both in the UK and internationally. These include:
Commercial Private Sector:
* Professional performers and composers, who will benefit from a new set of expressive digital tools. The platform resulting from this project will also give musicians greater ownership over the operation of their instruments, encouraging new forms of creative expression.
* Concert promoters and venues, who can use new instrument technology to appeal to new audiences.
* Musical instrument and audio equipment manufacturers, who can produce new designs tailored carefully to individual markets.
* Electronics manufacturers and distributors, who can develop and sell tools for musical interface creation. Open-source hardware makes the designs available freely to all, but it is also an important economic engine, as evidenced by successful companies like Arduino in Italy and SparkFun and AdaFruit in the United States. This project will help establish the UK as a major driver of the open hardware movement.
Public and Third Sectors:
* Educators, particularly at the secondary level and above, who can use musical instrument design to enhance instruction in scientific and mathematical fields.
* Independent organisations promoting STEM (science, technology, engineering and maths) to young people and underrepresented groups, by using music to generate interest in STEM careers.
* Museums, through new interactive exhibits that encourage audience creativity.
Wider Public:
* Amateur musicians, who can make use of the project tools to motivate learning and create personal means of expression.
* Students in both technical and musical fields, who can create links between scientific and musical domains to better understand both disciplines.
* Electronics and programming hobbyists, who in the past decade have formed a growing online 'maker' community devoted to creative open-source hardware design. The project will allow these individuals to apply their skills to the development and dissemination of new musical instruments. In turn, their activities building on project results will have the potential to benefit future digital musical instrument research.
Commercial Private Sector:
* Professional performers and composers, who will benefit from a new set of expressive digital tools. The platform resulting from this project will also give musicians greater ownership over the operation of their instruments, encouraging new forms of creative expression.
* Concert promoters and venues, who can use new instrument technology to appeal to new audiences.
* Musical instrument and audio equipment manufacturers, who can produce new designs tailored carefully to individual markets.
* Electronics manufacturers and distributors, who can develop and sell tools for musical interface creation. Open-source hardware makes the designs available freely to all, but it is also an important economic engine, as evidenced by successful companies like Arduino in Italy and SparkFun and AdaFruit in the United States. This project will help establish the UK as a major driver of the open hardware movement.
Public and Third Sectors:
* Educators, particularly at the secondary level and above, who can use musical instrument design to enhance instruction in scientific and mathematical fields.
* Independent organisations promoting STEM (science, technology, engineering and maths) to young people and underrepresented groups, by using music to generate interest in STEM careers.
* Museums, through new interactive exhibits that encourage audience creativity.
Wider Public:
* Amateur musicians, who can make use of the project tools to motivate learning and create personal means of expression.
* Students in both technical and musical fields, who can create links between scientific and musical domains to better understand both disciplines.
* Electronics and programming hobbyists, who in the past decade have formed a growing online 'maker' community devoted to creative open-source hardware design. The project will allow these individuals to apply their skills to the development and dissemination of new musical instruments. In turn, their activities building on project results will have the potential to benefit future digital musical instrument research.
People |
ORCID iD |
| Andrew McPherson (Principal Investigator) |
Publications
A. McPherson
(2016)
Action-Sound Latency: Are Our Tools Fast Enough?
McPherson A
(2016)
Designing for Exploratory Play with a Hackable Digital Musical Instrument
Morreale F
(2017)
Building a Maker Community Around an Open Hardware Platform
Pardue L
(2015)
A Low-Cost Real-Time Tracking System for Violin
in Journal of New Music Research
Topliss, J.
(2014)
Latency performance for real-time audio on BeagleBone Black
in Proc. Linux Audio Conference
Zappi V
(2018)
Hackable Instruments: Supporting Appropriation and Modification in Digital Musical Interaction
in Frontiers in ICT
Zappi, V.
(2014)
Dimensionality and Appropriation in Digital Musical Instrument Design
in Proc. New Interfaces for Musical Expression
| Title | C4DM Augmented Instruments concert |
| Description | On 4 November 2015, we held a free public concert at Queen Mary University of London featuring two digital instruments developed at QMUL (Laurel Pardue's augmented violin and the D-Box developed by Andrew McPherson and Victor Zappi). The programme also included performances by two other designer-performers: Dianne Verdonk (Utrecht, Netherlands) playing her instrument La Diantenne, and Tim Exile (London, UK) playing his Flow Machine. These performances were coupled with a research study of the audience experience in live electronic music performance. The audience contained 65 members not including the organisers (5-6 further people). This filled the venue to capacity. The audience took part in written surveys and live smartphone-based data feedback during the performance which was used as the basis for the research study. A paper has been submitted to the New Interfaces for Musical Expression 2016 conference (Brisbane, Australia) with a longer journal paper in preparation as of March 2016. |
| Type Of Art | Performance (Music, Dance, Drama, etc) |
| Year Produced | 2015 |
| Impact | The concert was well received by the audience and brought in some audience members who had not previously been to QMUL before. The data collected from the surveys has yielded several interesting insights on audience experience. For example, we discovered that the audience's level of technical familiarity with the instruments has no impact on their enjoyment of the performance, while musical style of the performance has a strong impact. This contributes important knowledge to the field of digital musical instrument design. |
| Title | Cube Instrument concert |
| Description | 10 musicians played a public concert using a deliberately very constrained instrument, exploring how simple tools are used in creative ways. |
| Type Of Art | Performance (Music, Dance, Drama, etc) |
| Year Produced | 2014 |
| Impact | Advanced the research by understanding reaction of performers and audience to highly constrained instruments. Also generated interest in the instrument amongst students and other researchers, who may adopt parts of the technical platform. |
| Title | D-Box concerts |
| Description | Public evening concert at QMUL with a 'hackable' digital musical instrument featuring 10 performers of varying backgrounds. Showed a range of creative techniques including circuit bending. Repeated in part one month later at a regular concert series at a pub near Goldsmiths College, London. |
| Type Of Art | Performance (Music, Dance, Drama, etc) |
| Year Produced | 2014 |
| Impact | Musicians will use the instruments in further performances of their own (including internationally, e.g. Japan tour by one musician). Interest by audiences in learning more about instrument design and technical platform. |
| Title | D-Box performance at Being Human Festival |
| Description | On 12 November 2015, a student ensemble from Kingston University performed a group improvisation on the D-Box "hackable instrument" developed on the EPSRC First Grant of the same name. This performance took place at Senate House, University of London, as part of the opening celebration for the Being Human festival. |
| Type Of Art | Performance (Music, Dance, Drama, etc) |
| Year Produced | 2015 |
| Impact | Raised the profile of the D-Box work amongst an audience consisting to a large degree of humanities scholars, a different interdisciplinary connection than I usually find in my digital music research. |
| URL | http://beinghumanfestival.org/event/being-human-late/ |
| Title | D-Box performance at Stereo |
| Description | In February 2017 at the University of Glasgow, we delivered an invited workshop using Bela, the open-source low-latency audio platform created in our research group. Following the workshop, we were invited to perform with the D-Box, a hackable digital musical instrument created as part of research into appropriation of music technology on the EPSRC Hackable Instruments project (EP/K032046/1). The concert was held at Stereo, a popular club in Glasgow, and also featured a performance using Bela by Sebastian Lexer of the University of Glasgow. |
| Type Of Art | Performance (Music, Dance, Drama, etc) |
| Year Produced | 2017 |
| Impact | The workshop and concert led to increased interest in Bela and the D-Box, and the trip also led to discussions about collaboration with the University of Glasgow on future funding proposals and joint research projects. |
| Title | Guthman Musical Instrument Competition |
| Description | The D-Box hackable musical instrument was named a finalist in the 2015 Guthman Musical Instrument Competition, an annual competition for new musical instruments held at Georgia Tech (USA) which attracts dozens of entries from around the world each year. Semifinalists are chosen via video submissions, and finalists are chosen after a performance and demonstration. As part of this competition we gave two performances on the D-Box which were live-streamed and covered by local news media. |
| Type Of Art | Performance (Music, Dance, Drama, etc) |
| Year Produced | 2015 |
| Impact | The event generated online media attention for the D-Box and the hackable instruments project. Connections made at the event are likely to result in future collaborations on the Bela embedded audio platform (another outcome of the same research project). |
| URL | http://guthman.gatech.edu |
| Description | Our research explored how musicians use technology, including digital musical instruments, in creative ways that the designer did not expect. We conducted a study which showed the value of constraints in eliciting creativity. 10 performers were given an extremely limited instrument and asked to prepare 2 performances; a surprising variety of unusual playing techniques emerged including many we did not explicitly design for. An important result of the study was to challenge the notion that more complex instruments are needed to produce richer musical outcomes. In our study, half of the performers received an instrument with two major controls (pitch and timbre) while the other half received an instrument with only one control (timbre). Remarkably, performers who received a one-dimensional instrument found it to be richer and more capable than those with the two-dimensional instrument, and they showed a corresponding greater range of interaction techniques. This demonstrates that sometimes strict limitations can be a musical asset. We also investigated the motivations and methods behind the practices of musical instrument hacking and circuit bending, interviewing several practitioners. We applied these findings to the creation of a new instrument, the D-Box, which is open to modification by the performer. It can be completely rewired internally to produce a wide range of new sounds. We demonstrated its usability in a second user study with 10 musicians. Experienced circuit benders were able to apply their techniques to the instrument, and musicians without hacking experience were also able to explore and discover new sounds. Each of the 10 performances was strikingly different in its musical content and interaction techniques, suggesting that we achieved our goal of making an instrument which is open to creative reinterpretation. Following the funded period, we have continued to explore this line of enquiry including through public workshops with the D-Box where we investigate the concept of exploratory play in relation to human-computer interaction. In the process of building these instruments, we developed Bela, a new ultra-low-latency audio platform based on the BeagleBone Black embedded computer. The platform captures real-time audio and sensor data and is suitable for creating high-performance, self-contained digital musical instruments. We released this platform open source, subsequently spinning it out into a company (Augmented Instruments Ltd). We regularly hold events and workshops around the world to introduce the platform, and we feature creative outputs that others have made with it on http://blog.bela.io. |
| Exploitation Route | Our findings on how the capabilities and constraints of a musical instrument influence the creative decisions of performers suggest new approaches for future instrument designers, both academic and industrial. In particular, we observed the value of clear, simple metaphors for interacting with the instrument, and the importance of avoiding dimensions of control which paradoxically make the performer feel more restricted. Instrument designers can also use our techniques for designing open, modifiable instruments which operate on different principles than commonly-used modular synthesis approaches. The low-latency audio platform we developed will be of use to a broad range of engineers, researchers and hobbyists, and it contributes to the growing open-source maker community. In the 18 months since we first showed early previews to people outside QMUL, the platform has gained substantial traction. As of the end of 2015, around 30 people (in academia and industry) had acquired the hardware from us to use in their own projects. On February 29, 2016 we launched a Kickstarter campaign which reached nearly 1100% of its original funding goal, £54,902 in total, from 530 backers. We subsequently formed a spinout company, Augmented Instruments Ltd, to continue sales and development of Bela. Performances with the D-Box instrument created on the Hackable Instruments project have continued in the 2015-17 period. Musicians in the final user study were able to keep their instruments at the end, and several more have acquired D-Boxes from our events and through the Kickstarter campaign. Finally, we ran a series of workshops introducing musicians and nonmusicians to circuit bending and hacking techniques as ways of creatively exploring electronics. The workshops have themselves led to further publications which investigate how people explore unfamiliar musical technology (DIS 2016), and how communities form around open-source tools (CHI 2017). Bela has now been used in over 35 publications by researchers across the world, in fields ranging from musical interaction to e-textiles to neuroscience. An updated list can be found at the following URLs: https://github.com/BelaPlatform/Bela/wiki/Publications-using-Bela and https://bela.io/education |
| Sectors | Creative Economy Digital/Communication/Information Technologies (including Software) Electronics |
| URL | http://instrumentslab.org/research/hackable-instruments.html |
| Description | The outcomes of our research are particularly relevant to the cultural sector, where the instruments we created have been used in several performances. These include three concerts in London and several independent events by the performers involved (in UK, USA, Canada, Japan). The continuing use and exploration of the instruments has been encouraging. Our findings on approaches to musical interface design have influenced an ongoing debate in the digital musical instrument research community about complexity, constraints and appropriation of technology. These topics featured prominently in a recent thesis at Goldsmiths University of London, and a group of students at QMUL repeated our study of constrained instruments. Challenging the notion that more complex interfaces produce richer musical outcomes may be one of the more important lasting results of this research. This idea seems to be gaining influence within the digital musical instrument community. More broadly, an understanding of the ways that musicians use technology in unexpected ways has informed current research directions on the EPSRC Fellowship "Design for Virtuosity" (EP/N005112/1). The most high-profile impact of this research has been the Bela low-latency audio platform which we created to support the musical instruments we designed for our research studies. During the funded period, we conducted workshops and hacking events, and since then the project has grown in both capability and profile. Many new features have been added to make it a high-performance general-purpose tool for creating interactive audio systems. The entire project is open source hardware and software. In April 2016, Bela completed a highly successful Kickstarter crowdfunding campaign, raising over £54k (nearly 1100% of its original goal) from 530 backers worldwide. The campaign grew the global user base of Bela by an order of magnitude and led several months later to the formation of a spinout company, Augmented Instruments Ltd (http://bela.io), in September 2016. As of March 2021, over £500k of Bela products have been sold, with sales growing 40% year-on-year. The company delivers industrial consultancy projects and has led two Innovate UK projects totalling over £100k. A component of one of these Innovate UK projects developed a touch sensor technology which was also originally created as part of the D-Box instrument on this EPSRC project. Those touch sensors, now branded Trill, were launched on Kickstarter in 2019, raising £25k. A new design featuring flexible PCB technology (Trill Flex) was released in 2021. More details on Trill can be found at http://bela.io/trill. Augmented Instruments Ltd supports a growing global community of makers, musicians, researchers and educators. The Bela blog (http://blog.bela.io) features projects made by the Bela community, including musical instruments, art installations, wearable technology and more. We released a free lecture series on YouTube teaching C++ audio programming (youtube.com/belaPlatform) with over 28,000 views as of March 2021. Bela is in use for classroom teaching at over 20 universities on three continents and is used for research at dozens more universities, as well as for internal R&D at dozens of companies. Since 2019, Augmented Instruments Ltd has been an industrial partner on the UKRI Centre for Doctoral Training in Artificial Intelligence and Music, contributing to the training programme and supporting PhD studentships. In 2021, the company became a partner on a UKRI Future Leaders Fellowship and a sponsor of a Royal Academy of Engineering Senior Research Fellowship. |
| First Year Of Impact | 2014 |
| Sector | Creative Economy,Digital/Communication/Information Technologies (including Software),Education,Electronics |
| Impact Types | Cultural Economic |
| Description | Arts and Technology Pilot Programme |
| Amount | £24,987 (GBP) |
| Organisation | Innovate UK |
| Sector | Public |
| Country | United Kingdom |
| Start | 02/2017 |
| End | 10/2017 |
| Description | Bela / Royal Academy of Engineering Senior Research Fellow in Embedded Music Computing |
| Amount | £208,774 (GBP) |
| Funding ID | RCSRF2021\1248 |
| Organisation | Royal Academy of Engineering |
| Sector | Charity/Non Profit |
| Country | United Kingdom |
| Start | 03/2021 |
| End | 02/2026 |
| Description | Centre for Public Engagement - Large Grants |
| Amount | £11,400 (GBP) |
| Organisation | Queen Mary University of London |
| Sector | Academic/University |
| Country | United Kingdom |
| Start | 09/2015 |
| End | 07/2016 |
| Description | Early Career Fellowships - ICT |
| Amount | £897,686 (GBP) |
| Funding ID | EP/N005112/1 |
| Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
| Sector | Public |
| Country | United Kingdom |
| Start | 01/2016 |
| End | 12/2020 |
| Description | InnovateUK 6478: Emerging and enabling round 3 |
| Amount | £100,000 (GBP) |
| Funding ID | 6478 |
| Organisation | Innovate UK |
| Sector | Public |
| Country | United Kingdom |
| Start | 03/2018 |
| End | 03/2019 |
| Description | Queen Mary Innovation Fund |
| Amount | £9,200 (GBP) |
| Organisation | Queen Mary University of London |
| Sector | Academic/University |
| Country | United Kingdom |
| Start | 04/2015 |
| End | 09/2015 |
| Description | Queen Mary Innovation Proof-of-Concept Fund |
| Amount | £50,000 (GBP) |
| Organisation | Queen Mary University of London |
| Department | Queen Mary Innovation |
| Sector | Private |
| Country | United Kingdom |
| Start | 01/2017 |
| End | 12/2017 |
| Description | Aalborg University Copenhagen |
| Organisation | Aalborg University |
| Country | Denmark |
| Sector | Academic/University |
| PI Contribution | This collaboration with Prof. Dan Overholt at Aalborg University Copenhagen explores electronic augmentation of the violin for teaching and performance purposes. This relates closely to the theme of our research on skills transfer and designing for expertise. A postdoc, Laurel Pardue, worked 50% time at QMUL and 50% time at Aalborg funded by an internal grant from that university. |
| Collaborator Contribution | Aalborg University funded an 18-month 50% FTE postdoc position through its "International post-doc" funding scheme. The postdoc, Laurel Pardue, worked the other 50% time at QMUL. She was supervised at Aalborg by Prof. Dan Overholt. The broader context of this project is a collaboration between our research lab and the Nordic Culture Fund project "Acoustically Active Augmented Instruments" (AAAI), which brings together researchers from Norway, Finland, Denmark and Iceland. This project has also funded a visit by our team to the University of Oslo to deliver a workshop. |
| Impact | L. Pardue, A. McPherson and D. Overholt, "Improving the Instrumental Learning Experience through Complexity Management", Proceedings of the International Conference on Sound and Music Computing, 2018. L. Pardue, K. Buys, M. Edinger, D. Overholt and A. McPherson, "Separating sound from source: sonic transformation of the violin through electrodynamic pickups and acoustic actuation", Proc. New Interfaces for Musical Expression, 2019. Under review (journal): L. Pardue, D. Overholt and A. McPherson, "Effects of limiting aural feedback on intonation during violin performance". The project resulted in a new augmented violin, the svampolin, which uses digital processing to transform the sound of the violin in real time, for example changing its pitch or timbre. The instrument has been performed in several concerts and won third place at the Guthman Musical Instrument Competition at Georgia Tech in March 2020 (Atlanta, USA - https://guthman.gatech.edu). The Bela open-source embedded audio platform (a creation of our lab resulting from two EPSRC grants) has been used regularly on the AAAI project, for example in the following paper by other members of the project: V. E. Gonzalez Sanchez, C. P. Martin, A. Zelechowska, K. A. V. Bjerkestrand, V. Johnson, and A. R. Jensenius, "Bela-based augmented acoustic guitars for sonic microinteraction," in Proceedings of the international conference on new interfaces for musical expression, Blacksburg, Virginia, USA, 2018, p. 324-327. This is a multidisciplinary collaboration involving computer science, electronic engineering, musical performance and composition. |
| Start Year | 2017 |
| Description | Drake Music |
| Organisation | Drake Music |
| Country | United Kingdom |
| Sector | Charity/Non Profit |
| PI Contribution | Drake Music is a charity dedicated to provide access to disabled musicians through the use of technology. Our lab has contributed to this mission by providing the Bela platform (described elsewhere on Researchfish) to members of the Drake Music community. We also hosted an Accessible Music Hackathon at QMUL which was sponsored by Drake. |
| Collaborator Contribution | Drake Music has provided valuable feedback on using Bela for instrument design, and contributed to the Accessible Music Hackathon by enlisting the participation of their enthusiastic developer and user community. They also purchased parts to be used in the hackathon and covered costs including prizes and food. |
| Impact | The partnership has resulted in a public event (the Accessible Music Hackathon) which produced 16 project demonstrations by the end of a single day. It has also resulted in ongoing participation by QMUL staff and PhD students in the monthly Drake Music "DMlab" development sessions. |
| Start Year | 2015 |
| Description | Mixed Reality Lab |
| Organisation | University of Nottingham |
| Department | School of Computer Science |
| Country | United Kingdom |
| Sector | Academic/University |
| PI Contribution | I have collaborated with members of the Mixed Reality Laboratory (led by Steve Benford) in Nottingham's computer science department on studies involving the D-Box hackable musical instrument. We have co-organised a series of free public workshops where people hack the D-Box instrument. I lead the workshop activities while the MRL researchers collect interviews and video footage as the event goes along, which we use for later analysis. Subsequently, we worked with MRL on a workshop at the STEIM electro-acoustic music centre in Amsterdam, using the Bela audio platform created in our lab. We led the technical aspects of the workshop, while MRL contributed to ethnographic study of how people used the technology. A paper on the results was accepted to CHI 2017. In 2019 we began preparing an ethnographic study of digital musical instruments in a community of learning-disabled individuals. Our team provided the instrument technology and the background in music and disability studies, while MRL contributed expertise in HCI and ethnography. This led to a late-breaking work paper at CHI 2019 and my participation in a CHI 2019 paper led by the University of Nottingham. In January-March 2020, I hosted PhD student Juan Martinez Avila, who has been working on augmented guitar research, and collaborated on a workshop on somaesthetic design of musical instruments. This led to a paper at the New Interfaces for Musical Expression 2020 conference ("Soma Design for NIME"). |
| Collaborator Contribution | MRL contributed their Artcodes research to the project. Artcodes are a form of information encoding in images based on patterns of open and closed shapes; notably, nearly any physical design can be turned into an artcode through careful management of light and dark regions. Artcodes are useful for linking physical objects to their virtual records in an aesthetically elegant manner. MRL also led the data collection and transcription for the workshops. We jointly prepared papers on the results. In 2019, MRL led work on studying the practice habits of musicians for a paper on which I am a co-author, and has contributed expertise on somaesthetic design and ethnography which has informed our joint work. |
| Impact | A. McPherson, A. Chamberlain, A. Hazzard, S. McGrath and S. Benford. Designing for Exploratory Play with a Hackable Digital Musical Instrument. Proc. DIS, Brisbane, Australia, 2016. F. Morreale, G. Moro, A. Chamberlain, S. Benford and A. McPherson. Building a Maker Community around an Open Hardware Platform. Proc. CHI, Denver, USA, 2017. J. Harrison, A. Chamberlain and A. McPherson. Accessible Instruments in the Wild: Engaging with a Community of Learning-Disabled Musicians. Proc. CHI Late Breaking Work, Glasgow, UK, 2019. A. Hazzard, C. Greenhalgh, M. Kallionpaa, S. Benford, A. Veinberg, Z. Kanga and A. McPherson. Failing with style: designing for aesthetic failure in interactive performance. Proc. CHI 2019. J. Martinez Avila, V. Tsaknaki, P. Karpashevich, C. Windlin, N. Valenti, K. Höök, A. P. McPherson and S. Benford. Soma Design for NIME. Proc. New Interfaces for Musical Expression, 2020. |
| Start Year | 2015 |
| Title | Bela |
| Description | Trademarks granted in UK and the USA for "Bela". Bela is an open-source embedded platform for audio and sensor processing. The trademark has been granted within the UK the USA in related areas including computer equipment and peripherals. |
| IP Reference | |
| Protection | Trade Mark |
| Year Protection Granted | 2017 |
| Licensed | Yes |
| Impact | Licensed to spinout company Augmented Instruments Ltd, whose activities are reported in a separate section. |
| Title | Bela Audio Platform |
| Description | Bela (formerly called BeagleRT) is an ultra-low-latency real-time audio platform for the BeagleBone Black embedded computer. It can be used to build musical instruments, including the D-Box hackable musical instrument developed for the EPSRC Hackable Instruments project. The software is of particular interest to audio and music researchers, especially those building real-time systems. With less than 1ms of latency between action and sound, Bela performs faster than any other computer-based environment on the market, including high-spec laptops. It also features audio-rate sampling of every analog and digital input which makes design of sensor systems convenient. In 2015, a built-in browser-based IDE was added along with support for the Pure Data graphical computer music language widely used in the digital music community. Since 2016, we have continued to add support for other programming languages and hardware accessories, as well as extending the documentation and online resources available to the community. |
| Type Of Technology | Software |
| Year Produced | 2014 |
| Open Source License? | Yes |
| Impact | Bela has gained significant traction amongst researchers, musicians and hobbyists in the 2014-17 period, with the number of users and community members growing every month. On 29 February 2016, we launched a Kickstarter crowdfunding campaign for Bela, seeking £5k to build and distribute the hardware. We exceeded our funding goal in less than 4 hours (out of a 32-day campaign), ultimately raising £55k from over 500 backers. Following the campaign, we received invitations to collaborate from major academic institutions (IRCAM in France, STEIM in the Netherlands, University of Virginia, University of Edinburgh, University of Glasgow, Goldsmiths University and several others) and industry (Cycling74, makers of the popular Max/MSP software). Meanwhile, since 2014 we have held 10+ workshops on Bela and/or the D-Box instrument including 2 workshops at the Sonar music festival, a workshop at the NIME 2015 conference (Baton Rouge, LA, USA), an Audio Music Hackathon sponsored by Harman Audio, and an Accessible Music Hackathon with the charity Drake Music (held at QMUL). Following the successful Kickstarter campaign, we spun out a new company, Augmented Instruments Ltd, to commercialise the platform, and relaunched a public web shop in October 2016. We maintain several community resources, including a blog, wiki, web forum and several social media accounts. Our users actively contribute to these resources and frequently share projects they make using Bela. Community development itself has become an area of research for us (e.g. Morreale et al., "Building a maker community around an open hardware platform", Proc. CHI 2017), and we have received a grant from Innovate UK (Feb-Oct 2017) to further develop the platform and build our community. For more information see the entry for spinout company Augmented Instruments Ltd. |
| URL | http://bela.io |
| Title | Trill touch sensors |
| Description | In 2014, as part of the EPSRC project Hackable Instruments (EP/K032046/1), I created a new capacitive touch sensor as part of a musical instrument called the D-Box. The sensor measured the position of up to 5 fingers along a single linear axis. Later, on the Design for Virtuosity project (EP/N005112/1), I and others in my lab further developed the touch sensor technology, using it in other instruments and producing expanded designs that sensed in two axes. In 2019, as part of an Innovate UK funded project in collaboration with QMUL spinout Augmented Instruments Ltd, this technology was developed to the point of commercial readiness. While capacitive touch position sensing is commonly found in consumer devices, the design of these sensors has often been limited to just a few experts. By contrast, these touch sensors are designed to be easy to use for makers, artists and researchers who are not expert embedded engineers. Accordingly, a set of libraries and example materials for popular maker platforms such as Arduino and Bela were developed. |
| Type Of Technology | New/Improved Technique/Technology |
| Year Produced | 2019 |
| Impact | The technology has been licensed to QMUL spinout Augmented Instruments Ltd, which also commercialises the Bela embedded platform which has come out of my research. The touch sensors have been branded "Trill" and are available in five formats: Bar (a one-axis slider), Square (a two-axis touch pad), Hex (two-axis tileable hexagons), Ring (a circular slider) and Craft (individual pins exposed for designers to make their own interfaces). The company ran a successful Kickstarter campaign in September 2019, raising £25k toward the production of Trill sensors. Shipping commenced in April 2020 and sales have continued at a steady pace since then. In March 2021, the company introduced Trill Flex, based on a flexible PCB design I co-created during the time of the Kickstarter campaign. |
| URL | http://bela.io/trill |
| Company Name | Augmented Instruments |
| Description | Augmented Instruments operates a digital music studio. |
| Year Established | 2016 |
| Impact | Augmented Instruments Ltd launched Bela for public sale in October 2016. Its annual turnover is around £130k. The company is profitable without the need for investment funding. During the first year, it employed one full-time staff member and several part-time positions. Income came from sales and from an extended technical consultancy with a major European musical instrument manufacturer. In the second year, the company expanded with the support of grants from Innovate UK, increased sales, and additional industrial consultancy. The company hired three members of staff with support from Innovate UK. It launched a new product, Bela Mini, in February 2018. It also continues to support a growing open-source community, providing resources including a repository of code and hardware designs, a tech support wiki, an online discussion forum, a blog featuring user-created projects, a YouTube channel and other social media links. We estimate that 2500+ unique users on 6 continents have acquired their own Bela boards since the project began. Bela sales to date exceed £500k. We studied the characteristics of our early adopter community in [3.1]. It contains a diversity of profiles and interests: makers (i.e. technology hobbyists and independent creators); musical performers; musical instrument and synthesiser designers; interactive artists; teachers; students; researchers in fields ranging from human-computer interaction to digital signal processing to neuroscience. As of early 2019, AIL also had around 200 institutional customers, including universities (45%), companies ranging from SMEs to large multinationals (25%), design and art consultancies (17%) and arts/maker organisations (6%). Bela is being used for teaching in 20 universities beyond QMUL including Aalborg University Copenhagen (Denmark), University of Kiel (Germany), University of Virginia (USA), Griffith University (Australia) and NTNU (Norway). Since 2016, Bela has been used in dozens of public workshops across Europe, North America and Australia, including events at maker spaces, museums, trade shows, music festivals, conferences, universities, secondary schools, and many other public venues. Many of these were led by the Bela creators (first at QMUL, then also at Augmented Instrumenst Ltd), and some are led spontaneously by our community. Total participation across all events is hard to precisely quantify, but is easily in excess of 1500 people. Bela maintains a committed and enthusiastic online community who contribute back to the project through porting software tools (e.g. computer music languages SuperCollider, Csound, FAUST, pyo) and through writing up and sharing projects built with Bela. In 2019, the company completed another successful Kickstarter campaign for Trill, a new type of capacitive touch sensor aimed at makers and artists. The campaign raised £25k and sales have continued since then. |
| Website | http://instrumentslab.org |
| Description | Accessible Music Hackathon |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Third sector organisations |
| Results and Impact | On 6 February 2016, my lab and the charity Drake Music jointly organised an Accessible Music Hackathon, a daylong event for creating new music technology for the disabled. The event was held at QMUL; it was free and open to the public, and advertised to Drake Music's community of supporters as well as local interest group mailing lists (e.g. Music Hackspace). At least 35 people attended (filling the venue to capacity) often working in small groups. At least 16 working projects were demonstrated at the end of the day. |
| Year(s) Of Engagement Activity | 2016 |
| Description | Ars Electronica presentation |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | Yes |
| Geographic Reach | International |
| Primary Audience | Public/other audiences |
| Results and Impact | Talk generated questions and discussion. No further impact at this point |
| Year(s) Of Engagement Activity | 2014 |
| Description | Audio Music Hackathon |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Public/other audiences |
| Results and Impact | On 18-19 July 2015, an Audio Music Hackathon was held at QMUL with sponsorship from Harman Audio. The general organisation of the event was led by Dr Josh Reiss at QMUL. I contributed through a series of tutorials and support sessions featuring the Bela embedded audio platform developed in my lab. Many hackers at the event used Bela in their projects, and following the event several of them purchased Bela hardware to continue using in their own work. |
| Year(s) Of Engagement Activity | 2015 |
| URL | http://c4dm.eecs.qmul.ac.uk/news/news.2015-06-13.Audio_and_Music_Hackathon.html |
| Description | Augmented Instruments Lab at Hackoustic Village |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Public/other audiences |
| Results and Impact | We presented a series of demonstrations and a talk at the We Are Robots Festival in London on 2 November 2017. The event was part of a larger exhibition called the Hackoustic Village, named after the artists collective Hackoustic. The event drew a large audience, several follow-up visits to our lab and plans for further work together. A visit from Hackoustic members to the lab is scheduled for late spring 2018. |
| Year(s) Of Engagement Activity | 2017 |
| URL | http://hackoustic.org/hackoustic-village-at-we-are-robots/ |
| Description | BeagleRT workshop C4DM |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Postgraduate students |
| Results and Impact | We organised a workshop for PhD students in the Centre for Digital Music and those on the Media & Arts Technology programme introducing the new BeagleRT real-time embedded audio environment. It showed how the environment could be used for the students' research and introduced basic concepts of real-time audio programming. Many participants expressed interest in using the platform in their own work. |
| Year(s) Of Engagement Activity | 2014 |
| Description | BeagleRT workshop Imperial |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Other academic audiences (collaborators, peers etc.) |
| Results and Impact | Organised audio programming workshop at Imperial College Advanced Hackspace, to happen 14 November. n/a (upcoming) |
| Year(s) Of Engagement Activity | 2014 |
| Description | Bela and TouchKeys at Superbooth |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Industry/Business |
| Results and Impact | In April 2017, we presented two research spinout projects at the Superbooth trade show in Berlin. Superbooth is an annual 3-day event aimed at makers and users of modular synthesisers and related music technology, which draws thousands of attendees each year. We hosted a booth featuring two projects: Bela, an open-source embedded platform for audio and sensor processing, which launched on Kickstarter in 2016 and subsequently spun out into a company, Augmented Instruments Ltd; and TouchKeys, a sensor kit which installs onto a piano-style keyboard to add capacitive touch sensing of the location of the fingers. The event also included a workshop where 15 people received a hands-on introduction to using Bela. The response to the event was very positive, with significant media publicity and many conversations with other industry professionals and members of the public. We plan to return for Superbooth 2018. |
| Year(s) Of Engagement Activity | 2017 |
| URL | http://2017.superbooth.com/en/ |
| Description | Bela at Heart n Soul charity events |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Public/other audiences |
| Results and Impact | Heart n Soul is a UK-based charity catering to children and adults with learning disabilities. Part of their activities involve a series of "SoundLab" events featuring music technology accessible to people with such disabilities. Each event includes participation from several companies or other music technology creators. In November 2016, we presented Bela (our low-latency embedded audio platform -- http://bela.io) at the Beautiful Octopus Club, a large-scale evening festival held at the London Southbank Centre. The SoundLab was part of the Beautiful Octopus Club event; over the course of an evening, we demonstrated several accessible musical instruments to a crowd of hundreds of people of all ages, both disabled and not. Later, in February 2017, we were invited to present Bela at another SoundLab event in South London, which reached an estimated 30 people. Following that event, we have made plans to meet further with some of the participants to better understand how to support high-level music making by people with disabilities. |
| Year(s) Of Engagement Activity | 2016,2017 |
| URL | http://www.heartnsoul.co.uk/category/taking_part/details/beautifuloctopusclub |
| Description | Bela at STEIM |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Study participants or study members |
| Results and Impact | In August 2016, my research lab organised a 3-day hands-on instrument design workshop at the Studio for Electro-Instrumental Music (STEIM) in Amsterdam. The workshop featured Bela, an open-source embedded platform for ultra-low-latency audio and sensor processing, which was developed originally as part of the EPSRC-funded Hackable Instruments project. The workshop had 8 participants who were digital musical instrument designers, plus several other people attending as observers. The event was facilitated by two technical organisers and two other researchers. The instrument designers each brought an existing instrument design which they ported to Bela with the help of the organisers. We recorded audio and video from the session, including regular interviews with each participant. This event led to a paper in CHI 2017 (Denver, USA) on community formation around open-source hardware tools. It also led to several participants using Bela in their own music and art installations at a wider scale. |
| Year(s) Of Engagement Activity | 2016 |
| URL | http://steim.org/event/bela-workshop-call/ |
| Description | Bela workshops |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Public/other audiences |
| Results and Impact | On 4-5 December 2015 we held two tutorial workshops on the Bela low-latency audio platform. These were free and open to the public. Originally the plan was to hold a single event, but high demand resulted in it filling up quickly, so we scheduled a second one the day before. In all we estimate the attendance at 40-50 people. Feedback was positive and many people bought hardware from us to continue using Bela in their own projects. This event was one of several that laid the groundwork for our highly successful Kickstarter crowdfunding campaign in March 2016. Since the original workshop, we have held several additional workshops throughout 2016 and 2017, together reaching several hundred people. Events have taken place in London, Glasgow, Edinburgh, Brighton, Berlin, Toronto, Baton Rouge and Brisbane, among others. Some events have targeted hobbyists and makers; others have targeted musicians and industry professionals; still others have targeted undergraduate and postgraduate students. Collectively, these events have significantly expanded the profile of the Bela platform, leading to research collaborations, increased public profile of our work and commercial benefits (hardware sales through a spinout company). |
| Year(s) Of Engagement Activity | 2015,2016,2017 |
| URL | http://bela.io |
| Description | D-Box Sonar workshop |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | Yes |
| Geographic Reach | International |
| Primary Audience | Public/other audiences |
| Results and Impact | Two workshops led to positive comments in person and by email and requests for further design information. Met the director of a UK charity for disabled musicians with whom I may collaborate in the future. Also raised profile of research and refined workshop activity for presenting again in other contexts. |
| Year(s) Of Engagement Activity | 2014 |
| URL | http://sonar.es/en/2014/prg/sm/queen-mary-university-of-london:-hack-in-the-box_109 |
| Description | D-Box presentation FabLab Cardiff |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Postgraduate students |
| Results and Impact | Presented the D-Box hackable instrument at FabLab Cardiff (Cardiff School of Art and Design), with an audience including arts students and members of the "maker" community. Followed by questions, discussions and hands-on demos. Introduced our research to a broader maker community outside science and engineering, and may lead to further collaborative projects e.g. a placement project for a PhD student. |
| Year(s) Of Engagement Activity | 2014 |
| Description | D-Box workshop Music Hackspace |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Public/other audiences |
| Results and Impact | Organised a workshop for electronic music hackers through London Music Hackspace. Interest led to workshop running well past scheduled time, including questions and further email contacts. Musicians interested in using research platform for their own artistic purposes. |
| Year(s) Of Engagement Activity | 2014 |
| Description | D-Box workshops FACT/JW3 |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Public/other audiences |
| Results and Impact | Two free public workshops were conducted on the D-Box hackable musical instruments, one at FACT (Foundation for Art and Creative Technology) in Liverpool and one at the JW3 Jewish community centre in London. At the half-day workshops, each participant (17 total) received a D-Box to play and modify. Feedback was positive from the participants on both the instrument and the workshop format, with many expressing continuing interest in the instrument. The workshops were jointly conducted with members of the Mixed Reality Laboratory at the University of Nottingham. A paper on the findings of the workshop has been accepted to ACM Designing Interactive Systems 2016. |
| Year(s) Of Engagement Activity | 2015 |
| Description | Heart n Soul SoundLab events |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Third sector organisations |
| Results and Impact | SoundLab events are run by Heart n Soul to connect music technology companies and research groups with members of the learning disabilities community. The Augmented Instruments Lab research group has attended several of these recurring events to showcase our instruments made using the Bela embedded controller which was also developed by members of the research group. |
| Year(s) Of Engagement Activity | 2017,2018 |
| Description | Media (BBC Click) |
| Form Of Engagement Activity | A magazine, newsletter or online publication |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Public/other audiences |
| Results and Impact | Hackable Instruments (EPSRC) project features on BBC Click radio programme. Some communication from people who had heard the programme. |
| Year(s) Of Engagement Activity | 2014 |
| URL | http://www.bbc.co.uk/programmes/p01rpnvz |
| Description | Media (New Scientist) |
| Form Of Engagement Activity | A magazine, newsletter or online publication |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Public/other audiences |
| Results and Impact | Article in New Scientist: "Dream Job: Digital Instrument Creator" published March 2014, covering research activities in the lab with a focus on the Hackable Instruments (EPSRC) project. Contacts from charities using music for special-needs children; may lead to future collaborations. |
| Year(s) Of Engagement Activity | 2014 |
| URL | http://www.newscientist.com/article/dn25154-dream-job-digital-instrument-creator.html?full=true#.VFu... |
| Description | OHMI judging panel |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | Yes |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Participated in judging panel for instrument design competition. Chose the top instrument designs enabling one-handed performance for musicians with disabilities, from an international pool of submissions. Invited to present my research at the Ars Electronica festival. |
| Year(s) Of Engagement Activity | 2014,2015,2016 |
| URL | http://www.ohmi.org.uk/the-competition.html |
| Description | Presentation at Aalto University Helsinki |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Postgraduate students |
| Results and Impact | On 29 November 2018 I gave an invited talk on my research to academics and postgraduate students at Aalto University (Helsinki). Following the talk I spent two days advising individual students on creative projects they were working on in the area of interactive systems. The students were on a course taught using Bela (bela.io), an open-source embedded platform developed in my research starting in 2014 and spun out into a company in 2016. |
| Year(s) Of Engagement Activity | 2018 |
| Description | School visit (Sutton Grammar School) |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Schools |
| Results and Impact | Talk on electronic musical instrument design led to questions and discussion, and was followed by a hands-on session in class about building new instruments in which students proposed designs of their own. My lab and department was invited to participate in the Big Bang Science Festival organised by the school. |
| Year(s) Of Engagement Activity | 2014 |