Design for Virtuosity: Modelling and Supporting Expertise in Digital Musical Interaction
Lead Research Organisation:
Queen Mary University of London
Department Name: Sch of Electronic Eng & Computer Science
Abstract
Musical performers spend many years achieving proficiency on their instruments. Newly-created digital musical instruments (DMIs) face a significant barrier to adoption in that few performers are willing to repeat these years of training to develop expertise on an unknown instrument. Without expert players, evaluating the success of a DMI design is challenging, and establishing its place in a broader musical community is nearly impossible. As a result, while many digital instruments have been created over the past decade, few have achieved lasting impact beyond the first few performances.
This fellowship proposes a new approach to DMI design which repurposes the existing skills and experience of trained musicians, providing them with a rapid path to virtuosity without years of retraining. The research programme is organised around two complementary themes: study of performer-instrument interaction and creation of new instruments which capture the full richness and subtlety of virtuosic performance.
First, models will be developed of the interaction between performer and instrument. Instrumental performance can be considered a special case of human-machine interaction which is interesting both for its complexity and for the common experience that the musical instrument becomes an extension of the body: while playing, the performer is often not consciously thinking about the instrument. Controlled experiments and participatory design exercises will establish how an instrument's design affects the development of expertise, and how existing expertise can be transferred to newly-created instruments.
Second, the resulting models will be applied to the DMI creation process, taking a holistic approach unifying hardware design, digital signal processing, human-computer interaction (HCI) and artistic considerations. Existing DMIs often implicitly prioritise the convenience of the computer over the experience of the human player. On acoustic instruments, the entire physical object contributes to the sound, however subtly, but the choice of sensors in a DMI typically reduces the performer's actions to just a few machine-tractable dimensions. This fellowship will create instruments which deliberately oversample the interaction, using more sensors and higher sampling rates than apparently necessary, not to create a more complicated instrument, but rather to capture the subtle nuances that experts prize. Evaluation of the new DMIs will help refine the original models of performer-instrument interaction.
This project focuses on musical interaction, but the principle of repurposing expertise is widely applicable within HCI. The capabilities of human users cannot be modelled only through generic cognitive and motor processes; real people have specialist skills developed over years of practice, and new technologies which connect to those skills are far more likely to find acceptance than ones which must be learned from scratch. Music is a good test case since instrumental training is widespread and reasonably standardised, but the findings will be relevant to other expert domains.
This fellowship supports the time of the PI and two postdoctoral researchers. One postdoc will focus on performer studies and interaction, the other on digital signal processing and data mapping strategies. Close collaboration with musicians throughout the research will ensure its relevance to that community. This research integrates hardware design, digital signal processing, human-computer interaction, cognitive science, musicology and arts practice. The PI, with background and professional activities in music composition, electronic engineering and HCI, is ideally placed to lead this multidisciplinary project.
This fellowship proposes a new approach to DMI design which repurposes the existing skills and experience of trained musicians, providing them with a rapid path to virtuosity without years of retraining. The research programme is organised around two complementary themes: study of performer-instrument interaction and creation of new instruments which capture the full richness and subtlety of virtuosic performance.
First, models will be developed of the interaction between performer and instrument. Instrumental performance can be considered a special case of human-machine interaction which is interesting both for its complexity and for the common experience that the musical instrument becomes an extension of the body: while playing, the performer is often not consciously thinking about the instrument. Controlled experiments and participatory design exercises will establish how an instrument's design affects the development of expertise, and how existing expertise can be transferred to newly-created instruments.
Second, the resulting models will be applied to the DMI creation process, taking a holistic approach unifying hardware design, digital signal processing, human-computer interaction (HCI) and artistic considerations. Existing DMIs often implicitly prioritise the convenience of the computer over the experience of the human player. On acoustic instruments, the entire physical object contributes to the sound, however subtly, but the choice of sensors in a DMI typically reduces the performer's actions to just a few machine-tractable dimensions. This fellowship will create instruments which deliberately oversample the interaction, using more sensors and higher sampling rates than apparently necessary, not to create a more complicated instrument, but rather to capture the subtle nuances that experts prize. Evaluation of the new DMIs will help refine the original models of performer-instrument interaction.
This project focuses on musical interaction, but the principle of repurposing expertise is widely applicable within HCI. The capabilities of human users cannot be modelled only through generic cognitive and motor processes; real people have specialist skills developed over years of practice, and new technologies which connect to those skills are far more likely to find acceptance than ones which must be learned from scratch. Music is a good test case since instrumental training is widespread and reasonably standardised, but the findings will be relevant to other expert domains.
This fellowship supports the time of the PI and two postdoctoral researchers. One postdoc will focus on performer studies and interaction, the other on digital signal processing and data mapping strategies. Close collaboration with musicians throughout the research will ensure its relevance to that community. This research integrates hardware design, digital signal processing, human-computer interaction, cognitive science, musicology and arts practice. The PI, with background and professional activities in music composition, electronic engineering and HCI, is ideally placed to lead this multidisciplinary project.
Planned Impact
This fellowship holds benefits for a wide variety of groups beyond academic researchers, both in the UK and internationally:
Industry:
* Musical instrument manufacturers, who will benefit from new tools and approaches to instrument design. Selling the same instruments year-upon-year risks market saturation, but conversely, uptake of unfamiliar instruments is a challenge for the industry owing to long learning curves. This fellowship will demonstrate how to create instruments which are creatively distinct from previous designs but still connect to existing skills.
* The broader technology industry, through creation and sales of computer interfaces which integrate into a user's existing workflows.
* Concert venues and promoters, who will benefit from performances involving new creative digital tools.
* Electronics companies serving the open-source and maker communities, who can develop and sell tools based on the open-source hardware released during this project. Open-source hardware makes the designs available freely to all, but it is also an important economic engine, as evidenced by successful companies like Arduino in Italy and SparkFun and Adafruit in the United States. This project will support the UK's growing role as a driver of the open hardware movement.
Musicians:
* Professional performers, who will benefit from instruments which connect to their existing training. By reducing relearning, instruments on the fellowship will make better use of their time, which holds a direct economic benefit.
* Composers, who will benefit in two ways: from immediate opportunities to write for new instruments during the fellowship, and from continuing opportunities when new instruments become established in the musical community.
* Instrumental teachers and students, for whom a deeper understanding of performer-instrument interaction will be relevant to teaching the subtle details of performance technique.
* Disabled and special-needs musicians, who will benefit from instruments which make more effective use of their abilities (with assistance from project partner The One-Handed Musical Instrument Trust).
Schools and Third Sector:
* School music teachers, who can use new digital instruments to promote student interest in music learning.
* Science and technology teachers, who already use the CS4Fn/EE4Fn (Computer Science for Fun; Electronic Engineering for Fun) project at QMUL to promote student interest in computing/engineering careers. This fellowship will contribute new articles and activities on digital musical instruments to the project and assist with the project's dissemination into schools.
* Independent groups promoting STEM (Science/Technology/Engineering/Maths) education, who can use digital musical instruments as the basis of workshop and outreach activities for students.
* Charities promoting music accessibility, for whom the workshop organised by partner OHMI can serve as a model, and for whom the technologies developed on the fellowship can later be applied to create accessible musical interfaces.
Wider Public:
* Amateur musicians, who will benefit from new instruments which make effective use of their training time.
* Audiences, who will benefit from engaging digital musical instrument performances where the expertise of the performer is clearly evident.
* Electronics hobbyists, who can use the open-source tools created on this project; these tools will be designed to use commonly-available fabrication techniques such as 3D printing to make them more accessible.
In addition to the fellowship's primary research activities, a targeted programme of workshops, concerts, recordings, artist residencies, schools talks, industry events and release of open-source designs will ensure the fellowship reaches the beneficiaries named above. See Pathways to Impact for full details.
Industry:
* Musical instrument manufacturers, who will benefit from new tools and approaches to instrument design. Selling the same instruments year-upon-year risks market saturation, but conversely, uptake of unfamiliar instruments is a challenge for the industry owing to long learning curves. This fellowship will demonstrate how to create instruments which are creatively distinct from previous designs but still connect to existing skills.
* The broader technology industry, through creation and sales of computer interfaces which integrate into a user's existing workflows.
* Concert venues and promoters, who will benefit from performances involving new creative digital tools.
* Electronics companies serving the open-source and maker communities, who can develop and sell tools based on the open-source hardware released during this project. Open-source hardware makes the designs available freely to all, but it is also an important economic engine, as evidenced by successful companies like Arduino in Italy and SparkFun and Adafruit in the United States. This project will support the UK's growing role as a driver of the open hardware movement.
Musicians:
* Professional performers, who will benefit from instruments which connect to their existing training. By reducing relearning, instruments on the fellowship will make better use of their time, which holds a direct economic benefit.
* Composers, who will benefit in two ways: from immediate opportunities to write for new instruments during the fellowship, and from continuing opportunities when new instruments become established in the musical community.
* Instrumental teachers and students, for whom a deeper understanding of performer-instrument interaction will be relevant to teaching the subtle details of performance technique.
* Disabled and special-needs musicians, who will benefit from instruments which make more effective use of their abilities (with assistance from project partner The One-Handed Musical Instrument Trust).
Schools and Third Sector:
* School music teachers, who can use new digital instruments to promote student interest in music learning.
* Science and technology teachers, who already use the CS4Fn/EE4Fn (Computer Science for Fun; Electronic Engineering for Fun) project at QMUL to promote student interest in computing/engineering careers. This fellowship will contribute new articles and activities on digital musical instruments to the project and assist with the project's dissemination into schools.
* Independent groups promoting STEM (Science/Technology/Engineering/Maths) education, who can use digital musical instruments as the basis of workshop and outreach activities for students.
* Charities promoting music accessibility, for whom the workshop organised by partner OHMI can serve as a model, and for whom the technologies developed on the fellowship can later be applied to create accessible musical interfaces.
Wider Public:
* Amateur musicians, who will benefit from new instruments which make effective use of their training time.
* Audiences, who will benefit from engaging digital musical instrument performances where the expertise of the performer is clearly evident.
* Electronics hobbyists, who can use the open-source tools created on this project; these tools will be designed to use commonly-available fabrication techniques such as 3D printing to make them more accessible.
In addition to the fellowship's primary research activities, a targeted programme of workshops, concerts, recordings, artist residencies, schools talks, industry events and release of open-source designs will ensure the fellowship reaches the beneficiaries named above. See Pathways to Impact for full details.
Organisations
- Queen Mary University of London (Lead Research Organisation)
- Aalborg University (Collaboration)
- UNIVERSITY OF NOTTINGHAM (Collaboration)
- Kiel University of Applied Sciences (Collaboration)
- Princeton University (Collaboration)
- Iceland Academy of the Arts (Collaboration)
- One-Handed Musical Instrument Trust (Collaboration)
- University of Lethbridge (Collaboration)
- Aalto University (Collaboration)
- University of Surrey (Project Partner)
- Roli (United Kingdom) (Project Partner)
- One-Handed Musical Instrument (Project Partner)
- Imperial College London (Fellow)
People |
ORCID iD |
| Andrew McPherson (Principal Investigator / Fellow) |
Publications
Moro G
(2020)
Performer Experience on a Continuous Keyboard Instrument
in Computer Music Journal
Morreale F
(2017)
Building a Maker Community Around an Open Hardware Platform
Morreale F
(2019)
Designing Creative Tensions between Concept and Embodied Practice
Morreale F
(2019)
Magpick: an Augmented Guitar Pick for Nuanced Control
Morreale F
(2018)
Effect of Instrument Structure Alterations on Violin Performance
in Frontiers in Psychology
Morreale F
(2018)
NIME Identity from the Performer's Perspective
N Robson
(2021)
On mediating space, sound and experience: interviews with situated sound art practitioners
in Organised Sound
Pardue L
(2019)
Real-Time Aural and Visual Feedback for Improving Violin Intonation
in Frontiers in Psychology
Pardue LS
(2019)
Real-Time Aural and Visual Feedback for Improving Violin Intonation.
in Frontiers in psychology
Pigrem J
(2018)
Do We Speak Sensor? Cultural Constraints of Embodied Interaction
Reed C
(2020)
Surface Electromyography for Direct Vocal Control
Robson N
(2022)
On Mediating Space, Sound and Experience: Interviews with situated sound art practitioners
in Organised Sound
S. M. Astrid Bin
(2017)
Hands where we can see them! Investigating the impact of gesture size on audience perception
S. M. Astrid Bin
(2018)
Risky business: Disfluency as a design strategy
Turchet L
(2018)
Real-Time Hit Classification in a Smart Cajón
in Frontiers in ICT
Zappi V
(2018)
Hackable Instruments: Supporting Appropriation and Modification in Digital Musical Interaction
in Frontiers in ICT
| Title | Answer Machine Tape, 1987 by Philip Venables |
| Description | Zubin Kanga, pianist and UKRI Future Leadership Fellow at Royal Holloway, commissioned and premiered a new piece by Philip Venables entitled "Answer Machine Tape, 1987" related to the height of the AIDS crisis in the 1980's. Kanga premiered the piece in July 2022 at Time of Music, Finland. The piece uses an optical scanner for the piano keyboard developed in the research of Andrew McPherson and Giulio Moro, measuring the motion of the keys so that a pianist can control sounds and graphics from an acoustic piano. |
| Type Of Art | Composition/Score |
| Year Produced | 2022 |
| Impact | This is one of a number of collaborations between the Augmented Instruments Laboratory (led by Andrew McPherson) and the Cyborg Soloists project (led by Zubin Kanga) to create new musical works featuring technologies developed out of UKRI-supported research. |
| URL | https://twitter.com/ZubinKanga/status/1542481911951540225 |
| Title | Concert - Beep/Bang |
| Description | This concert was organised in support of my second study. As well as the experimental performances it involved a 5 other practitioners working in innovative digital percussion. |
| Type Of Art | Performance (Music, Dance, Drama, etc) |
| Year Produced | 2016 |
| Impact | The concert was well-received and contributed to raising awareness of the work going on at QMUL in musical interface design. |
| Title | Concert - Music from the Augmented Instruments Lab |
| Description | This concert was organised for my third study, but also featured music and musical interfaces produced by my research group, the Augmented Instruments Lab. |
| Type Of Art | Performance (Music, Dance, Drama, etc) |
| Year Produced | 2017 |
| Impact | The concert was well-received and contributed to raising awareness of the work going on at QMUL in musical interface design. |
| Title | Concert Series: Music, Installations & Designs from the Augmented Instruments Lab at Iklectik |
| Description | On 14 June 2019, our lab held a concert of new music and new instruments at the Iklectik Art Lab in south London. The principal organiser of the concert was Lia Mice, a PhD student in the lab, with support from several other members. The concert featured a headline performance by Robyn Steward (https://www.robynsteward.com/music), a trumpeter and electronic musician who is autistic and travels widely to promote autism awareness. Other performances included newly created instruments and projects-in-progress from the research underway in the lab, including a hybrid acoustic-electronic violin (the svampolin) created to repurpose the skills of expert violin performers. A second concert took place on 27 November 2019, again featuring the svampolin alongside a different set of instruments including a new electronic tabla created by London startup Keda Music, where some alums of the lab are now employed. |
| Type Of Art | Performance (Music, Dance, Drama, etc) |
| Year Produced | 2019 |
| Impact | Both concerts were sold out, with an audience of 100 people. The audiences consist of members of the general public, professional musicians and technologists, and academics (post-graduate, post-doctorate, lecturers and professors) from the Centre for Digital Music, Queen Mary University of London. The June concert led to interest in the lab's research and most directly to an article in the Guardian which was published on 10 July 2019 (https://www.theguardian.com/careers/2019/jul/10/we-need-to-put-inclusion-at-the-start-of-the-process-the-disabled-musicians-making-their-own-instruments). |
| Title | D-Box performance at Stereo |
| Description | In February 2017 at the University of Glasgow, we delivered an invited workshop using Bela, the open-source low-latency audio platform created in our research group. Following the workshop, we were invited to perform with the D-Box, a hackable digital musical instrument created as part of research into appropriation of music technology on the EPSRC Hackable Instruments project (EP/K032046/1). The concert was held at Stereo, a popular club in Glasgow, and also featured a performance using Bela by Sebastian Lexer of the University of Glasgow. |
| Type Of Art | Performance (Music, Dance, Drama, etc) |
| Year Produced | 2017 |
| Impact | The workshop and concert led to increased interest in Bela and the D-Box, and the trip also led to discussions about collaboration with the University of Glasgow on future funding proposals and joint research projects. |
| Title | Hybrid acoustic-electronic violin at FARM 2019 conference |
| Description | On 23 August 2019, postdoc Laurel Pardue and PhD student Jack Armitage performed at the 7th ACM SIGPLAN Workshop on Functional Art, Music, Modeling and Design (FARM). Laurel performed on the svampolin, a new hybrid acoustic-electronic violin developed on my research fellowship with the aim of repurposing expert skills of trained violinists, while Jack performed with live coding (algorithmically generated sounds programmed in real time and projected for the audience to see the code). |
| Type Of Art | Performance (Music, Dance, Drama, etc) |
| Year Produced | 2019 |
| Impact | The svampolin has gone on to be selected as a semi-finalist in the Guthman Musical Instrument Competition, held annually at Georgia Tech in Atlanta, USA. The competition will take place in March 2020. |
| URL | https://functional-art.org/2019/performance.html |
| Title | Keyboard scanner used in CodeKlavier |
| Description | I have created a prototype sensor device for measuring the continuous motion of the keys of a piano keyboard. The device installs on any keyboard, including acoustic pianos, to provide information about the motion of the keys. I have supplied a prototype of this device to two musicians working on a project entitled CodeKlavier, where piano performance is used to create a live coding performance (i.e. live real-time programming of algorithmic music). Patterns played on the piano are translated into code fragments which run and in turn create new musical patterns. The scanner has been used in several CodeKlavier performances around the world including the 2019 International Conference on Live Coding in Madrid. |
| Type Of Art | Performance (Music, Dance, Drama, etc) |
| Year Produced | 2019 |
| Impact | Working with the CodeKlavier team has helped improve my research by making the device more robust and giving me useful feedback on its artistic utility. According to the CodeKlavier team, the performances have received good responses from the audience (though I have not been able to attend one in person) and they consistently attract on attention on social media, especially Twitter. |
| URL | https://codeklavier.space |
| Title | Live-stream: music from the Augmented Instruments Laboratory |
| Description | On 10 March and 23 November 2021, my research group (the Augmented Instruments Lab) hosted online streaming concerts of music with new digital instruments. The March concert included a performance on the svampolin, a hybrid acoustic-electronic violin created by postdoctoral researcher Laurel Pardue on the EPSRC Design for Virtuosity project. The November concert included performances on the Chaos Bells, a new digital musical instrument created by PhD researcher Lia Mice (UKRI Media and Arts Technology CDT), who is investigating the effects of large scale instruments on embodiment and use of technology. |
| Type Of Art | Performance (Music, Dance, Drama, etc) |
| Year Produced | 2021 |
| Impact | Between live and subsequent viewers, the concerts have been seen by over 1000 people, and have raised the visibility of research in the lab. Plans are underway for further live streamed and in-person concerts. |
| URL | https://www.youtube.com/c/AugmentedInstrumentsLab/videos |
| Title | MRP featured on album The See Within by Echo Collective |
| Description | In 2020, the Brussels chamber ensemble Echo Collective released a commercial album on the Berlin-based 7K label entitled "The See Within". The album heavily features the magnetic resonator piano, an augmented piano from my research which actuates the piano strings with electromagnets to produce novel sonic effects controlled continuously from the keyboard. |
| Type Of Art | Artefact (including digital) |
| Year Produced | 2020 |
| Impact | An album release tour and several artistic residencies are planned for 2021, though these plans are currently in flux due to the ongoing pandemic situation. The MRP will be featured at each of these events. |
| URL | https://www.echocollective.be |
| Title | Magnetic Resonator Piano at Inter/Sections |
| Description | The magnetic resonator piano (MRP) was featured in an installation and performance as part of the Inter/Sections festival in September 2016, a multiday event in London organised by PhD students on the EPSRC and AHRC Centre for Doctoral Training in Media and Arts Technology. The MRP is an electromagnetically augmented acoustic piano which extends the capabilities of the traditional piano. It was set up as an interactive installation for the duration of the festival, and was featured in an evening performance with performances by pianist Kate Ryder. |
| Type Of Art | Artistic/Creative Exhibition |
| Year Produced | 2016 |
| Impact | Following the exhibition I have been invited to show the MRP in other venues, including an opera theatre in Rome where it will be used as part of a newly composed opera by Maria Kallionpää. |
| URL | https://2016.intersections.io |
| Title | Magnetic Resonator Piano at International Conference on Live Interfaces |
| Description | The magnetic resonator piano (MRP), an electronically augmented acoustic piano developed as part of my research, was featured in a 2-day composition and improvisation workshop at the International Conference on Live Interfaces in Brighton, UK in June 2016. I was invited to deliver this workshop by the conference organisers. During the workshop, 6 composers used the MRP in new short pieces which were played by pianist Kate Ryder at the end of the event. |
| Type Of Art | Performance (Music, Dance, Drama, etc) |
| Year Produced | 2016 |
| Impact | The event looks likely to lead to future collaborations with some of the involved composers and other pianists who attended, and more immediately led to Kate Ryder participating in another concert at the Inter/Sections festival in September 2016. |
| Title | Magnetic resonator piano at Approximation Festival, Duesseldorf |
| Description | The magnetic resonator piano (an augmented piano developed in my research) was used in a performance at the Approximation Festival, a 3-day festival of new music held in Duesseldorf, Germany in February 2019. The instrument was performed by pianist, composer and improviser Xenia Pestova to a sold-out audience of approximately 100 people. Prior to the performance I gave a talk on the technical and musical aspects of the instrument. |
| Type Of Art | Performance (Music, Dance, Drama, etc) |
| Year Produced | 2019 |
| Impact | The hall was sold out for the performance and the feedback from the audience was very positive, with many people asking questions about the instrument and the music. My participation with the instrument was prominently featured in the festival's publicity materials, which included billboards and posters all around Duesseldorf. |
| URL | http://www.approximation-festival.de/festival.html |
| Title | Magnetic resonator piano at Cafe Oto and Royal Birmingham Conservatoire |
| Description | The magnetic resonator piano (a product of my research) was used in two performances by pianist Xenia Pestova (University of Nottingham) and composer John Young (De Montfort University). The first performance was at the Royal Birmingham Conservatoire, the second at Cafe Oto, a leading experimental music venue in Dalston, London. |
| Type Of Art | Performance (Music, Dance, Drama, etc) |
| Year Produced | 2018 |
| Impact | Both concerts were well attended and received enthusiastic audience response to both the music and the instrument. The Cafe Oto performance was attended by several well-known composers and by writers for leading music technology publications. As a recent event the long-term impact is hard to assess but it appears to have raised the profile of my research and led to new connections into the arts community. |
| URL | http://cafeoto.co.uk/events/magnetic-string-resonance-xenia-pestova/ |
| Title | Magnetic resonator piano at Daylight Music and New Music Dublin |
| Description | On 29 June 2019, pianist-composer Xenia Pestova Bennett performed a concert at Union Chapel (Islington, London, UK) as part of their weekly Daylight Music series. Xenia played the magnetic resonator piano, a hybrid acoustic-electronic instrument developed in my research. She played a solo piece for the instrument and a second piece for magnetic resonator piano and string quartet, together with the Ligeti Quartet, one of the UK's leading quartets specialising in contemporary music. |
| Type Of Art | Performance (Music, Dance, Drama, etc) |
| Year Produced | 2019 |
| Impact | The concert was well attended (precise figures not gathered but probably 100+) and there was strong interest in the instrument from the audience members. Xenia and the Ligeti Quartet went on to record the same pieces for a vinyl release from Diatribe Records, which has in turn led to broadcasts on BBC Radio 3 and invitations to perform internationally, including at New Music Dublin, where the programme was repeated on 1 March 2020 (https://www.newmusicdublin.ie/events/diatribe-stage-5-xenia-pestova-bennett-atomic-legacies). |
| URL | http://daylightmusic.co.uk/event/311-magnetic-string-resonance/ |
| Title | Magnetic resonator piano at Fish Factory studios |
| Description | On 20 November 2019, the magnetic resonator piano, a hybrid acoustic-electronic piano developed as part of my research, was featured in a daylong studio session by the label Trestle Records as part of their "One-Day Band" series. The session featured a duo performance by pianists Sam Beste and Gwilym Gold. The session was recorded and will be released by Trestle Records later in 2020. |
| Type Of Art | Performance (Music, Dance, Drama, etc) |
| Year Produced | 2019 |
| Impact | No follow-up yet as the session has not been publicly released. |
| Title | Magnetic resonator piano at Occupy the Pianos |
| Description | The magnetic resonator piano (an augmented acoustic piano developed in my research) was performed at St Johns Smith Square (London) on 20 April 2018 as part of the Occupy the Pianos festival. The festival was organised by pianist and composer Rolf Hind, who wrote and performed a piece on the instrument as part of a programme of contemporary piano music. |
| Type Of Art | Performance (Music, Dance, Drama, etc) |
| Year Produced | 2018 |
| Impact | The instrument attracted a great deal of interested from the audience in attendance, and the collaboration with Rolf Hind has continued with another event at QMUL in November 2018 and a planned event as part of a music festival in Deal in summer 2019. |
| URL | https://www.sjss.org.uk/events/occupy-pianos |
| Title | Magnetic resonator piano at Opera Theatre of Rome |
| Description | On 11-12 October 2017, the magnetic resonator piano (an augmented instrument that is the product of my research) was featured in a newly commissioned opera, "She", written by Maria Kallionpaa. The opera was commissioned and performed by the Teatro dell'opera di Roma in Rome's Teatro Nazionale as part of the Fabbrica young artists programme. |
| Type Of Art | Performance (Music, Dance, Drama, etc) |
| Year Produced | 2017 |
| Impact | The performance was well received by the local audience and the instrument attracted considerable interest. Maria Kallionpaa plans to use it further in her composition and performance. |
| URL | http://www.operaroma.it/spettacoli/on-off-she/ |
| Title | Magnetic resonator piano in Christopher Robin movie score |
| Description | The magnetic resonator piano (an augmented instrument I have developed in my research) was featured in the major motion picture Christopher Robin (2018) by Disney Studios. The score was written and performed by high-profile Hollywood composer Jon Brion (https://www.imdb.com/name/nm0109726/?ref_=nv_sr_2), and recorded in May 2018 at Metropolis Studios in London. The film was released in August 2018 and has grossed $100M since its August 2018 release. The MRP is credited at the end of the film, and I am credited as its creator. The film received positive critical reviews, and the score was nominated for an International Film Music Critics Award in 2019. |
| Type Of Art | Composition/Score |
| Year Produced | 2018 |
| Impact | Aside from the commercial impact of the film itself, in which the magnetic resonator piano does not play a substantial part, the biggest impact from this work has been attention on the instrument and my research from studio executives and high-profile musicians in the film industry. For example, several months later, I was contacted by the conductor of the Los Angeles studio orchestra which had recorded the rest of the parts of Jon Brion's score. This conductor is also a composer and wanted to record the MRP for his own film project. We held an initial informal recording session in my lab in late 2018, and further collaborations may follow. |
| URL | https://www.imdb.com/title/tt4575576/?ref_=nv_sr_1 |
| Title | Performance at Iklectik Art Lab |
| Description | On 11 May 2018 the members of the Augmented Instruments Laboratory, a 10-person research team led by Andrew McPherson, held an evening performance at Iklectik, a performance space in south London. The programme featured instruments created in our research and performances and compositions by members of the lab. Contemporary pianist and improviser Xenia Pestova and digital musician D. Andrew Stewart were the featured guests. Pestova played her own music on the magnetic resonator piano, an instrument developed in the lab; Stewart played a piece for digital controllers that he had worked on while in residency in the lab. |
| Type Of Art | Performance (Music, Dance, Drama, etc) |
| Year Produced | 2018 |
| Impact | The venue was full for the concert (audience of around 100) including many people active in digital musical instrument research and performance outside of our institution. The concert increased interest in the lab and led to further conversations and visits. Another concert is planned for summer 2019. |
| URL | http://iklectikartlab.com/augmented-instruments-lab/ |
| Title | Vinyl release: Atomic Legacies by Xenia Pestova Bennett |
| Description | On 1 March 2020, pianist-composer Xenia Pestova Bennett released Atomic Legacies, an album featuring two of her new compositions for magnetic resonator piano. The magnetic resonator piano is an augmented acoustic-electronic piano developed in my research. It extends the capabilities of the traditional piano to include infinite sustain, crescendos, pitch bends, harmonics and new timbres, all produced acoustically by the strings. |
| Type Of Art | Artefact (including digital) |
| Year Produced | 2020 |
| Impact | Atomic Legacies was released by Diatribe Records with vinyl and digital formats. One of the pieces also features the Ligeti Quartet, one of the UK's leading new-music quartets. It has generated strong reviews in the press, significant social media attention and excerpts and interviews on BBC Radio 3. |
| URL | https://shop.diatribe.ie/album/atomic-legacies |
| Title | With/Without |
| Description | With/Without was a concert held on 24 June 2016 at Schott Music in London, featuring the magnetic resonator piano (MRP), an electromagnetically augmented acoustic piano developed as part of my research. Prof. Elaine Chew (QMUL) played a new piece by Dr Oden Ben-Tal (Kingston University) and several other pieces on the MRP. |
| Type Of Art | Performance (Music, Dance, Drama, etc) |
| Year Produced | 2016 |
| Impact | The performance increased awareness of my research among the general public, and I expect further collaborations with Elaine Chew and Oded Ben-Tal on this and other pieces. |
| URL | http://elainechew-piano.blogspot.co.uk/2016/06/withwithout-excerpts.html |
| Description | In the first year of the fellowship, the preliminary findings divided into two categories: the influence of musical instrument design on the performer experience, and the ways that technical tools influence the actions and experience of the designer. In the first category, we have conducted studies with violinists to understand how modifying the violin affects the existing expertise of the performer. We have found that violinists can easily adapt to small changes in the instrument, but that certain changes, such as reversing the order of the strings, result in a nearly complete inability to play fluently. This simple experiment demonstrates the importance of highly developed sensorimotor processes on musical performance. This led to publications in the journal Frontiers in Psychology and the New Interfaces for Musical Expression conference. We are also investigating how to develop new instruments which make use of established sensorimotor skills, a topic which began in the second year of the grant and will remain ongoing through its conclusion. There are several lines of enquiry in this project. One looks at modelling, parameterising and ultimately extending the signals generated in traditional musical instruments. Over the past three years we have developed new sensing hardware and a real-time signal model of the string vibrations of the acoustic violin which has a more nuanced representation than current industry standard music data protocols like MIDI, while also containing a richer understanding of the acoustics of the instrument a typical audio signal. We also conducted early experiments using deep learning techniques to parametrise bowed string signals in real time. A second line of enquiry has focused on instrument accessories which address questions of expert playing technique. We developed a new control device for electric guitar: the Magpick, a plectrum which senses the ambient magnetic fields around the pickups and allows the performer to manipulate the sound of the guitar through subtle hand motions over the strings which are easily learned and compatible with existing technique. This work has led to an ongoing collaboration with the University of Auckland. A third line of enquiry looks at acoustic actuation of instruments. Performers often emphasise the importance of an instrument that produces its own acoustic sound locally to the performer, as opposed to electronic instruments whose sound is exclusively produced through external speakers. We have developed a new hybrid acoustic-electronic violin where the strings and bridge are acoustically isolated from the body, and the body is instead driven by electronic actuators. This allows transformations of the sound to be applied between performer and acoustic output. In this area of enquiry, we also continue to develop the possibilities of actuation of acoustic piano strings, working closely with performers and composers to understand both sensorimotor and cultural implications of new instrument technologies. We have also conducted experiments with percussion instruments, showing how latency and the constraints of an instrument change the gestures that performers choose to interact with the instrument. Our results suggest that the relationship between the performer and instrument goes both ways: not only does the performer control the instrument, the instrument design implicitly influences what actions the performer chooses to take. A final line of enquiry take a broad view of recent work in digital musical instrument design, both within academic communities (e.g. the conference New Interfaces for Musical Expression) and within industry (especially crowdfunding campaigns). We have identified common traits of successful instruments which are still being used and made recommendations for designers in papers published or submitted at conferences in the field. This has led to several collaborations in various stages of the pipeline, producing joint papers and workshop proposals on what factors promote instrument or how the tools designers use affect their creative choices. This work has led to wide-ranging discussions in our core research community about how best to archive and distribute knowledge about new instruments. In the category of how tools influence the designer, we have studied how open source tools can be used to support musical instrument design, and we have investigated the process by which communities form around tools. For this work, we used the Bela audio platform developed in our lab (which has been significantly enhanced and expanded since the beginning 2016). A paper on open source community building was published at CHI 2017, with a further paper at a CHI 2017 "HCI tools" workshop. We regularly host public and academic workshops on these topics, and we also host regular external visitors of durations from a few days to several months. In 2019 returned to CHI for a workshop on research through design, showing some of our recent tools and instrument technologies and examining their implications for a wider HCI community. |
| Exploitation Route | This work has academic, commercial and cultural implications. In terms of core research, the project has opened up interesting questions around sensorimotor skill and its transfer from one domain to the other. One area of enquiry that might be productively explored in future research is "auditory imagery", or the mental imagination of sound without actually hearing it. Our research has suggested that auditory imagery plays an important role in instrumental performance, and that the successful transfer of expertise between instruments may be dependent on coherent auditory and motor imagery. This work may be of interest to psychology and human-computer interaction researchers. The research on new sensor and actuator technologies for musical instruments will also be of interest to several communities. Already we are seeing uptake of our research amongst a set of collaborators in our core research community: for example, a string instrument sensor technology we published at the NIME 2019 conference is being used by other researchers and artists. Beyond the musical domain, some of these technologies could also be used more widely in human-computer interaction research; one PhD project outside the project has used them to study the working processes of skilled craftspeople through augmented sonic an haptic feedback. The sensor and actuator technologies also have commercial potential. Already a capacitive touch sensor technology first created in an earlier research project but further developed on this project has been licensed to QMUL spinout company Augmented Instruments Ltd, where it was the subject of a successful Kickstarter crowdfunding campaign. Aside from the commercial implications, these sensors will enable makers and artists to easily create sophisticated interfaces they would not have otherwise been able to make. We also expect that complete instruments developed on this project may have commercial potential. One instrument, a hybrid acoustic-electronic violin, won third place in the annual Guthman Musical Instrument Competition at Georgia Tech. While commercial readiness might take several years, if successful the commercialisation could deliver both economic and cultural impacts. Finally, the instruments and technologies we have developed on the project are finding increasing use amongst composers and performers. We have observed that devices that begin as focused research probes end up finding rich and varied use in the wild. Sometimes this can be a complete instrument, or other times it may be a technology that is adopted by instrument designers outside the research domain. The pandemic and consequent lockdowns led to a delays in the parts of the project engaging with performers, but we have picked up some of these activities as performance events have become possible again, with future activities still planned. This project began with a focus mainly around sensorimotor skill in musical performance, and from this initial focus, several other interrelated areas emerged which can be taken forward in future research. One is around the creation of hardware and software systems with sufficient bandwidth of interaction, and sufficiently low latency, to capture and process the rich and nuanced physical language of musical performance. The need for high-performance embedded hardware was already clear going into the project, but several years of research revealed specific directions which are still in need of further investigation, especially around increasing the total amount of sensor data captured and finding ways of intelligently processing that data. The other stream of research to be taken forward concerns the cultural and aesthetic values which are embedded into every technology. Future research could seek a more transparent account of these values and how they influence designers. |
| Sectors | Creative Economy,Digital/Communication/Information Technologies (including Software),Education,Culture, Heritage, Museums and Collections |
| URL | http://instrumentslab.org/research/ |
| Description | Commercial: In 2016, we spun out a company Augmented Instruments Ltd to commercialise work coming out of my laboratory. Its flagship project is Bela, an open-source embedded hardware platform for creating musical instruments and other interactive systems, which is targeted at makers and artists who may not have a formal engineering background. Sales of Bela exceed £500k since its launch, and the company also receives income from several commercial consultancies. In 2018-19 the company led a £70k grant from Innovate UK on reaching new communities with interactive technology. In 2019, a capacitive touch sensor technology developed in our research was launched by Augmented Instruments Ltd in a Kickstarter crowdfunding campaign. Trill, as the sensor is know, raised over £25k on the campaign, and attracted significant interest from makers and artists worldwide. The company continues to grow organically and steadily through sales and consultancy. Artistic: my lab has hosted over 15 musical performance events since the start of the fellowship. These events have featured new instruments developed on the project alongside instruments developed by PhD students I supervise. Several performances have had invited external guest artists. A pre- and post-lockdown concert series at the Iklectik Art Lab in south London has been routinely sold out (capacity ca. 100), and these concerts have led to publicity in the Guardian and other media. Streaming concerts were held during the Covid lockdowns. Beyond that, the instruments created in my research have been adopted by a growing community of composers and performers. The magnetic resonator piano, an augmented acoustic piano that uses electromagnets to actuate the strings, has been performed dozens of times over the past 5 years. Pianist-composer Xenia Pestova Bennett has engaged in an extended project with the instrument that has included a commercial album released on Diatribe Records, performances in Germany, Ireland and the UK, and featured segments on BBC Radio 3. The instrument has also been used in a major motion picture film score (released 2018), two operas (2018-20), and jazz and chamber music albums released in 2020. An augmented violin created on the project in 2019 has been performed in several concerts and won third place in the Guthman Musical Instrument Competition at Georgia Tech (March 2020). An optical keyboard scanner developed on the project is now in regular use by several composers and pianists. Charity: We are increasingly working with third-sector organisations on public engagement, including the OHMI Trust, a charity dedicated to accessible music-making. The partnership with OHMI predates the award but has significantly strengthened in the past year to include several new activities including the organisation of a conference on "Music and Physical Disability" which took place in September 2018. QMUL has become a founding member of the OHMI Research Partnership (ORP), a collaborative effort to maintain an active research base on musical instruments for the physically disabled. I became a trustee of OHMI in 2021. We have also worked closely with the charity Heart n Soul, which provides community activities for children and adults with learning disabilities. This work is also supported by the EPSRC and AHRC CDT in Media and Arts Technology. PhD students in the lab have created new guitar-like instruments which were initially used to investigate research questions around sensorimotor skill and cultural form, but which have subsequently found regular use in Heart n Soul's musical activities for their community. We have worked with dozens of learning-disabled people and several facilitators. Following on from this, in 2019-2020, we have had a small research project funded by the Chelsea and Westminster Hospital Foundation (CW+) on the design of new instruments for stroke patients in a recovery ward. An initial design is now being tested with patients and clinicians. Other parters include the music charities Heart n Soul and Drake Music, the community interest company Conductive Music, and larger companies like Ableton (Berlin, Germany) in whose Loop Festival we have participated in in 2017, 2018. Ableton has gone on to fund further research in my lab on accessible music technology, considering use of music production software by visually impaired musicians. Other public engagement: We regularly host workshops and exhibitions on technology from the lab. The Bela embedded platform is often a focus of these events, which range from science fairs aimed at teens to industry-targeted technical workshops. I have given talks aimed at the general public at venues ranging from the IET to the Ars Electronica festival. We have also hosted artistic residencies by several composers, performers and designers, providing new interactive technologies for them to use in their projects and often co-organising performances or exhibitions. |
| First Year Of Impact | 2016 |
| Sector | Creative Economy,Digital/Communication/Information Technologies (including Software),Electronics,Culture, Heritage, Museums and Collections |
| Impact Types | Cultural,Societal,Economic |
| Description | Arts and Technology Pilot Programme |
| Amount | £24,987 (GBP) |
| Organisation | Innovate UK |
| Sector | Public |
| Country | United Kingdom |
| Start | 02/2017 |
| End | 10/2017 |
| Description | Bela / Royal Academy of Engineering Senior Research Fellow in Embedded Music Computing |
| Amount | £208,774 (GBP) |
| Funding ID | RCSRF2021\1248 |
| Organisation | Royal Academy of Engineering |
| Sector | Charity/Non Profit |
| Country | United Kingdom |
| Start | 03/2021 |
| End | 02/2026 |
| Description | Bridging the Gap - visually impaired and sighted music producers working side by side |
| Amount | £576,879 (GBP) |
| Funding ID | AH/V011340/1 |
| Organisation | Arts & Humanities Research Council (AHRC) |
| Sector | Public |
| Country | United Kingdom |
| Start | 09/2021 |
| End | 09/2024 |
| Description | Centre for Public Engagement Large Grants |
| Amount | £9,300 (GBP) |
| Organisation | Queen Mary University of London |
| Sector | Academic/University |
| Country | United Kingdom |
| Start | 01/2018 |
| End | 07/2018 |
| Description | Designing new musical technologies for older adults' wellbeing |
| Amount | £946,672 (GBP) |
| Funding ID | MR/T040580/1 |
| Organisation | Medical Research Council (MRC) |
| Sector | Public |
| Country | United Kingdom |
| Start | 02/2021 |
| End | 01/2025 |
| Description | InnovateUK 6478: Emerging and enabling round 3 |
| Amount | £100,000 (GBP) |
| Funding ID | 6478 |
| Organisation | Innovate UK |
| Sector | Public |
| Country | United Kingdom |
| Start | 04/2018 |
| End | 03/2019 |
| Description | Investigating Mental Models Of DAW Software Use By Visually-impaired Musicians |
| Amount | £97,000 (GBP) |
| Organisation | Ableton AG |
| Sector | Private |
| Country | Germany |
| Start | 04/2022 |
| End | 10/2023 |
| Description | Musical instruments for upper limb rehabilitation |
| Amount | £25,000 (GBP) |
| Organisation | CW+ |
| Sector | Charity/Non Profit |
| Country | United Kingdom |
| Start | 10/2019 |
| End | 03/2020 |
| Description | RUDIMENTS: Reflective Understanding of Digital Instruments as Musical Entanglements |
| Amount | £1,723,186 (GBP) |
| Funding ID | EP/X023478/1 |
| Organisation | United Kingdom Research and Innovation |
| Sector | Public |
| Country | United Kingdom |
| Start | 02/2023 |
| End | 01/2027 |
| Description | Aalborg University Copenhagen |
| Organisation | Aalborg University |
| Country | Denmark |
| Sector | Academic/University |
| PI Contribution | This collaboration with Prof. Dan Overholt at Aalborg University Copenhagen explores electronic augmentation of the violin for teaching and performance purposes. This relates closely to the theme of our research on skills transfer and designing for expertise. A postdoc, Laurel Pardue, worked 50% time at QMUL and 50% time at Aalborg funded by an internal grant from that university. |
| Collaborator Contribution | Aalborg University funded an 18-month 50% FTE postdoc position through its "International post-doc" funding scheme. The postdoc, Laurel Pardue, worked the other 50% time at QMUL. She was supervised at Aalborg by Prof. Dan Overholt. The broader context of this project is a collaboration between our research lab and the Nordic Culture Fund project "Acoustically Active Augmented Instruments" (AAAI), which brings together researchers from Norway, Finland, Denmark and Iceland. This project has also funded a visit by our team to the University of Oslo to deliver a workshop. |
| Impact | L. Pardue, A. McPherson and D. Overholt, "Improving the Instrumental Learning Experience through Complexity Management", Proceedings of the International Conference on Sound and Music Computing, 2018. L. Pardue, K. Buys, M. Edinger, D. Overholt and A. McPherson, "Separating sound from source: sonic transformation of the violin through electrodynamic pickups and acoustic actuation", Proc. New Interfaces for Musical Expression, 2019. Under review (journal): L. Pardue, D. Overholt and A. McPherson, "Effects of limiting aural feedback on intonation during violin performance". The project resulted in a new augmented violin, the svampolin, which uses digital processing to transform the sound of the violin in real time, for example changing its pitch or timbre. The instrument has been performed in several concerts and won third place at the Guthman Musical Instrument Competition at Georgia Tech in March 2020 (Atlanta, USA - https://guthman.gatech.edu). The Bela open-source embedded audio platform (a creation of our lab resulting from two EPSRC grants) has been used regularly on the AAAI project, for example in the following paper by other members of the project: V. E. Gonzalez Sanchez, C. P. Martin, A. Zelechowska, K. A. V. Bjerkestrand, V. Johnson, and A. R. Jensenius, "Bela-based augmented acoustic guitars for sonic microinteraction," in Proceedings of the international conference on new interfaces for musical expression, Blacksburg, Virginia, USA, 2018, p. 324-327. This is a multidisciplinary collaboration involving computer science, electronic engineering, musical performance and composition. |
| Start Year | 2017 |
| Description | Aalto University - Koray Tahiroglu |
| Organisation | Aalto University |
| Country | Finland |
| Sector | Academic/University |
| PI Contribution | In November 2018 I visited Aalto University in Helsinki, Finland, for a short research collaboration with Dr Koray Tahiroglu, an Academy Research Fellow in the university's School of Media. I presented a talk on my work, mentored students in a class on interactive system design, and we worked together on a paper related to the aesthetic influence of technologies in human-computer interaction. This collaboration continued across 2019, with a visit from Koray to QMUL and regular Skype meetings, resulting in a paper accepted to the journal Organised Sound, published early 2020. |
| Collaborator Contribution | Koray arranged and hosted my visit to Aalto and worked together with me on a journal paper. He led much of the data gathering for this research, including running an online survey of musical instrument designers. |
| Impact | McPherson, Andrew, and Koray Tahiroglu. "Idiomatic Patterns and Aesthetic Influence in Computer Music Languages." Organised Sound 25(1), 2020. |
| Start Year | 2018 |
| Description | CTAG - Kiel University of Applied Sciences |
| Organisation | Kiel University of Applied Sciences |
| Country | Germany |
| Sector | Academic/University |
| PI Contribution | We worked with the Creative Technologies working group (CTAG) at the Kiel University of Applied Sciences in the first half of 2018. We hosted a Master's student, Henrik Langer, and met regularly with his supervisor Prof. Robert Manzke. Langer and Manzke are working on embedded hardware for audio processing, and we assisted them in technical development using our knowledge from the Bela project (http://bela.io), our own open-source hardware platform for ultra-low-latency audio and sensor processing. |
| Collaborator Contribution | Henrik's visit was supported by his institution. They supplied expertise in Linux audio development and hardware prototyping toward an effort to develop a joint product for small scale commercial release. |
| Impact | The project resulted in the public commercial release of the CTAG FACE audio interface. This is a hardware board for multichannel audio I/O which can be used with embedded Linux systems or with Bela (http://bela.io). The project is now managed by QMUL spinout company Augmented Instruments Ltd, which sells the CTAG FACE products on its online shop (shop.bela.io). The CTAG boards have been used by engineers, researchers and artists worldwide for creating a variety of interactive audio projects. |
| Start Year | 2018 |
| Description | Intelligent Instruments Lab, Iceland University of the Arts |
| Organisation | Iceland Academy of the Arts |
| Country | Iceland |
| Sector | Academic/University |
| PI Contribution | Starting in 2021 I have working with the Intelligent Instruments Lab (IIL), a newly-founded research team at the Iceland University of the Arts, led by Prof. Thor Magnusson. We are collaborating on an ERC Consolidator Grant awarded to Prof. Magnusson ("INTENT") which focuses on artificial intelligence technologies in musical instruments. As part of the collaboration, IIL commissioned a magnetic resonator piano, an augmented acoustic instrument which comes from my previous research. I travelled to set up the instrument in Reykjavik as part of a research workshop in December 2021. |
| Collaborator Contribution | Members of IIL will be working with the magnetic resonator piano to create new compositions and performances, and to explore extensions of its technical capabilities. We also co-organised a workshop on embedded AI technologies for music at the 2022 International Conference on New Interfaces for Musical Expression, gathering short papers and presentations from authors around the world on the motivations and practicalities of using embedded AI in musical instruments. |
| Impact | Research workshop, 'Embedded AI for NIME: Challenges and Opportunities' (NIME 2022) Several joint papers under review as of March 2023 |
| Start Year | 2021 |
| Description | Mixed Reality Lab |
| Organisation | University of Nottingham |
| Department | School of Computer Science |
| Country | United Kingdom |
| Sector | Academic/University |
| PI Contribution | I have collaborated with members of the Mixed Reality Laboratory (led by Steve Benford) in Nottingham's computer science department on studies involving the D-Box hackable musical instrument. We have co-organised a series of free public workshops where people hack the D-Box instrument. I lead the workshop activities while the MRL researchers collect interviews and video footage as the event goes along, which we use for later analysis. Subsequently, we worked with MRL on a workshop at the STEIM electro-acoustic music centre in Amsterdam, using the Bela audio platform created in our lab. We led the technical aspects of the workshop, while MRL contributed to ethnographic study of how people used the technology. A paper on the results was accepted to CHI 2017. In 2019 we began preparing an ethnographic study of digital musical instruments in a community of learning-disabled individuals. Our team provided the instrument technology and the background in music and disability studies, while MRL contributed expertise in HCI and ethnography. This led to a late-breaking work paper at CHI 2019 and my participation in a CHI 2019 paper led by the University of Nottingham. In January-March 2020, I hosted PhD student Juan Martinez Avila, who has been working on augmented guitar research, and collaborated on a workshop on somaesthetic design of musical instruments. This led to a paper at the New Interfaces for Musical Expression 2020 conference ("Soma Design for NIME"). |
| Collaborator Contribution | MRL contributed their Artcodes research to the project. Artcodes are a form of information encoding in images based on patterns of open and closed shapes; notably, nearly any physical design can be turned into an artcode through careful management of light and dark regions. Artcodes are useful for linking physical objects to their virtual records in an aesthetically elegant manner. MRL also led the data collection and transcription for the workshops. We jointly prepared papers on the results. In 2019, MRL led work on studying the practice habits of musicians for a paper on which I am a co-author, and has contributed expertise on somaesthetic design and ethnography which has informed our joint work. |
| Impact | A. McPherson, A. Chamberlain, A. Hazzard, S. McGrath and S. Benford. Designing for Exploratory Play with a Hackable Digital Musical Instrument. Proc. DIS, Brisbane, Australia, 2016. F. Morreale, G. Moro, A. Chamberlain, S. Benford and A. McPherson. Building a Maker Community around an Open Hardware Platform. Proc. CHI, Denver, USA, 2017. J. Harrison, A. Chamberlain and A. McPherson. Accessible Instruments in the Wild: Engaging with a Community of Learning-Disabled Musicians. Proc. CHI Late Breaking Work, Glasgow, UK, 2019. A. Hazzard, C. Greenhalgh, M. Kallionpaa, S. Benford, A. Veinberg, Z. Kanga and A. McPherson. Failing with style: designing for aesthetic failure in interactive performance. Proc. CHI 2019. J. Martinez Avila, V. Tsaknaki, P. Karpashevich, C. Windlin, N. Valenti, K. Höök, A. P. McPherson and S. Benford. Soma Design for NIME. Proc. New Interfaces for Musical Expression, 2020. |
| Start Year | 2015 |
| Description | OHMI |
| Organisation | One-Handed Musical Instrument Trust |
| Country | United Kingdom |
| Sector | Charity/Non Profit |
| PI Contribution | We have contributed to the design of new musical interfaces accessible to musicians who lack the use of a hand or arm, following OHMI's core mission to enable "full and undifferentiated participation" in musical performance. I continue to serve as a judge on the annual One-Handed Musical Instrument competition and presented at the Ars Electronica festival where their awards were held in 2014. In 2018 I was one of the paper chairs for the conference Music & Physical Disability: From Instrument to Performance (Birmingham, UK, September 2018). Since that time I have been involved in discussions for creating a new OHMI Research Partnership to develop original research and translational activities to increase the accessibility of musical performance to people with physical impairments. I continue to advise OHMI on new technologies for accessible music performance. |
| Collaborator Contribution | OHMI has provided guidance on design principles, access to professional instrument builders and feedback from teachers and students. They have provided financial support for ordering parts for PhD student projects. Each year OHMI hosts a PhD student on the EPSRC and AHRC Centre for Doctoral Training in Media and Arts Technology. I am the academic supervisor for those placement projects. OHMI is a partner on my EPSRC Fellowship (EP/N005112/1), providing expertise on performance and education, and connections to instrument makers and other practitioners in the field of music and disability. |
| Impact | As a result of earlier collaborations, OHMI is now a project partner on my EPSRC Early Career Fellowship. The collaboration is multi-disciplinary: OHMI is primarily an arts and cultural organisation which also contributes to policy on music education for the disabled (e.g. through a 2017 House of Lords event hosted by the Lord Lipsey). This collaboration crosses from these areas over to science and engineering. Other outcomes include: several awards from OHMI for student projects I supervise, including Duncan Menzies (one-handed bagpipe chanter, 2017); Jacob Harrison (adapted bass, 2017). Duncan's chanter has gone on to be featured on BBC and Radio Scotland. In September 2018, I helped organise a conference Music & Physical Disability: from Instrument to Performance, contributing on the academic programme. OHMI acted as central organisers of the conference, with further assistance from researchers at Birmingham City University. The conference brought together researchers, teachers, artists, musical instrument makers and industrial partners from around the world to start an important conversation around the creation of accessible instruments for musicians with physical disabilities. |
| Start Year | 2014 |
| Description | Princeton Global Network |
| Organisation | Princeton University |
| Country | United States |
| Sector | Academic/University |
| PI Contribution | We are participants in the Princeton Global Network for New Electronic Musical Instrument Design, an international partnership supported by internal Princeton University funding which brings together designers and musicians around the world. Other participating universities include Stanford (USA), UCSD (USA), Goldsmiths University (UK), Aalborg University Copenhagen (Denmark) and the University of Gothenburg (Sweden). The network launched recently and we will be attending seminars and hosting residencies, collectively aimed at the design and performance of new musical instruments. |
| Collaborator Contribution | The network is funded by Princeton University and led by Dr Jeff Snyder and Prof Dan Trueman in the music department. $125k in total funding is provided, including support for travel and accommodation for us to attend events at Princeton and other locations. The project runs from 2018 to 2021. |
| Impact | Covid disruption meant that many of the planned meetings weren't possible, but plans have been made to develop the network further in the future. |
| Start Year | 2018 |
| Description | Research visit by D. Andrew Stewart |
| Organisation | University of Lethbridge |
| Country | Canada |
| Sector | Academic/University |
| PI Contribution | From February to May 2018, performer and researcher D. Andrew Stewart, a professor in music at the University of Lethbridge (Canada), visited the Centre for Digital Music at Queen Mary University of London. He was hosted by Andrew McPherson, leader of the Augmented Instruments Laboratory, a 10-person research team specialising in digital musical instruments. |
| Collaborator Contribution | During his residency, D. Andrew Stewart developed compositions on digital musical instruments, one of which he performed at a concert on 11 May 2018 which was organised by the lab and held in the Iklectik art space in south London. He gave a seminar and met regularly with PhD students and postdocs to discuss their research, providing advice from a musician's perspective on how to promote good performance practice on digital instruments. |
| Impact | Performance at Iklectik Art Lab, 11 May 2018, described in Artistic Outputs section. Possibility for future joint papers, for example at the New Interfaces for Musical Expression conference. This was a multi-disciplinary collaboration, combining music, engineering and human-computer interaction. D. Andrew Stewart is based in a music department, and we are based in an engineering and computer science department. |
| Start Year | 2018 |
| Title | Bela |
| Description | Trademarks granted in UK and the USA for "Bela". Bela is an open-source embedded platform for audio and sensor processing. The trademark has been granted within the UK the USA in related areas including computer equipment and peripherals. |
| IP Reference | |
| Protection | Trade Mark |
| Year Protection Granted | 2017 |
| Licensed | Yes |
| Impact | Licensed to spinout company Augmented Instruments Ltd, whose activities are reported in a separate section. |
| Title | Multi-Touch Piano Keyboard |
| Description | A musical keyboard touch sensor pad having a plurality of integrated sensors disposed in an array is disclosed. The plurality of integrated sensors represents a keyboard key. A processor is electronically coupled to the plurality of integrated sensors. The processor is programmed to receive signals from several of the plurality of integrated sensors and to generate a sound signal based on input from the several of the plurality of integrated sensors. The output is variable based on the number in the plurality and the location of the plurality in the array. An audio output device is electronically coupled to the processor to generate a sound based on the sound signal. A keyboard using a plurality of the sensor pads as well as a method of generating sounds from the keyboard are also disclosed. |
| IP Reference | US2014083281 |
| Protection | Patent granted |
| Year Protection Granted | 2014 |
| Licensed | Yes |
| Impact | In 2015, a spinout company TouchKeys Instruments Ltd was formed to commercialise the TouchKeys musical instrument. The company focuses on direct consumer sales of kits and keyboards plus the exploration of commercial licensing opportunity to established keyboard manufacturers. In December 2016, a new TouchKeys website and online shop launched to the public, which has generated sales and media attention. |
| Title | Bela Audio Platform |
| Description | Bela (formerly called BeagleRT) is an ultra-low-latency real-time audio platform for the BeagleBone Black embedded computer. It can be used to build musical instruments, including the D-Box hackable musical instrument developed for the EPSRC Hackable Instruments project. The software is of particular interest to audio and music researchers, especially those building real-time systems. With less than 1ms of latency between action and sound, Bela performs faster than any other computer-based environment on the market, including high-spec laptops. It also features audio-rate sampling of every analog and digital input which makes design of sensor systems convenient. In 2015, a built-in browser-based IDE was added along with support for the Pure Data graphical computer music language widely used in the digital music community. Since 2016, we have continued to add support for other programming languages and hardware accessories, as well as extending the documentation and online resources available to the community. |
| Type Of Technology | Software |
| Year Produced | 2014 |
| Open Source License? | Yes |
| Impact | Bela has gained significant traction amongst researchers, musicians and hobbyists in the 2014-17 period, with the number of users and community members growing every month. On 29 February 2016, we launched a Kickstarter crowdfunding campaign for Bela, seeking £5k to build and distribute the hardware. We exceeded our funding goal in less than 4 hours (out of a 32-day campaign), ultimately raising £55k from over 500 backers. Following the campaign, we received invitations to collaborate from major academic institutions (IRCAM in France, STEIM in the Netherlands, University of Virginia, University of Edinburgh, University of Glasgow, Goldsmiths University and several others) and industry (Cycling74, makers of the popular Max/MSP software). Meanwhile, since 2014 we have held 10+ workshops on Bela and/or the D-Box instrument including 2 workshops at the Sonar music festival, a workshop at the NIME 2015 conference (Baton Rouge, LA, USA), an Audio Music Hackathon sponsored by Harman Audio, and an Accessible Music Hackathon with the charity Drake Music (held at QMUL). Following the successful Kickstarter campaign, we spun out a new company, Augmented Instruments Ltd, to commercialise the platform, and relaunched a public web shop in October 2016. We maintain several community resources, including a blog, wiki, web forum and several social media accounts. Our users actively contribute to these resources and frequently share projects they make using Bela. Community development itself has become an area of research for us (e.g. Morreale et al., "Building a maker community around an open hardware platform", Proc. CHI 2017), and we have received a grant from Innovate UK (Feb-Oct 2017) to further develop the platform and build our community. For more information see the entry for spinout company Augmented Instruments Ltd. |
| URL | http://bela.io |
| Title | Trill touch sensors |
| Description | In 2014, as part of the EPSRC project Hackable Instruments (EP/K032046/1), I created a new capacitive touch sensor as part of a musical instrument called the D-Box. The sensor measured the position of up to 5 fingers along a single linear axis. Later, on the Design for Virtuosity project (EP/N005112/1), I and others in my lab further developed the touch sensor technology, using it in other instruments and producing expanded designs that sensed in two axes. In 2019, as part of an Innovate UK funded project in collaboration with QMUL spinout Augmented Instruments Ltd, this technology was developed to the point of commercial readiness. While capacitive touch position sensing is commonly found in consumer devices, the design of these sensors has often been limited to just a few experts. By contrast, these touch sensors are designed to be easy to use for makers, artists and researchers who are not expert embedded engineers. Accordingly, a set of libraries and example materials for popular maker platforms such as Arduino and Bela were developed. |
| Type Of Technology | New/Improved Technique/Technology |
| Year Produced | 2019 |
| Impact | The technology has been licensed to QMUL spinout Augmented Instruments Ltd, which also commercialises the Bela embedded platform which has come out of my research. The touch sensors have been branded "Trill" and are available in five formats: Bar (a one-axis slider), Square (a two-axis touch pad), Hex (two-axis tileable hexagons), Ring (a circular slider) and Craft (individual pins exposed for designers to make their own interfaces). The company ran a successful Kickstarter campaign in September 2019, raising £25k toward the production of Trill sensors. Shipping commenced in April 2020 and sales have continued at a steady pace since then. In March 2021, the company introduced Trill Flex, based on a flexible PCB design I co-created during the time of the Kickstarter campaign. In January 2023, the company launched Gliss, a synthesiser module featuring the Trill technology, raising over $70k on the crowdfunding site Crowd Supply. |
| URL | http://bela.io/trill |
| Company Name | Augmented Instruments Ltd |
| Description | Augmented Instruments Ltd is a spinout from the Centre for Digital Music at Queen Mary University of London, incorporated on 8 September 2016. The aim of the company is to develop and commercialise digital musical instrument research developed in the Augmented Instruments Laboratory, a research team within the Centre for Digital Music led by Dr Andrew McPherson. Technology is licensed from QMUL to Augmented Instruments Ltd, which develops and sells direct to consumers and conducts business consultancy related to the technologies. The company's flagship product is Bela (http://bela.io), an open-source embedded computing platform for ultra-low-latency processing of audio and sensor data. Bela, originally known as BeagleRT, was developed starting in 2014 in the Augmented Instruments Laboratory. Since its creation for a specific musical instrument, it has gradually grown into a general-purpose platform aimed at the maker and musician communities. It couples high performance (1GHz processor, submillisecond latency, high-bandwidth, synchronous gathering of audio and sensor data) with ease of use (a browser-based development environment and support for popular computer music programming languages). On 1 April 2016, Bela completed a highly successful Kickstarter campaign which raised £54,902 (out of an original goal of £5k). This provided the seed funding to spin out the company, which has subsequently been supported by sales of Bela and industrial consultancy. |
| Year Established | 2016 |
| Impact | Augmented Instruments Ltd launched Bela for public sale in October 2016. Its annual turnover was around £200k in 2021. The company is profitable without the need for investment funding. As of early 2022, it employed two full-time staffs member and several part-time positions. Income came from sales and from several industrial consultancy projects. In the second year, the secure two grants from Innovate UK. It launched a new product, Bela Mini, in February 2018. It also continued to support a growing open-source community, providing resources including a repository of code and hardware designs, a tech support wiki, an online discussion forum, a blog featuring user-created projects, a YouTube channel and other social media links. As of early 2022, these resources continue to grow in scope and audience. We estimate that 3000+ unique users on 6 continents have acquired their own Bela boards since the project began. Bela sales to date exceed £600k. We studied the characteristics of our early adopter community in [3.1]. It contains a diversity of profiles and interests: makers (i.e. technology hobbyists and independent creators); musical performers; musical instrument and synthesiser designers; interactive artists; teachers; students; researchers in fields ranging from human-computer interaction to digital signal processing to neuroscience. As of early 2019, AIL also had around 200 institutional customers, including universities (45%), companies ranging from SMEs to large multinationals (25%), design and art consultancies (17%) and arts/maker organisations (6%). Bela is being used for teaching in 20 universities beyond QMUL including Aalborg University Copenhagen (Denmark), University of Kiel (Germany), University of Virginia (USA), Griffith University (Australia) and NTNU (Norway). Since 2016, Bela has been used in dozens of public workshops across Europe, North America and Australia, including events at maker spaces, museums, trade shows, music festivals, conferences, universities, secondary schools, and many other public venues. Many of these were led by the Bela creators (first at QMUL, then also at Augmented Instrumenst Ltd), and some are led spontaneously by our community. Total participation across all events is hard to precisely quantify, but is easily in excess of 1500 people. Bela maintains a committed and enthusiastic online community who contribute back to the project through porting software tools (e.g. computer music languages SuperCollider, Csound, FAUST, pyo) and through writing up and sharing projects built with Bela. In 2019, the company completed another successful Kickstarter campaign for Trill, a new type of capacitive touch sensor aimed at makers and artists. The campaign raised £25k and sales have continued since then. In January 2023, the company launched Gliss, a synthesiser module featuring the Trill technology, raising over $70k on the crowdfunding site Crowd Supply. As of 2023, the company continues to grow steadily based on sales of Bela and Trill products and industrial consultancy services, and the Bela community continues to expand in industry, education, research and creative practice. |
| Website | http://bela.io |
| Description | 2017-07-19 Workshop on Digital Musical Instrument Design at Queen Mary University of London |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Industry/Business |
| Results and Impact | 20-30 people attended this workshop about designing digital musical instruments at Queen Mary University of London |
| Year(s) Of Engagement Activity | 2017 |
| Description | Absurd Musical Instrument Hackathon at QMUL |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | On 8-9 November 2019, our lab hosted "Unuseless Music Designs: A Seriously Absurd Hackathon". This two-day event was led by QMUL PhD student Giacomo Lepri with support from other members of my lab. A call for participants led to applicants from three continents (Europe, N. America, S. America) and about 30 participants. The two-day event included talks from invited mentors John Bowers (Professor of Creative Practice at Newcastle University) and Hannah Perner-Wilson (a leading practitioner of e-textiles based in Berlin) plus short introductory workshops on e-textiles and the Bela embedded audio/sensor platform. The main part of the event was the open development of absurd new musical interfaces, which also served a serious purpose as an investigation and critique of the values found in the musical instrument community. The event concluded with short presentations from each of the designers which were videoed and shared online. Participants expressed interest in developing their ideas further following the event, and discussions began about hosting a second similar event in the near future. One immediate outcome was the submission of a workshop proposal to the New Interfaces for Musical Expression 2020 conference, which was accepted and subsequently won the best workshop award. |
| Year(s) Of Engagement Activity | 2019 |
| URL | http://instrumentslab.org/research/absurd-music-design.html |
| Description | Accessible Music Hackathon |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Third sector organisations |
| Results and Impact | On 6 February 2016, my lab and the charity Drake Music jointly organised an Accessible Music Hackathon, a daylong event for creating new music technology for the disabled. The event was held at QMUL; it was free and open to the public, and advertised to Drake Music's community of supporters as well as local interest group mailing lists (e.g. Music Hackspace). At least 35 people attended (filling the venue to capacity) often working in small groups. At least 16 working projects were demonstrated at the end of the day. |
| Year(s) Of Engagement Activity | 2016 |
| Description | AudioMostly paper sensor workshop |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Postgraduate students |
| Results and Impact | On 17 September 2019, Jack Armitage (PhD student in my lab), Becky Stewart (Lecturer in the Design School at Imperial College London) and I led a workshop on building paper sensors with Bela, the open-source embedded platform developed in my lab which simplifies the creation of rich and responsive interactive systems. The paper sensor activity was developed by Becky Stewart as part of an Innovate UK grant led by Augmented Instruments Ltd, a spinout company from my research. Compared to traditional electronic assembly using breadboards or printed circuit boards, paper and copper tape can make circuits much more rapidly, building on skills that most people already have. It is therefore an ideal medium to introduce electronic making to broader audiences. |
| Year(s) Of Engagement Activity | 2019 |
| URL | https://audiomostly.com/2019/program/workshop-program/ |
| Description | Augmented Instruments Lab at Hackoustic Village |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Public/other audiences |
| Results and Impact | We presented a series of demonstrations and a talk at the We Are Robots Festival in London on 2 November 2017. The event was part of a larger exhibition called the Hackoustic Village, named after the artists collective Hackoustic. The event drew a large audience, several follow-up visits to our lab and plans for further work together. A visit from Hackoustic members to the lab is scheduled for late spring 2018. |
| Year(s) Of Engagement Activity | 2017 |
| URL | http://hackoustic.org/hackoustic-village-at-we-are-robots/ |
| Description | Bela and BBC Audio R&D |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Industry/Business |
| Results and Impact | On 15 October 2019, I led a team presenting an invited workshop for the BBC Audio R&D group. The workshop introduced Bela, an open-source embedded platform coming out of my research which has industry-leading low-latency audio performance. In comparison to other workshops we have run which focus on makers and the general public, this workshop was aimed at a more technical audience of industry developers and included lower-level programming languages such as C++. The workshop led to ongoing interest in Bela amongst the BBC team. |
| Year(s) Of Engagement Activity | 2019 |
| Description | Bela and TouchKeys at Superbooth |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Industry/Business |
| Results and Impact | In April 2017, we presented two research spinout projects at the Superbooth trade show in Berlin. Superbooth is an annual 3-day event aimed at makers and users of modular synthesisers and related music technology, which draws thousands of attendees each year. We hosted a booth featuring two projects: Bela, an open-source embedded platform for audio and sensor processing, which launched on Kickstarter in 2016 and subsequently spun out into a company, Augmented Instruments Ltd; and TouchKeys, a sensor kit which installs onto a piano-style keyboard to add capacitive touch sensing of the location of the fingers. The event also included a workshop where 15 people received a hands-on introduction to using Bela. The response to the event was very positive, with significant media publicity and many conversations with other industry professionals and members of the public. We plan to return for Superbooth 2018. |
| Year(s) Of Engagement Activity | 2017 |
| URL | http://2017.superbooth.com/en/ |
| Description | Bela at Big Bang Science Fair |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Schools |
| Results and Impact | On 12 July 2019, I and a PhD student (Andrea Guidi) presented an exhibition at the Big Bang Science Fair at Sutton Grammar School, Surrey. We presented Bela, an open-source embedded maker board developed in our lab which simplifies the design of powerful and versatile new digital musical instruments. The audience consisted mainly of schoolchildren (of a fairly wide age range, with teens predominant) plus teachers and other caregivers. The event also included participation by 10+ companies and se |
| Year(s) Of Engagement Activity | 2019 |
| Description | Bela at Heart n Soul charity events |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Public/other audiences |
| Results and Impact | Heart n Soul is a UK-based charity catering to children and adults with learning disabilities. Part of their activities involve a series of "SoundLab" events featuring music technology accessible to people with such disabilities. Each event includes participation from several companies or other music technology creators. In November 2016, we presented Bela (our low-latency embedded audio platform -- http://bela.io) at the Beautiful Octopus Club, a large-scale evening festival held at the London Southbank Centre. The SoundLab was part of the Beautiful Octopus Club event; over the course of an evening, we demonstrated several accessible musical instruments to a crowd of hundreds of people of all ages, both disabled and not. Later, in February 2017, we were invited to present Bela at another SoundLab event in South London, which reached an estimated 30 people. Following that event, we have made plans to meet further with some of the participants to better understand how to support high-level music making by people with disabilities. |
| Year(s) Of Engagement Activity | 2016,2017 |
| URL | http://www.heartnsoul.co.uk/category/taking_part/details/beautifuloctopusclub |
| Description | Bela at STEIM |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Study participants or study members |
| Results and Impact | In August 2016, my research lab organised a 3-day hands-on instrument design workshop at the Studio for Electro-Instrumental Music (STEIM) in Amsterdam. The workshop featured Bela, an open-source embedded platform for ultra-low-latency audio and sensor processing, which was developed originally as part of the EPSRC-funded Hackable Instruments project. The workshop had 8 participants who were digital musical instrument designers, plus several other people attending as observers. The event was facilitated by two technical organisers and two other researchers. The instrument designers each brought an existing instrument design which they ported to Bela with the help of the organisers. We recorded audio and video from the session, including regular interviews with each participant. This event led to a paper in CHI 2017 (Denver, USA) on community formation around open-source hardware tools. It also led to several participants using Bela in their own music and art installations at a wider scale. |
| Year(s) Of Engagement Activity | 2016 |
| URL | http://steim.org/event/bela-workshop-call/ |
| Description | Bela at The Raw and the Cooked event |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Public/other audiences |
| Results and Impact | On 29 September, I led a team of 4 people delivering two hands-on workshops featuring Bela (http://bela.io), an open-source embedded platform for audio and sensor processing which we developed in our research over the past several years. The workshops were part of an event entitled The Raw and the Cooked at Cafe 1001 on London's Brick Lane. The overall event was a 2-day festival of live electronic music and DIY music technology activities. I estimate that 40 people participated in the workshops and a further 20 observed for part of the time. The main result was an increase in interest and awareness in using Bela for creating interactive audio systems. |
| Year(s) Of Engagement Activity | 2018 |
| URL | https://therawandthecooked.intersections.io/cooked.html |
| Description | Bela workshops |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Public/other audiences |
| Results and Impact | On 4-5 December 2015 we held two tutorial workshops on the Bela low-latency audio platform. These were free and open to the public. Originally the plan was to hold a single event, but high demand resulted in it filling up quickly, so we scheduled a second one the day before. In all we estimate the attendance at 40-50 people. Feedback was positive and many people bought hardware from us to continue using Bela in their own projects. This event was one of several that laid the groundwork for our highly successful Kickstarter crowdfunding campaign in March 2016. Since the original workshop, we have held several additional workshops throughout 2016 and 2017, together reaching several hundred people. Events have taken place in London, Glasgow, Edinburgh, Brighton, Berlin, Toronto, Baton Rouge and Brisbane, among others. Some events have targeted hobbyists and makers; others have targeted musicians and industry professionals; still others have targeted undergraduate and postgraduate students. Collectively, these events have significantly expanded the profile of the Bela platform, leading to research collaborations, increased public profile of our work and commercial benefits (hardware sales through a spinout company). |
| Year(s) Of Engagement Activity | 2015,2016,2017 |
| URL | http://bela.io |
| Description | C4DM at Sonar MarketLab 2016 |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Industry/Business |
| Results and Impact | C4DM presented a public exhibition of its research at the Sonar+D 2016 festival in Barcelona, 16-18 June 2016. Sonar+D is a high-profile annual exhibition of music technology attended by thousands of industry professionals, musicians and members of the general public. C4DM was chosen by competitive application for a booth on the exhibition floor. We showed 9 recent research projects supported by EPSRC funding (EP/K009559/1, EP/K032046/1, EP/G03723X/1). The booth attracted significant interest from a broad cross-section of the Sonar audience: it was nearly full for most of the three days, and anecdotally appeared to have more traffic than most of the booths at the exhibition. In parallel with this in-person exhibition, Sonar+D featured C4DM in their social media feeds, and we were interviewed for a broadcast on Spanish television. Aside from public engagement, the exhibition led to several new industry contacts, and it provided valuable feedback to the attending PhD students and postdocs on how to showcase their research in public. The projects shown were: 1) Bela, an open-source platform for ultra-low-latency audio and sensor processing, which launched on Kickstarter in 2016; 2) TouchKeys, a transformation of the piano-style keyboard into an expressive multi-touch control surface, launched on Kickstarter in 2013 and spun out into a company in 2016; 3) Collidoscope, a collaborative audio-visual musical instrument which was a viral hit online with over 10M views; 4) RTSFX (Real-Time Sound Effects), an online library of real-time synthesised (rather than sampled) sound effects for a variety of different objects and environmental sounds; 5) Moodplay, a mobile phone-based system that allows users to collectively control music and lighting effects to express desired emotions; 6) Augmented Violin, a sensor-based extension of the traditional violin to give students constructive feedback on their playing 7) Tape.pm, an interactive object which explores novel ways to record and share an improvisation that is created on a musical instrument; 8) Aural Character of Places, an interactive online demo of soundwalks conducted around London; 9) MixRights, a demo of how content reuse is enabled by emerging MPEG standards, such as IM AF format for interactive music apps and MVCO ontology for IP rights tracking, driving a shift of power in the music value chain. Several potential collaborations may result from this visit, including commercial opportunities related to the TouchKeys and Bela projects, and possible joint projects with other university labs who attended (e.g. IRCAM and the Music Technology Group at Universitat Pompeu Fabra in Barcelona). The publicity surrounding this exhibition may also serve as a recruitment tool for future research students in the Music and Acoustic Technology research area. A particularly valuable outcome, especially for the PhD student attendees, was experience in showcasing their research to the public. Several students commented that they received valuable feedback on their projects and learned general skills in public engagement over the course of the three days. In one case (Tape.pm), interactions with the public helped generate a dataset which will be analysed in future research. Finally, attendees at Sonar+D also had the opportunity to see other booths and talks at the event, generating new ideas and connections for future projects. |
| Year(s) Of Engagement Activity | 2016 |
| URL | https://sonar.es/en/2016/ |
| Description | Guardian article on accessible musical instruments |
| Form Of Engagement Activity | A magazine, newsletter or online publication |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Public/other audiences |
| Results and Impact | Our lab's research was featured in a Guardian article about new music technology and the possibilities it brings to make musical performance more accessible to people with disabilities. The article generated significant interest in the lab and contacts from potential future collaborators. |
| Year(s) Of Engagement Activity | 2019 |
| URL | https://www.theguardian.com/careers/2019/jul/10/we-need-to-put-inclusion-at-the-start-of-the-process... |
| Description | Hackoustic Presentation |
| Form Of Engagement Activity | Participation in an open day or visit at my research institution |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Public/other audiences |
| Results and Impact | On 5 April 2018 I hosted a visit to my research lab by Hackoustic, an artist collective working on new musical instruments. I and other lab members showed our latest research to the attendees. The visit was subsequently written up in a blog post on the music industry site Kitmonsters. |
| Year(s) Of Engagement Activity | 2018 |
| Description | Heart n Soul SoundLab events |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Third sector organisations |
| Results and Impact | SoundLab events are run by Heart n Soul to connect music technology companies and research groups with members of the learning disabilities community. The Augmented Instruments Lab research group has attended several of these recurring events to showcase our instruments made using the Bela embedded controller which was also developed by members of the research group. |
| Year(s) Of Engagement Activity | 2017,2018 |
| Description | IET Christmas Lecture at QMUL |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Schools |
| Results and Impact | On 14 December 2016, I gave the annual IET Christmas Lecture in the School of Electronic Engineering and Computer Science at Queen Mary University of London. The event was entitled "The Sound of Computing" and discussed how computing and electronics could be used to augment familiar musical instruments. The talk was pitched to schools and was attended by students of a wide range of ages. It led to further invitations to speak at IET and other events, and an invitation to develop a school engineering design challenge with IET, which is planned for 2018. |
| Year(s) Of Engagement Activity | 2016 |
| URL | http://www.eecs.qmul.ac.uk/news/view/music-transformed-by-technology-at-qmuls-christmas-lecture |
| Description | Lab visit from Heart n Soul charity |
| Form Of Engagement Activity | Participation in an open day or visit at my research institution |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Third sector organisations |
| Results and Impact | On 23 October 2019, our lab (the Augmented Instruments Laboratory, part of QMUL's Centre for Digital Music) hosted a visit from the SoundLab at the Heart n Soul charity. Heart n Soul are an award-winning creative arts company and charity that work with young people and adults with learning disabilities. The SoundLab project is an innovative digital music making project, with goal of exploring ways of using music technology to encourage people with learning disabilities to create music and sound experiences. The visit included several young people with disabilities and several caregivers, who spent around 2 hours trying out new instruments created in our research. Some of these instruments have been deployed in Heart n Soul's own SoundLab sessions as part of the PhD research of Jacob Harrison. |
| Year(s) Of Engagement Activity | 2019 |
| URL | http://instrumentslab.org/news/2019/10/23/SoundLabVisit.html |
| Description | Magnetic resonator piano at Hearing Colour event, Being Human Festival |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Public/other audiences |
| Results and Impact | On 20 November 2021, the magnetic resonator piano (MRP), an augmented acoustic piano which was a product of my research, was featured in a performance at the Iklectik Art Lab in south London. The event, entitled 'Hearing Colour', was organised by Dr Charalampos Saitis as part of the Being Human Festival. It featured an MRP performance by pianist Xenia Pestova Bennett accompanied by a mobile app and projection system developed by Dr Saitis's students in which the audience could indicate sound-colour associations in real time during the performance. The performance was followed by a panel discussion including C Saitis, A McPherson and X Pestova Bennett. |
| Year(s) Of Engagement Activity | 2021 |
| URL | https://iklectikartlab.com/hearing-colour-sonic-worlds-and-other-senses/ |
| Description | Magnetic resonator piano at Keyboard Evolution |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Public/other audiences |
| Results and Impact | On 19-20 November 2018, I held a workshop on new keyboard instruments as part of an event at Queen Mary University of London entitled Keyboard Evolution (organised by QMUL music director Paul Edlin). At this workshop I gave presentations and hands-on demos of two pieces of keyboard-related research I have done: the magnetic resonator piano, an electromagnetically-augmented acoustic grand piano, and TouchKeys, capacitive touch sensors which transform the keyboard into a multi-touch control surface. The workshop was followed by an evening performance by pianist and composer Rolf Hind, featuring music for magnetic resonator piano by Hind and by Paul Edlin, amongst other pieces on the programme. |
| Year(s) Of Engagement Activity | 2018 |
| Description | Magnetic resonator piano on BBC Radio 3 |
| Form Of Engagement Activity | A broadcast e.g. TV/radio/film/podcast (other than news/press) |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Public/other audiences |
| Results and Impact | As part of the release of a new album "Atomic Legacies" by pianist-composer Xenia Pestova Bennett, the magnetic resonator piano, a new instrument created in my research, was featured on the Music Matters programmed on BBC Radio 3. A segment entitled "The secret life of musical instruments" broadcast on 22 February 2020, featuring an interview with Xenia and with lab researcher Giulio Moro, plus clips of the instrument. Tracks from Xenia's new album have also been broadcast on Radio 3 on 18 January 2020 and 4 February 2020. A further broadcast on the MRP on Radio 3 occurred on 26 April 2020. |
| Year(s) Of Engagement Activity | 2020 |
| URL | https://www.bbc.co.uk/programmes/m000fnc8 |
| Description | Magnetic resonator piano on Radio Eins (Berlin) |
| Form Of Engagement Activity | A broadcast e.g. TV/radio/film/podcast (other than news/press) |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Media (as a channel to the public) |
| Results and Impact | In February 2019, my augmented instrument research, and in particular the magnetic resonator piano (an augmented acoustic grand piano I developed) was featured on the "Strom und Drang" science programme in Radio Eins Berlin. The piece includes an interview with me (translated to German). |
| Year(s) Of Engagement Activity | 2019 |
| URL | https://www.radioeins.de/programm/sendungen/modo1316/strom-und-drang/augmented-instruments-laborator... |
| Description | Music and Ideas: Developing Repertoire for New Instruments |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Postgraduate students |
| Results and Impact | On 23 November 2017 I presented a talk and concert on the magnetic resonator piano, an electromagnetically augmented acoustic piano I continue to develop, at the Royal College of Music in London. The event featured a performance of a new piece by composer Jonathan Pitkin performed by pianist Kate Ryder. The event led to increased awareness of augmented instrument research among students and staff of a top UK conservatoire. Plans are forming to repeat the programme in other venues later. |
| Year(s) Of Engagement Activity | 2017 |
| URL | http://www.rcm.ac.uk/events/listings/details/?id=1272862 |
| Description | OHMI Awards 2017 |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Professional Practitioners |
| Results and Impact | On 1 July 2017 we participated in the annual OHMI Awards, presented by the OHMI Trust, a charity devoted to musical instruments accessible to people with physical disabilities. Each year, OHMI runs a competition to identify and create adaptations of traditional instruments which can be played by disabled musicians. In 2017, we won two awards from OHMI, one for Duncan Menzies' PhD research project on digital bagpipes, which he adapted for one-handed playing, and other on an adaptation of the electric bass by PhD student Jacob Harrison. The event was attended by teachers, students, members of other charities, policymakers interested in disability, and members of the public. Following the event, Duncan Menzies has been featured on BBC and Radio Scotland for his work teaching the bagpipes to children who only have the use of one hand. |
| Year(s) Of Engagement Activity | 2017 |
| URL | http://www.ohmi.org.uk/previous-winning-instruments.html |
| Description | Online talk at Music Hackspace |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Public/other audiences |
| Results and Impact | On 22 June 2020, I gave an invited live-streaming presentation hosted by Music Hackspace (musichackspace.org). I gave a presentation followed by Q&A by Music Hackspace founder Jean-Baptiste Thiebaut to an estimated audience of 50-100 live viewers and >500 total subsequent views on YouTube. |
| Year(s) Of Engagement Activity | 2020 |
| URL | https://www.youtube.com/watch?v=6I-tsINIe_w |
| Description | Presentation at Aalto University Helsinki |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Postgraduate students |
| Results and Impact | On 29 November 2018 I gave an invited talk on my research to academics and postgraduate students at Aalto University (Helsinki). Following the talk I spent two days advising individual students on creative projects they were working on in the area of interactive systems. The students were on a course taught using Bela (bela.io), an open-source embedded platform developed in my research starting in 2014 and spun out into a company in 2016. |
| Year(s) Of Engagement Activity | 2018 |
| Description | Presentation at ReSounds symposium |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Undergraduate students |
| Results and Impact | In February 2022, I gave an online talk about digital musical instrument design at ReSounds, a two-day symposium organised by researchers at Cornell University (NY, USA). The symposium featured invited speakers from North America and Europe around a central theme of new musical instruments. |
| Year(s) Of Engagement Activity | 2022 |
| Description | Presentation at the IET: Electronic Augmentation of Traditional Musical Instruments |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Professional Practitioners |
| Results and Impact | On 15 December 2017, I gave an invited talk on my research at the IET in London. The talk covered how technology could be used to extend the capabilities of traditional musical instruments. The audience consisted of members of the public, especially professionally practicing engineers. It was eligible for CPD credits under the IET's CPD monitoring scheme. The talk received significant interest from the attendees and several follow-up proposals afterward, including an invitation to guest edit a journal special issue (deferred to a later date because of time commitments). |
| Year(s) Of Engagement Activity | 2017 |
| URL | http://theiet.org/events/local/249080.cfm |
| Description | Presentation at the International Symposium on Performance Science, Reykjavik |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Postgraduate students |
| Results and Impact | Laurel Pardue (PDRA on Design for Virtuosity) presented a talk on an augmented violin for assisting learning at the International Symposium on Performance Science. The talk has led to an invited journal article currently under review. |
| Year(s) Of Engagement Activity | 2017 |
| URL | http://www.performancescience.org/ISPS2017/ |
| Description | Talk at the Lifecycle of Musical Instruments event |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Professional Practitioners |
| Results and Impact | Fabio Morreale (PDRA on the Design for Virtuosity project) presented a talk "Ephemeral Objects and Enduring Knowledge in the Creation of Digital Musical Instruments" at the Lifecycle of Musical Instruments event organised by the Musical Instruments Research Network, 12 October 2017. This event led to an emerging collaboration with the curator of a soon-to-open musical instrument museum at the Royal College of Music. We may contribute digital musical instruments to this collection or advise on their choice of other instruments. |
| Year(s) Of Engagement Activity | 2017 |
| URL | https://mirnblog.files.wordpress.com/2016/10/full-programme-abstracts-and-speakers.pdf |
| Description | Workshop and talk at Ars Electronica festival |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Public/other audiences |
| Results and Impact | On 8 September 2019, I presented a workshop at the annual Ars Electronica festival in Linz, one of the largest annual festivals of electronic art which takes over the whole city for a period of several days. Our workshop took place in the main exhibition building at the festival. It introduced Bela, the open-source embedded platform developed in our lab which simplifies the process of making rich and nuanced interactive systems, including new musical instruments. The workshop lasted a half day and included a series of example hands-on projects involving Bela. Following the workshop, I presented a talk (in the AIxMusic sub-stream of the festival) about the Centres for Doctoral Training hosted by QMUL: the EPSRC and AHRC Centre for Doctoral Training in Media and Arts Technology and the UKRI Centre for Doctoral Training in Artificial Intelligence and Music. Both events led to contacts and enquiries for further information on our work and the CDTs. |
| Year(s) Of Engagement Activity | 2019 |
| Description | YouTube lecture series on C++ audio programming |
| Form Of Engagement Activity | Engagement focused website, blog or social media channel |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Public/other audiences |
| Results and Impact | Beginning in March 2020, I created a free online course, "C++ Real-Time Audio Programming with Bela". The course comprises 20+ video lectures on YouTube with companion slides and code examples hosted on GitHub. The course introduces the fundamentals of programming real-time audio on embedded systems and builds up to progressively more advanced topics in audio signal processing. As of February 2021, the videos had collectively received 28,000 views and 1100 likes, with highly positive feedback in comments. |
| Year(s) Of Engagement Activity | 2020 |
| URL | https://github.com/BelaPlatform/bela-online-course |