Learn to Play: Computational Assessment of Musical Playability for Users' Practice

Lead Research Organisation: Goldsmiths College
Department Name: Computing Department

Abstract

This Follow-On Funding for Impact and Engagement proposal is based on research from the AHRC Digital Transformations project, 'Transforming Musicology' (AH/L006820/1), and the Electronic Corpus of Lute Music project, most recently as 'Lute Music in the Open (ECOLM III)', AH/H037829/1. It explores the concept of 'playability' of music. By developing a system to assess the difficulty of a displayed piece, and then using this system to create on demand a set of practice exercises based on passages within the music judged to be tricky by the system, it will help students learning to play an instrument (flute, guitar, or renaissance lute).

The guitar is the most widespread instrument in the world today, and the internet provides a literally bewildering number of 'tabs' (scores notated in the format known as tablature) requiring no formal knowledge of music notation. Tablature provides instructions about the placement of fingers to form chords or melodies and the sequence in which they should be played. It is a system that has stood the test of time, and has been used for hundreds of years, at least since the 15th century, and is particularly useful for instrumental teaching, especially in the early stages.

There is a vast amount of music available online and the system we create will help musicians find music to suit their skill level. The system will analyse the playability of tablature versions of pieces of music for guitar (classical and other styles) and for renaissance lute (we already have a corpus of c10,000 pieces in ECOLM). Using measures based on hand-stretches and position-shifts indicated in the tablature we'll compute indexes of playability of individual chords and transitions between them.

The flute is another very popular instrument among self-learners and young people, especially in schools; based on figures from the Hackney Music Service, we estimate that over 3,000 non-beginner flute students take lessons in London schools alone. We'll build on earlier work carried out by co-I Fiebrink on the modelling of difficulty in flute music, a very useful starting point, since the simpler texture of the music allows us to focus on its melodic aspects rather than on chords (as on guitar or lute).

We'll then use standard machine-learning techniques to build models of playability to identify difficult passages in unknown flute, guitar and lute pieces. They will also be used to grade pieces (based on the difficulty of the most technically-challenging passages) and the results compared with the grades listed by music publishers in their catalogues. The proof-of-concept demonstrator forming the main output of the project will then use simple algorithms to generate entirely new exercises derived from these passages for practising by a student.

All the above will be evaluated by our user community - players at various levels and flute, guitar and lute teachers.

The music will be presented within a high-quality graphical user interface provided by our music-industry partner, Tido Music. Currently used for a number of educational packages, mostly aimed at amateur pianists, it will be adapted to communicate remotely with the playability estimation and exercise generation back-end developed and maintained by Goldsmiths. This way our models can be tested from the outset with a professional user-interface, and use musical scores from the Tido music library (access restrictions entirely under Tido's control), or from elsewhere, without compromising rights ownership.

The lessons learned will be applied directly in two ways. We shall hold a workshop for professional and amateur musicians, including those involved as beta-testers, to discuss their assessment of the system with its designers and developers. This feedback will then be used as material for a full proposal to Innovate UK for funds to carry out further research and development to take this work beyond proof of concept to a commercially viable product.

Planned Impact

The guitar is the most popular instrument in the world today, and hundreds of thousands of players at every level learn and exchange music online. Millions of pieces can be found notated in tablature, an ideal system for learners (e.g., a single site, www.ultimate-guitar.com, has tens of thousands of regular users and offers 800,000 songs in tab), but finding those within a player's capability is a difficult and tedious task. This project aims to make that easier and to provide automatically-generated exercises for tricky passages in chosen pieces.

In musicological research carried out within the Transforming Musicology project and elsewhere, we have found that an important feature of historical repertories not hitherto studied extensively is the 'playability' of instrumental pieces (especially in arrangements), and we developed proof-of-concept measures to locate hard passages within pieces for renaissance lute notated in tablature. These can be applied to the vast modern repertory of music for guitar in tablature found on the internet. Co-I Fiebrink has also developed a method for grading pieces for flute (in normal staff-notation) according to their technical difficulty.

This proposal builds on that work in a system to help self-learning musicians, and their teachers, find music to suit their level of attainment from online resources, and - a feature unprecedented as far as we are aware - which generates a set of original practice exercises based on hard passages identified by the system.

Generating exercises from difficult passages of music will have a definite medium-term impact on the teaching and learning of the flute and the guitar (our principal focuses). Practising short exercises is far more enjoyable than struggling with complete pieces. Automated generation of original exercises based on a piece being studied (as are often written out by a teacher) provides a new and enjoyable experience for beginners and intermediate learners, and will help advanced students (up to conservatory level) make best use of their practice time.

Our music industry partner, Tido Music, have expressed a need to respond to a demand from their users for automated playability measurements of this sort. With them we shall be able to trial the method at an early stage of development, providing user feedback into the design process. We'll develop a demonstrator to show the potential for this approach which will be submitted to Tido's team of beta-testers. This is a basis for rapid development of a working system that will be offered as a service in conjunction with their educational materials. The 'grading' methods we use will enhance the experience of searching for music suitable for their level of ability for online customers of companies distributing such music. Once the proof-of-concept demonstrator produced here is developed to industry standards, and its capabilities extended to other instruments (especially the piano) it is likely to have a wide uptake among instrumental teachers as well as self-learning online customers.

We'll publish academic papers in the normal way throughout the project to ensure immediate academic impact for the project, further promoted by the use of posters and demos at music/computing, music education and musicology conferences. Our project workshop will report on activities, inviting interested communities and stakeholders to try the system and hear about our vision of how it offers a transformative enhancement for instrumental teaching yet preserves traditional standards and values.

This project seizes an opportunity presented by the application of a novel musicological concept developed in T-Mus (that of 'playability' as a determinant of historical repertoire) in a public-facing role with potential as a commercial venture. It thus further contributes to the stated belief of the T-Mus project that "Musicology need not - and should not - be an 'ivory-tower' discipline."

Publications

10 25 50
 
Description This Follow-On Funding grant mainly devoted to an exploration of the possibilities around the automated assessment of 'playability' of certain kinds of instrumental music. To this aim, user-experiences are paramount, and the main effort so far has been in the creation of a web-based user-interface allowing ranking, or more detailed evaluation, of the technical difficulty of individual chords, sequences of chords, and more diverse passages in music that can be expressed or represented in the form of tablature, a graphical notation system used for the renaissance lute and - to a far greater extent - for the guitar via internet downloads. This interface will very shortly be released online so that we can rapidly acquire human judgements (by expert professional to self-taught amateur players) of a test set of pieces. The human judgements thus gathered will then be compared with the automated assessments we can already make on the same music to see to what extent the latter is realistic. As such we are not in a position yet to announce 'findings' per se, as these will only become apparent during the next phase of the research (April-July 2018). In our bid we did not propose to publish a great deal from the project, as we regard it as a scoping exercise, hoping to spend some effort during the same period to writing a proposal for enterprise funding from Innovate UK with our partners, Tido Music; this will begin shortly. However, we have one paper accepted (Music Encoding Conference 2018, Maryland Univ, US, May 22-25) and intend to submit at least one more (possibly a poster) based on this work during the project duration or shortly afterwards.
Exploitation Route Since we have not yet published anything, the work cannot be taken forward by others immediately. However, in our active discussions with musical colleagues (local, national and international) we are beginning to see raising interest in the concept of 'playability' as a feature of instrumental music which has hitherto been discussed in completely subjective terms, yet which can - albeit to a limited extent - be described to some extent objectively. We expect that this will be taken further by ourselves and others to explore research questions such as: 'To what extent did playability affect the choices of repertory in historical publications, either for didactic or leisure use?' or 'How might objective measures of playability be enhanced and refined in future?' or 'Can the use of advanced techniques such as video-analysis or movement-capture contribute to a more robust automatic playability assessment process?'
Sectors Digital/Communication/Information Technologies (including Software),Education,Leisure Activities, including Sports, Recreation and Tourism,Culture, Heritage, Museums and Collections,Other

 
Description A web interface presenting chords in tablature for the renaissance lute and modern guitar has been developed. This involved several innovative aspects which will find further use in other projects such as the EU Horizon 2020-funded TROMPA, in which the PI acts as the UK PI. Data collected in the crowd-sourcing operation this enables will be used in upcoming research papers and conference presentations for the Music Encoding Conference, and ISMIR 2019. These will be submitted in due course as Publications on ResearchFish We have developed ways to parse the informally-structured tablatures submitted in their hundreds of thousand by enthusiasts to various web-sites, so that they can be analysed using the software framework Music21, developed at MIT, with whole we shall be collaborating in the future. Our data-analysis will permit the building of models of the relative difficulty of pieces of music in these formats. This will be used by commercial music-publishers, such as our partner, Tido Music, to help customers find music suitable for them to play.
First Year Of Impact 2018
Sector Digital/Communication/Information Technologies (including Software),Education,Leisure Activities, including Sports, Recreation and Tourism,Culture, Heritage, Museums and Collections
Impact Types Cultural,Societal,Economic