An Integrated Audio-Symbolic Model of Music Similarity

Similarity is a central aspect of music. It can help as a benchmark for testing theories of music perception and cognition, it is useful to answer for practical applications, such as music retrieval and recommendation, and it is an overarching question that relates to many aspects of music. However, theories of musical similarity have mostly focused on symbolic representations, where musical structures such as melodic and harmonic development are addressed but the aspects or expressive performance, such as micro-timing and timbre are ignored. On the other hand, audio based models, typically distances based on audio features, can capture details of the performance such as timbre, dynamics and tempo changes, but little of the musical structure as it unfolds over time.

Recent progress in audio transcription and alignment and the availability of music analysis tools for music collections with audio and symbolic content, which are being developed in the Digital Music Lab (DML) project, are changing the landscape of research in music. Large datasets of acoustic and symbolic music data that are generated or unveiled through the DML and "Optical Music Recognition from Multiple Sources" Big Data projects encourage an approach that can combine symbolic and audio based analyses into a joint similarity mode. This opens a great potential to new tools for research in music information retrieval and musicology, as the interaction between symbolic structure and acoustic information such as timbral texture has rarely been addressed and it could reveal aspects that have been unnoticed or unexplained so far. If successful, it might contribute to breaking the glass ceiling in music recommendation.

The aim of this project is to develop an initial framework and conduct experiments on an integration of symbolic melodic and structural similarity models with audio based models. The models ability to capture various notions of similarity will be evaluated on cultural information in music collections (e.g. genre, style, composer) as well as user annotations or ratings of similarity. This work will build on the experience of the participants in modelling audio similarity (Wolff & Weyde 2013), audio transcription and the integration of symbolic and audio based models (Benetos, Ewert & Weyde 2014), symbolic melodic similarity (Marsden 2012) and probabilistic music and performance modelling (Spiro, Gold & Rink 2010, Abdallah et al 2012).

This project offers an opportunity to transform musicological research, specifically the research areas of music similarity and structure analysis. The proposed work will enable access to musicologists, technologists, British Library users, and the wider music listening public to large collections of music recordings and scores, and tools for comparing, classifying, and clustering music collections, enabling large-scale systematic musicology research. In specific, the beneficiaries of this project include those directly involved, and those involved through our partner the British Library (BL), and from any likely commercialisation of project outcomes.
Those directly involved are:
(1) Users of the British Library, in particular of the Digital Music Collections
(2) The British Library, in particular the British Library Labs and the British Library Sound Archive
(3) Musicologists, musicians and music enthusiasts accessing the project-created online resources
Those indirectly involved are:
(4) Music technologists, in particular developers working on music recommendation and similarity
(5) Potential licensees and adopters of the music analysis tools showcased by this project
(6) Customers of licensees and adopters, i.e. the wider music listening public

These different constituents benefit in differing ways. Users of the BL will have access to large collections of audio recordings and scores (the latter also automatically aligned to the recordings), as well as high-level annotations of musical structure. From a collection-level, users will be able to visualise collections, as well as compare and cluster large groups of music recordings and scores. The BL will also be able to improve their service and infrastructure, and will be able to exploit the vast amount of data which already exist in the BL Sound Archive, and also link it to its large corpus of transcribed scores (from printed sources) that are being made available through the Optical Music Recognition project.

Musicologists will be able to access this resource online from BL services but also from the online services developed as part of the DML project, and will benefit from automated tools for accessing audio recordings, scores, and high-level information regarding structure of individual music pieces, as well as collection-level analysis tools for music similarity. Musicians will have the opportunity to practice and study using an online dataset providing recordings, scores, and structural information. Music technologists will realise the impact that audio-score integration can lead to improved systems for music recommendation. Potential licensees of music analysis tools will benefit from software that jointly analyse symbolic and audio information for music, and provides reliable similarity measures for music recommendation applications. Customers of music technology tools will benefit from a more insightful organisation of their music collections and will be able to more easily locate music that matches their interests.

We expect beneficiaries (1) and (2) to gain significant benefit during the lifetime of the project, while beneficiaries (3) will receive increased benefit as the main project outcomes are disseminated. Finally, beneficiaries (4)-(6) should see benefit after the end of the project.

The project includes regular interaction between researchers and partners, as well as partners on the DML and Optical Music Recognition projects. We will organise a workshop towards the end of the project, aimed at musicologists and music technologists, where the created datasets and tools for music similarity will be presented. Project documentation and press items for engaging with users and the wider public will also be created, along with a project-specific website, blog, press, and social media presence. Mechanisms to present the project to the public will be sought in conjunction with the Press and Publications Offices of City, UCL and Lancaster.