An Integrated Audio-Symbolic Model of Music Similarity

Lead Research Organisation: City University London
Department Name: Computing

Abstract

Similarity is a central aspect of music. It can help as a benchmark for testing theories of music perception and cognition, it is useful to answer for practical applications, such as music retrieval and recommendation, and it is an overarching question that relates to many aspects of music. However, theories of musical similarity have mostly focused on symbolic representations, where musical structures such as melodic and harmonic development are addressed but the aspects or expressive performance, such as micro-timing and timbre are ignored. On the other hand, audio based models, typically distances based on audio features, can capture details of the performance such as timbre, dynamics and tempo changes, but little of the musical structure as it unfolds over time.

Recent progress in audio transcription and alignment and the availability of music analysis tools for music collections with audio and symbolic content, which are being developed in the Digital Music Lab (DML) project, are changing the landscape of research in music. Large datasets of acoustic and symbolic music data that are generated or unveiled through the DML and "Optical Music Recognition from Multiple Sources" Big Data projects encourage an approach that can combine symbolic and audio based analyses into a joint similarity mode. This opens a great potential to new tools for research in music information retrieval and musicology, as the interaction between symbolic structure and acoustic information such as timbral texture has rarely been addressed and it could reveal aspects that have been unnoticed or unexplained so far. If successful, it might contribute to breaking the glass ceiling in music recommendation.

The aim of this project is to develop an initial framework and conduct experiments on an integration of symbolic melodic and structural similarity models with audio based models. The models ability to capture various notions of similarity will be evaluated on cultural information in music collections (e.g. genre, style, composer) as well as user annotations or ratings of similarity. This work will build on the experience of the participants in modelling audio similarity (Wolff & Weyde 2013), audio transcription and the integration of symbolic and audio based models (Benetos, Ewert & Weyde 2014), symbolic melodic similarity (Marsden 2012) and probabilistic music and performance modelling (Spiro, Gold & Rink 2010, Abdallah et al 2012).

Planned Impact

This project offers an opportunity to transform musicological research, specifically the research areas of music similarity and structure analysis. The proposed work will enable access to musicologists, technologists, British Library users, and the wider music listening public to large collections of music recordings and scores, and tools for comparing, classifying, and clustering music collections, enabling large-scale systematic musicology research. In specific, the beneficiaries of this project include those directly involved, and those involved through our partner the British Library (BL), and from any likely commercialisation of project outcomes.
Those directly involved are:
(1) Users of the British Library, in particular of the Digital Music Collections
(2) The British Library, in particular the British Library Labs and the British Library Sound Archive
(3) Musicologists, musicians and music enthusiasts accessing the project-created online resources
Those indirectly involved are:
(4) Music technologists, in particular developers working on music recommendation and similarity
(5) Potential licensees and adopters of the music analysis tools showcased by this project
(6) Customers of licensees and adopters, i.e. the wider music listening public

These different constituents benefit in differing ways. Users of the BL will have access to large collections of audio recordings and scores (the latter also automatically aligned to the recordings), as well as high-level annotations of musical structure. From a collection-level, users will be able to visualise collections, as well as compare and cluster large groups of music recordings and scores. The BL will also be able to improve their service and infrastructure, and will be able to exploit the vast amount of data which already exist in the BL Sound Archive, and also link it to its large corpus of transcribed scores (from printed sources) that are being made available through the Optical Music Recognition project.

Musicologists will be able to access this resource online from BL services but also from the online services developed as part of the DML project, and will benefit from automated tools for accessing audio recordings, scores, and high-level information regarding structure of individual music pieces, as well as collection-level analysis tools for music similarity. Musicians will have the opportunity to practice and study using an online dataset providing recordings, scores, and structural information. Music technologists will realise the impact that audio-score integration can lead to improved systems for music recommendation. Potential licensees of music analysis tools will benefit from software that jointly analyse symbolic and audio information for music, and provides reliable similarity measures for music recommendation applications. Customers of music technology tools will benefit from a more insightful organisation of their music collections and will be able to more easily locate music that matches their interests.

We expect beneficiaries (1) and (2) to gain significant benefit during the lifetime of the project, while beneficiaries (3) will receive increased benefit as the main project outcomes are disseminated. Finally, beneficiaries (4)-(6) should see benefit after the end of the project.

The project includes regular interaction between researchers and partners, as well as partners on the DML and Optical Music Recognition projects. We will organise a workshop towards the end of the project, aimed at musicologists and music technologists, where the created datasets and tools for music similarity will be presented. Project documentation and press items for engaging with users and the wider public will also be created, along with a project-specific website, blog, press, and social media presence. Mechanisms to present the project to the public will be sought in conjunction with the Press and Publications Offices of City, UCL and Lancaster.
 
Title Musical Composition and Production: Beyond the Fence 
Description The musical Beyond the Fence was the first mainly computer generated musical, and the DML infrastructure with ASymMus similarity analysis was used to conduct research that supported the generation. 
Type Of Art Composition/Score 
Year Produced 2016 
Impact The musical Beyond the Fence was performed in the Arts Theatre in London's West End 22 Feb - 5 March 2016. 
URL http://beyondthefencemusical.com/
 
Title TV programme: Computer Says Show 
Description 'Computer Says Show' is a two part television program on the creation of the world's first computer-generated musical 'Beyond the Fence'. 
Type Of Art Film/Video/Animation 
Year Produced 2016 
Impact This program was broadcast on Sky Arts on the 25th Feb and 3rd March 2016. 
URL http://www.wingspanproductions.co.uk/news-and-awards/read/48/Beyond-the-Fence-the-world-s-first-comp...
 
Description Key findings of this project include an information theoretic approach to music similarity estimation based on diverse representations (including audio and symbolic).

We also produced an updated version of the DML system which includes similarity features and provides visualisations for over 300k audio recordings from the British Library and I Like Music.
Exploitation Route The findings of the ASymMus project can help to analyse music collections for their internal and external structure, e.g. when analysing the development of styles as is planned in the Deep History of Music project.
Sectors Creative Economy,Digital/Communication/Information Technologies (including Software),Education,Leisure Activities, including Sports, Recreation and Tourism,Culture, Heritage, Museums and Collections

URL http://dml.city.ac.uk/category/asymmus/
 
Description A Machine Learning Framework for Audio Analysis and Retrieval
Amount £404,470 (GBP)
Funding ID RF/128 
Organisation Royal Academy of Engineering 
Sector Learned Society
Country United Kingdom
Start 03/2015 
End 03/2020
 
Description Automatic segmentation of audio recordings to speech and music
Amount £6,000 (GBP)
Organisation City, University of London 
Sector Academic/University
Country United Kingdom
Start 04/2015 
End 07/2015
 
Description Musical theatre project
Amount £6,000 (GBP)
Organisation Wingspan Productions Ltd 
Sector Private
Country Unknown
Start 06/2015 
End 07/2015
 
Title DML audio features, high and mid level analysis, and similarity data 
Description The features of the audio tracks analysed by the DML and ASymMus projects, including low, mid, and high-level features and similarity data, are available for download through our SPARQL endpoint. Also on request in other formats. 
Type Of Material Database/Collection of data 
Year Produced 2015 
Provided To Others? Yes  
Impact This data is the basis for musicological work with the DML interface, which continues to take place. 
URL http://mirg.city.ac.uk/cp/home
 
Description British Library 
Organisation The British Library
Country United Kingdom 
Sector Public 
PI Contribution The British Library has been a research partner in the Digital Music Lab and has supported the ASymMus AHRC projects, and the just started Making Sense of Sounds project by Prof Plumbley (EPSRC) and the RAEng fellowship for Dr. Benetos. In these collaborations, the British Library provides access to audio and other media data, the curation and/or digitisation of collections and the creation of metadata.
Collaborator Contribution Researchers at City University London and other contributing universities (Queen Mary University of London, University College London, Lancaster University) have created hard and software infrastructure and applications that support the British Library in serving their users with advanced services and remote access.
Impact The collaboration has extended through out the the DML and ASymMus project and is ongoing in the above mentioned projects. We have also applied for new projects with the European Commission (under review) and are planning to apply for a research project with the EPSRC.
Start Year 2014
 
Description EPFL collaboration 
Organisation Swiss Federal Institute of Technology in Lausanne (EPFL)
Country Switzerland 
Sector Public 
PI Contribution The Metamedia Centre at the EPFL hosts the audiovisual archive of the Montreux Jazz Festival. The Metamedia Institute has applied for a grant (CHF 100k) where Prof Simon Dixon and Dr Tillman Weyde will be active as advisors and the DML system will be used as part of the planned system for copyright-compliant research on the Metamedia archive.
Collaborator Contribution The Metamedia archive will be a user of the our system and give access to their data.
Impact No outputs yet.
Start Year 2018
 
Description I Like Music Ltd 
Organisation I Like Music Ltd
Country United Kingdom 
Sector Private 
PI Contribution The DML and ASymMus project provide the hard- and software infrastructure for music analysis.
Collaborator Contribution I Like Music provides access to over 1 million audio tracks of commercial music and several hundred thousand tracks of production music.
Impact The ILM data is an important contribution to enable music analysis on sizeable collections.
Start Year 2014
 
Title DML Research Information and Result Management System 
Description https://code.soundsoftware.ac.uk/hg/dml-open-cliopatria. 
Type Of Technology Software 
Year Produced 2015 
Open Source License? Yes  
Impact This software implements the back-end information management developed in the DML and ASymMus projects that enables data analysis with the DML. It provides the API for the Web interface and access via SPARQL and Prolog. 
URL https://code.soundsoftware.ac.uk/hg/dml-open-cliopatria
 
Title DML Visualisation Framework 
Description The visualisation layer of the DML, allowing the analysis of music data collections. 
Type Of Technology Software 
Year Produced 2015 
Open Source License? Yes  
Impact This software was used in the DML and it's follow-on project ASymMus, as well as in the recent Musical Theatre project. 
URL https://code.soundsoftware.ac.uk/hg/dml-open-vis
 
Description ASyMMuS Workshop on Audio-Symbolic Music Similarity Modelling 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact The AHRC funded project on An Integrated Audio-Symbolic Model of Music Similarity (ASyMMuS) aims to integrate aspects of audio and symbolic representations, such as scores or MIDI data, in a joint model. By building on the Digital Music Lab structure, the project's aim is to promote a data driven approach to music similarity. This workshop brought together researchers with different approaches to promote discussions on what constitutes and what contributes to music similarity.
Year(s) Of Engagement Activity 2015
URL http://dml.city.ac.uk/workshops/asymmus-workshop/
 
Description ASyMMuS at Lorentz Center Leiden Workshop on Music Similarity 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Several researchers from the ASyMMus and DML projects prominently contributed to the high-profile international workshop "Music Similarity: Concepts, Cognition and Computation".

The workshop gathered experts on music similarity from Computer Science, Musicology, Music Psychology and related scientific fields. In a highly-motivated series of workgroups and talks, our researchers collaborated with other experts in the field in theoretical concepts and computer models of music similarity.
Year(s) Of Engagement Activity 2015
URL http://www.lorentzcenter.nl/lc/web/2015/669/description.php3?wsid=669&venue=Oort
 
Description ASyMMus at MTG Seminar, Universitat Pompeu Fabra Barcelona 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Other audiences
Results and Impact The ASyMMus project and its integration into the DML web interface were presented by Daniel Wolff during his departmental talk on music similarity.
Year(s) Of Engagement Activity 2015
URL http://mtg.upf.edu/node/3393
 
Description DML and ASyMMuS projects at DMRN+9 workshop 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact Current progress on the DML and ASyMMuS projects were presented at the Digital Music Research Network Workshop 2014 (DMRN+9), taking place on Tuesday 16th December at Queen Mary University of London. The list of project-related presentations is as follows:

"The ASyMMuS project: An integrated audio-symbolic model of music similarity", Emmanouil Benetos, Daniel Wolff, Tillman Weyde (City University London), Nicolas Gold, Samer Abdallah (University College London) and Alan Marsden (Lancaster University)
"Towards analysing big music data - Progress on the DML research project", Tillman Weyde, Stephen Cottrell, Jason Dykes, Emmanouil Benetos, Daniel Wolff, Dan Tidhar, Alexander Kachkaev (City University London), Mark Plumbley, Simon Dixon, Mathieu Barthet, Steven Hargreaves (Queen Mary University of London), Nicolas Gold, Samer Abdallah (University College London), Aquiles Alancr-Brayner, Mahendra Mahey and Adam Tovell (The British Library)
Year(s) Of Engagement Activity 2014
URL http://c4dm.eecs.qmul.ac.uk/dmrn/events/dmrnp9/
 
Description DML and ASymMus projects at FMA 2015 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Work carried out on analysing world and traditional music as part of the DML and ASymMus projects was presented at the 5th International Workshop on Folk Music Analysis (FMA 2015). FMA took place on 10-12 June in Paris, France. Project-related papers are listed below:

S. Abdallah, A. Alencar-Brayner, E. Benetos, S. Cottrell, J. Dykes, N. Gold, A. Kachkaev, M. Mahey, D. Tidhar, A. Tovell, T. Weyde, and D. Wolff, "Automatic transcription and pitch analysis of the British Library World & Traditional Music Collection"
A. Leroi, M. Mauch, P. Savage, E. Benetos, J. P. Bello, M. Panteli, J. Six, and T. Weyde, "The deep history of music project"
Year(s) Of Engagement Activity 2015
URL http://fma2015.sciencesconf.org
 
Description DML and ASymMus projects at Musical Timbre Workshop 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Recent work on instrumentation recognition and on music that was carried out as part of the DML project was presented at the Workshop on Musical Timbre, that took place on 14th November at Télécom ParisTech, in Paris, France.
Year(s) Of Engagement Activity 2014
URL https://musictimbre.wp.mines-telecom.fr
 
Description DML project at THATCamp British Library Labs 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact DML project members participated at the THATCamp British Library Labs, which took place on 13th February 2015 at the British Library. THATCamp stands for "The Humanities and Technology Camp", that is an open, inexpensive meeting where humanists and technologists of all skill levels learn and build together in sessions pitched and voted on at the beginning of the day.
Year(s) Of Engagement Activity 2015
URL http://britishlibrary.typepad.co.uk/digital-scholarship/2015/02/thatcamp-british-library-labs.html
 
Description Transforming Musicology blog post on music similarity 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact Dr Alan Marsden (Co-I for the ASyMMuS project) wrote a post on the Transforming Musicology blog entitled "Similarity: haven't we heard this before somewhere?". The post mentions the ASyMMuS project and its connections with other AHRC-funded projects:
Year(s) Of Engagement Activity 2015
URL http://transforming-musicology.org/blog/2015-04-09_similarity-havent-we-heard-this-before-somewhere/