Predicting musical choices using computational models of cognitive and neural processing

Lead Research Organisation: Queen Mary, University of London
Department Name: Sch of Electronic Eng & Computer Science

Abstract

Music consumption has shifted dramatically in recent years towards streaming from vast music libraries, overloading the user with the enormity of possible musical choices. This new landscape makes it imperative to develop intelligent tools to help listeners choose music to listen to. Research in music technology has traditionally followed a pure engineering approach, which has taken the field some way. However, progress is being hindered by the lack of a robust, scientifically grounded model of the listener, which can be used to inform digital music players (e.g., iTunes, Spotify, Last FM) about users' preferences for selecting music.

The proposed research addresses this gap by developing the scientific knowledge needed to create computational models which can predict listeners' musical choices from features of the music and electrical brain responses recorded using Electroencephalography (EEG). The principal idea is to develop a scientific understanding of the psychological and neural processes involved when a listener chooses music to listen to. The hypothesis is that accurate predictions of a listener's musical choices can be made using a combination of psychological principles, musical features and electrical brain responses recorded from the scalp. This research has two foundations: first, to conduct listener studies to identify those psychological principles, musical features and brain responses; and second, to use that knowledge to build a computational model that predicts a listener's choice of music.

The modelling approach includes three components to capture features of music that have an impact on musical choices. The first component uses acoustic features such as dissonance and temporal regularity extracted from the audio using signal processing methods. The second component takes a higher-level cognitive approach, extracting measures of complexity using information-theoretic models based on note-level representations of music. The third component extracts the emotional intentions of the music from affective textual analysis of the lyrical content. Understanding the exact nature and weighting of these features and how they impact on musical choice requires the detailed examination of the choices that listeners make when listening to music.

Therefore, investigating the behaviour of actual listeners is central to this research and two studies will be performed. The first focuses on collecting EEG data while participants listen to musical excerpts and select them for future listening. Machine learning methods will be used to predict the listeners' decisions using features of the time-varying neural response recorded prior to the choice being made. The purpose of the second user study is to collect data for predict modelling of choices from features of the music itself and attributes of the listener. This will involve a larger range of musical excerpts and a wider range of listeners than is practical for the EEG study. The objective is to understand the psychological processes involved in musical choice and to use this knowledge to refine, parameterise and optimise the computational models of musical choice.

The final stage of the research will develop an integrated predictive model of musical choice by combining predictive models using the musical signal with those making predictions from the neural signal. The development of such an integrated model is highly innovative. The advent within the last two years of affordable, multi-channel, wireless EEG headsets for the consumer market makes the possibility of using these devices to control media players and other interactive systems a real possibility. Therefore, the time is ripe to combine research in neuroscience, music cognition and machine learning, reflecting the unique interdisciplinary expertise of the PI, to understand the mapping between neural signals, musical structure and song selections.

Planned Impact

By investigating and modelling the psychological and neural processes involved in musical preference and choice, this project holds benefits to several groups beyond academic researchers, both in the UK and internationally. These include:

1. Commercial Private Sector:
* Digital music software industry: understanding the psychological and neural processes involved in listeners making musical choices, will help the digital music industry develop better software for supporting and enhancing those choices, especially since this research will implement computational models that aim to predict choice;
* Advertising industry: Understanding the psychological and neural principles of musical preference and decision making can help the advertising industry understand how to use music most effectively for marketing products;
* Musicians and concert organisers: who can use the principles of musical choice to help design better concert programmes and playlists for releases.

2. Public and third sectors:
* music educators: understanding of the psychological principles involved in musical preference and choice can facilitate the selection of repertoire and help design software for inspiring young children in musical education
* Independent organisations promoting STEM (science, technology, engineering and maths) to young people and underrepresented groups, by using music to generate interest in STEM careers.

3. The wider public:
* musical listeners will benefit from better software tools that match their intuitive preferences when making musical choices;
* Students and school children in technical, musical and psychological fields can benefit by creating links between scientific, technical and musical domains to better understand these disciplines.

The ways in which this research will target these beneficiaries is outlined in the Pathways to Impact document. For academic beneficiaries see Section 2 in the Case for Support.

Publications

10 25 50
publication icon
Pearce M (2017) Compression-based Modelling of Musical Similarity Perception in Journal of New Music Research

publication icon
Sears DR (2019) Expectations for tonal cadences: Sensory and cognitive priming effects. in Quarterly journal of experimental psychology (2006)

 
Description The research conducted by the project shows that listeners preferences for music vary as a function of the activity that they are undertaking concurrently (e.g., driving, working, exercising). Furthermore, we have established information-theoretic measures of musical complexity related specifically to harmonic movement, going beyond previous work on melody. Finally, we established a general-purpose information-theoretic measure of musical similarity based on compression distance and showed that it simulates listeners' similarity perception very accurately. Ongoing research, drawing on this project, is examining whether context-related preferences are related to such information-theoretic measures of complexity and similarity.

We have also discovered that overall people prefer music that expresses a calm or peaceful mood (compared with happy, sad and angry). In some situations listeners choose to listen to music that is mood congruent (e.g., a listener in a peaceful mood listening to peaceful music) while in other situations they choose music that is mood-incongruent (e.g., a listener in an angry mood choosing music that is peaceful). Ongoing follow-on research is investigating what might explain these different listening strategies.
Exploitation Route Understanding why people like to listen to music raises profound questions for academic research and exciting opportunities for practical application in the non-academic world.

Regarding academic developments, further research is required on how the context (e.g., social or task-related factors) interacts with individual factors (e.g., personality, genre preferences, current emotional state, motivations, goals and desires) and attributes of the music (e.g., loudness, dissonance, complexity, familiarity, prototypicality) to determine the musical choices that individuals make in particular situations. In particular, it seems that in some situations individuals make mood-congruent (e.g., a person feeling sad choosing to listen to sad music) choices and in others they make mood-incongruent choices (e.g,. the same person choosing to listen to happy music) but currently we don't understand why this is. Research using EEG has focused to date on emotional responses to music and there is very little scientific understanding of neural correlates of musical pleasure or liking, let alone preference decisions.

Although there is much research to be done, understanding the factors - psychological, neuroscientific, musical and contextual - that determine musical preference decisions would be of great benefit beyond academia. It would allow the development of more sophisticated music recommendation engines which could be used in industries ranging from online radio stations, personal listening, retail and advertising. It may also be useful in music therapy allowing a therapist to select appropriate music for particular patients.
Sectors Digital/Communication/Information Technologies (including Software),Education,Healthcare,Leisure Activities, including Sports, Recreation and Tourism,Culture, Heritage, Museums and Collections

 
Description We have run a Live Science Residency called "Why do you like the music you like" at the Science Museum, London. The residency ran over 5 weeks in Nov-Dec 2015 and engaged visitors to the museum as participants in research on musical emotion and preferences. The residency included both psychological tasks and neural investigation using a mobile EEG headset, which recorded visitors electrical brain responses while listening to music. In addition to collecting data, we also actively engaged with visitors explaining our research to them and asking for their opinions on our research. In total, we engaged with 372 people ranging in age from 8 to 78 (308 adults and 64 children), yielding more than 480 sets of data across both tasks. Many people came to the museum specifically to take part in our residency and some brought their whole family. About 10% of participants completed feedback forms. These indicated that the participants enjoyed the experience (mean rating of 6 out 7) and 71% of participants said they would be more likely to engage with science in the future as a result of taking part in our experiments.
First Year Of Impact 2015
Sector Education,Leisure Activities, including Sports, Recreation and Tourism,Culture, Heritage, Museums and Collections
Impact Types Cultural,Societal

 
Description QMUL Centre for Public Engagement Small Grants Award
Amount £966 (GBP)
Organisation Queen Mary University of London 
Sector Academic/University
Country United Kingdom
Start 11/2015 
End 02/2016
 
Description Live Science Residency at the Science Museum 
Organisation Science Museum Group
Department The Science Museum
Country United Kingdom 
Sector Public 
PI Contribution We developed three experiments, adapted from our lab-based studies to run within a 15-20 minute session with visitors to the museum. We successfully applied for QMUL ethical consent for the project, prepared the text for the promotional materials, consent forms and information sheets and ran the residency over a five week period in the Science Museum. We also sent a report to the Science Museum and we are currently analysing over 470 datasets that we collected.
Collaborator Contribution The Science Museum provided the gallery space, computers, public engagement training for our team and produced final copies of the promotional materials, consent forms and information sheets along with other advice, guidance and support.
Impact The collaboration is multi-disiplinary involving psychology, neuroscience, computer science and music. Full details of the outcomes can be found in Narrative Impact.
Start Year 2015
 
Description Live Science Residency, Science Museum 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact During a Live Science residency at the Science Museum, we engaged with 372 visitors over five weeks people ranging in age from 8 to 78. Visitors took part in short experiments and we engaged with them about our research on musical preferences. About 10% of participants completed feedback forms which indicated that they enjoyed the experience (mean rating of 6 out 7) and 71% said they would be more likely to engage with science in the future as a result of taking part in our experiments. See Narrative Impact for full details.
Year(s) Of Engagement Activity 2015
 
Description Workshop on Auditory Neuroscience, Cognition & Modelling (WANCM 2016) 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact We organised a one-day workshop in Auditory Neuroscience, Cognition and Modelling, taking place on 17 February 2016 at QMUL (Charterhouse Square Campus and attended by over 96 participants from the UK and abroad. There were 32 presentations: 3 keynote talks, 6 oral presentations, 23 poster presentations (oral and poster presentations following a public call for abstracts and selection process). The workshop website includes PDF book of abstracts and video recorded oral presentations: http://c4dm.eecs.qmul.ac.uk/wancm2016/.
Year(s) Of Engagement Activity 2016
URL http://c4dm.eecs.qmul.ac.uk/wancm2016/