EnhanceMusic: Machine Learning Challenges to Revolutionise Music Listening for People with Hearing Loss

Lead Research Organisation: University of Salford
Department Name: Sch of Science,Engineering & Environment

Abstract

Every culture has music. It brings people together and shapes society. Music affects how we feel, tapping into the pleasure circuits of the brain. In the UK each year, the core music industry contributes £3.5bn to the economy (UK Music 2012) with 30 million people attending concerts and festivals (UK Music 2017). Music listening is widespread in shops, movies, ceremonies, live gigs, on mobile phones, etc.

Music is important to health and wellbeing. As a 2017 report by the All-Party Parliamentary Group on Arts, Health & Wellbeing demonstrates, "The arts can help keep us well, aid our recovery and support longer lives better lived. The arts can help meet major challenges facing health and social care: ageing, long-term conditions, loneliness and mental health. The arts can help save money in the health service and social care."

1 on 6 people in the UK has a hearing loss, and this number will increase as the population ages (RNID). Poorer hearing makes music harder to appreciate. Picking out lyrics or melody lines is more difficult; the thrill of a musician creating a barely audible note is lost if the sound is actually inaudible, and music becomes duller as high frequencies disappear. This risks disengagement from music and the loss of the health and wellbeing benefits it creates.

We need to personalise music so it works better for those with a hearing loss. We will consider:
1. Processing and remixing mixing desk feeds for live events or multitrack recordings.
2. Processing of stereo recordings in the cloud or on consumer devices.
3. Processing of music as picked up by hearing aid microphones.

For (1) and (2), the music can be broadcast directly to a hearing aid or headphones for reproduction.
For (1), having access to separate tracks for each musical instrument gives greater control over how sounds are processed. This is timely with future Object-Based Audio formats allowing this approach.
(2) is needed because we consume much recorded music. It's more efficient and effective to pre-process music than rely on hearing aids to improve the sound, as this allows more sophisticated signal processing.
(3) is important because hearing aids are the solution for much live music. But, the AHRC Hearing Aids for Music project found that 67% of hearing-aid users had some difficulty listening to music with hearing aids. Hearing aid research has focussed mostly on speech with music listening being relatively overlooked.

Audio signal processing is a very active and fast-moving area of research, but typically fails to consider those with a hearing loss. The latest techniques in signal processing and machine learning could revolutionise music for those with a hearing impairment. To achieve this we need more researchers to consider hearing loss and this can be achieved through a series of signal processing challenges. Such competitions are a proven technique for accelerating research, including growing a collaborative community who apply their skills and knowledge to a problem area.

We will develop tools, databases and objective models needed to run the challenges. This will lower barriers that currently prevent many researchers from considering hearing loss. Data would include the results of listening tests into how real people perceive audio quality, along with a characterisation of each test subject's hearing ability, because the music processing needs to be personalised. We will develop new objective models to predict how people with a hearing loss perceive audio quality of music. Such data and tools will allow researchers to develop novel algorithms.

The scientific legacy will be new approaches for mixing and processing music for people with a hearing loss, a test-bed that readily allows further research, better understanding of the audio quality required for music, and more audio and machine learning researchers considering the hearing abilities of the whole population for music listening.

Publications

10 25 50
 
Title Muddy, Muddled, or Muffled? Understanding the Perception of Audio Quality in Music by Hearing Aid Users 
Description Underlying primary research data for a sensory evaluation study, focussing on the development of perceptual attributes for music audio quality from the perspective of hearing aid users. Research data include three .csv datasets corresponding to an individual elicitation task (single-word descriptors to describe music samples), background questionnaire (e.g., demographics, other characteristics), and a follow-up task (quantitative ratings relating to use of perceptual attributes of audio quality). These files are accompanied by a guidance document to navigate dataset variables. Finally, the research data include music samples (original and processed) utilised within the study. Part of the Cadenza ProjectPlease cite:Bannister S., Greasley A. E., Cox T. J., Akeroyd M. A., Barker J., Fazenda B., Firth J., Graetzer S. N., Roa Dabike G., Vos R. R., & Whitmer W. M. (2024). Muddy, muddled, or muffled? Understanding the perception of audio quality in music by hearing aid users. Frontiers in Psychology, 15: 1310176. Doi: 10.3389/fpsyg.2024.1310176.Music samples (original and processed) included in this research data are taken from the Medley DB database, with the following citations:Bittner, R., Salamon, J., Tierney, M., Mauch, M., Cannam, C., and Bello, J. P. (2014). MedleyDB: A Multitrack Dataset for Annotation-Intensive MIR Research. Taipei, Taiwan: 15th International Society for Music Information Retrieval Conference..Bittner, R., Wilkins, J., Yip, H., & Bello, J. (2016). MedleyDB 2.0: New Data and a System for Sustainable Data Collection. New York, NY, USA: International Conference on Music Information Retrieval (ISMIR-16).Medley DB materials are available under a CC-BY-NC-SA 4.0 International License. However, inclusion of original and processed music samples in this data release was also given consent by the authors of the database, on 2nd Monday October 2023. 
Type Of Material Database/Collection of data 
Year Produced 2024 
Provided To Others? Yes  
URL https://salford.figshare.com/articles/dataset/Muddy_Muddled_or_Muffled_Understanding_the_Perception_...
 
Title Muddy, Muddled, or Muffled? Understanding the Perception of Audio Quality in Music by Hearing Aid Users 
Description Underlying primary research data for a sensory evaluation study, focussing on the development of perceptual attributes for music audio quality from the perspective of hearing aid users. Research data include three .csv datasets corresponding to an individual elicitation task (single-word descriptors to describe music samples), background questionnaire (e.g., demographics, other characteristics), and a follow-up task (quantitative ratings relating to use of perceptual attributes of audio quality). These files are accompanied by a guidance document to navigate dataset variables. Finally, the research data include music samples (original and processed) utilised within the study. Part of the Cadenza ProjectPlease cite:Bannister S., Greasley A. E., Cox T. J., Akeroyd M. A., Barker J., Fazenda B., Firth J., Graetzer S. N., Roa Dabike G., Vos R. R., & Whitmer W. M. (2024). Muddy, muddled, or muffled? Understanding the perception of audio quality in music by hearing aid users. Frontiers in Psychology, 15: 1310176. Doi: 10.3389/fpsyg.2024.1310176.Music samples (original and processed) included in this research data are taken from the Medley DB database, with the following citations:Bittner, R., Salamon, J., Tierney, M., Mauch, M., Cannam, C., and Bello, J. P. (2014). MedleyDB: A Multitrack Dataset for Annotation-Intensive MIR Research. Taipei, Taiwan: 15th International Society for Music Information Retrieval Conference..Bittner, R., Wilkins, J., Yip, H., & Bello, J. (2016). MedleyDB 2.0: New Data and a System for Sustainable Data Collection. New York, NY, USA: International Conference on Music Information Retrieval (ISMIR-16).Medley DB materials are available under a CC-BY-NC-SA 4.0 International License. However, inclusion of original and processed music samples in this data release was also given consent by the authors of the database, on 2nd Monday October 2023. 
Type Of Material Database/Collection of data 
Year Produced 2024 
Provided To Others? Yes  
URL https://salford.figshare.com/articles/dataset/Muddy_Muddled_or_Muffled_Understanding_the_Perception_...
 
Title pyclarity 
Description pyclarity is a software suite for machine learning challenges to enhance hearing-aid signal processing and to better predict how people perceive speech-in-noise (Clarity) and speech-in-music (Cadenza). Files can be accessed via the Related Materials links below. 
Type Of Technology Software 
Year Produced 2023 
Open Source License? Yes  
URL https://figshare.shef.ac.uk/articles/software/pyclarity/23230694
 
Description Patient group workshop 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Study participants or study members
Results and Impact Ran two sensory panels of 12 people with hearing loss as upstream engagement to discover the issues important to them and shape the research.
Year(s) Of Engagement Activity 2022,2023
URL http://cadenzachallenge.org
 
Description The 1st Cadenza Workshop on Machine Learning Challenges to Improve Music for People with Hearing Loss (Cadenza-2023) 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact The Cadenza workshops are designed to stimulate the development of systems to improve music for people with a hearing loss. The aim of this virtual workshop was to report on the First Cadenza Challenge, the first ever machine learning challenge targeted to improve music for those with a hearing impairment.
Year(s) Of Engagement Activity 2023
URL https://cadenzachallenge.org/cadenza2023-workshop/
 
Description Workshop at the 4th Virtual Conference on Computational Audiology (VCCA2023) 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Demos and discussion around how can we process and remix music, so it sounds better for those with a hearing loss. This upstream engagement shaped the machine learning challenges in the Cadenza project (EPSRC project EnhanceMusic). Getting input from hearing aid experts, audiologists and academics.
Year(s) Of Engagement Activity 2023
URL https://computationalaudiology.com/events/vcca2023/