How the brain detects patterns in sound sequences

Lead Research Organisation: University College London
Department Name: Ear Institute

Abstract

This project employs state of the art neuroimaging (EEG, MEG and fMRI) and behavioural experimentation (psychophysics) to reveal how human listeners discover patterns and statistical regularities in rapidly unfolding sound sequences. Accumulating experimental evidence suggests that the brain is sensitive to statistical regularities in sensory input, at multiple time scales, and that this sensitivity plays a key role in our ability to understand, efficiently interact with, and survive in the environment. However, a key question - the process through which patterns are detected in the first place - has largely eluded investigation. My proposal focuses on this crucial missing link. I propose a paradigm, recently developed and validated in my laboratory, that allows us to observe these processes, with great detail, within the auditory system. The method, based on measuring behavioural and time-locked brain responses to rapid tone-pip sequences governed by specifically controlled rules aims to uncover (1) how the brain discovers patterns in sound sequences, (2) which neural mechanisms are involved, (3) to what degree the process is automatic or susceptible to attentional state and behavioural goals of the listener. The results will reveal an important missing link in understanding audition, and perception more generally, and inform the current debate in systems neuroscience surrounding sensitivity to input statistics and predictive coding. The project draws upon, and will extend, my expertise in psychophysics, electrophysiology, and brain imaging of auditory function. It is backed by the strong multidisciplinary scientific and clinical environment of the UCL Ear Institute and the Wellcome Trust Centre for Neuroimaging.

Technical Summary

This project employs state of the art neuroimaging (EEG, MEG and fMRI) and behavioural experimentation (psychophysics) to reveal how human listeners discover patterns and statistical regularities in rapidly unfolding sound sequences. Accumulating experimental evidence suggests that the brain is sensitive to statistical regularities in sensory input, at multiple time scales, and that this sensitivity plays a key role in our ability to understand, efficiently interact with, and survive in the environment. However, a key question - the process through which patterns are detected in the first place - has largely eluded investigation. My proposal focuses on this crucial missing link. I propose a paradigm, recently developed and validated in my laboratory, that allows us to observe these processes, with great detail, within the auditory system. The method, based on measuring behavioural and time-locked brain responses to rapid tone-pip sequences governed by specifically controlled rules aims to uncover (1) how the brain discovers patterns in sound sequences, (2) which neural mechanisms are involved, (3) to what degree the process is automatic or susceptible to attentional state and behavioural goals of the listener. The results will reveal an important missing link in understanding audition, and perception more generally, and inform the current debate in systems neuroscience surrounding sensitivity to input statistics and predictive coding. The project draws upon, and will extend, my expertise in psychophysics, electrophysiology, and brain imaging of auditory function. It is backed by the strong multidisciplinary scientific and clinical environment of the UCL Ear Institute and the Wellcome Trust Centre for Neuroimaging.

Planned Impact

The present project is classified as 'basic research' with the goal of understanding how normal hearing listeners detect the emergence of patterns and statistical regularities within rapid, stochastic sound sequences. The major implications of this work are in the academic domain, however it also has the potential of generating impact in the clinical, industry and arts spheres.


(1) Sensitivity to patterning is a fundamental aspect of listening. Understanding the range of sound patterns to which listeners are particularly sensitive, and the neural signature corresponding to their detection, will inform work efforts in all aspects of auditory-related technologies. This including the design of hearing aids and of brain-computer interfaces and any technology that produces sound for the purpose of conveying information, providing feedback or creating ambience. Understanding sensitivity to sound patterns could inform the design of these systems for example by programming them so that they produce events that listeners are particularly sensitive to. This could benefit individuals and the commercial private sector.

(2) Our results could also find an interesting outlet in the arts field including sound-scape design and composition.


(3) Our experimental paradigms, designed to study acoustic pattern processing in the normal (healthy) brain may be used to understand and characterize deficits in clinical populations (e.g. 'Auditory processing disorder', Schizophrenia) which exhibit deficits in complex sensory processing. This will benefit individuals, health professionals as well as NHS policy makers.


(4) Analysis tools developed in the course of this project might be of use to brain-machine applications thus benefiting government agencies, and the commercial private sector.
 
Description This grant was focused at understanding the machinery that is related to discovering patterns in sound sequences.
We are currently finishing up analysis/writing of the remaining projects. The key findings are listed below:
(1) Memory for patterns. Manuscript published (Bianco et al, 2020). We demonstrate a remarkable human ability to implicitly encode rapid auditory patterns into long term memory. Our results reveal the heuristics utilized by the brain in determining how to reinforce retained representations of statistical structure in the sensory environment.

(2) Modelling memory for patterns. Manuscript published (Harrison et al, 2020). Statistical learning and probabilistic prediction are fundamental processes in auditory cognition. A prominent computational model of these processes is Prediction by Partial Matching (PPM), a variable-order Markov model that learns by internalizing n-grams from training sequences. However, PPM has limitations as a cognitive model: in particular, it has a perfect memory that weights all historic observations equally, which is inconsistent with memory capacity constraints and recency effects observed in human cognition. We address these limitations with PPM-Decay, a new variant of PPM that introduces a customizable memory decay kernel.

(3) A key role for Norepinephrine (NE) in tracking the statistics of rapid sound sequences. Paper published. Zhao et al, 2019. We implicated the neurotransmitter Norepinephrine in coding for abrupt changes of the statistics of sound sequences, suggesting that in addition to its role in tracking the statistics of reward environments, NE also plays a role in tracking the statistics of sensory signals in the context of perception. This finding is exciting because it opens the door to investigating the consequence of impairment in the LC-NE system on auditory perception.

(4) Remarkable sensitivity to rapid temporal patterns. Paper published. Southwell & Chait (2018). The brain draws on knowledge of statistical structure in the environment to facilitate detection of new events. Understanding the nature of this representation is a key challenge in sensory neuroscience. Specifically, it is has been unknown whether real-time perception of rapidly-unfolding sensory signals is driven by a coarse or detailed representation of the proximal stimulus history. We recorded electroencephalography brain responses to frequency outliers in regularly-patterned (REG) versus random (RAND) tone-pip sequences which were generated anew on each trial. REG and RAND sequences were matched in frequency content and span, only differing in the specific order of the tone-pips. Stimuli were very rapid, limiting conscious reasoning in favour of automatic processing of regularity. Listeners were naïve and performed an incidental visual task. Outliers within REG evoked a larger response than matched outliers in RAND. These effects arose rapidly (within 80 msec) and were underpinned by distinct sources from those classically associated with frequency-based deviance detection. These findings are consistent with the notion that the brain continually maintains a detailed representation of ongoing sensory input and that this representation shapes the processing of incoming information. Predominantly auditory-cortical sources code for frequency deviance whilst frontal sources are associated with tracking more complex sequence structure.

(5) Pupil responses reflected the predictability of rapid sound sequence (Milne et al, 2021). To understand the impact of stimulus predictability on perception it is important to determine how the detection of predictable structure influences processing and attention. Here we use pupillometry to gain insight into the effect of sensory regularity on arousal. Pupillometry is a commonly used measure of salience and processing effort, with more perceptually salient or perceptually demanding stimuli consistently associated with larger pupil diameters. In two experiments we tracked human listeners' pupil dynamics while they listened to sequences of 50ms tone pips of different frequencies. The order of the tone pips was either random, contained deterministic (fully predictable) regularities (experiment 1, n = 18, 11 female) or a probabilistic regularity structure (experiment 2, n = 20, 17 female). The sequences were rapid, preventing conscious tracking of sequence structure thus allowing us to focus on automatic extraction of different types of regularities. We hypothesized that if regularity facilitates processing by reducing processing demands, a smaller pupil diameter would be seen in response to regular relative to random patterns. Conversely, if regularity is associated with heightened arousal and attention (i.e. engages processing resources) the opposite pattern would be expected. In both experiments we observed a smaller sustained (tonic) pupil diameter for regular compared with random sequences, consistent with the former hypothesis and confirming that predictability facilitates sequence processing.
(6) Our planned brain imaging work was interrupted due to the onset of the covid pandemic but completed just before grant end in December 2021. Data analysis is close to finishing on two outstanding MEG projects. In addition to the analysis of available data, we have shifted our attention to adapting our behavioural paradigms to be run using online platforms. This effort yielded a new efficient test to check for headphone use (paper published, Milne et al, 2020); and development of methods to keep online participants motivated (Bianco et al, 2021).
(7) Two further online experiments which were conducted in 2021 have now been submitted for publication (Binaco & Chait, 2023; Bianco et al, 2023). This work, which was also partly funded by a grant from ARUK to Roberta Bianco, used the pattern detection task we developed to understand how auditory memory and sensitivity to structure in unfolding sound sequences is affected during aging. The main discoveries are that older people exhibit reduced short- and longer term auditory memory performance. But this does not correlate with other measures of auditory (e.g. Speech in Noise perception) or cognitive performance.
Exploitation Route the results of our experiments have implications to understanding hearing and memory more broadly and for models of the brain as a statistical learning machine.
The analysis of the final experiments (delayed due to covid, but completed in end of 2021) is nearing the end. We have to brain imaging works being written up for publication currently. Preprints will be available by May 2023. Both projects were presented at the recent ARO conference (Feb 2023) and attracted substantial interest in the community.
Sectors Digital/Communication/Information Technologies (including Software),Education

 
Description BRC equipment grant
Amount £120,000 (GBP)
Organisation NHS England 
Sector Public
Country United Kingdom
Start 02/2018 
 
Description Development of a non-target hearing test for hearing-impaired listeners
Amount 2,730,000 kr. (DKK)
Organisation William Demant Foundation 
Sector Charity/Non Profit
Country Denmark
Start 06/2022 
End 06/2025
 
Description UCL-Peking U international partnership award
Amount £10,000 (GBP)
Organisation University College London 
Sector Academic/University
Country United Kingdom
Start 05/2018 
 
Title a new headphone test to faciliate conudcting audio experiments online 
Description Online experimental platforms can be used as an alternative to, or complement, lab-based research. However, when conducting auditory experiments via online methods, the researcher has limited control over the participants' listening environment. We offer a new method to probe one aspect of that environment, headphone use. Headphones not only provide better control of sound presentation but can also "shield" the listener from background noise. Here we present a rapid (< 3 min) headphone screening test based on Huggins Pitch (HP), a perceptual phenomenon that can only be detected when stimuli are presented dichotically. We validate this test using a cohort of "Trusted" online participants who completed the test using both headphones and loudspeakers. The same participants were also used to test an existing headphone test (AP test; Woods et al., 2017, Attention Perception Psychophysics). We demonstrate that compared to the AP test, the HP test has a higher selectivity for headphone users, rendering it as a compelling alternative to existing methods. Overall, the new HP test correctly detects 80% of headphone users and has a false-positive rate of 20%. Moreover, we demonstrate that combining the HP test with an additional test-either the AP test or an alternative based on a beat test (BT)-can lower the false-positive rate to ~ 7%. This should be useful in situations where headphone use is particularly critical (e.g., dichotic or spatial manipulations). Code for implementing the new tests is publicly available in JavaScript and through Gorilla (gorilla.sc). 
Type Of Material Technology assay or reagent 
Year Produced 2020 
Provided To Others? Yes  
Impact currently used widely for online experiments 
URL https://link.springer.com/article/10.3758/s13428-020-01514-0
 
Title PPM-Decay: A Computational Model of Auditory Prediction with Memory Decay 
Description Statistical learning and probabilistic prediction are fundamental processes in auditory cognition. A prominent computational model of these processes is Prediction by Partial Matching (PPM), a variable-order Markov model that learns by internalizing n-grams from training sequences. However, PPM has limitations as a cognitive model: in particular, it has a perfect memory that weights all historic observations equally, which is inconsistent with memory capacity constraints and recency effects observed in human cognition. We address these limitations with PPM-Decay, a new variant of PPM that introduces a customizable memory decay kernel. In three studies - one with artificially generated sequences, one with chord sequences from Western music, and one with new behavioral data from an auditory pattern detection experiment - we show how this decay kernel improves the model's predictive performance for sequences whose underlying statistics change over time, and enables the model to capture effects of memory constraints on auditory pattern detection. The resulting model is available in our new open-source R package, ppm (https://github.com/pmcharrison/ppm). 
Type Of Material Computer model/algorithm 
Year Produced 2020 
Provided To Others? Yes  
Impact this model has just been made available to the field. We expect it to have a broad impact in modelling and understanding human memory performance across a range of domains. 
URL https://www.biorxiv.org/content/10.1101/2020.01.09.900266v2
 
Description Continued ongoing collaboration with Dr Roberta Bianco 
Organisation Italian Institute of Technology (Istituto Italiano di Tecnologia IIT)
Department Neuroscience and Brain Technologies IIT
Country Italy 
Sector Academic/University 
PI Contribution Dr Roberta Bianco was the post doctoral fellow funded by this award. She is now a PI in IIT Rome. We are continuing to work together to finalize work she conducted at UCL and develop new projects related to this work. New work on animal models (primates) and in the context of aging and music is currently being developed.
Collaborator Contribution Roberta is also involved in supervising MSc and PhD students in my lab whose work forms a continuation of this project
Impact Experiments are being developed/conducted now. no outputs yet. An initial poster on the primate work has been presented at the recent ARO conference (Orlando, USA Feb 2023)
Start Year 2022
 
Description AHRC network "Beyond the visual: non sighted modes of engaging art". 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact this is an ongoing project funded by the AHRC with project partners: Tate, Henry Moore Institute, VocalEyes, Shape Arts, The DisOrdinary Architecture Project.
The purposes is to discuss how audio can be used to promote engagement with art specifically for non-sighted individuals.
Year(s) Of Engagement Activity 2022
 
Description Collaboration with Ford automotors on a campaign to increase awareness of using headphones when cycling/scooting 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact This activity is ongoing. i have advised Ford on their campaign (including design of an app to demonstrate the risk of using headphones whilst on the road). The campaign with launch in a few months
Year(s) Of Engagement Activity 2021
 
Description Resonant frequencies event 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact one-day summit of talks, presentations and performances around sound. Brigning together artists and researchers.
more info is on the website:
http://davidrobertsartfoundation.com/live/draf-x-goldsmiths-resonant-frequencies-summit-7-dec-2019-at-draf-london/
Year(s) Of Engagement Activity 2019
URL http://davidrobertsartfoundation.com/live/draf-x-goldsmiths-resonant-frequencies-summit-7-dec-2019-a...
 
Description Speaker and Panellist. "How music contributes to productivity and focus". Discussion section during a "Crick Late" public event. The Crick Institute, London UK 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact Speaker and Panellist. "How music contributes to productivity and focus". Discussion section during a "Crick Late" public event. The Crick Institute, London UK
Year(s) Of Engagement Activity 2019
 
Description World Hearing Day (March 2nd) stall at Kings Cross stations 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Public/other audiences
Results and Impact To Mark World Hearing Day (March 2nd) I help organize, set up, and run a public engagement event at Kings Cross station. We spent the whole date (7 AM till 4 PM) at the station engaging the passers by, distributing leaflets and raising awareness of hearing and hearing protection.
Year(s) Of Engagement Activity 2018
 
Description talk at the University of the 3rd Age (U3A), Hounslow Branch 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact I gave a presentation on our ongoing work.
Year(s) Of Engagement Activity 2019
 
Description talk at the University of the 3rd Age (U3A), Upminster Branch 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact a talk about auditory processing and music at the Upminster U3A branch.
Year(s) Of Engagement Activity 2020