Brain-to-brain coupling during dialogue: What sentence fragments can reveal about 'joint' mental representations

Lead Research Organisation: University of Edinburgh
Department Name: Sch of Philosophy Psychology & Language

Abstract

Talking to one another is one of the cornerstones and joys of human experience. Imagine how difficult our lives would be if we couldn't use language to communicate, have a conversation, ask questions or easily say what we mean. But actually, we might not even have to say or pronounce all the things that are meant or understood in a conversation (full meanings shown for the reader in brackets):
A: Someone called for you. B: [Did someone] Really [call for me]?
A: Yes, [someone] really [called for you]. B: Who [called for me]?
A: John [called for you]. B: About what [did John call]?
Everyday interaction would be completely different if we had to repeat all the information we mean and understand, word for word, during dialogue - it would sure take a lot longer to get anywhere. Although we are not aware of it, most conversation is reduced or simplified into sentence fragments that "stand in" for full formal linguistic structure. Nonetheless, without being pronounced, complex structure is easily understood by both people. This fact clashes with a defining property of human language - a formal correspondence between sound and meaning, and seems to challenge the rules we use to combine linguistic units into larger ones. These seemingly innocuous sentence fragments may be why dialogue is such an easy, effective form of communication. Fragments naturally tap into information exchange during dialogue because they require speakers and listeners to refer to and access a shared memory of what has been said.
But how do we form and update this shared memory? Speakers must formulate utterances with the mental states of addressees in mind. The reduced fragments people end up producing and hearing operate as a natural "compression algorithm", making conversation easy and productive most of the time. But how can dialogue be so fragmented without people losing the plot more often? It may well be that aspects of linguistic structures and human memory allow people to 'jointly' activate information, or align their dialogue. In current theory, people align via priming, or passive activation of mental representations between speakers - which is why people often repeat each other's words and phrases. What is rarely noted is how, in order to converse, speaker and listener have to tacitly 'agree on' material that is not overtly expressed, as in the brackets above.
A series of experiments investigates how fragments are represented and transmitted between speakers, and whether the brain treats externally-sourced representations the same as internally-generated ones during dialogue. A second line of research develops an innovative new method that can index the 'flow' of information between speakers' brains during spontaneous, unscripted dialogue. Humans are fundamentally interactive - a fact only recently impacting theories of human cognition, highlighting the necessity of understanding classic cognitive questions in terms of an interaction between two actors in an environment. The basic notion is that real-time interaction, such as dialogue, is made possible by brain-to-brain coupling; where one brain's perceptual system is coupled to another brain's motor or speech output, deriving from a related ability of brains to bind to exogenous environmental signals. These signals, including those emanating from other brains, e.g., speech, are converted into internal signals the brain uses to infer, in turn, about its environment. A brain-to-brain coupling approach for dialogue requires methodological innovation in language research: we need to observe brain activity from two people interacting in real time. My research combines methods and theories from distinct domains of psychology and cognitive neuroscience to develop a novel paradigm for studying real-time interactive dialogue. This funding, if awarded, would allow me to develop my skill set to a degree that I would otherwise not be able to, adding value to my academic training and growth.

Planned Impact

Who will benefit from this research?
Because talking to each other and making conversation is such a big part of human life, research on language and dialogue is a natural jumping-off point for increasing public engagement in science. I believe the proposed research, especially the study of language and the brain during interactive dialogue, could have substantial impact outside academia and engage people of all walks of life to learn more about psychology and the science of the human mind. The simple fact that dialogue is a powerful tool for the exchange of information and knowledge transfer, enabling us to transmit information about events that are temporally and spatially remote, highlights an amazing ability in the natural world.
The key stakeholders (besides the Academic Beneficiaries listed separately) are the general public, the media, and science teachers. Specifically, the end goal for impact would be to make the science of the human mind resonate better with science teachers, children, and young people. This resonance would happen by using everyday daily activities, like having a conversation or asking a question, and unpacking the complex and fascinating human cognition underlying them. Dialogue is a clear input for highlighting the centrality of language and the human mind in everyday life, especially for audiences for whom conversation often happens via texting or online chatting.

How will they benefit?
Seeing 'everyday' things differently. The potential benefits to the key stakeholders from increasing public engagement with language research and the science of the mind come primarily from seeing everyday behaviours, events, and human interactions in a new, compelling and vibrant light. Linking scientific research to tangible, accessible, human behaviours, such as having a conversation or asking and answering questions, will promote the science of mind. I believe this can be accomplished by communicating how human language is unique, and demonstrating how a complex computational system underlies simple human moments. The main benefit would be that people have a deeper understanding of our place in the natural world.

Specifically, I envision three main avenues for promoting the public engagement with the science of the mind and language research:
(1) Presenting research on language and dialogue at science festivals.
(2) Generating press releases and lay articles
(3) Building creative partnerships with schools and science museums to feature research on language and the human mind by creating simple interactive presentations and activities.
Please see the Pathways to Impact statement for my specific goals for increasing public understanding of language and mind research, and my plans on how to achieve them.

Impact outcomes. An outcome of impact would be increasing public interest in the human mind and language, via the understanding of: (1) the uniqueness of human language among naturally occurring communication systems and computations and (2) how our cognitive abilities pervade and enable everyday activities, such having a conversation or asking a question, in ways we never stop to realise. These outcomes could be assessed via science festival attendance numbers, website visits and download numbers for online articles, school activity use and museum installation attendance. Please see the Pathways to Impact statement for more details.

Publications

10 25 50
 
Description The unanticipated disruptions to the project led to a new focus on the development of a research program that has fundamentally changed my approach to understanding human language processing. My new research program uses computational models to test language processing mechanisms in the brain and uncover the 'nuts and bolts' of human language ability. If brain responses (fluctuations in energy use) can be predicted from one model architecture, but not another, this helps us infer the kind of 'program' that the brain runs to produce and comprehend language. This novel approach allows us to develop theories both about the meaning we experience during language use that also account for the way our brains expend energy during language processing. The intellectual, technical, and career development objectives were met - I have grown in my expertise and learned to use computational models and to work with neural oscillations (Martin & Doumas, 2017; Nieuwland & Martin, 2017). I proposed my own processing theory of language that was only possible because of results I obtained in the grant project (Martin, 2016). Thus, Objectives (iv) and (v) were met. The main objective "to understand how the human mind and brain enable the efficient exchange of information during interactive dialogue" was perhaps too grand, and in retrospect, I see that it could never have been met in a single 3-year project with the employ of only a part-time research assistant in addition to myself. However, despite unforeseen facilities disruptions (see below), staff shortages, recruiting limitations, and methodological challenges, I was able to address aspects of Objectives (i)- (iii) and determined that speaker identity mitigates memory interference effects in dialogue compared to monologue. The objective of developing a novel interactive paradigm for brain-to-brain coupling during dialogue was not possible with facilities disruption, staff shortages/ recruitment issues, and such a small team (the PI and 0.5 FTE research assistant).
Exploitation Route My initial findings are relevant for psychologists, linguistics, and neuroscientists. A larger team of cognitive neuroscientists and psycholinguistics who had access to magnetoencephalogralphy would be able to cope with the signal processing issues I faced and extend the project and its principles, I believe. The preliminary findings minimally suggest that speaker identity information such be included in model of language comprehension as a factor that affects how language is understood. The role of speaker voice, identity, and other information is woefully underspecified in our theories of how our brains represent and process language. The fact that we all have different voices and our brains can exploit that fact to keep information separate can act as a natural jumping off point for engagement of the general public with language and brain research. Furthermore, the incorporation of speaker voice and identity information could improve artificial intelligence systems for human language and speech recognition.
Sectors Digital/Communication/Information Technologies (including Software),Other

 
Description Humboldt Fellowship - team member
Amount € 100,000 (EUR)
Organisation German Research Foundation 
Sector Charity/Non Profit
Country Germany
Start 12/2017 
End 12/2019
 
Description Research Grant
Amount £125,000 (GBP)
Funding ID RPG-2017-049 
Organisation The Leverhulme Trust 
Sector Charity/Non Profit
Country United Kingdom
Start 09/2017 
End 09/2019
 
Title Time-frequency decoding data pipeline 
Description I and my collaborator Phillip Alday have developed a new way to analyze EEG and relate it to language representations, including during dialogue. 
Type Of Material Improvements to research infrastructure 
Provided To Others? No  
Impact We are presenting an example of this method at conferences in March and April 2017 and are at work at writing a paper. 
 
Description Abstraction in time: Finding hierarchical linguistic structure in a model of relational processing 
Organisation University of Edinburgh
Department Department of Psychology
Country United Kingdom 
Sector Academic/University 
PI Contribution My theoretical process model is being tested and implemented using both brain data and computational modelling.
Collaborator Contribution Expertise in computational modelling.
Impact We have published this conference paper and have another paper under revision. Doumas, L. A. A., & Martin, A. E. (2016). Abstraction in time: Finding hierarchical linguistic structure in a model of relational processing. In Proceedings of the 38th Annual Conference of the Cognitive Science Society. Austin, TX: Cognitive Science Society.
Start Year 2015
 
Description Decoding cortical linguistic representations 
Organisation Max Planck Society
Department Max Planck Institute for Psycholinguistics
Country Germany 
Sector Academic/University 
PI Contribution I conceived of the theoretical question and conceptual design of the methods
Collaborator Contribution My collaborator implemented the data processing stream
Impact Alday, P. & Martin, A. E. (2017) Decoding linguistic representations in the time-frequency domain. Presentation to be given at the Annual Meeting of the Cognitive Neuroscience Society in March 2017.
Start Year 2016
 
Description Deriving dynamic cortical models of interactive language processing 
Organisation University College London
Department Division of Infection and Immunity
Country United Kingdom 
Sector Academic/University 
PI Contribution My process model (Martin, 2016)
Collaborator Contribution Jeremy Skipper's neurobiological model
Impact None yet
Start Year 2016
 
Description Hierarchy in cognition and brain function through the lenses of memory and language 
Organisation Koc University
Country Turkey 
Sector Academic/University 
PI Contribution My process model will be tested via a series of fMRI experiments we are designing. I will make the stimuli, and help write the papers.
Collaborator Contribution My collaborator will train me in machine learning approaches to fMRI data analysis and will collect data and give me access to data.
Impact We have applied for a Newton Advanced fellowship from the Royal Society but we were not successful.
Start Year 2016
 
Description Oscillations and sentence processing 
Organisation Max Planck Society
Department Max Planck Institute for Empirical Aesthetics
Country Germany 
Sector Academic/University 
PI Contribution Collaborative magentoencephalography (MEG) experiments are being planned which will be conducted at the MPI for Empirical Aesthetics in Frankfurt. I am developing and planning the experiments and will present them to collaborators early in May 2017 at a workshop. I will also analyse, model and write up data for the experiments.
Collaborator Contribution The expertise of the Director of the Neuroscience department has informed my thinking and theorising. I will have access to MEG data from his lab.
Impact There are no outputs or outcomes yet.
Start Year 2015
 
Description Oscillations and sentence processing 
Organisation New York University
Department Department of Psychology
Country United States 
Sector Academic/University 
PI Contribution Collaborative magentoencephalography (MEG) experiments are being planned which will be conducted at the MPI for Empirical Aesthetics in Frankfurt. I am developing and planning the experiments and will present them to collaborators early in May 2017 at a workshop. I will also analyse, model and write up data for the experiments.
Collaborator Contribution The expertise of the Director of the Neuroscience department has informed my thinking and theorising. I will have access to MEG data from his lab.
Impact There are no outputs or outcomes yet.
Start Year 2015
 
Description Prospective memory in parsing 
Organisation University of Edinburgh
Department Department of Psychology
Country United Kingdom 
Sector Academic/University 
PI Contribution I helped design experiments and interpret results. I also wrote half of the stimuli.
Collaborator Contribution My partner wrote stimuli, help design the studies, collected the data and analysed it.
Impact Grant submitted to Leverhulme is through to the 2nd round. We presented this poster: Sturt P., & Martin, A. E. (2016). Using grammatical features to forecast incoming structure: The processing of Across-the-board extraction. The 29th annual CUNY. Gainesville, FL, USA.
Start Year 2014
 
Description EEG Open day 
Form Of Engagement Activity Participation in an open day or visit at my research institution
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Public/other audiences
Results and Impact Around 50 students and research participants attended an open day to visit the research facility, we gave a demo on EEG using headsets, this sparked questions and discussion and many people enjoyed using the headsets. We had a spike in student projects/inquiries following the demo.
Year(s) Of Engagement Activity 2015
 
Description Invited talk MPI Frankfurt 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact I given two invited talks at the MPI for Empirical Aesthetics in Frankfurt in 2015 and 2016 leading to ongoing collaboration.
Year(s) Of Engagement Activity 2015,2016
 
Description Invited talk and collaboration visit MPI AE Frankfurt 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact Gave a lecture and met with MSc and PhD students
Year(s) Of Engagement Activity 2017
 
Description Language and Attention Conference (Mainz, Germany) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact 50 people attended an invited talk I gave at a conference on Language and Attention.
Year(s) Of Engagement Activity 2013
 
Description Neurobiology of Language Conference (Amsterdam) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact I presented 3 posters at the Society for the Neurobiology of Language Conference in Amsterdam in 2014.
Year(s) Of Engagement Activity 2014
 
Description Sentence Processing Conference "Architectures and Mechanisms for Language Processing" (AMLaP, Edinburgh and Malta) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact About 350 people attended a talk I gave at the Edinburgh AMLaP in 2014, I also presented a poster there that had about 40 attendees. I presented a poster that had about 30 attendees at the Malta AMLaP conference in 2014.
Year(s) Of Engagement Activity 2014,2015
 
Description Sentence Processing Conference of the City University of New York (CUNY, Los Angeles, CA; Gainesville, FL) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact I presented a talk and poster at the CUNY sentence processing conference in Los Angeles in 2015, and I will present a poster at the CUNY conference in Gainesville, FL in March 2016.
Year(s) Of Engagement Activity 2015,2016