When words speak off the page: Covert emotional prosodic processing in silent reading of direct quotations

Lead Research Organisation: University of Manchester
Department Name: School of Biological Sciences

Abstract

Written communication (e.g., emails, news reports, social media) is a major form of social information exchange in today's world. However, it is sometimes difficult to interpret the intended meaning of a written message without hearing prosody (rhythm, stress, and intonation of speech) that is instrumental in understanding the writer's feelings, attitudes, and intentions. For example, a prosody-less "thank you" email can be confusing as to whether the sender is being sincere or sarcastic (Kruger et al., 2005). Emails like these are often misinterpreted as being more negative or neutral than intended; such miscommunications can damage social cohesiveness and group identity within organisations and communities, thereby undermining economic performance and societal stability (Byron, 2008).

Interestingly, written words may not be entirely "silent" after all. My recent research showed that we mentally (or covertly) simulate speech prosody (or "inner voices") during silent reading of written direct quotations (Mary gasped: "This dress is beautiful!") as if we were hearing someone speaking (Yao et al., 2011, 2012). For example, Yao and colleagues (2011) observed that silent reading of direct quotations elicited higher neural activity in voice-selective areas of the auditory cortex as compared to silent reading of meaning-equivalent indirect speech (Mary gasped that the dress was beautiful.).

Can such covert prosody compensate for the lack of overt speech prosody in written language and thus enhance written communication? To address this question, the proposed project will systematically examine the nature (is covert prosody sound- or action-based in nature?), mechanisms (what information processing systems are engaged?) and emotional consequences (does covert prosody induce emotions and thereby influence behaviour?) of covert prosodic processing in silent reading of written direct quotations.

Theoretically motivated by the working neural models for "overt" emotional prosodic processing in speech (e.g., Schirmer & Kotz, 2006), the current proposal will probe "where" and "when" in the brain covert prosodic cues of various natures are mentally simulated and integrated into coherent covert prosodic representations and how these representations consequently induce emotional responses and aid in inferring the quoted speaker's mental state. Using complementary neuroimaging techniques, it will localise the neural substrates of systems engaged in covert emotional prosodic processing (fMRI), specify the time courses of the information processes within these systems (EEG, MEG), and integrate this information to form a unified spatio-temporal neural model for covert emotional prosodic processing.

The findings of this project have clear implications for the theoretical development of emotional prosody-based social communication, embodied cognition, and speech pragmatics, and will be of interest to all written language users (e.g., communication-based enterprises, social services, and the wider public). This research also has potential impact on early language education and diagnosis of Parkinson's disease (PD). For example, understanding direct quotations requires the reader to take the quoted speaker's perspective and attribute emotions and mental states to them. A quotation-rich teaching method thus may effectively enhance children's Theory of Mind ability (ability to attribute mental states) that is crucial in their cognitive development and social cognition. Moreover, PD patients may struggle in simulating covert emotional prosody due to their motor (articulation) dysfunction. Consequently, they may display difficulty in understanding figurative speech quotations (e.g., they may not detect the sarcasm in - She rolled her eyes, grumbling: "What a sunny day!"). This research could thus motivate the development of a low-cost quotation-based diagnostic tool for monitoring PD progression.

Planned Impact

EARLY LANGUAGE EDUCATORS
Making sense of a direct speech quotation entails taking the quoted speaker's perspective and interpreting their prosody. A quotation-rich teaching method thus may benefit children with poor perspective taking skills (especially deaf children and second language learners who are not explicitly taught to attend to speech prosody). For example, a teacher may direct pupils to analyse the sarcastic prosody of utterance "The weather is lovely today" and guide them to infer the actual meaning it conveys. My current and follow-up research will illuminate the role of covert prosody and perspective taking in the processing of direct quotations. With the help of the ESRC International Centre for Language and Communicative Development (LuCiD) at the University of Manchester, the produced scientific knowledge has the potential for influencing policy makers in designing curriculum and textbooks for early language education.

CLINICAL PRACTITIONERS AND USERS
Understanding the role of the motor (articulation) system in simulating covert prosody has the potential for developing low-cost, flexible paradigms for monitoring the progression of Parkinson's disease (PD). PD is a progressive degenerative disorder that begins with motor system malfunctioning, resulting in a range of cognitive and emotional problems that can lead to social isolation of PD patients. Among other chronic degenerative diseases, PD alone costs health services 13.9 billion euros (2010) and the number of patients is expected to double by 2030 (European Parkinson Disease Association). PD progression and treatment outcome are difficult and costly to monitor because it progresses at variable rates and manifests in a range of deficits. A low-cost and flexible (implemented in pen-and-paper, mobile apps, etc.) quotation comprehension paradigm could potentially address this challenge. For example, the severity of PD symptoms may be reflected in patients' difficulty in simulating and detecting covert emotional prosody (e.g., sarcasm) of a written quotation (e.g., She rolled her eyes, grumbling: "What a sunny day!") due to deterioration in their motor (articulation) system. Once established and validated, it can greatly benefit the current healthcare system where flexibility and individualised profiling are simply not practical.
My research can also be of interest to clinicians treating auditory verbal hallucinations (AVH). AVH affects 50-80% of schizophrenic patients and negatively impacts their lives. Recent development of an "avatar therapy" indicates that building a computer-based avatar (3D face model) to embody the voices in AVH can effectively reduce the frequency and severity of AVH, superior than current pharmaceutical treatments. My work on voice-hearing in direct speech quotations can help develop an economical, text-based avatar therapy to embody voices in AVH.

WRITTEN LANGUAGE USERS
Written communication (e.g., emails, social media, news reports) is a major form of social information exchange across all facets of society (e.g., social services, commercial sector, the wider public). Albeit highly efficient and economical, it can easily be misunderstood due to the lack of speech prosody. Specialist knowledge on covert prosodic processing in written language comprehension can help develop strategies for enhancing communication across different sectors of society. For example, effective use of linguistic cues and quotations can enhance the impact of important public messages regarding health, education and the economy. Understanding the emotional impact of direct quotations also helps identify and prevent biased reporting of two-sided political debates (e.g., Scottish independence, EU membership). Thus, my research will make a timely contribution to the rapidly evolving communication needs of our society with considerable benefits for the economic performance and societal stability of the UK and beyond.
 
Description This project has two objectives: 1) to understand the neural circuits underlying inner speech processing in silent reading of direct quotes; 2) to understand the emotional consequences of hearing inner speech.

Inner speech is the experience of hearing speech without any actual speakers. It is an ubiquitous psychological phenomenon observed in a wide range of cognitive tasks, including thinking, problem solving, working memory, reading and writing. The dysfunctions of inner speech are often associated with symptoms in mental health disorders such as rumination in depression, auditory verbal hallucinations in schizophrenia and associated disorders. The current literature has recognised the importance of inner speech, but has not fully understood its neural and cognitive underpinnings.

There are many types of inner speech. Some emphasise on inner speaking (covert articulation) and some focus on inner hearing (a form of auditory imagery). In terms of its phonological details, some inner speech are vivid and resemble audible speech whereas the others are more abstract and condensed. In the current project, we focused on the vivid form of inner speech during silent reading of fictional stories as it is closest to . In silent reading, we used direct speech quotes (She said, "The brain is amazing!") to experimentally induce vivid inner speech and used indirect speech quotes (She said the brain was amazing) as a linguistically matched control. We have used a combination of behavioural measures, eye tracking, functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) to understand the mechanisms underlying inner speech and its emotional consequences.

In this project, we have explored the mechanisms of inner speech in both time and space. In terms of time scales, our key finding is that inner speech may be governed by neural oscillations at theta frequencies (4-7Hz) in particular. Neural oscillations are rhythmic cycles of neuronal excitability. They are instrumental in the segmentation and integration of overt speech. Neurons in the auditory cortex naturally oscillate at theta frequencies which match the typical syllabic rate of human languages (4-7 syllables per second). In overt speech perception, the brain's neuronal activity "synchronises" with the rhythms of speech to optimise the encoding of speech signal. Our research shows that inner speech, at least in silent reading of speech quotes, is linked to increased phase-locked neural activity in theta frequencies. This phase-locked theta activity originates from the cortical areas that are involved in overt speech processing, which is in line with previous fMRI studies on inner speech. The exact nature of this phase-locked theta activity remains inconclusive, however. In our control study, we have ruled out the possibility that it results from differential eye movements in reading. It is more likely to reflect increased phase resetting of ongoing theta oscillations at the onset of inner speech. Although future research is still needed to further elucidate this neurophysiological signature, it opens up exciting possibilities that we may be able to monitor inner speech via real-time EEG recordings and that we may be able to control inner speech via neurostimulation (e.g., with alternating currents).

As for the spatial loci (neural correlates) of inner speech, our efforts were dedicated to exploring the wider neural networks beyond the speech processing areas. By contrasting silent reading of direct and indirect speech quotes in fMRI, we have replicated previous findings of increased neural activity in the right superior temporal sulcus - an area for speech prosody processing. Importantly, we have additionally identified a Theory of Mind network and a identity representation network to be associated with inner speech processing. Theory of Mind is the brain's ability to attribute mental states to oneself and others. Silent reading of direct speech quotes was associated with increased activity in bilateral temproparietal junctions, bilateral dorsolateral prefrontal cortex, right medial STS and left cerebellum (related to increased Theory of Mind processing and correlated with individual differences in perspective taking), as well as in bilateral inferior frontal areas and anterior temporal lobes (linked to others' voice identity representations and correlated with individuals' tendencies to attribute other people's voices). These findings are in line with the contemporary thinking that inner speech is not a mere auditory experience, but one that may be shaped and driven by social understanding and communication intends.

To further test the social relevance of inner speech, we have pursued the second objective of this project by examining the emotional consequences of hearing inner speech in silent reading. In a series of behavioural experiments, we have successfully demonstrated a consistent facilitation of emotional processing in silent reading of direct (vs. indirect) speech quotes. This quotation-induced emotional advantage cannot be explained by variation in reading times and perhaps surprisingly, cannot be reliably suppressed by concurrent articulation (e.g. saying bah bah bah during reading). It therefore remains debatable whether emotional facilitation is caused by inner speech, or that inner speech is a consequence of increased emotional state attribution in the first place. Although the causal relationships between inner speech and emotional processing still need to be ironed out, we have gathered convincing evidence that inner speech is correlated with the attribution of the emotional states to the speakers.
Exploitation Route Academic impact:
1) Journal publications and data sharing: Our findings will eventually reach out to other researchers via conference presentations and journal publications. We have already presented our results in Society of Neuroscience of Language conference, Embodied and Situated Language Processing conference, Architectures and Mechanisms of Language Processing conference and Experimental Psychology Society conference. Two papers have been published. One paper reports a set of studies on how inner speech may bias syntactic interpretations in silent reading of speech quotations. The other paper reports a set fo studies on how inner speech in silent reading may be phonetically detailed, as evident in visual tongue twister effects (increased reading times) using eye tracking. Our EEG paper on theta phase-locking has undergone several rounds of reviews and will be published in 2021. Our manuscripts on the fMRI results are under preparation, pending results from complementary EEG source estimation analyses. Our behavioural data on the emotional consequences of inner speech are to be presented at the AMLaP conference this year and will be published once we have incorporated follow-up results regarding the causal relationships between inner speech and emotional processing. All our data will be made public in Reshare so anyone could exploit secondary values from the data.

2) New research avenues: This research has attracted several follow-up funding by the Medical Research Council, the British Academy and the Bial Foundation (Portugal) to study the neural mechanisms of auditory verbal hallucinations.

Non-academic impact:
1) Auditory verbal hallucinations: Our findings on neural signatures of inner speech may help to improve the detection and monitoring of auditory verbal hallucinations. We currently focus on hallucinatory experiences in the general population and will soon extend the research to patients.

2) Creative industries and education: Our findings may also inform communication-based industries and the education sector about the impacts of quotations and inner speech on emotional processing. We have so far disseminated our findings to school pupils and the general public via a series of national and regional public engagement events. We are in discussion with our university's business engagement team to develop potential Knowledge Transfer Partnerships with relevant industries.
Sectors Creative Economy,Digital/Communication/Information Technologies (including Software),Education,Healthcare

 
Description Cortical tracking of inner speech in silent reading
Amount £9,942 (GBP)
Funding ID SRG1819\191229 
Organisation The British Academy 
Sector Academic/University
Country United Kingdom
Start 05/2019 
End 09/2020
 
Description Does inner speech modulate word meaning retrieval in silent reading?
Amount £3,500 (GBP)
Organisation Experimental Psychology Society (EPS) 
Sector Charity/Non Profit
Country United Kingdom
Start 04/2021 
End 03/2023
 
Description MRC DTP PhD Scholarship
Amount £84,130 (GBP)
Organisation Medical Research Council (MRC) 
Sector Public
Country United Kingdom
Start 09/2018 
End 03/2022
 
Description Testing a neurophysiological model of inner speech processing
Amount € 45,000 (EUR)
Funding ID 284/2018 
Organisation BIAL Foundation 
Sector Public
Country Portugal
Start 09/2019 
End 08/2020
 
Title Direct speech quotations promote low relative-clause attachment in silent reading of English 
Description The implicit prosody hypothesis (Fodor, 1998, 2002) proposes that silent reading coincides with a default, implicit form of prosody to facilitate sentence processing. Recent research demonstrated that a more vivid form of implicit prosody is mentally simulated during silent reading of direct speech quotations (e.g., Mary said, "This dress is beautiful"), with neural and behavioural consequences (e.g., Yao, Belin, & Scheepers, 2011; Yao & Scheepers, 2011). In this study, we explored the relation between 'default' and 'simulated' implicit prosody in the context of relative-clause (RC) attachment in English. English RC-attachment structures were embedded in direct speech, indirect speech or narrative sentences. Participants either completed sentence fragments ending in incomplete RCs (Experiment 1) or rated the felicity of unambiguous low vs. high RC-attachments in silent reading (Experiment 2) and in oral reading (Experiment 3), respectively. In this data collection, you will find task instructions, data and R scripts for each of the three experiment. 
Type Of Material Database/Collection of data 
Year Produced 2018 
Provided To Others? Yes  
Impact It has been viewed 64 times and downloaded 23 times. 
URL http://reshare.ukdataservice.ac.uk/853105/
 
Title Mental simulations of phonological representations are causally linked to silent reading of direct versus indirect speech 2016-2019 
Description In three experiments, this project explored the phonological aspect and the causal role of speech simulations in silent reading of tongue twisters in direct speech, indirect speech and non-speech sentences. Embodied theories propose that language is understood via mental simulations of sensory states related to perception and action. Given that direct speech (e.g., She says, "It's a lovely day!") is perceived to be more vivid than indirect speech (e.g., She says (that) it's a lovely day) in perception, recent research shows in silent reading that more vivid speech representations are mentally simulated for direct speech than for indirect speech. This 'simulated' speech is found to contain suprasegmental prosodic representations (e.g., speech prosody) but its phonological detail and its causal role in silent reading of direct speech remain unclear. The results demonstrated greater visual tongue-twister effects (phonemic interference) during silent reading (Experiment 1) but not oral reading (Experiment 2) of direct speech as compared to indirect speech and non-speech. The tongue-twister effects in silent reading of direct speech were selectively disrupted by phonological interference (concurrent articulation) as compared to manual interference (finger tapping) (Experiment 3). The results replicated more vivid speech simulations in silent reading of direct speech, and additionally extended them to the phonological dimension. Crucially, they demonstrated a causal role of phonological simulations in silent reading of direct speech, at least in tongue-twister reading. The findings are discussed in relation to multidimensionality and task dependence of mental simulation and its mechanisms. 
Type Of Material Database/Collection of data 
Year Produced 2020 
Provided To Others? Yes  
Impact n/a 
URL http://reshare.ukdataservice.ac.uk/id/eprint/854460
 
Title Reading Direct Speech Quotes Increases Theta Phase-locking: Evidence for Theta Tracking of Inner Speech 
Description Growing evidence shows that theta-band (4-7Hz) activity in the auditory cortex phase-locks to rhythms of overt speech. Does theta activity also encode the rhythmic dynamics of inner speech? Previous research established that silent reading of direct speech quotes (e.g., Mary said: "This dress is lovely!") elicits more vivid inner speech than indirect speech quotes (e.g., Mary said that the dress was lovely). As we cannot directly track the phase alignment between theta activity and inner speech over time, we used EEG to measure the brain's phase-locked responses to the onset of speech quote reading. We found that direct (vs. indirect) quote reading was associated with increased theta phase synchrony over trials at 250-500 ms post-reading onset, with sources of the evoked activity estimated in the speech processing network. An eye-tracking control experiment confirmed that increased theta phase synchrony in direct quote reading was not driven by eye movement patterns, and more likely reflects synchronous phase resetting at the onset of inner speech. These findings suggest a functional role of theta phase modulation in reading-induced inner speech. 
Type Of Material Database/Collection of data 
Year Produced 2021 
Provided To Others? Yes  
Impact The dataset has facilitated the development of new research, funded by the British Academy. 
URL https://reshare.ukdataservice.ac.uk/854892/
 
Description Keynote speaker at the Symposium on Neuroscience by the Society of Spanish Researchers in the United Kingdom 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Public/other audiences
Results and Impact I presented my research on inner speech to around 70-100 people and was on the expert panel for a public debate on neuroscience in education. The talk and the debate sparked many questions and interests on neuroscience and education, especially surrounding innovative teaching approaches and ethical issues in studying neuroscience in educational settings.

The event was organised by the Society of Spanish Researchers in the United Kingdom. The audience were from all walks of life, including students, health care professionals, educators and researchers, and are from many different countries.
Year(s) Of Engagement Activity 2019
URL https://sruk.org.uk/events/nwneuroscience/
 
Description Participation in large-scale public engagement event The Brain Box 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact The science event The Brain Box attracted more than 5000 visitors on the day. Our stand attracted more than 100 participants in our activities. Many of our visitors were very interested in our findings using functional magnetic resonance imaging and shared their experience in hearing "voices" during reading with enthusiasm. Others indicated that they didn't experience "inner voices" when they read, suggesting that there is individual differences in the population. Our visitors reported that our activities were very interesting; they learned that neuroimaging can 'photograph' brain activity and help us study subjective experiences such as inner voices. We, as hosts, also learned what aspects of our activities worked and what aspects didn't. This helps us plan and improve our future engagement activities.
Year(s) Of Engagement Activity 2016
URL https://mcrbrainbox.wordpress.com/learn/
 
Description Science Exhibition at University of Manchester Community Festival 2017 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact Our stand attracted more than 100 participants in our activities. In addition to showcasing our MRI results on inner voice processing with posters and cards, we also designed a fun activity to bring arts and science together. We ask our visitors to listen to voices in different emotions (e.g., happy, sad, angry, etc.) and then depict them on a piece of paper with colour pens. The drawings we collected seem to accurately reflect the emotions the authors heard. It suggests that we might mentally r
Year(s) Of Engagement Activity 2017
URL https://www.youtube.com/watch?v=UkWGYikJvz0
 
Description Science Spectacular at Manchester Museum, as part of the Manchester Science Festival. 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Public/other audiences
Results and Impact Science Spectacular was part of the Manchester Science Festival, organised by The University of Manchester. We participated in this event and held a exhibition stand inside Manchester Museum. We set up posters, information cards, a model brain, and computerised mini-experiments to showcase our research on inner voices to our visitors. Our adult visitors really enjoyed the information and activities. They agreed that different inner voices gave rise to distinct emotional experiences. Teenage visitors seemed to be particularly interested in this topic. Perhaps it is the age when they start to experience inner voices in reading and writing. After learning about inner voices, one teenage boy was inspired and came up a writing method by describing a scene through characters' voices.

However, we also realised that some of our activities were not suitable for younger visitors (~6 years old). Many of them did not recognise the TV characters we used and some of them were not yet able to read. Though kids enjoyed the written stories (read by their parents) nonetheless, we've learned that we needed to adapt our activities for younger visitors.
Year(s) Of Engagement Activity 2016
URL http://brainsciences.wixsite.com/epiclab/single-post/2016/11/23/Witnessing-the-little-voice-Science-...
 
Description Science Uncovered at European Researcher's Night 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact Hundreds of scientists right across Europe took part in Researcher's Night - an event that gets scientists out of the lab and into public spaces to talk about their work. Manchester's own event was the Science Uncovered evening at Manchester Museum, where we set up a stand to quiz people about their experiences with inner speech and talk to them about our research.

Over the course of the night, we spoke to over a hundred visitors who inspired some really interesting conversations. These inclu
Year(s) Of Engagement Activity 2016
URL https://www.flickr.com/photos/mcrmuseum/29523535233/in/album-72157673591998441/
 
Description What Do Psychologists Do? - Widening participation event for local schools in Greater Manchester 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Schools
Results and Impact 300 pupils visited The University of Manchester and experienced what psychologists do in their research and teaching at the university. I gave a talk on my current research on inner voices and demonstrated the magical techniques we use to study the brain. The feedback was extremely positive and many pupils reported increased interested in the subject of psychology and neuroscience.
Year(s) Of Engagement Activity 2017,2018