Embodied Language Production

Lead Research Organisation: University of East Anglia
Department Name: School of Psychology

Abstract

We use language to record, preserve and communicate about events, (e.g., reporting a kicking event by saying "A waitress kicks a clown"). But how do speakers encode an event into speech sounds? To do so, speakers need to start from a message and then find words for the message and arrange them into a sentence. However, how speakers construct a message out of, e.g., seeing an event, rarely has been empirically addressed. A standard view assumes that message construction mainly concerns searching concepts in long-term memory for event participants (e.g., CLOWN, WAITRESS) and their relation (e.g., KICK). One problem with this account of language production is that the message is devoid of the rich perceptual, motor and affective (i.e., sensorimotor) content we perceive of an event. For example, the perception of the kicking event above offers the visual imagery of a waitress, a clown and the act of kicking. It has been shown that when people *comprehend* a sentence describing an event, they mentally replay or simulate the sensorimotor content implied in the event, but it yet remains to be explored whether similar sensorimotor simulation is used when they *produce* a sentence to describe an event.
We propose that in language production, speakers use the sensorimotor content to mentally simulate or replay an event they want to speak about. Since sensorimotor content is perspective-specific (e.g., the sensorimotor experience of a kicking event is different for the kicker and the kickee), we hypothesise that speakers prefer to take the perspective of a more similar event participant and to choose a grammatical structure (e.g., active vs. passive) in which the perspective participant is realised as the subject of the sentence. Such an account explains the general finding across languages that speakers tend to use a more human/animate concept as the subject of a sentence (e.g., "A clown is hit by a stone" rather than "A stone hits a clown").
The proposal presents the first attempt to explore language production in terms of embodied cognition (i.e., cognition as mental simulation of sensorimotor content). We examine whether speakers employ perspective-specific sensorimotor simulation in constructing a message out of an event and whether sensorimotor simulation in the message affects the subsequent choice of grammar. In the first series of studies, we ask speakers to describe a picture (e.g., "The waitress passes the clown a ball") while tracking their computer mouse trajectories as a measure of whether they mentally simulate the movement in an event while speaking. In the second series of studies, we aim to explore whether different perspectives of the same event (e.g., a waitress kicking a clown) result in different sensorimotor simulations (let/foot movement from the waitress' perspective but pain from the clown's perspective) by recording speakers' electrophysiological activities. In the third series of studies, we investigate whether speakers tend to take the perspective of an event participant that is more similar to them in gender or emotion and realise that participant as the subject of the sentence (e.g., "The clown is kicked by the waitress" by male speakers). We aim to achieve two main objectives via these studies. Theoretically, the project presents an empirical investigation of conceptualisation, an important but rarely investigated area, and will provide a new perspective (embodied cognition) on language production. Methodologically, we will develop novel paradigms for investigating experienced-based knowledge/thoughts and how they drive symbolic behaviours (e.g., sentences).
Through the project, I aim to develop my skills in theoretical construction in language and cognitive sciences, in using of neuroscientific research methods to study language and cognition, and in using computational models to implement my theoretical proposals.

Planned Impact

Language production (speaking and writing) is one of the most basic skills and probably one that is the most frequently used in our daily life; furthermore, it is an important part of schooling as many professions rely on the creative use of language to describe thoughts and events (e.g., students, journalists, novelists). Thus, besides academic implications for many fields in cognitive science, our research will also have practical implications for the society especially in the areas below.

Early language education
Children vary in their capacity of using language to express a thought/event (e.g., in story-telling), which requires good lexical expressions and grammatical choices. Our work will give us precious information on how speakers encode events for speaking, thus can lead to teaching implications. For example, if we find that speakers replay the sensorimotor content of a to-be-talked-about event from a particular perspective, and then use this simulation to retrieve words and organize the words in a sentence, classroom strategies in which children are encouraged to use their imagery to run such simulations may lead to better linguistic production. It would also be desirable that policy makers for early language education take such a learning strategy into consideration when designing curriculum and textbooks.

Foreign language education
Learning a second language (L2) has been a universal educational practice world-wide. Learning to speak (or write) in an L2 is notoriously more difficult than learning to comprehend (or read). For instance, learners often find it difficult to use a certain grammatical construction (e.g., the passive construction) or an appropriate word (e.g., using the verb "bring" when "take" or "fetch" is intended). While clearly frequency of use matters, it may also be the case the L2 words and grammatical constructions lack sufficient sensorimotor content due to common practices of L1-L2 association in foreign language teaching and learning. Thus, it is important that in teaching words and constructions in a foreign language, teachers create an environment in which learners can internalise the rich sensorimotor content associated with the things/events these words and constructions refer to (e.g., by combining real-world experience with task-based teaching).

UK-China research collaboration
The current project presents opportunities for developing UK-China collaboration and for promoting the UK as a centre of international excellence in the psychology of language. First, the PI has already established good collaborative relations with China. A prominent Chinese psycholinguist, Ruiming Wang from South China Normal University, has expressed interest in collaborating on the proposed project (see SCNU Visit Letter). We have discussed extending the current proposal to bilingualism and second language acquisition (see above) and applying for a grant from the National Natural Science Foundation of China (with the PI as a co-applicant) as an extension to the current project. Second, there is currently much teaching and research in the language sciences in China, but it is failing to make an impact internationally, because of inexperience and lack of appropriate collaboration with international mentors. By collaborating in research, presenting our research findings and presenting the broader state of the art in the language sciences to Chinese researchers, our activities will have impact in terms of raising the profile of UCL (and the UK in general) as a leader for language sciences.

Publications

10 25 50

Related Projects

Project Reference Relationship Related To Start End Award Value
ES/L010224/1 01/03/2015 23/03/2016 £149,161
ES/L010224/2 Transfer ES/L010224/1 24/03/2016 23/03/2018 £87,583
 
Description The project aims to investigate the role of embodiment in language production. While I initially was able to find that people simulate the event they are speaking about, later experiments failed to replicate this finding. In particular, I failed to produce positive evidence in the following three strands of research.

1) Motor simulation when conceptualising an action event. In quite a few experiments (with slightly different paradigms), I failed to produce evidence that speakers simulate an outward-going action when they say something like "I give a book to the sailor" or an inward-going action when they say something like "The professor gives a book to me".
2) Gender simulation in language production. In several experiments (with slightly different paradigms), I failed to produce evidence that speakers tend to place in the subject position a concept of the same gender as the speaker (e.g., female speakers more likely to say "The waitress pushes the tailor" while male speakers more likely to say "The tailor is pushed by the waitress").
3) Perspective simulation in producing noun-phrase conjunctions. In several experiments (with slightly different paradigms), I failed to produce evidence that speakers simulate an event from a particular perspective and produce noun-phrase conjunction accordingly to the adopted perspective. For instance, I failed to show that speakers are more likely to say "Sitting up there on the branch of a tree is a man with a tie and a hat" (where the spatial arrangement from "tie" to "hat" is consistent with looking up) than "Sitting up there on the branch of a tree is a man with a hat and a tie" (where the spatial arrangement from "hat" to "tie" is inconsistent with looking up).

I also failed to show positive evidence for some studies associated with the current grant:
1) Action-sentence compatibility (ACE) effect. In this part of a many-lab replication, I (and other labs) failed to replicate the ACE effect in sentence comprehension. That is, we did not observe evidence that people tend to simulate an outward-going action when comprehending a sentence such as "I give a book to the sailor" or an inward-going action when comprehending a sentence such as "The professor gives a book to me".
2) In an fMRI study, I failed to observe common neural activation between emotion words and space words, despite that emotion words are argued to be spatially grounded (e.g., happiness is up while sadness is down).

In another associated study, we found positive evidence that listeners simulate the linguistic background of the speaker while understanding word meanings. For instance, we showed that British speakers tend to retrieve the hat meaning of "bonnet" more often when they listen to an American English speaker than a British English speaker.

I have developed my neuroscientific and computational skills during the grant period. In particular, I have learned how to design and conduct EEG and fMRI studies and how to analyse the data. I have also learned to run Bayesian models to simulate behavioural data.
Exploitation Route Academically, my research on embodied language production can contribute to the current debate regarding the effects of embodiment in language comprehension and processing in general. The lack of embodiment in language production is, in a hindsight, not surprising at the failure to replicate the embodiment effect in language comprehension. However, it is important to note that the failure to observe embodiment effects might due to the inability to exactly pinpoint when conceptualisation begins.
Sectors Education

 
Description I had a discussion about the potential practical implications of embodied cognition for language teaching with Bin Zeng, a Chinese teacher in Portsmouth. Bin had expressed interest in whether he could develop more theory-informed teaching methods and one of the potential impact that I outlined in the grant application was to disseminate possible findings from the current project (and indeed the whole field in general) to Bin and colleagues so that they could come up with a more theory-informed pedagogy.
First Year Of Impact 2017
Sector Education
Impact Types Societal

 
Description General programme
Amount ¥253,800 (CNY)
Funding ID 31571142 
Organisation National Natural Science Foundation of China 
Sector Public
Country China
Start 01/2016 
End 12/2017
 
Title A large-scale behavioural study and a database on Chinese character handwriting 
Description We reported a psycholinguistic database of Chinese character handwriting based on large-scale study that involved 203 participants each handwriting 200 characters randomly sampled from a cohort of 1600 characters. We collected writing latencies, durations and accuracy. To explore what lexical factors impact handwriting, we conducted regression analyses on the three collected measures using 10 lexical variables: character count frequency, character context frequency, stroke number, radical number, homophone density, phonogram status, spelling regularity, sound radical order, character composition, and age of acquisition (AoA). The results show that access to orthographic codes in handwriting (as reflected by writing latencies) is affected by character count frequency, stroke number, homophone density, spelling regularity, character composition, and AoA, while the motor execution of handwriting (as reflected by writing durations) was affected by character frequency and stroke number, composition and AoA. Handwriting accuracy is dependent on a character's frequency, stroke number, homophone density, spelling regularity and AoA. Further analyses showed that frequency, stroke number and AoA are the most important determinants of handwriting. In addition, the results also speak to the conflicting findings from small-sample lab studies concerning the role of phonology in orthographic access: orthographic access is slower and less accurate when a character has more homophonous neighbours and lower spelling regularity. As the first of its kind, the database from this study can be used as a source of secondary data analyses and a tool for stimulus construction in handwriting research. 
Type Of Material Database/Collection of data 
Year Produced 2018 
Provided To Others? Yes  
Impact This database is currently under review in a research method journal. We have made it public in the Open Science Framework and others can make use of it to investigate cognitive processes underlying (Chinese) handwriting. 
URL https://osf.io/7s9kq/
 
Description Embodied bilingualism 
Organisation South China Normal University
Country China 
Sector Academic/University 
PI Contribution I will contribute to the research design, data analysis and write-up of the collaborative research.
Collaborator Contribution contribute to the research design, data collection, data analysis and write-up of the collaborative research.
Impact No output yet
Start Year 2016
 
Description A practical course on English academic writing 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact This was a course to help young researchers, and PhD and MSc students to develop their English writing skill, developed by Professor Martin Pickering, Professor Holly Branigan and Dr Zhenguang Cai, and organised by Shanghai Jiaotong University.
Year(s) Of Engagement Activity 2017
URL http://mp.weixin.qq.com/s/ja2uQBrO9kS1f-h8GlSTFw
 
Description A practical course on English academic writing 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Postgraduate students
Results and Impact This was a course to help young researchers, and PhD and MSc students to develop their English writing skill, developed by Professor Martin Pickering, Professor Holly Branigan and Dr Zhenguang Cai, and organised by Prof Pickering and Dr Cai, under the sponsorship of the Guangdong Psychological Association.
Year(s) Of Engagement Activity 2016
 
Description A practical course on English academic writing (Xi'an Jiaotong University) 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Postgraduate students
Results and Impact This was a course to help young researchers, PhD and MSc students to develop their English writing skills, developed by Professor Martin Pickering, Professor Holly Branigan and Dr Zhenguang Cai, and organised by Xi'an Jiaotong University.
Year(s) Of Engagement Activity 2017
URL http://gs.xjtu.edu.cn/info/1009/3927.htm
 
Description A presentation at the 39th Annual Meeting of the Cognitive Science Society 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact About 500 people attended the conference and the presentation (on Bayesian inferences in magnitude perception) attracted a lot of interest and questions among the audience.
Year(s) Of Engagement Activity 2017
URL http://www.cognitivesciencesociety.org/conference/cogsci2017/
 
Description A presentation at the First Meeting of Timing Research Forum 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact About 200 people attended the conference and the presentation (on Bayesian inferences in magnitude perception) attracted a lot of interest and questions among the audience.
Year(s) Of Engagement Activity 2017
URL http://timingforum.org/trf-1/
 
Description A presentation at the International Conference on the Processing of East Asian Languages (ICPEAL) in Dec 2016 in Guangzhou, China 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact About 300 people attended the conference and the presentation (on the use of a speaker model in language comprehension) attracted a lot of interest and questions among the audience.
Year(s) Of Engagement Activity 2016
URL http://www.icpeal2016.com/
 
Description A talk at Xi'an Jiaotong University 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Postgraduate students
Results and Impact This was an invited talk on a research topic (the role of context in language comprehension)
Year(s) Of Engagement Activity 2016
 
Description An invited talk at the Department of Linguistics and Modern Languages at the Chinese University of Hong Kong 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Other audiences
Results and Impact About 30 staff members/research students attended the talk (on the mental representations underlying language production and compression); they expressed a lot of interest in the talk and some expressed interest in collaboration.
Year(s) Of Engagement Activity 2017
URL http://www.cuhk.edu.hk/lin/new/doc/seminars/2nd_1617/Poster_20170605.pdf
 
Description Attendance of AMLaP 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact I gave an oral presentation at the annual conference of AMLaP (Europe's leading conference on language processing). The talk was on how accent can modulate people understanding of word meanings and was well received by the audience. The feedback they provided was incorporated into the paper we wrote on the basis of the talk.
Year(s) Of Engagement Activity 2015
URL https://www.um.edu.mt/events/amlap2015
 
Description Discussion of implications of embodied cognition for the foreign/second language teaching 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Professional Practitioners
Results and Impact I had a discussion about the potential practical implications of embodied cognition for language teaching with Bin Zeng, a Chinese teacher in Portsmouth. Bin had expressed interest in whether he could develop more theory-informed teaching methods and one of the potential impact that I outlined in the grant application was to disseminate possible findings from the current project (and indeed the whole field in general) to Bin and colleagues so that they could come up with a more theory-informed pedagogy.
Year(s) Of Engagement Activity 2017
 
Description EPS conference, Jan 2015 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact I gave a oral presentation of some findings from experiments that were supported by the grant. Some of the audience (experimental psychologists) expressed great interest in the findings and in the paradigm that I pioneered. They also gave very useful comments for my further research along that line.
Year(s) Of Engagement Activity 2015
URL http://www.eps.ac.uk/images/epsfiles/2014/jan%20-%202015%20-%20draftfinal.pdf
 
Description Research visit to Xi'an Jiaotong University, China 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Postgraduate students
Results and Impact I was invited to give an introduction to my research to staff and students in both the social sciences department and the foreign language department. I introduced my research on language processing and some staff expressed interest in future collaboration.
Year(s) Of Engagement Activity 2015