From icon to abstraction in sign language: how iconicity shapes the lexicon in the visual modality
Lead Research Organisation:
University of Birmingham
Department Name: Department of English Literature
Abstract
Language is a unique human ability, yet the forces shaping vocabulary are little understood. A central tenet of linguistic theory is that the lexicon is arbitrary, with word and meaning related by convention alone1. However, recent research gives a more nuanced view: arbitrary form-meaning relationships co-exist with and complement iconic and systematic relationships2,3. Iconicity, the direct relationship between form and meaning, has been shown to play an important role in language learning and processing, offering a concrete grounding to our experience in the world3-6. As a result, iconicity has been characterised as having a limited capacity to refer to abstract concepts and thus as remaining limited in its scope within the lexicon of spoken languages7. Compared to the spoken modality, the visual modality of sign languages affords a high degree of iconicity. In this project, we explore how iconicity constrains form-meaning mappings in the visual modality and shapes a lexicon for both concrete and abstract concepts.
We capitalise on the notion that words develop abstract meaning through the figurative use of concrete concepts while maintaining links to their origin, e.g. the concrete and abstract meaning of anchor share a core meaning of "preventing something from moving"8. We investigate sign languages, the natural languages of deaf communities. These are full-fledged languages on all levels of linguistic organisation and which exhibit a high prevalence of iconicity in their lexicon. Iconicity patterns in systematic ways across semantic domains9,10, and very often signs converge in the way concepts are expressed (e.g. the sign TO-WRITE). Importantly, sign languages exhibit instances of abstract concepts arising from concrete, iconic referents. E.g., in German Sign Language (DGS), the sign SCHOOL shares an iconic base with the sign TO-WRITE (Fig. 1). This suggests that groups of iconic signs share elements that are exploited as a springboard for meaning diversification while retaining some degree of semantic relatedness (i.e. process of colexification11). The iconic base of signs may be shared across sign languages (e.g. DGS and British Sign Language (BSL)), and the gestures produced by speaking communities may also have similar iconic forms (Fig. 1) due to shared conceptual representations and the shared modality. By comparing the form and meaning of signs across semantic domains in two unrelated sign languages (DGS and BSL), it will be possible to understand the underlying mechanisms by which iconicity shapes and constrains a lexicon. Comparing signs with silent gestures produced for the same concepts by non-signers in Germany and the UK will reveal the extent of shared cognitive and/or cultural bases and how these may diversify lexical forms and meanings for concrete and abstract concepts.
We capitalise on the notion that words develop abstract meaning through the figurative use of concrete concepts while maintaining links to their origin, e.g. the concrete and abstract meaning of anchor share a core meaning of "preventing something from moving"8. We investigate sign languages, the natural languages of deaf communities. These are full-fledged languages on all levels of linguistic organisation and which exhibit a high prevalence of iconicity in their lexicon. Iconicity patterns in systematic ways across semantic domains9,10, and very often signs converge in the way concepts are expressed (e.g. the sign TO-WRITE). Importantly, sign languages exhibit instances of abstract concepts arising from concrete, iconic referents. E.g., in German Sign Language (DGS), the sign SCHOOL shares an iconic base with the sign TO-WRITE (Fig. 1). This suggests that groups of iconic signs share elements that are exploited as a springboard for meaning diversification while retaining some degree of semantic relatedness (i.e. process of colexification11). The iconic base of signs may be shared across sign languages (e.g. DGS and British Sign Language (BSL)), and the gestures produced by speaking communities may also have similar iconic forms (Fig. 1) due to shared conceptual representations and the shared modality. By comparing the form and meaning of signs across semantic domains in two unrelated sign languages (DGS and BSL), it will be possible to understand the underlying mechanisms by which iconicity shapes and constrains a lexicon. Comparing signs with silent gestures produced for the same concepts by non-signers in Germany and the UK will reveal the extent of shared cognitive and/or cultural bases and how these may diversify lexical forms and meanings for concrete and abstract concepts.
| Description | The preliminary findings of the data collection thus far shows that British Sign Language (BSL) and German Sign Language (DGS) have significant overlap because some signs have almost the same form. For example, the sign TABLE in both sign languages is produced by tracing a flat surface. However, there are some other signs (mainly abstract concepts) that do not have the same degree of overlap. For instance, the signs CULTURE in both sign languages have very distinct forms. This is an important finding because BSL and DGS are unrelated sign languages so there is not genealogical relationship. After we quantify and carry out the next stage of the analysis we will be able to confirm our original hypothesis. Namely, that sign languages have similar forms for concrete concepts (e.g., table) but quite distinct when it comes to abstract concepts (e.g., culture). Once we can continue with data collection from deaf signers we will be able to continue with the next stage of the study (gestures from hearing people in Germany and UK). |
| Exploitation Route | These findings are important because the data generated in this study will help us make accurate estimations about language emergence and evolution. Scientists specialised in evolution can use the data and make approximations about the timescale and trajectory of the origins of sign languages. Also, by understanding how signs and their multiple meanings (polysemy) it will be possible to device interventions for deaf children learning a sign language as first language. It will also be possible to create a cognitive map of deaf people and their vocabulary to develop assessment tools needed to explore conditions such as Alzheimer and dementia. |
| Sectors | Education,Healthcare,Culture, Heritage, Museums and Collections |
| Title | Database of gestures by hearing speakers |
| Description | This database consists of 109 videos of silent gestures that can be regarded as highly systematic within a community of Dutch speakers. These gestures were the most commonly produced by minimally 12 out of 20 participants and thus can be generalised to the Dutch population. Appendix I includes a description of each silent gesture according to their four features: hand configuration, orientation, movement, and place of articulation (Bressem, 2013). Appendix III includes the iconicity ratings of these gestures by a different group of participants. Appendix III also reports the mode of representation (Muller, 2013; 2016), i.e., the type of iconic depiction, of each gesture. Appendix II includes a list of concepts that did not elicit a systematic gesture within our criteria. The zip folders include videos of the 109 concepts that elicited systematic gestures in the formats .mpeg and .mp4. These videos are freely available for public use for non-commercial purposes. The data in this repository is reported in: Ortega, G. & Özyürek, A. (2019). Systematic mappings between semantic categories and types of iconic representations in the manual modality: a normed database of silent gesture. Research and Behavior Methods. DOI: 10.3758/s13428-019-01204-6 |
| Type Of Material | Database/Collection of data |
| Year Produced | 2018 |
| Provided To Others? | Yes |
| Impact | A database of silent gestures as produced by a community of speakers can help in the design of experiments that are more ecologically valid. It has also been suggested that although patients with aphasia struggle to retrieve lexical labels as a result of their condition, they can still communicate through silent gesture, with some gestural forms being more accessible than others (van Nispen, van de Sandt-Koenderman, Mol, & Krahmer, 2016; van Nispen et al., 2017). Similarly, children with specific language impairment have been reported to compensate for their inability to produce lexical labels in speech by replacing them with gestures. |
| URL | https://osf.io/w4apb/ |
| Description | Iconicity in sign and gesture in Germany and UK |
| Organisation | University of Cologne |
| Country | Germany |
| Sector | Academic/University |
| PI Contribution | Dr Gerardo Ortega (PI in the UK) - hold weekly meetings to manage the binational project - daily supervision of British team - providing training sessions on linguistic analysis - teaching lessons on the use of the linguistic annotation software ELAN - budget allocation - supervision during data collection - ethical application of the study and overviewing good scientific practice Carla Pol and Rachel England (research assistants in the UK) - creation of COVID-safe protocols for data collection - purchase of equipment - creation of coding scheme for linguistic analysis - recruitment of deaf participants - implementation of software for presentation of stimulus materials |
| Collaborator Contribution | Prof. Pamela Perniss - hold weekly meetings to manage the binational project - daily supervision of German team - setting up the digital archive and shared folders - coordination of interpreters across sites (British Sign Language/ German Sign Language) - providing training sessions on data collection and linguistic analysis - leading journal clubs Nia Lazarus (PhD student), Annika Schiefner (PhD student) - data collection from deaf signers - annotation of data - programming the code on PsychoPy (psycholinguistic software) - coordination of sign language interpreters - archiving of resources for all studies |
| Impact | The current grant was awarded as part of the binational scheme between the AHRC-DFG where two partners (one in the UK and one in Germany) collaborate in a joint research project. The PI in Germany is Prof. Pamela Perniss who has hired two PhD students (Nia Lazarus, Annika Schiefner) and both Prof. Perniss and myself give joint supervision. Due to the current COVID restrictions, both PhD students had to start their contracts in autumn. In addition, Miss Lazarus is an American citizen and she experienced several delays in her arrival to Europe because of frequent cancellations of her flights. For these reasons, both PhD students couldn't start data collection until September. However, we've been in constant communication remotely and they have been analysing videos of German Sign Language. I am certain that they will generate some concrete results in 2021. |
| Start Year | 2020 |
| Description | 71. StuTS and 31. Tacos Leeuwarden, Netherlands |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Postgraduate students |
| Results and Impact | This presentation took place in Leeuwarden, Netherlands. In this talk, our team presented preliminary results of our project in a postgraduate conference, which is a student-led academic outlet. |
| Year(s) Of Engagement Activity | 2022 |
| Description | Comparing sign language lexica at the intersection of phonology and iconicity |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Postgraduate students |
| Results and Impact | Our team attended the 14th edition of the conference on Theoretical Issues on Sign Language Research (TISLR14) in Osaka, Japan (26-Sep - 01-Oct-2022). This is the most important conference on sign language linguistics and it allows to present the latest scientific work. |
| Year(s) Of Engagement Activity | 2022 |
| URL | https://www.tislr2022.jp/ |
| Description | Iconicity in concrete and abstract concepts: a cross-linguistic and cross-cultural comparison of gestures and signs |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Postgraduate students |
| Results and Impact | Our team attended, the 9th conference of the International Society for Gesture Studies (ISGS9) in Chicago, IL, USA (13 - 15- July-2022). In this conference, we presented our work comparing two signs languages (British and German Sign Language) as well as gestures of British and German speakers. This is the most prestigious academic conference on bodily communication. |
| Year(s) Of Engagement Activity | 2022 |
| URL | https://isgs9conference.wixsite.com/isgs9 |
| Description | Online talk to deaf community in Mexico |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Public/other audiences |
| Results and Impact | I was invited to give a talk at the Workshop at the Human Rights Commission in Mexico City (March 3, 2021). The title of my talk was: Aplicaciones y retos de las investigaciones en las lenguas de señas (Applications and challenges in sign language research). In this talk I addressed members of the deaf community and explained how our research can be translated to interventions in teaching and education. The talk is available (in Spanish) in my YouTube channel. |
| Year(s) Of Engagement Activity | 2021 |
| URL | https://bit.ly/3eRvPaI |
| Description | Presentation day of the gradschool |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Postgraduate students |
| Results and Impact | In this talk, postgraduates students heard about sign language linguistics and how to conduct research with deaf communities. |
| Year(s) Of Engagement Activity | 2022 |
| Description | Talk at the UK Cognitive Linguistics Conference |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Other audiences |
| Results and Impact | I was invited to give a talk about iconicity in sign languages and gesture. I was the sole presenter and I gave the theoretical framework of our study (we hadn't carried out data collection by then). This workshop consisted of 5 talks from international speakers who specialise on the topic of iconicity. The workshop was organised by Mutsumi Imai (Keio University) and Marcus Perlman (University of Birmingham). This was a very important event because the most notable academics on the topic discussed their most recent findings. |
| Year(s) Of Engagement Activity | 2020 |
| URL | https://www.ukclc2020.com/pre-conference |
| Description | When iconicity diverges - evidence from the lexicon of two sign languages |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Our team had the opportunity to present the first results of the project at13th conference of the Iconicity in Language and Literature 13 (ILL13) in Paris, France (31- May - 02-June-2022). This is one of the most important academic outlets that investigate iconicity from a literary and linguistic perspective. |
| Year(s) Of Engagement Activity | 2022 |
| URL | https://ill13.sciencesconf.org/ |
| Description | Why rain and water don't go together, but friends and elasticity do: An exploratory study of colexification in two sign languages |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Postgraduate students |
| Results and Impact | Our team presented an invited talk at PhD Candidate Workshop: Current issues in sign language research at the University of Cologne, Cologne, Germany (6 May 2022). |
| Year(s) Of Engagement Activity | 2022 |
| Description | Why rain and water don't go together, but friends and elasticity do: An exploratory study of colexification in two sign languages |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Postgraduate students |
| Results and Impact | In this conference, our team presented a peer-reviewed paper presented on our preliminary findings at the 13th International Symposium on Iconicity in Language and Literature (ILL13), Paris, France (31 May 2022). |
| Year(s) Of Engagement Activity | 2022 |
| URL | https://ill13.sciencesconf.org/ |
| Description | Why rain and water don't go together, but friends and elasticity do: An exploratory study of colexification in two sign languages |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Postgraduate students |
| Results and Impact | In this conference, our team presented a peer-reviewed poster at the 9th Conference of the International Society for Gesture Studies (ISGS9) in Chicago, Illinois, United States (14 July 2022). |
| Year(s) Of Engagement Activity | 2022 |
| URL | https://isgs9conference.wixsite.com/isgs9 |
| Description | Why rain and water don't go together, but friends and elasticity do: An exploratory study of colexification in two sign languages. |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Postgraduate students |
| Results and Impact | In this conference, our team presented a peer-reviewed paper presented at the 14th International Conference on Theoretical Issues in Sign Language Research (TISLR14) in Osaka, Japan. |
| Year(s) Of Engagement Activity | 2022 |
| URL | https://www.tislr2022.jp/ |
| Description | Why rain and water don't go together, but friends and elasticity do: An exploratory study of colexification in two sign languages. |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Postgraduate students |
| Results and Impact | Our PhD student Nia Lazarua was a invited to give a talk at the Bedny Lab for Neuroplasticity and Development at Johns Hopkins University, Baltimore, Maryland, United States (1 - August -2022). |
| Year(s) Of Engagement Activity | 2022 |