Categories and gradience in intonation: Evidence from linguistics and neurobiology.

Lead Research Organisation: University of Cambridge
Department Name: English and Applied Linguistics

Abstract

Abstracts are not currently available in GtR for all funded research. This is normally because the abstract was not required at the time of proposal submission, but may be because it included sensitive information such as personal details.
 
Description Intonation, or the melody of speech, plays a central role in human communication. It is one of many elements in the speech signal that needs to be decoded to translate speech sound into meaning, but its role in speech understanding is crucial, since it can give us immediate cues to the start of a new word or phrase in the speech stream, and to the meaning of utterances. As a consequence, when the intonation is wrong, communication often breaks down.

In spite of its importance, intonation is still very poorly understood. It is notoriously difficult to analyse because it is carried by a continuous sound signal, it has multiple functions, and it interacts with other elements in the speech signal that convey meaning. We know that at some stage in the comprehension process, some of this continuous information is interpreted categorically and decoded into distinct meaningful units, such as a rising pattern that marks a question. However, sometimes it makes a more gradient contribution to meaning, when gradual increases or decreases in a particular feature like pitch convey, for instance, a more angry or less timid tone of voice. These variations in form and their contribution to meaning are closely intertwined, and difficult to disentangle. To make matters worse, they are ignored in virtually all cognitive, neuropsychological and neurobiological studies of intonation. As a result, it is unclear exactly how intonation is realised in speech, what units are involved, how it contributes to speech comprehension, and how it is processed in the brain. This lack of understanding holds back progress in the speech sciences.

The main objective of this research is to test the central principle of the currently predominant theoretical framework for intonation analysis (the 'Autosegmental-Metrical approach'). This principle allows us to analyse the intonational features of the speech signal in a discrete, insightful way by disentangling the interaction between the categorical and gradient information mentioned above. Since it helps us identify the critical features of intonation patterns, we can use it to pin down their role in language processing and the neural architecture that supports it, which is the second aim of this research.

Combining a number of well-established speech production and perception tasks with the latest scanning techniques, we examined the acoustic and perceptual correlates of the relevant categorical and gradient information in the speech signal, and we identified the brain systems that are involved in processing the different types of intonation.

The speech production and perception experiments confirmed that the categorical information in question is signalled, perceived, and processed in a systematically different way from gradient information as proposed in Autosegmental-Metrical (AM) theory. We also found that the listener immediately perceives and exploits the cues to the categorical information in the incoming signal in online interpretation.

The neuroimaging results also provided evidence for AM's central principle, since we found that distinct neural systems were engaged in processing intonation depending on the type of information it conveyed. Also, the subsystems which were active in processing categorical intonational information were the same as those that have been observed for other types of categorical linguistic information in the speech signal (e.g. consonants and vowels).

This project has done much to reinforce the key theoretical construct of a separation between categorical and gradient information in prosody, thus supporting the key principle that underlies almost all current linguistic research in the area; to date, decisive evidence had been elusive. The research also shows, for the first time, that hierarchically organised processing is a universal characteristic of speech processing, encompassing both segmental and intonational properties.

The findings also confirm that intonation is not merely a side effect of biological imperatives related to animal communication (e.g. high pitch being associated with questions or with an uncertain attitude just as high pitch signals fear or danger in small and frightened animals). Instead, we can conclude that much of intonation is in fact properly part of the linguistic system.

This research is also innovative in its methodology in combining acoustic analysis, statistical modelling, and auditory analysis with fMRI, EEG, and perception tests, as well as developing or further enhancing existing analysis techniques. In this way, the project has provided the tools to develop a unified model of prosody which can integrate insights from different research disciplines (i.e. phonetics, linguistic typology, cognitive psychology, and neurobiology), and it offers a template for further research.
Exploitation Route The fundamental research carried out in this project will only have scientific impacts, but indirectly, it may benefit non-academic communities in the longer term.

The findings allow linguists, cognitive psychologists, neuropsychologists, speech engineers and computer scientists to improve their theories and models of language and speech processing. Secondary beneficiaries are applied researchers in neuropsychology, language teaching and speech synthesis and recognition. In these disciplines it has long been known that deficiencies in prosodic performance are a major factor in impeding communication (eg cross-culturally, or in aphasic speech). This is not surprising given the crucial role that prosody and intonation have been shown to play in (human) speech segmentation and recognition.

In all of these disciplines, computer-aided programmes are being developed to optimise speech processing conditions for their respective user groups. For the neuropsychologist, knowing which areas of the brain are crucially implicated in processing intonation for different functions will allow fine-tuning of treatments for stroke rehabilitation and speech therapy. In second language teaching, a better understanding of which aspects of an intonation contour crucially affect how they are processed and perceived by a listener can be used improve teaching methods by helping learners' attention focus on the most critical aspects of the spoken L2. Similarly, in speech synthesis and recognition, a better understanding of the categorical phonological and gradient phonetic information in intonation can be used to help lift success rates which are no longer showing the improvements that are offered by computational and engineering techniques alone.

More generally, the findings advance our insight into the integration of the cues to channels of communication that simultaneously convey linguistic messages, but also information about the speaker's state of mind (such as their attitude or feelings). Our understanding of human communication and the neural and cognitive systems that support it crucially hinges on such insights.

The relevant audience is being reached through publications in international peer-reviewed journals and edited volumes, as well as presentations at specialist workshops and conferences.
Sectors Education,Healthcare

URL http://www.mml.cam.ac.uk/dtal/research/CGI/index.html
 
Description A. The project has had two main kinds of impact: research-based (i) and (ii), and enabling (iii) (i) The project findings have the key implication that any formal linguistic model of intonation needs to distinguish between linguistic and paralinguistic meaning, while accommodating the complexities of the mapping relations between cues, meanings and various types of linguistic structures in the language. The production work which we carried out in this context on French phrasing and intonation as well as the work on the cueing of discourse structure is used as a starting point in a number of research projects - D (i) below. (ii) The second key finding that a specialised neural system supports the processing of linguistic phonological information in intonation as distinct from paralinguistic phonetic information has started to attract interest among neurolinguists - D (ii). (iii) The work on data collection and transcription which integrated project insights relating to intonational segmentation, abstraction, and categorisation is enabling the further development of semi-automated analysis of prosodic phenomena - D (iii). B. The findings and outputs from the project which have had the impacts are the following: (i) The impacts from the speech production and perception experiments concerned the finding that linguistic information in intonation is signalled, perceived, and processed in a systematically different way from paralinguistic information, as has been hypothesised in linguistic theory ('Autosegmental-Metrical theory'): a) multiple phonetic parameters (pitch, duration, loudness, tempo, voice quality and spectral properties) covary in different ways to communicate different intonational meanings, which is true for different types of categorical intonational information: 'grammatical' meaning (e.g. statement, question, or listing; phrasing; Post 2011a, 2011b, Delais-Roussarie et al. 2011, Hudson et al. 2010, Post et al. in prep), as well as discourse meaning (Zellers et al. 2009, 2010, Zellers and Post 2010, 2012) b) the listener immediately perceives and exploits these multiple cues in the incoming signal in online interpretation (Zellers and Post 2010, Post et al. in prep) (ii) Elsewhere, impact was achieved with the findings from the neuroimaging studies which, in confirmation of the findings mentioned in B(i) above, showed differential processing for linguistic and paralinguistic meaning in intonation (Post et al. 2009, 2010, 2011, 2012a, 2012b, 2013, in prep. b; Post et al. under revision): a) distinct, but overlapping neural systems are engaged depending on type of meaning (i.e. linguistic or paralinguistic) b) the subsystems which are engaged in the linguistic interpretation of intonational information are the same as those which support abstraction and categorisation for other types of categorical linguistic information in the speech signal (e.g. consonants and vowels); dissociations in lower-level auditory and higher-level linguistic subprocesses reflect distinctions made in current intonational theory (iii) See C for outputs related to enabling impact. C. Impact was achieved through the usual channels: publications, and a substantial number of conference presentations as listed under publications and engagement activities: (i)-a On cueing 'grammatical meaning', e.g.: - Post, B (2011). Metrical structure and the prosodic hierarchy. In Language and Music as Cognitive Systems. Oxford: Oxford University Press, 32-42. - Post, B (2011). The multi-faceted relation between phrasing and intonation in French. In Hamburger Studies in Multilingualism 10. Amsterdam: John Benjamins, 44-74. - Disentangling linguistic and paralinguistic intonational information: An fMRI study. BAAP, Leeds, Mar 2012. (i)-b On cueing discourse categories, e.g.: - Zellers, M and B Post (2012). Combining Formal and Functional Approaches to Discourse Structure. Language and Speech 55: 119-139. - Zellers, M and B Post (2010). Aperiodicity at Topic Structure Boundaries. Speech Prosody 2010, 100845: 1-4. - Zellers, M, M Gubbian and B Post (2010). Redescribing Intonational Categories with Functional Data Analysis. 11th Interspeech, 1141-1144. (ii) On neural processing, e.g.: - Post, B, E Stamatakis, I Bohr, F Nolan, and C Cummins (2013). Categories and gradience in intonation. An fMRI study. In J. Romero and M. Riera (eds.) Phonetics and Phonology in Iberia. - Post, B, F Nolan, E Stamatakis and T Hudson (2009). Categories and gradience in intonation: Evidence from linguistics and neurobiology. 10th Interspeech. Brighton: Causal Productions Pty Ltd., 2307-2310. (iii) On data collection and transcription, e.g.: - Delais-Roussarie, É and B Post (2013). Corpus annotation: methodology and transcription systems. In The Handbook of Corpus Phonology. OUP. - Post, B and F Nolan (2012). Data collection for prosodic analysis of continuous speech and dialectal variation. In The Oxford Handbook of Laboratory Phonology. OUP, 538-547. - Prosodic transcription, Invited workshop at 'The Prosody-Discourse Interface', Salford, Sept 2011 D. The research has been taken up by: (i)-a Dr Mathieu Avanzi (Université de Neuchâtel) in a Swiss National Science Foundation Post-doctoral Fellowship awarded for a visit to our Lab (2014-2015) to apply our work on phrasing and accentuation to dialect variation (i)-a Aline Russ (University of Konstanz) and Nadine Richter (Freie Universität Berlin) in undergraduate projects on polar questions and focus in French (i)-a Dr Simone Graetzer (University of Melbourne) in a study of the role of prosody in coarticulation, thus testing predictions of Post (2011a, b; planned for 2014-2016). (i)-b Ms Imme Lammertink (Radboud University & Max Planck Institute Nijmegen) during a Lab visit (Jan-July 2014) for an MA dissertation on the processing of intonational cues to conversational discourse by infants (using eyetracking), applying our research in the domain of acquisition. (i)-a&b Calbert Graham (University of Cambridge) in the doctoral dissertation The Phonetics and Phonology of Late Bilingual Prosodic Acquisition: A Cross-linguistic Investigation (2013), separating gradient phonetic and categorical phonological aspects of intonation in the acquisition of grammatical and discourse meaning by L2 learners of English (ii) Joan Borràs Comes (Universitat Pompeu Fabrà) in the doctoral dissertation The role of intonation and facial gestures in conveying interrogativity (2012) in an exploration of the neural traces for intonation-based discourse categories (using ERP) (ii) Dr Kai Alter & team (University of Newcastle; establishing a new research network) in an ERP study showing evidence of different timecourses for the processing of linguistic and paralinguistic intonational information. (ii) indirectly, the Standing Committee of the international conference Phonetics and Phonology in Europe, who asked us to host the conference in 2015 after hearing our contributions at TIE 2010 and PaPI 2011. (iii) Dr Avanzi (see (i)-a above) in his work on semi-automated annotation and analysis of intonation. Unexpected and Potential Future Impacts Unexpected Impacts We expected the production work to garner the usual interest among Linguists, but we had not anticipated that it would form an important basis for Avanzi's successful external application for post-doctoral funding (see D(i)-a). Also, we had not anticipated the work on discourse at the application stage, in particular the results obtained with the multiple methods approach advocated in Zellers and Post (2012). The role of the neuroimaging research in attracting a major conference to Cambridge, and in sparking off ERP work on the same questions, was wholly unforeseen. Potential Future Impacts Scientific impacts Increasing uptake of further outcomes through (invited) lectures and forthcoming publications, in particular: - Intonation has a dual function which is supported by distinct cognitive and neural mechanisms, the one being encoded in the linguistic system, and the other reflecting biological imperatives much more directly (cf. Biological Codes explaining correlations between pitch cues and speaker attitudes/feelings). - Hierarchically organised processing is a universal characteristic of speech processing; dissociations in lower-level auditory and higher-level linguistic subprocesses reflect distinctions made in current intonational theory. In e.g.: - Post et al. (under revision). Question or surprise? J Cog Neurosci. - Post et al. (in prep). The effect of intonational function on cue dependencies in intonational rises. JASA. - Post (2013). Intonation: from animal communication to grammatical encoding. International Conference Language Sciences in the 21st century, Cambridge, UK. Economic and societal impacts The research could also find application in language therapy, language teaching and speech synthesis and recognition. In all of these disciplines, computer-aided programmes are being developed to optimise speech processing conditions for their respective user groups. For the neuropsychologist, knowing which areas of the brain are crucially implicated in processing intonation for different functions will allow fine-tuning diagnostics and treatments for stroke rehabilitation and speech therapy (e.g. schizophrenia and Parkinson's disease). In second language teaching, a better understanding of form-function mappings in intonation can be used to help learners focus on the most critical aspects to acquire, as prefigured in Graham's PhD. In speech synthesis and recognition, it can be used to help improve naturalness and lift success rates that cannot be achieved with computational techniques alone.
First Year Of Impact 2012
 
Description Cambridge Humanities Research Grants Scheme
Amount £8,665 (GBP)
Organisation University of Cambridge 
Sector Academic/University
Country United Kingdom
Start 10/2014 
End 04/2016
 
Description Collaborative grant from the French Agence Nationale de Recherche
Amount € 9,000,000 (EUR)
Organisation National Agency for Research 
Sector Public
Country France
Start 08/2011 
End 09/2020
 
Description An ERP investigation of categorical linguistic and gradient paralinguistic intonation 
Organisation Newcastle University
Country United Kingdom 
Sector Academic/University 
PI Contribution Collaboration with Kai Alter (University of Newcastle) to carry out an ERP study investigating the time course differences in the neural processing of categorical as opposed to gradient information in intonation. The research addresses the primary and secondary objectives of the research proposal. We contributed expertise in the area of intonational phonology and neural processing of this information.
Collaborator Contribution Dr Alter and his team provided the facilities and expertise to design, run, and analyse the data.
Impact Keynote lecture 'Intonation: from animal communication to grammatical encoding'. Language Sciences in the 21st century: The interdisciplinary challenge, Cambridge, UK. 3 - 4 October 2013.
Start Year 2012
 
Description Annotation in the analysis of intonational information in speech 
Organisation Paris Diderot University
Country France 
Sector Academic/University 
PI Contribution Collaboration with Elisabeth Delais-Roussarie (Université Paris VII/CNRS) to review, develop, and apply transcription systems for prosodic information, which crucially revolves around the Categories and Gradience project theme of segmentation, abstraction, and categorisation in intonation.
Collaborator Contribution Prof Delais specifically contributed expertise on formalisation, syntax-phonology interfaces, and corpus analysis.
Impact Delais-Roussarie, Élisabeth and Brechtje Post (2014). Corpus annotation: methodology and transcription systems. In J. Durand, U. Gut and G. Kristoffersen (eds.), The Handbook of Corpus Phonology. Oxford: Oxford University Press. Post, Brechtje and Élisabeth Delais-Roussarie (in press). L'intonation en français. In A. Abeillé, F. Gadet, and D. Godard (eds.), Grande grammaire du français. Paris: Éditions Actes Sud. The collaboration also played an important part in the development of a transcription system for French prosody (reference below), and the French part of an online interactive atlas of Romance prosody: http://prosodia.upf.edu/iari/index.html Delais-Roussarie, Élisabeth, Brechtje Post, Mathieu Avanzi, Caroline Buthke, Albert Di Cristo, Ingo Feldhausen, Sun-Ah Jun, Philippe Martin, Trudel Meisenburg, Annie Rialland, Rafèu Sichel-Bazin, and Hi-Yon Yoo (accepted). Developing a ToBI system for French. In Frota, Sonia and Pilar Prieto (eds.) Intonational Variation in Romance. Oxford University Press.
Start Year 2009
 
Description Functional data analysis of multiple cues to paralinguistic and linguistic intonation 
Organisation Radboud University Nijmegen
Country Netherlands 
Sector Academic/University 
PI Contribution Collaboration with Michele Gubbian (University of Nijmegen) to apply a powerful statistical technique, Functional Data Analysis, to analyse acoustic cues to intonational structure. The envisaged outcome directly addresses the primary objective of the research project (i.e. to provide evidence for the hypothesis that categorical, linguistically used information should be distinguished from gradiently varying information in intonation)
Collaborator Contribution Dr Gubbian contributed his expertise on FDA in the context of prosodic analysis
Impact Zellers, Meg, Michele Gubbian and Brechtje Post (2010). Redescribing Intonational Categories with Functional Data Analysis. Proceedings of 11th Interspeech, Makuhari, Japan: 1141-1144.
Start Year 2010
 
Description Investigation of final rises in Australian English 
Organisation Macquarie University
Department Department of Linguistics
Country Australia 
Sector Academic/University 
PI Contribution Design and materials used in British English studies that were part of the Categories and Gradience project allowed direct comparison with AusEn use of rises; my contribution was at stages of design, development, and interpretation.
Collaborator Contribution The experiment was carried out by the Macquarie team, and their lead drafted the resulting paper.
Impact Schmidt, Elaine, Brechtje Post, Yvan Yuen, Carmen Kung, and Katherine Demuth (2015). The effect of listener and speaker gender on the perception of rises in AusE. Proceedings of the 18th ICPhS, Glasgow.
Start Year 2014
 
Description Categories and gradience : neural systems for speech communication 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact We organised an in international workshop on Neural Systems for Speech Communication at the Department of Theoretical and Applied Linguistics, University of Cambridge, UK, on 26th June 2012. The focus of the workshop was on representations and neural mechanisms for segmentation, abstraction, and categorisation in speech processing.
Our invited speakers were
• Kai Alter (University of Newcastle)
• Matt Davis (MRC Cognition and Brain Sciences Unit, Cambridge)
• Aditi Lahiri (University of Oxford)
The workshop provided a forum for discussion between researchers who approach the neurocognitive processing of suprasegmental and segmental information in speech from different intellectual contexts (notably linguistics, audition, language psychology, and cognitive neuroscience), and with different methodologies.

New research collaboration using EEG to investigate neural processing of intonational information
Year(s) Of Engagement Activity 2012
 
Description Getting a handle on category membership in intonation 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Other academic audiences (collaborators, peers etc.)
Results and Impact School of Linguistics & English Language established seminar series; talk sparked discussion and suggestions

Colleagues in Bangor have invited me since to participate in a workshop, and now, as external examiner.
Year(s) Of Engagement Activity 2010
 
Description Intonation in discourse : gradient or categorical behavior? 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? Yes
Geographic Reach International
Primary Audience Other academic audiences (collaborators, peers etc.)
Results and Impact A main problem in prosody research is the attempt to separate out the linguistic from the

paralinguistic aspects of intonation. Ladd (1996:36) has argued that gradience is the "defining

characteristic" of paralanguage, whereas linguistic structures are categorical. However, especially

within the realm of intonational phonology, categorical structures have proved problematic to

identify. In particular, the attempt to isolate such structures within the context of discourse above

the level of the intonational phrase (IP) has thus far received relatively little attention in

intonational phonology.

Talk sparked discussion
Year(s) Of Engagement Activity 2009
 
Description Intonation in the brain 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? Yes
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact Presentation of project to the wider public at the 2009 Cambridge Festival of Ideas; an annual public engagement event which celebrates the arts, humanities and social sciences. The Public Engagement team at the University of Cambridge co-ordinate a series of talks, hand-on activities and debates together with academics. BBC radio were present for some of the afternoon.

Sparked many questions in the audience
Year(s) Of Engagement Activity 2009
 
Description Intonation: from animal communication to grammatical encoding 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other academic audiences (collaborators, peers etc.)
Results and Impact Lecture at 'Language Sciences in the 21st century: The interdisciplinary challenge' Cambridge, UK. 3 - 4 October 2013

Several members of the audience made valuable suggestions about the methodology and interpretation of my data, coming from very different disciplines within the language sciences.
Year(s) Of Engagement Activity 2013
 
Description Phonology in speech processing 2009/10 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Postgraduate students
Results and Impact RCEAL, University of Cambridge; scheduled MPhil and PhD weekly seminar series (8 one-hour sessions) taught by Brechtje Post throughout the Lent term 2009/2010 in the context of research on project. Students learned about cognitive and neural processing of speech.

Enhanced knowledge and interest among students
Year(s) Of Engagement Activity 2009,2010
 
Description Prosodic transcription 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? Yes
Geographic Reach International
Primary Audience Other academic audiences (collaborators, peers etc.)
Results and Impact Invited half-day workshop on prosody and prosodic annotation at the international conference The Prosody-Discourse Interface. Audience reported greater understanding of issues and technicalities.

Enhanced interest and understanding among academic audience; workshop sparked new collaborations with members of the audience (e.g. Piet Mertens, Univerity of Louvain).
Year(s) Of Engagement Activity 2011
 
Description Quand frontières prosodiques et frontières syntaxiques se rencontrent 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Other academic audiences (collaborators, peers etc.)
Results and Impact Invited lecture at workshop "Unités syntaxiques et unités prosodiques" in which we explore the interrelatedness between morphosyntactic structure, prosodic phrasing, and intonation contours. Sparked discussion .

Led to invited publication in A-rated journal
Year(s) Of Engagement Activity 2010
 
Description Radio interview 'Tuning into the melody of speech' 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? Yes
Geographic Reach Regional
Primary Audience Media (as a channel to the public)
Results and Impact Interview with PI on BBC Radio Cambridgeshire Breakfast programme

Further invitation for interview broadcast online
Year(s) Of Engagement Activity 2013
 
Description Sound to sense : the melody of speech 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Public/other audiences
Results and Impact The Festival of Ideas is an annual public engagement event which celebrates the arts, humanities and social sciences. The Public Engagement team at the University of Cambridge co-ordinate a series of talks, hand-on activities and debates together with academics, targeting the general audience and professionals (e.g. language teachers).

Talk sparked many questions from the public.
Year(s) Of Engagement Activity 2011
 
Description Tuning into the melody of speech 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact Research Horizons publication about the project targeted at a general audience (via University front page and hard copy).

Invitation for radio interview
Year(s) Of Engagement Activity 2013
 
Description Working with annotated speech data for quantitative prosodic analysis 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other academic audiences (collaborators, peers etc.)
Results and Impact Invited lecture at workshop on developing standards for phonological corpora which sparked extensive discussion

Further collaboration (e.g. on Handbook of Corpus Phonology)
Year(s) Of Engagement Activity 2009