Modeling the Development of Phonetic Representations

Listeners differ cross-linguistically in the cues, or perceptual dimensions, they rely on when perceiving speech. For example, Japanese listeners categorizing English [l] and [r] do not rely on the same acoustic features of the speech signal (e.g., the third formant) that native English listeners do. These cross-linguistic differences are typically attributed to listeners' knowledge of sound categories. For example, English listeners know that [l] and [r] are two categories, whereas Japanese listeners know that [l] and [r] are part of the same category; and this knowledge is hypothesized to affect their reliance on dimensions.

The proposed research tests the hypothesis that category knowledge is not necessary for perceptual dimension learning to occur. Drawing on representation learning methods that have performed well in low-resource automatic speech recognition, where extensive labeled training data are not available, two models are proposed that learn dimensions without relying on knowledge of sound categories. The first relies on temporal information as a proxy for category knowledge, while the second relies on top-down information from similar words, which infants have been shown to use. These models are evaluated on their ability to predict listeners' discrimination judgments from speech perception experiments on native and non-native contrasts, when trained on the same language background as the listeners.

Building models of perceptual dimension learning can significantly impact science and health. For example, greater insight into dimension learning can help us to understand the difficulties that adults face when acquiring a second language, and possibly to develop better methods for teaching second languages. In addition, research suggests that dyslexia and some language learning impairments can be caused by problems with low-level speech processing. A better understanding of perceptual learning could lead to better diagnosis and treatment in these cases.

Our work could also lead to improved speech technology for low-resource languages. State-of-the-art systems rely on hundreds of hours of hand-transcribed data, which is time-consuming and expensive to create. Consequently, high-quality systems are only available for a few languages, and speech recognition researchers are increasingly looking for ways to develop systems that learn from audio alone. Our proposed work draws on existing methods, but explores these in ways that have not been done within the ASR community, including more carefully controlled comparisons between methods and comparisons against human perceptual data. We anticipate that our investigation will lead to insights and perhaps new techniques that can transfer directly to the field of speech recognition and ultimately lead to systems that learn more effectively using little or no transcribed audio. Such systems could become important tools for documenting and analyzing endangered and minority languages, and could help make speech technology more universally available, not just to majority language speakers in rich countries, as most systems are today.