Computational neuroscience of natural language learning in the brain

Understanding how information processing is done in the visual and the auditory pathways in the brain is important for understanding speech recognition. The current project will aim to create a biologically plausible model of the auditory system and, after further investigations with a biologically plausible model of the visual system, VisNet, bring the two pathways together to investigate how word learning might operate in the brain. Our approach is distinctive in that we will investigate processing across these two areas through the development of biologically accurate neuro-dynamical models of these pathways, and explore how representations develop in these areas as the models are exposed to realistic visual and auditory sensory input. Our model of the auditory system will be based on the neurophysiology of the brain by modeling it using biologically realistic neural networks with associative learning rules. Although neural networks have been used in speech recognition before, previous models have typically used supervised learning, which is not biologically plausible. We expect that the behaviour of our model will be critically dependent on the precise dynamical properties of the neurons, synapses and learning rules that constitute the network model. It is expected that the model will be able to cope with such problems as different speeds of speech and different speakers by creating representations of words, which are reasonably robust to these dimensions of variability. Another issue we will explore is the possible development of representations of word categories during learning. The development of such categories may be guided by a number of factors. For example, the statistics of where words occur in sentences (syntax) may drive representations reflecting the parts of speech. There is a lot of evidence for audio and visual information integration in the brain. For example, neurophysiological studies have shown that visual and auditory information is integrated in the superior colliculi. There is also evidence suggesting that the two pathways can affect each other. Probably the most extensively studied example of this is the McGurk effect. A biologically plausible neural network model of the ventral visual stream, VisNet, will be used to study the visual pathway. In order to model the interaction between the visual and language areas of the brain, we need to understand how visual scenes are represented in the later stages of the visual pathway. For example, how might activity within the output layer of the visual pathway represent individual objects, multiple objects, and the parts of objects? How might the identity and emotional expression of faces be represented? The way in which such information is represented in the visual system may have a fundamental effect on the nature of the interaction between vision and language during development. Following the proposed separate investigations for the visual and the auditory pathways, the two models will be combined to investigate language acquisition constraints such as the taxonomic, the whole object and the mutual exclusivity constraints. These constraints will be modelled by introducing bilateral modifiable connections between the output layers of the auditory and the visual pathways. Once the model can explain how the constraints operate, it can be used to study how further word learning happens, where these constraints are broken and children learn to associate new words with individual instances of categories, or parts of objects or have more than one label for an objects. This is expected to happen because of the statistical properties of the stimuli. Through our dual-pathway model, we aim to understand how word learning actually happens in the infant brain, and how children's vocabulary gets expanded during development. We hope this research will ultimately lead to more powerful speech recognition models than the existing engineering approaches.