Exploring structural constraints in neural network approaches for Natural Language Processing.

Building effective and efficient models from data that map input structures to output structures is the lifeblood of Statistical Natural Language Processing (NLP). As an example, machine translation maps an input sequence in one language to an output sequence in another language, while syntactic and semantic analysers map input sequences to graphs, which express the structure of the sentence. In all of these cases, outputs that are predicted for a subset of an input sequence severely affect which remaining predictions are plausible. Namely, the outputs are structured.

Neural network models have brought great advances to NLP modelling, since they boast of multiple alluring properties, such as scalable learning on web-scale datasets, learning reusable linguistic representations from raw text and having the ability to combine submodules from multiple modalities (text, image, speech) to learn complex mappings from inputs to outputs. However, the adoption of neural network models for NLP has come with some compromises. It is less clear what neural network models learn about structure. Furthermore, it is generally assumed that a large amount of annotated training data is readily available, an assumption that for many languages and domains (healthcare) is not realistic.

In this project we aim to revisit the benefits of earlier structured approaches and intersect them with the beneficial traits of neural network models. Can we leverage neural models that are structure-aware to obtain effective and efficient models using less data? To this end, we will strive to better understand how neural network models effectively capture structure, despite the fact that most neural models don't explicitly model it. In addition, we will compare approaches that embed structure and constraints in neural models and contrast the dependence of these models on the amount of available training data.