Joint Text Understanding and Coreference Resolution: Augmenting Recurrent Neural Networks with Entity-Centric Discourse Memory

This research project falls within the natural language processing, artificial intelligence technologies, and human-computer interaction research areas of EPSRC primarily within the ICT theme.

Despite recent advances in machine learning and natural language processing, accurate text comprehension has remained a challenging problem. Prior works often used recurrent neural networks that embed the question, passage, and each answer candidate into real-valued vectors, optionally with an additional attention mechanism.

Such approaches have two drawbacks. First, the finite capacity of the neural network hidden states means that the model has to compress all the relevant information within the network capacity. For instance, the model needs to make the connection that mentions of "Hillary Clinton" and "Clinton" refer to the same person (i.e. coreference resolution task), along with the relation between different entities. Second, the resulting vector embedding is often not easily interpretable to humans.

This research area is an important one, since any meaningful interaction between humans and computers through text or speech requires the ability to properly understand and reason about meaning and context of natural language. Designing intelligent systems that can recognize entities and the relations between them as part of text comprehension would give rise to more accurate and interpretable text comprehension systems.

The aim of this research project is to design statistical models and algorithms that can jointly perform coreference resolution and text comprehension, using an entity-centric memory that stores long-range discourse features. Recent work have demonstrated that reasoning about entities and references is beneficial for language modeling and text comprehension, although the proposed approach learns such coreference information automatically and jointly with the text comprehension task. A second goal of this approach is to improve model interpretability by associating each information with the relevant entity, and explicitly making predictions about coreference links and entity relations.
Our proposed approach similarly leverages the strength of recurrent neural networks and attention mechanisms, but also augments the model with a "discourse memory" that keeps track of each entity and its various mentions within the text. This approach has the benefit of integrating all pertinent information regarding an entity into the same file in a computationally efficient manner, even though each information is mentioned separately and may occur distantly in the passage. While some prior work have similarly investigated an entity-centric approach to text comprehension with discourse-level features, the novelty of our approach additionally lies in performing the coreference resolution jointly with the text comprehension task, which remains a challenging task.