Inference mechanisms using pragmatics and detailed knowledge for Commonsense reasoning

Lead Research Organisation: University of Leeds
Department Name: Sch of Computing

Abstract

Commonsense reasoning (CSR) is a key component of any truly artificially intelligent agent; moreover the ability to interpret questions posed in natural language (NL) is important for many applications of AI. This project aims to investigate and make progress in this important but challenging area.

A key aspect of CSR involved in question answering tasks is the ability to draw on a wealth of knowledge as well as appeal to common assumptions about the world and interpret terms in a defeasible manner. The task of structuring such knowledge is a huge challenge and is an active area of research. Well-developed cognitive theories exist involving pragmatics, dealing with common assumptions and interpretations of terms; however little is said on how to extract rules from these theories. The question then is, how can we leverage existing knowledge bases while maintaining ideas from pragmatics in order to solve question answering benchmarks?

NL can be very informationally dense, allowing speakers to convey a lot of information in only a few words. There is a wealth of research relating to theories of discourse and how humans manage to deal with this semantic under specification in communication. Only with pragmatic inferences can a sentence be transformed into a formal representation of the intended proposition of the speaker. Therefore any system attempting to interpret NL should incorporate mechanisms for inferring this hidden information. Further, being able to account for these sorts of inferences can be very important in CSR for AI, in particular in the Winograd Schema Challenge (WSC). However, the difficulty of the task means that little work exists on formalising this pragmatic knowledge or integrating it into reasoning systems.

Though there exist attempts to formalize some aspects of pragmatics, as pointed out by Bunt and Black some parts of pragmatics "do not enjoy a wealth of representational formalisms". One interesting avenue from the perspective of pragmatics is how to deal with prototypes. As I discuss in my paper, appealing to prototypes is an important aspect of CSR. This raises two particular questions, firstly how do we identify prototypes? Secondly, how do we identify when using prototypes is inappropriate and in these cases how do we change/loosen the definitions we are using? For instance, when one reasons about the sentence "The trophy doesn't fit into the suitcase because it is too large", what is too large? It is not necessary to worry about a precise semantic commitment for the notion of 'large', but instead to evaluate the sentence considering an interpretation of large which satisfies most notions of large. There is also the interesting case of spatial prepositions. A spatial preposition like 'on' is heavily underspecified; it can be used to denote a variety of spatial configurations - the glass on the table, the picture on the wall etc. Though it appears to have a prototypical definition - being above and in contact with - it often just denotes a salient spatial relationship between two objects and then from knowledge of the two objects and context we infer a more precise configuration. Understanding when to use a prototypical 'on' is a similar problem to 'large' - interpreting the intention of the speaker and world knowledge is required.

The aim of this research is to develop systems for interpreting NL while incorporating detailed knowledge and using the principles of pragmatics to guide this. The WSC helps motivates this discussion, however formalizing the necessary aspects of reasoning to tackle the WSC and integrating them into one system is notoriously hard. Therefore, the WSC is not necessarily a suitable problem to tackle. As we will be trying to extract complex pragmatic information it makes sense to work in an area where semantic representations are well developed and understood, therefore I intend to restrict to work on problems in the spatial domain.

Studentship Projects

Project Reference Relationship Related To Start End Student Name
EP/N509681/1 01/10/2016 30/09/2021
1961029 Studentship EP/N509681/1 01/05/2017 31/10/2021 Adam Louis Richard-Bollans
 
Description There has been much work in linguistics and cognitive science regarding the semantics of spatial prepositions ('in', 'on', 'above' etc..), highlighting the complexity of representing their meaning. However, much of this insight has not yet been translated into a satisfactory computational model. Most existing models of spatial prepositions take a very simple approach and ignore many features that may be salient as well as the various different ways the terms may be understood. So far we have provided methods for formalising some of these often ignored features and shown that models can be generated from data which account for a wide variety of features. This work will also hopefully lead to a method which allows the various different senses a spatial preposition may express to be accounted for.
Exploitation Route Modelling spatial prepositions is crucial for many applications involving human-robot interaction. It is possible that the outcomes of this project are implemented to help robot agents better manage spatial language.
Sectors Digital/Communication/Information Technologies (including Software)

 
Title Investigating the Dimensions of Spatial Language: A Preliminary Study 
Description In order to collect rich data on spatial prepositions, we have created a framework for gathering user annotations in 3D virtual environments. The framework is built using the Blender 3D modelling software and game engine. The framework initially provides two distinct tasks with which our preliminary study was conducted - a Selection Task and a Description Task. In both tasks, participants are given a first person view of a scene which they can navigate using the mouse and keyboard. In the Selection Task participants are given a preposition on screen and asked to select all figure-ground pairs in the scene which fit the preposition. Once they have selected all pairs they believe to be admissible they are shown another preposition and asked to repeat the process. In our preliminary study, we limited this task to 'in', 'inside', 'on', 'on top of', 'against', 'over', 'under', 'above' and 'below'. In the Description Task objects are highlighted and participants are able to type in a spatial description of the object. In our preliminary study we asked participants to give descriptions of the object locations using a definite description, in the format figure + preposition + ground e.g. 'the guitar by the bookshelf'. We also allowed the use of multiple prepositions if the participant deemed it necessary e.g. 'the cup on the table near the lamp'. To make the annotations more relevant to our research we asked participants to only use the prepositions in the Selection Task plus 'to the right of', 'to the left of', 'in front of', 'behind', 'near', 'next to', 'at'. The current dataset is from a session we hosted in one of our computing labs where we invited participants from across the university campus to take part. We asked participants to complete either the Selection Task or Description Task in one of our scenes. Participants were given a brief introduction to the study and instructions on how to complete the given task along with explanations of key terminology. All participants were asked to create annotations for ten minutes. 23 native English speakers participated, of which 13 performed the Selection Task and 10 performed the Description Task 
Type Of Material Database/Collection of data 
Year Produced 2019 
Provided To Others? Yes  
Impact Spatial prepositions in the English language can be used to denote a vast array of configurations which greatly diverge from any typical meaning and there is much discussion regarding how their semantics are shaped and understood. Though there is general agreement that non-geometric aspects play a significant role in spatial preposition usage, there is a lack of available data providing insight into how these extra semantic aspects should be modelled. This paper is aimed at facilitating the acquisition of data that supports theoretical analysis and helps understand the extent to which different kinds of features play a role in the semantics of spatial prepositions. We first consider key features of spatial prepositions given in the literature. We then introduce a framework intended to facilitate the collection of rich data; including geometric, functional and conventional features. Finally, we describe a preliminary study, concluding with some insights into the difficulties of modelling spatial prepositions and gathering meaningful data about them. 
URL http://archive.researchdata.leeds.ac.uk/558/
 
Title Study on the Semantics of Spatial Language 
Description Building on a previous study to collect rich data on spatial prepositions, we have conducted a study to gather annotations related to spatial language in 3D virtual environments. The data collection environment is built using the Unity3D modelling software and game engine. The study comprised two tasks --- a Preposition Selection Task and a Comparative Task. The Preposition Selection Task allows for the collection of categorical data while the Comparative Task provides typicality judgements. In the Preposition Selection Task participants are shown a figure-ground pair (highlighted and with text description) and asked to select all prepositions in the list which fit the configuration. Users may select `None of the above' if they deem none of the prepositions to be appropriate. In the Comparative Task a description is given with a single preposition and ground object where the figure is left ambiguous. Participants are asked to select an object in the scene which best fits the description. Again, participants can select none if they deem none of the objects appropriate. In both tasks, participants are given a first person view of an indoor scene which they can navigate using the mouse and keyboard. To allow for easy selection, objects in the scene are indivisible entities e.g. a table in the scene can be selected but not a particular table leg. We limited both tasks to prepositions: `in', `inside', `on', `on top of', `against', `over', `above', `under' and `below'. The current dataset is from an online study where participants were recruited via internal mailing lists along with recruitment of friends and family. For the study 67 separate scenes were created in order to capture a variety of tabletop configurations. Each participant performed first the Preposition Selection Task on 10 randomly selected scenes and then the Comparative Task on 10 randomly selected scenes, which took participants roughly 10 minutes. Some scenes were removed towards the end of the study to make sure each scene was completed at least 3 times for each task. 32 native English speakers participated in the Preposition Selection Task providing 635 annotations, and 29 participated in the Comparative Task providing 1379 annotations. As the study was hosted online we first asked participants to show basic competence. This was assessed by showing participants two simple scenes with an unambiguous description of an object. Participants are asked to select the object which best fits the description in a similar way to the Comparative Task. If the participant makes an incorrect guess in either scene they are taken back to the start menu. 
Type Of Material Database/Collection of data 
Year Produced 2020 
Provided To Others? Yes  
Impact In cognitive accounts of concept learning and representation three modelling approaches provide methods for assessing typicality: rule-based, prototype and exemplar models. The prototype and exemplar models both rely on calculating a weighted semantic distance to some central instance or instances. However, it is not often discussed how the central instance(s) or weights should be determined in practice. The current dataset allowed us to explore how to automatically generate prototypes and typicality measures of concepts from data, introducing a prototype model and discussing and testing against various cognitive models. We conclude that our model provides significant improvement over the other model and also discuss the improvements given by a novel inclusion of functional features in our model. 
URL https://doi.org/10.5518/764
 
Description COSIT Doctoral Colloquium Presentation 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Presented current research project and approaches to the challenge. Promoted interesting discussions afterwards
Year(s) Of Engagement Activity 2019
URL https://cosit2019.ur.de/index.php
 
Description Commonsense Presentation 2017 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Presented our paper 'The Role of Pragmatics in Solving the Winograd Schema Challenge' to researchers at the conference, leading to discussion of related topics and potential future collaborations.
Year(s) Of Engagement Activity 2017
URL http://eprints.whiterose.ac.uk/122937/
 
Description ESSLLI Student Session 2018 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact Gave a talk to summer school attendees outlining my research project, this led to lots of discussions of related topics.
Year(s) Of Engagement Activity 2018
URL http://esslli2018.folli.info/student-session/
 
Description Machine Intelligence 21 workshop Presentation 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Presented current research project and approaches to the challenge. Promoted interesting discussions afterwards
Year(s) Of Engagement Activity 2019
URL http://mi21-hlc.doc.ic.ac.uk/
 
Description Speaking of Location 2019: Communicating about Space Presentation 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Presented our paper 'Investigating the Dimensions of Spatial Language'
Year(s) Of Engagement Activity 2019
URL http://eprints.whiterose.ac.uk/148335/8/Investigating_the_Dimensions_of_Spatial_Language.pdf