Explainable Natural Language Inference over Cancer Clinical Trial Texts
Lead Research Organisation:
University of Manchester
Department Name: Computer Science
Abstract
The project aims to develop specialized Natural Language Inference (NLI) techniques to support inference over cancer clinical trial (CCT) reports. The project aims to address the following research questions:
1: How can we construct explanation-based multi-hop NLI models capable of supporting inference over CCT reports? An NLI CCT dataset called NLI4CT was designed and constructed to test various models on this task, a shared task was run on NLI4CT, and a second iteration of this shared task will be run on an updated version of NLI4CT.
1.1: How can we encode medical definitions and CCT reports to support explanation-based multi-hop NLI? Relations and definitions were extracted from an oncology ontology, and encoded into a hyperbolic and Euclidean model, currently under evaluation, with additional experimentation on the retrieval of CCT reports for patient profile queries using Large Language Models (LLM).
1.2: How can we construct high quality explanations to support model predictions over multi-hop NLI for CCTs? We will experiment with Differentiable Convex Optimization for explanation construction, building graphs from relevant domain facts, and extracting constraint-based subgraphs to simulate natural explanations.
1.3: How can we develop models capable of explainable numerical NLI for CCTs?
Experiments will be carried out on numerical NLI4CT instances, testing Auto formalisation, LLMs and SymPy solvers.
2: How can we quantitatively and qualitatively characterise the behaviour of our model with regards to RQ1-1.3? Quantitatively we have tested with F1 score and Mean Average Precision, as well as defining two novel qualitative measures, Consistency and Faithfulness, designed for causal analyses of NLI models. Qualitatively, we explore the traversal of the hyperbolic embedding space, performing midpoint analysis, and studying additive/multiplicative behaviours. Additional experiments will include probing, categorical analysis, ablation, generalisation and adversarial studies
1: How can we construct explanation-based multi-hop NLI models capable of supporting inference over CCT reports? An NLI CCT dataset called NLI4CT was designed and constructed to test various models on this task, a shared task was run on NLI4CT, and a second iteration of this shared task will be run on an updated version of NLI4CT.
1.1: How can we encode medical definitions and CCT reports to support explanation-based multi-hop NLI? Relations and definitions were extracted from an oncology ontology, and encoded into a hyperbolic and Euclidean model, currently under evaluation, with additional experimentation on the retrieval of CCT reports for patient profile queries using Large Language Models (LLM).
1.2: How can we construct high quality explanations to support model predictions over multi-hop NLI for CCTs? We will experiment with Differentiable Convex Optimization for explanation construction, building graphs from relevant domain facts, and extracting constraint-based subgraphs to simulate natural explanations.
1.3: How can we develop models capable of explainable numerical NLI for CCTs?
Experiments will be carried out on numerical NLI4CT instances, testing Auto formalisation, LLMs and SymPy solvers.
2: How can we quantitatively and qualitatively characterise the behaviour of our model with regards to RQ1-1.3? Quantitatively we have tested with F1 score and Mean Average Precision, as well as defining two novel qualitative measures, Consistency and Faithfulness, designed for causal analyses of NLI models. Qualitatively, we explore the traversal of the hyperbolic embedding space, performing midpoint analysis, and studying additive/multiplicative behaviours. Additional experiments will include probing, categorical analysis, ablation, generalisation and adversarial studies
Organisations
People |
ORCID iD |
| Andre Freitas (Primary Supervisor) | |
| Mael Jullien (Student) |
Studentship Projects
| Project Reference | Relationship | Related To | Start | End | Student Name |
|---|---|---|---|---|---|
| EP/T517823/1 | 01/10/2020 | 30/09/2025 | |||
| 2859087 | Studentship | EP/T517823/1 | 01/10/2021 | 30/09/2024 | Mael Jullien |