Measuring the variability of AI and manual delineations for radiotherapy

1. Brief description of the context of the research including potential impact:
Radiotherapy is a crucial component of cancer treatment, but its effectiveness is often compromised by uncertainties stemming from inaccurate anatomical delineations. These inaccuracies can have significant implications for both tumour control and the risk of normal tissue toxicity. Traditionally, clinicians performed these delineations manually; however, the emergence of AI-based software for automated delineation offers a promising alternative. These AI solutions have the potential to enhance efficiency by saving time and reducing variability in delineations. Therefore, there is a pressing need to devise methods for the assessment and comparison of manual and AI-generated delineations.

2. Aims and objectives:
The core objective of this research is to develop cutting-edge machine learning methods for quantifying and characterising variability in delineations across a diverse patient population. These methods will be capable of evaluating both manual and AI-generated delineations. The specific aims are as follows:
- To create state-of-the-art machine learning models that can assess and parameterise variability in anatomical delineations.
- To facilitate fair and objective comparisons between manual and AI-generated delineations for various patient cohorts.
- To critically evaluate delineations for individual patients and determine if they fall within the range of variability observed in the broader population.
- To streamline the quality assurance process for delineations in clinical trials and routine clinical practice, thereby enhancing treatment efficacy and patient safety.

3. Novelty of research methodology:
Precisely assessing the variability in delineations is a difficult task. Historically, studies on contour variability usually followed the same method: a limited number of scans were used with various individuals tasked to delineate the same structures on them. Once delineated, metrics were employed to quantify differences between the contours, facilitating the measurement of inter-observer variability (while intra-observer variability could be assessed by having the same individual delineate a scan multiple times). However, this approach suffers from several limitations. It is impractical for use on many scans, and participants' awareness of being part of a study may lead to conscious or subconscious efforts to perform better than they would in a clinical setting. Furthermore, this methodology is unsuitable for evaluating AI systems, as one would expect AI to consistently produce identical results for the same scan.
One of the primary innovations in this project is therefore our intention to devise methodologies for quantifying variability when each scan undergoes delineation only once. Our approach builds upon recent advancements in the domains of Machine Learning and Deep Learning, particularly focusing on uncertainty modelling and normative modelling.

4. Alignment to EPSRC's strategies and research areas:
Our research aligns closely with the strategic priorities of the EPSRC by advancing the integration of data science and artificial intelligence in healthcare. Specifically, our work addresses the EPSRC's emphasis on transformative healthcare technologies, data-driven research, and the development of innovative methodologies for improving patient outcomes.

5. Companies or collaborators involved:
- Centre for Medical Imaging Computing (CMIC) at UCL
- National Physical Laboratory (NPL)
- Radiotherapy Trials Quality Assurance Group (RTTQA)
- Mount Vernon Cancer Centre (MVCC)"