Measuring the variability of AI and manual delineations for radiotherapy

1) Brief description of the context of the research including potential impact

Radiotherapy is a crucial component of cancer treatment. Accurate treatment planning is essential, as the radiation affects both the tumour and surrounding healthy cells. During the planning process, a clinician delineates the tumour volume (TV) and the surrounding organs at risk (OARs) on a patient's CT scan and determines the appropriate radiation dose for the TV. The treatment is then optimised to deliver the prescribed dose to the TV while ensuring the dose to the OARs remains below the known tolerance value for each organ, minimising damage to vital organs and healthy tissues.

Historically, clinicians manually delineate these volumes. Studies have shown significant variations in delineations among clinicians (inter-observer variability) and even within delineations performed by the same clinician (intra-observer variability). Variability in OAR delineations can lead to incorrect treatment plans, resulting in lower doses delivered to the TV and higher toxicities in healthy tissues. Recently, AI solutions have offered a promising alternative to the traditional manual delineation approach. These allow for significant time savings while producing more homogenous delineations [3]. However, these solutions still need to be accurately assessed and quality-assured.

2) Aims and Objectives

- To create tools to parameterise the variability in OAR delineations in the head and neck region.
- To critically evaluate delineations for individual patients and determine if they fall within the range of variability observed in the broader population.
- To facilitate comparisons between manual and AI-generated delineations or different AI delineations for various patient cohorts.
- To streamline the National Radiotherapy Trials Quality Assurance (RTTQA) Group's review of clinical trial cases by automating the detection of abnormal delineations.

3) Novelty of Research Methodology

Precisely assessing the variability in delineations is a difficult task. Past studies involved asking multiple clinicians (or the same clinician) to delineate the same structures on a unique scan. Once delineated, metrics were employed to quantify differences and measure interand intra-observer variability. This approach has several limitations: it is time-consuming and severely limits how many scans and clinicians can be assessed. Moreover, there is a potential for bias in the results, as clinicians may generate more accurate delineations when they are aware of participating in a study. A clinician might also learn the patient's anatomy if asked to delineate multiple times. It is also unsuitable to evaluate AI systems, as we would expect the AI to consistently deliver the same results when presented with the same scan multiple times.

Several new approaches use Deep Learning (DL) to evaluate segmentations. Among these methods is normative modelling, where we model the distribution of a set of data samples. This enables us to assess the probability that a specific data sample originates from that distribution. In addition to evaluating the position of a data sample in the distribution, this technique also enables the identification and exclusion of OOD samples. The network trained is a generative model, such as a Variational Auto-Encoder (VAE).

4) Alignment to EPSRC's strategies and research areas

Our research aligns closely with the strategic priorities of the EPSRC by advancing the integration of data science and artificial intelligence in healthcare. Specifically, our work addresses the EPSRC's emphasis on transformative healthcare technologies, data-driven research, and the development of innovative methodologies for improving patient outcomes.

5) Any companies or collaborators involved

- Centre for Medical Imaging Computing at UCL
- National Physical Laboratory
- Radiotherapy Trials Quality Assurance Group
- Mount Vernon Cancer Centre