Neural networks for joint modelling of medical images and radiology reports

Neural networks have achieved remarkable results for medical image analysis with potential to facilitate healthcare. Examples of common tasks where these models are applied include classification and segmentation of pathology in X-rays, CT and MRI. Image-only models have various limitations. For example, they can only learn from manual annotations made for an image, such the drawn outline of a pathology, which are limited because making them is expensive, time-consuming and requires clinical expertise. Learning from this limited data leads to models with suboptimal generalization. Moreover, models cannot explain the reason for making a prediction, as they have no mechanism of "communicating" with the human user.

In clinical practice, a radiologist produces a written report in natural language, describing contents of a medical image. Such a report can describe what pathology exists, its location, approximate size, appearance, and more. These reports are produced daily in large quantities in every Radiology department and hold rich information about the corresponding images and pathologies therein. Current generation of machine learning models, however, cannot exploit this wealth of data adequately.

Aim and Objectives

Aim of this project is to develop neural networks that can jointly process and learn from medical images and corresponding written radiology reports. Specifically, we will develop learning frameworks and models that are able to:

a) Process an input image and automatically generate text that describes its contents. This automatically generated report can be invaluable for accelerating radiology workflows.

b) Use information from written reports to learn how to detect a pathology in an image. The information from the numerous radiology reports will be combined with information from a smaller database of manually annotated images (e.g. drawn outlines of pathologies) to train disease detection models that are more potent than the current generation of models, which can only learn from the limited manual annotations.

c) Jointly modelling images and language will give models the capability to explain their predictions. Such models will be able to use natural language to describe to a human why a specific prediction is made about disease segmentation, and vice versa.

Importance and Contributions to Knowledge

This project falls within the EPSRC Medical Imaging research area and the Healthcare Technologies theme.
The proposed research will generate novel techniques for jointly modelling visual and textual information from medical databases, an area where current generation of algorithms is still inadequate. Such models will be able to extract wealth of information from routinely produced radiology reports to learn how to better detect pathologies in medical scans. When these models make a prediction for the existence of a pathology in a new scan, these models will also be able to generate explanations for their prediction in natural language that can be easily interpreted by the human user. These mechanisms will enhance predictive performance and explainability of the models, factors that are necessary for reliable adoption of such tools in clinical workflows.

In the same way that bioinformatics has transformed genomic research and clinical practice, health data science will have a dramatic and lasting impact upon the broader fields of medical research, population health, and healthcare delivery. The beneficiaries of the proposed training programme, and of the research that it delivers and enables, will include academia, industry, healthcare, and the broader UK economy.

Academia: Graduates of the training programme will be well placed to start their post-doctoral careers in leading academic institutions, engaging in high-impact multi-disciplinary research, helping to build training and research capacity, sharing their experience within the wider academic community.

Industry: Partner organisations will benefit from close collaboration with leading researchers, from the joint exploration of research priorities, and from the commercialisation of arising intellectual property. Other organisations will benefit from the availability of highly-qualified graduates with skills in big health data analytics.

Healthcare: Healthcare organisations and patients will benefit from the results of enabled and accelerated health research, leading to new treatments and technologies, and an improved ability to identify and evaluate potential improvements in practice through the analysis of real-world health data.

Economy: The life sciences sector is a key component of the UK economy. The programme will provide partner companies with direct access to leading-edge research. Graduates of the programme will be well-qualified to contribute to economic growth - supporting health research and the development of new products and services - and will be able to inform policy and decision making at organisational, regional, and national levels.