CardiacA.I.: Machine learning for the analysis of multimodal cardiac MR images used in the diagnosis of coronary heart disease

Lead Research Organisation: University of Edinburgh
Department Name: Sch of Engineering

Abstract

A sedentary lifestyle, poor diet, smoking, and genetic and other health factors are major contributors to coronary heart disease (CHD). Despite recent medical advances that have lowered the number of deaths compared to the past decades, CHD still remains the number 1 disease in mortality in the UK (73,000 deaths per year) with a tremendous economic burden: estimates put the cost to UK's economy at £6.7 billion per year. The overriding goal of this project is to take advantage of multimodal information within cardiac magnetic resonance images to improve their analysis and facilitate the diagnosis and improve treatment of CHD.

Magnetic Resonance Imaging (MRI) as an imaging diagnostic tool is uniquely positioned to help as it is non-invasive and does not use radiation. A typical cardiac protocol relies on several MR imaging sequences to provide images of different contrast, termed as modalities hereafter, to assess disease progression and status. As a result of this range of acquisitions, hundreds of multidimensional multimodal images are generated in a single patient exam leading to severe data overload.

Therefore, robust and automated analyses algorithms would help alleviate the clinical reading burden. Several algorithms have been proposed to segment and register the myocardium in the most commonly used modalities by considering them independently. However, the problem remains difficult and performance is not yet adequate. Currently, the analysis of cardiac imaging data still remains a manual, time consuming, and expensive process typically performed by clinical experts. As a result, despite the huge amount of data generated, not only in a clinical but also in a research setting, only a fraction is being analysed robustly, due to the vast amount of time required for the analysis of this data.

This proposal aims to address the above shortcomings by proposing mechanisms that take advantage of the shared information that exists across modalities to enable the jint analysis of cardiac imaging data and thus make a significant leap in how we approach their analysis. We propose new multimodal machine learning driven mechanisms to learn image features (i.e. how local image information is represented for an algorithm to use) that do not change between imaging modalities whilst preserving shared anatomical information. We will then use the learned features in multimodal patch-based myocardial segmentation and inter-modality non-linear registration (i.e. the non-linear registration between two images coming from different cardiac MR sequences) thus enabling us to relate images of the same patient across different modalities. To maximise impact, we will develop an inter-modality cardiac registration plugin for a commercial clinical package that is also offered as an open source variant for academic purposes.

We expect that when our complete framework is integrated into clinical tools and becomes widely available it can radically change current clinical reading workflow and decision-making. It will permit the propagation of annotations across multimodal images of a patient exam effortlessly and seamlessly, thus significantly reducing reading time and permitting the analysis of cardiac data on a larger scale.

Planned Impact

This interdisciplinary project combines our knowhow in machine learning, image processing and medical image analysis to solve an important societal and health challenge: the diagnosis of Coronary Heart Disease (CHD) on the basis of MRI images. We propose a new concept on how to represent images in order to facilitate the analysis of cardiac data and ultimately, alleviate the efforts required by clinicians to render a diagnosis and guide treatment.

In the UK alone more than 73,000 deaths per year are attributed to CHD. This demonstrates a significant psychological burden to the patients and their families. Not only that, but CHD is truly expensive. According to the British Heart Foundation the cost to UK's economy is close to £6.7 billion/year. This not only affects the NHS (27% direct NHS burden) but the economy overall directly (47% productivity loss has been considered) and also indirectly due to cost of immediate support and informal care of patients by their families or support circle (another 26%). MRI offers unique diagnostic information that is repeatable (being non-invasive and non-ionizing) and can thus be used to diagnose, stratify and treat (by follow-up) patients and alleviate the above psychological and economic cost. Thus, even the general patient population will have direct interest in our developed tools in improving cardiac analysis. By translating our methods to the clinic and in linking with and informing the public (by liaising and collaborating with Chest Heart & Stroke Scotland), we aim to cultivate this interest and maximize the benefit to the patient population.

Naturally, clinicians working in cardiac MRI from a research perspective would also benefit. For example, MRI is the imaging modality of choice for the UK Biobank: a £92M investment and currently the world's largest health imaging study (100k people). Image and data analysis tools to analyze this enormous repository of data automatically are still lacking and this project directladdresses this national need. For this project we will collaborate directly with clinicians but we will also liaise with several local clinician networks interested in these problems.

Since clinicians do rely on software to read images, we will collaborate with an SME, that develops a clinical cardiac analysis software platform, to translate one of the algorithms developed within this project. The exchange of knowhow with industry and particular small enterprises is a key industrial exploitation of the work carried here. The company will benefit by introducing a new plugin that solves a critical clinical problem: inter-modality cardiac registration. Clinical users will save time and produce more robust outcomes when using this plugin. As a result, it will create added value to the customer (and the SME). Plugins developed for the software are available as open source to further the development of new tools since also the software is open source (for academic use). Our plugin, and the corresponding algorithm, can certainly be used to inspire new developments by similar companies throughout the world.

Our methods deal with an important problem: relating images of different context and contrast in MRI such that we can unravel common information, but be resilient to nuisance factors. We propose machine learning methods, that build upon new developments in deep learning, that are able to discern useful from meaningless information. This is an important problem, in medical image analysis and in general. As a result, finding good solutions to it will have a direct impact to researchers working on multimodal data and aiming to find common patterns across them for whatever the application domain. The expected impact and uptake from the research will be maximised by publishing in top journals and conference proceedings, via our international collaborations, and by sharing code on our website.
 
Description We developed models that can learn from medical imaging data without significant amount of supervision.
We showed that we can learn models to solve key image analysis tasks such as segmentation and image synthesis by:
a) requiring only few paired data
b) having no paired data
c) without even the images being co-registered

Supporting open science all our models are publicly available to allow the community to reproduce our experiments.
Exploitation Route We have made available code of our models so researchers and other entities can safely reproduce findings.
One of the key findings we made was that inductive bias of the networks has a considerable potential of the difficulty or ease a solution can be found.
Sectors Digital/Communication/Information Technologies (including Software),Healthcare

URL http://tsaftaris.com/project_Cardiac_AI.html
 
Description Tutorial on Simulation and Synthesis
Geographic Reach Multiple continents/international 
Policy Influence Type Influenced training of practitioners or researchers
 
Description Workshop on Simulation and Synthesis in Medical Imaging
Geographic Reach Multiple continents/international 
Policy Influence Type Influenced training of practitioners or researchers
 
Description Canon Medical / Royal Academy of Engineering Senior Research Fellow in Healthcare AI
Amount £814,254 (GBP)
Organisation Royal Academy of Engineering 
Sector Learned Society
Country United Kingdom
Start 03/2019 
End 03/2024
 
Description Industrial Centre for Artificial Intelligence Research in Digital Diagnostics (iCAIRD)
Amount £10,000,000 (GBP)
Funding ID 104690 
Organisation Innovate UK 
Sector Public
Country United Kingdom
Start 01/2019 
End 12/2021
 
Title Doing more with less 
Description We developed a method to train neural networks to work with different input images, to solve different tasks and to learn to solve such tasks without need for significant annotation. 
Type Of Material Technology assay or reagent 
Year Produced 2018 
Provided To Others? Yes  
Impact The work has already been cited three times by the academic community. 
 
Title Learning without labels 
Description We developed methods to train neural networks without any supervised data. 
Type Of Material Technology assay or reagent 
Year Produced 2018 
Provided To Others? Yes  
Impact The work despite its infancy has already been cited twice by the academic community. 
URL https://openreview.net/forum?id=S11Xr-3iM
 
Title Factorisation model 
Description A method to factorise an input image into a segmentation and latent code that encodes the style of the input. 
Type Of Material Computer model/algorithm 
Year Produced 2018 
Provided To Others? Yes  
Impact The work has been cited already twice and the model was used by a team from Siemens Healthineers to demonstrate suitability for a task of learning anatomical maps. 
URL https://github.com/agis85/spatial_factorisation
 
Title Model synthesis 
Description This model learns to synthesize data given multimodal inputs employing deep neural networks. 
Type Of Material Computer model/algorithm 
Year Produced 2017 
Provided To Others? Yes  
Impact The work has been cited collectively 31 times and already 5 papers utilize the code to demonstrate performance on other data. 
URL https://github.com/agis85/multimodal_brain_synthesis
 
Description Medviso 
Organisation Medviso AB
PI Contribution We have several times with the CTO of Medviso and discussed the potential implications of integrating our solutions to their software.
Collaborator Contribution Medviso offered the open source version of their software and also provided assistance with integrating our approach in their API.
Impact n/a
Start Year 2012
 
Title Multimodal synthesis 
Description A python source code to create an output image given different multimodal inputs. Code to train the model used and to use the model is provided. 
Type Of Technology New/Improved Technique/Technology 
Year Produced 2018 
Impact Used several times by other researchers. 
URL https://github.com/agis85/multimodal_brain_synthesis
 
Title Semi-supervised cardiac segmentation 
Description Model (in Python) that can segment cardiac MRI and particularly the myocardium without requiring significant amount of training data. Code to train the model and also to infer output is provided. 
Type Of Technology New/Improved Technique/Technology 
Year Produced 2018 
Impact Led to development of even more advanced model. 
URL https://github.com/agis85/spatial_factorisation
 
Description Deep learning for undergraduates 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Postgraduate students
Results and Impact We presented a 4 hour tutorial to an audience comprising of undergraduate, MSc and PhD students on using machine learning and deep learning for image analysis tasks
Year(s) Of Engagement Activity 2018
 
Description Visit by Canon Medical Research Europe 
Form Of Engagement Activity Participation in an open day or visit at my research institution
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Industry/Business
Results and Impact Representatives from a local R&D division of an international conglomerate visited my lab and we discussed knowledge exchange via a series of presentations
Year(s) Of Engagement Activity 2018
 
Description Visit to Hospital 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Professional Practitioners
Results and Impact We visited the local hospital and discussed extensions of our work in the context of CT imaging as part of large national study.
Year(s) Of Engagement Activity 2018