CardiacA.I.: Machine learning for the analysis of multimodal cardiac MR images used in the diagnosis of coronary heart disease
Lead Research Organisation:
University of Edinburgh
Department Name: Sch of Engineering
Abstract
A sedentary lifestyle, poor diet, smoking, and genetic and other health factors are major contributors to coronary heart disease (CHD). Despite recent medical advances that have lowered the number of deaths compared to the past decades, CHD still remains the number 1 disease in mortality in the UK (73,000 deaths per year) with a tremendous economic burden: estimates put the cost to UK's economy at £6.7 billion per year. The overriding goal of this project is to take advantage of multimodal information within cardiac magnetic resonance images to improve their analysis and facilitate the diagnosis and improve treatment of CHD.
Magnetic Resonance Imaging (MRI) as an imaging diagnostic tool is uniquely positioned to help as it is non-invasive and does not use radiation. A typical cardiac protocol relies on several MR imaging sequences to provide images of different contrast, termed as modalities hereafter, to assess disease progression and status. As a result of this range of acquisitions, hundreds of multidimensional multimodal images are generated in a single patient exam leading to severe data overload.
Therefore, robust and automated analyses algorithms would help alleviate the clinical reading burden. Several algorithms have been proposed to segment and register the myocardium in the most commonly used modalities by considering them independently. However, the problem remains difficult and performance is not yet adequate. Currently, the analysis of cardiac imaging data still remains a manual, time consuming, and expensive process typically performed by clinical experts. As a result, despite the huge amount of data generated, not only in a clinical but also in a research setting, only a fraction is being analysed robustly, due to the vast amount of time required for the analysis of this data.
This proposal aims to address the above shortcomings by proposing mechanisms that take advantage of the shared information that exists across modalities to enable the joint analysis of cardiac imaging data and thus make a significant leap in how we approach their analysis. We propose new multimodal machine learning driven mechanisms to learn image features (i.e. how local image information is represented for an algorithm to use) that do not change between imaging modalities whilst preserving shared anatomical information. We will then use the learned features in multimodal patch-based myocardial segmentation and inter-modality non-linear registration (i.e. the non-linear registration between two images coming from different cardiac MR sequences) thus enabling us to relate images of the same patient across different modalities. To maximise impact, we will develop an inter-modality cardiac registration plugin for a commercial clinical package that is also offered as an open source variant for academic purposes.
We expect that when our complete framework is integrated into clinical tools and becomes widely available it can radically change current clinical reading workflow and decision-making. It will permit the propagation of annotations across multimodal images of a patient exam effortlessly and seamlessly, thus significantly reducing reading time and permitting the analysis of cardiac data on a larger scale.
Magnetic Resonance Imaging (MRI) as an imaging diagnostic tool is uniquely positioned to help as it is non-invasive and does not use radiation. A typical cardiac protocol relies on several MR imaging sequences to provide images of different contrast, termed as modalities hereafter, to assess disease progression and status. As a result of this range of acquisitions, hundreds of multidimensional multimodal images are generated in a single patient exam leading to severe data overload.
Therefore, robust and automated analyses algorithms would help alleviate the clinical reading burden. Several algorithms have been proposed to segment and register the myocardium in the most commonly used modalities by considering them independently. However, the problem remains difficult and performance is not yet adequate. Currently, the analysis of cardiac imaging data still remains a manual, time consuming, and expensive process typically performed by clinical experts. As a result, despite the huge amount of data generated, not only in a clinical but also in a research setting, only a fraction is being analysed robustly, due to the vast amount of time required for the analysis of this data.
This proposal aims to address the above shortcomings by proposing mechanisms that take advantage of the shared information that exists across modalities to enable the joint analysis of cardiac imaging data and thus make a significant leap in how we approach their analysis. We propose new multimodal machine learning driven mechanisms to learn image features (i.e. how local image information is represented for an algorithm to use) that do not change between imaging modalities whilst preserving shared anatomical information. We will then use the learned features in multimodal patch-based myocardial segmentation and inter-modality non-linear registration (i.e. the non-linear registration between two images coming from different cardiac MR sequences) thus enabling us to relate images of the same patient across different modalities. To maximise impact, we will develop an inter-modality cardiac registration plugin for a commercial clinical package that is also offered as an open source variant for academic purposes.
We expect that when our complete framework is integrated into clinical tools and becomes widely available it can radically change current clinical reading workflow and decision-making. It will permit the propagation of annotations across multimodal images of a patient exam effortlessly and seamlessly, thus significantly reducing reading time and permitting the analysis of cardiac data on a larger scale.
Planned Impact
This interdisciplinary project combines our knowhow in machine learning, image processing and medical image analysis to solve an important societal and health challenge: the diagnosis of Coronary Heart Disease (CHD) on the basis of MRI images. We propose a new concept on how to represent images in order to facilitate the analysis of cardiac data and ultimately, alleviate the efforts required by clinicians to render a diagnosis and guide treatment.
In the UK alone more than 73,000 deaths per year are attributed to CHD. This demonstrates a significant psychological burden to the patients and their families. Not only that, but CHD is truly expensive. According to the British Heart Foundation the cost to UK's economy is close to £6.7 billion/year. This not only affects the NHS (27% direct NHS burden) but the economy overall directly (47% productivity loss has been considered) and also indirectly due to cost of immediate support and informal care of patients by their families or support circle (another 26%). MRI offers unique diagnostic information that is repeatable (being non-invasive and non-ionizing) and can thus be used to diagnose, stratify and treat (by follow-up) patients and alleviate the above psychological and economic cost. Thus, even the general patient population will have direct interest in our developed tools in improving cardiac analysis. By translating our methods to the clinic and in linking with and informing the public (by liaising and collaborating with Chest Heart & Stroke Scotland), we aim to cultivate this interest and maximize the benefit to the patient population.
Naturally, clinicians working in cardiac MRI from a research perspective would also benefit. For example, MRI is the imaging modality of choice for the UK Biobank: a £92M investment and currently the world's largest health imaging study (100k people). Image and data analysis tools to analyze this enormous repository of data automatically are still lacking and this project directly addresses this national need. For this project we will collaborate directly with clinicians but we will also liaise with several local clinician networks interested in these problems.
Since clinicians do rely on software to read images, we will collaborate with an SME, that develops a clinical cardiac analysis software platform, to translate one of the algorithms developed within this project. The exchange of knowhow with industry and particular small enterprises is a key industrial exploitation of the work carried here. The company will benefit by introducing a new plugin that solves a critical clinical problem: inter-modality cardiac registration. Clinical users will save time and produce more robust outcomes when using this plugin. As a result, it will create added value to the customer (and the SME). Plugins developed for the software are available as open source to further the development of new tools since also the software is open source (for academic use). Our plugin, and the corresponding algorithm, can certainly be used to inspire new developments by similar companies throughout the world.
Our methods deal with an important problem: relating images of different context and contrast in MRI such that we can unravel common information, but be resilient to nuisance factors. We propose machine learning methods, that build upon new developments in deep learning, that are able to discern useful from meaningless information. This is an important problem, in medical image analysis and in general. As a result, finding good solutions to it will have a direct impact to researchers working on multimodal data and aiming to find common patterns across them for whatever the application domain. The expected impact and uptake from the research will be maximised by publishing in top journals and conference proceedings, via our international collaborations, and by sharing code on our website.
In the UK alone more than 73,000 deaths per year are attributed to CHD. This demonstrates a significant psychological burden to the patients and their families. Not only that, but CHD is truly expensive. According to the British Heart Foundation the cost to UK's economy is close to £6.7 billion/year. This not only affects the NHS (27% direct NHS burden) but the economy overall directly (47% productivity loss has been considered) and also indirectly due to cost of immediate support and informal care of patients by their families or support circle (another 26%). MRI offers unique diagnostic information that is repeatable (being non-invasive and non-ionizing) and can thus be used to diagnose, stratify and treat (by follow-up) patients and alleviate the above psychological and economic cost. Thus, even the general patient population will have direct interest in our developed tools in improving cardiac analysis. By translating our methods to the clinic and in linking with and informing the public (by liaising and collaborating with Chest Heart & Stroke Scotland), we aim to cultivate this interest and maximize the benefit to the patient population.
Naturally, clinicians working in cardiac MRI from a research perspective would also benefit. For example, MRI is the imaging modality of choice for the UK Biobank: a £92M investment and currently the world's largest health imaging study (100k people). Image and data analysis tools to analyze this enormous repository of data automatically are still lacking and this project directly addresses this national need. For this project we will collaborate directly with clinicians but we will also liaise with several local clinician networks interested in these problems.
Since clinicians do rely on software to read images, we will collaborate with an SME, that develops a clinical cardiac analysis software platform, to translate one of the algorithms developed within this project. The exchange of knowhow with industry and particular small enterprises is a key industrial exploitation of the work carried here. The company will benefit by introducing a new plugin that solves a critical clinical problem: inter-modality cardiac registration. Clinical users will save time and produce more robust outcomes when using this plugin. As a result, it will create added value to the customer (and the SME). Plugins developed for the software are available as open source to further the development of new tools since also the software is open source (for academic use). Our plugin, and the corresponding algorithm, can certainly be used to inspire new developments by similar companies throughout the world.
Our methods deal with an important problem: relating images of different context and contrast in MRI such that we can unravel common information, but be resilient to nuisance factors. We propose machine learning methods, that build upon new developments in deep learning, that are able to discern useful from meaningless information. This is an important problem, in medical image analysis and in general. As a result, finding good solutions to it will have a direct impact to researchers working on multimodal data and aiming to find common patterns across them for whatever the application domain. The expected impact and uptake from the research will be maximised by publishing in top journals and conference proceedings, via our international collaborations, and by sharing code on our website.
Publications
Wang C
(2021)
DiCyc: GAN-based deformation invariant cross-domain information fusion for medical image synthesis.
in An international journal on information fusion
Xia Tian
(2019)
Adversarial Pseudo Healthy Synthesis Needs Pathology Factorization
in arXiv e-prints
Spagnuolo A
(2022)
Characterizing passenger-ship emissions: towards improved sustainability for MedMar fleet (gulf of Naples).
in Energy efficiency
Campello VM
(2022)
Cardiac aging synthesis from cross-sectional data with conditional generative adversarial networks.
in Frontiers in cardiovascular medicine
Panayides AS
(2020)
AI in Medical Imaging Informatics: Current Challenges and Future Directions.
in IEEE journal of biomedical and health informatics
Oksuz I
(2017)
Unsupervised Myocardial Segmentation for Cardiac BOLD.
in IEEE transactions on medical imaging
Frangi AF
(2018)
Simulation and Synthesis in Medical Imaging.
in IEEE transactions on medical imaging
Valvano G
(2021)
Learning to Segment From Scribbles Using Multi-Scale Adversarial Attention Gates.
in IEEE transactions on medical imaging
Campello VM
(2021)
Multi-Centre, Multi-Vendor and Multi-Disease Cardiac Segmentation: The M&Ms Challenge.
in IEEE transactions on medical imaging
Chartsias A
(2021)
Disentangle, Align and Fuse for Multimodal and Semi-Supervised Image Segmentation.
in IEEE transactions on medical imaging
Chartsias A
(2018)
Multimodal MR Synthesis via Modality-Invariant Latent Representation.
in IEEE transactions on medical imaging
Xia T
(2020)
Pseudo-healthy synthesis with pathology disentanglement and adversarial learning.
in Medical image analysis
Liu X
(2022)
Learning disentangled representations in the imaging domain.
in Medical image analysis
Xia T
(2021)
Learning to synthesise the ageing brain without longitudinal data.
in Medical image analysis
Chartsias A
(2019)
Disentangled representation learning in cardiac image analysis.
in Medical image analysis
Xia T.
(2019)
Adversarial Pseudo Healthy Synthesis Needs Pathology Factorization
in Proceedings of Machine Learning Research
Chartsias A
(2017)
Simulation and Synthesis in Medical Imaging
Valvano G
(2022)
Biomedical Image Synthesis and Simulation
Joyce T
(2018)
Deep Multi-Class Segmentation Without Ground-Truth Labels
Description | We developed models that can learn from medical imaging data without significant amount of supervision. We showed that we can learn models to solve key image analysis tasks such as segmentation and image synthesis by: a) requiring only few paired data b) having no paired data c) without even the images being co-registered Supporting open science all our models are publicly available to allow the community to reproduce our experiments. |
Exploitation Route | We have made available code of our models so researchers and other entities can safely reproduce findings. One of the key findings we made was that the inductive bias of the networks has a considerable effect on the difficulty or ease of how an appropriate solution can be found. |
Sectors | Digital/Communication/Information Technologies (including Software),Healthcare |
URL | http://tsaftaris.com/project_Cardiac_AI.html |
Description | * The findings of this research has led to several open software that followed publications of the research (2018-2019). * Currently this research is translated to other organs by Canon Medical Research Europe (2020-). * Several keynotes and dissemination to research community (e.g. ICANN 2018, Phenome 2019, ICLR 2020) * PI represented the UK's machine learning community at a visit organised by Scottish Enterprise to Japan (2019) * PI invited to participate at the Global AI Summit, in Saudi Arabia (2020) * Follow on project (1.2million) funded by RAENg and Canon Medical Research Europe (2019-2023) |
First Year Of Impact | 2020 |
Sector | Digital/Communication/Information Technologies (including Software),Healthcare |
Impact Types | Economic |
Description | Tutorial on Simulation and Synthesis |
Geographic Reach | Multiple continents/international |
Policy Influence Type | Influenced training of practitioners or researchers |
Description | Workshop on Simulation and Synthesis in Medical Imaging |
Geographic Reach | Multiple continents/international |
Policy Influence Type | Influenced training of practitioners or researchers |
Description | Canon Medical / Royal Academy of Engineering Senior Research Fellow in Healthcare AI |
Amount | £814,254 (GBP) |
Organisation | Royal Academy of Engineering |
Sector | Charity/Non Profit |
Country | United Kingdom |
Start | 03/2019 |
End | 03/2024 |
Description | From trivial representations to learning concepts in AI by exploiting unique data |
Amount | £202,351 (GBP) |
Funding ID | EP/X017680/1 |
Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
Sector | Public |
Country | United Kingdom |
Start | 10/2022 |
End | 09/2024 |
Description | Industrial Centre for Artificial Intelligence Research in Digital Diagnostics (iCAIRD) |
Amount | £10,000,000 (GBP) |
Funding ID | 104690 |
Organisation | Innovate UK |
Sector | Public |
Country | United Kingdom |
Start | 01/2019 |
End | 12/2021 |
Title | Code implementations for papers |
Description | We are committed to open and reproducible science |
Type Of Material | Technology assay or reagent |
Year Produced | 2021 |
Provided To Others? | Yes |
Impact | Considerable number of stars and forks on githbub |
URL | https://vios.science/code_and_data/ |
Title | Doing more with less |
Description | We developed a method to train neural networks to work with different input images, to solve different tasks and to learn to solve such tasks without need for significant annotation. |
Type Of Material | Technology assay or reagent |
Year Produced | 2018 |
Provided To Others? | Yes |
Impact | The work has already been cited three times by the academic community. |
Title | Learning to Segment from Scribbles using Multi-scale Adversarial Attention Gates |
Description | Implementation for Learning to Segment from Scribbles using Multi-scale Adversarial Attention Gates G. Valvano, A. Leo, S.A. Tsaftaris IEEE Transactions on Medical Imaging, vol. 40, no. 8, pp. 1990-2001, Aug. 2021 |
Type Of Material | Technology assay or reagent |
Year Produced | 2021 |
Provided To Others? | Yes |
Impact | Code has 7 forks and 28 stars |
URL | https://github.com/gvalvano/multiscale-adversarial-attention-gates |
Title | Learning without labels |
Description | We developed methods to train neural networks without any supervised data. |
Type Of Material | Technology assay or reagent |
Year Produced | 2018 |
Provided To Others? | Yes |
Impact | The work despite its infancy has already been cited twice by the academic community. |
URL | https://openreview.net/forum?id=S11Xr-3iM |
Title | Factorisation model |
Description | A method to factorise an input image into a segmentation and latent code that encodes the style of the input. |
Type Of Material | Computer model/algorithm |
Year Produced | 2018 |
Provided To Others? | Yes |
Impact | The work has been cited already twice and the model was used by a team from Siemens Healthineers to demonstrate suitability for a task of learning anatomical maps. |
URL | https://github.com/agis85/spatial_factorisation |
Title | Model synthesis |
Description | This model learns to synthesize data given multimodal inputs employing deep neural networks. |
Type Of Material | Computer model/algorithm |
Year Produced | 2017 |
Provided To Others? | Yes |
Impact | The work has been cited collectively 31 times and already 5 papers utilize the code to demonstrate performance on other data. |
URL | https://github.com/agis85/multimodal_brain_synthesis |
Description | Medviso |
Organisation | Medviso AB |
Country | Sweden |
Sector | Private |
PI Contribution | We have several times with the CTO of Medviso and discussed the potential implications of integrating our solutions to their software. |
Collaborator Contribution | Medviso offered the open source version of their software and also provided assistance with integrating our approach in their API. |
Impact | n/a |
Start Year | 2017 |
Title | DATA PROCESSING APPARATUS AND METHOD |
Description | A medical image data processing apparatus comprises processing circuitry configured to:receive medical image data in respect of at least one subject;receive non-image data;generate a filter based on the non-image data; andapply the filter to the medical image data, wherein the filter is configured to limit a region of the medical image data. |
IP Reference | US2021279863 |
Protection | Patent application published |
Year Protection Granted | 2021 |
Licensed | Yes |
Impact | Canon Medical is a licensee and is being considered in their patent pool. |
Title | Multimodal synthesis |
Description | A python source code to create an output image given different multimodal inputs. Code to train the model used and to use the model is provided. |
Type Of Technology | New/Improved Technique/Technology |
Year Produced | 2018 |
Impact | Used several times by other researchers. |
URL | https://github.com/agis85/multimodal_brain_synthesis |
Title | Semi-supervised cardiac segmentation |
Description | Model (in Python) that can segment cardiac MRI and particularly the myocardium without requiring significant amount of training data. Code to train the model and also to infer output is provided. |
Type Of Technology | New/Improved Technique/Technology |
Year Produced | 2018 |
Impact | Led to development of even more advanced model. |
URL | https://github.com/agis85/spatial_factorisation |
Description | Deep learning for undergraduates |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Postgraduate students |
Results and Impact | We presented a 4 hour tutorial to an audience comprising of undergraduate, MSc and PhD students on using machine learning and deep learning for image analysis tasks |
Year(s) Of Engagement Activity | 2018 |
Description | Keynote at DART 2020 |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Postgraduate students |
Results and Impact | I gave a talk (virtually) in front of 90 students/postdocs of this workshop. |
Year(s) Of Engagement Activity | 2020 |
Description | Keynote talk STACOM |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Postgraduate students |
Results and Impact | I gave a talk at a workshop organised yearly, STACOM 2019 |
Year(s) Of Engagement Activity | 2019 |
Description | Panel participation for EXPO 2020 |
Form Of Engagement Activity | A formal working group, expert panel or dialogue |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | I participated in a panel invited by Canon medical as part of their media activity for EXPO 2020 dubai (occurred virtually in Jan 2022) |
Year(s) Of Engagement Activity | 2022 |
URL | https://www.canon.co.uk/view/expo-dubai-talks-artificial-intelligence-healthcare/ |
Description | Presentation at Onassis Health Day |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Regional |
Primary Audience | Patients, carers and/or patient groups |
Results and Impact | I presented at the Onassis Health day celebrating 30 years since the start of the cardiac transplant hospital in Greece. I discussed the impact of AI on cardiac health at present and the immediate fture. |
Year(s) Of Engagement Activity | 2023 |
URL | https://www.onassis.org/whats-on/onassis-health-day-2023 |
Description | Visit by Canon Medical Research Europe |
Form Of Engagement Activity | Participation in an open day or visit at my research institution |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Industry/Business |
Results and Impact | Representatives from a local R&D division of an international conglomerate visited my lab and we discussed knowledge exchange via a series of presentations |
Year(s) Of Engagement Activity | 2018 |
Description | Visit to Hospital |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Professional Practitioners |
Results and Impact | We visited the local hospital and discussed extensions of our work in the context of CT imaging as part of large national study. |
Year(s) Of Engagement Activity | 2018 |