Graph Convolutional Neural Networks for Medical Image Analysis
Lead Research Organisation:
Heriot-Watt University
Department Name: Sch of Engineering and Physical Science
Abstract
Deep Learning-based Artificial Intelligence, in particular Convolutional Neural Networks (CNNs), has seen exponential growth in many pattern recognition challenges since the early 2010s. While CNNs were originally developed to act on data defined on regular cartesian grids, such as images, audio, and text data, they have recently been extended to work on irregularly connected data, such as graphs. This opens the door to a whole new set of potential applications where graphs arise, either naturally (e.g. surface or volumetric meshes representing shapes) or as a useful intermediate representation (e.g. superpixels representing a medical image). The project will apply graph CNNs to medical image analysis. For example, to identify and outline (segment) a specific structure of interest in a medical image. This is useful in many clinical areas, including cancer (e.g. tumour segmentation to determine treatment effect), neurodegenerative disorders (e.g. parcellation to detect early dementia), cardiovascular diseases (e.g. quantifying coronary artery disease), and musculoskeletal problems.
Planned Impact
Complementing our Pathways to Impact document, here we state the expected real-world impact, which is of course the leading priority for our industrial partners. Their confidence that the proposed CDT will deliver valuable scientific, engineering and commercial impact is emphasized by their overwhelming financial support (£4.38M from industry in the form of cash contributions, and further in-kind support of £5.56M).
Here we summarize what will be the impacts expected from the proposed CDT.
(1) Impact on People
(a) Students
The CDT will have its major impact on the students themselves, by providing them with new understanding, skills and abilities (technical, business, professional), and by enhancing their employability.
(b) The UK public
The engagement planned in the CDT will educate and inform the general public about the high quality science and engineering being pursued by researchers in the CDT, and will also contribute to raising the profile of this mode of doctoral training -- particularly important since the public have limited awareness of the mechanisms through which research scientists are trained.
(2) Impact on Knowledge
New scientific knowledge and engineering know-how will be generated by the CDT. Theses, conference / journal papers and patents will be published to disseminate this knowledge.
(3) Impact on UK industry and economy
UK companies will gain a competitive advantage by using know-how and new techniques generated by CDT researchers.
Companies will also gain from improved recruitment and retention of high quality staff.
Longer term economic impacts will be felt as increased turnover and profitability for companies, and perhaps other impacts such as the generation / segmentation of new markets, and companies receiving inward investment for new products.
(4) Impact on Society
Photonic imaging, sensing and related devices and analytical techniques underpin many of products and services that UK industry markets either to consumers or to other businesses. Reskilling of the workforce with an emphasis on promoting technical leadership is central to EPSRC's Productive Nation prosperity outcome, and our CDT will achieve exactly this through its development of future industrially engaged scientists, engineers and innovators. The impact that these individuals will have on society will be manifested through their contribution to the creation of new products and services that improve the quality of life in sectors like transport, dependable energy networks, security and communications.
Greater internationalisation of the cohort of CDT researchers is expected from some of the CDT activities (e.g. international summer schools), with the potential impact of greater collaboration in the future between the next generations of UK and international researchers.
Here we summarize what will be the impacts expected from the proposed CDT.
(1) Impact on People
(a) Students
The CDT will have its major impact on the students themselves, by providing them with new understanding, skills and abilities (technical, business, professional), and by enhancing their employability.
(b) The UK public
The engagement planned in the CDT will educate and inform the general public about the high quality science and engineering being pursued by researchers in the CDT, and will also contribute to raising the profile of this mode of doctoral training -- particularly important since the public have limited awareness of the mechanisms through which research scientists are trained.
(2) Impact on Knowledge
New scientific knowledge and engineering know-how will be generated by the CDT. Theses, conference / journal papers and patents will be published to disseminate this knowledge.
(3) Impact on UK industry and economy
UK companies will gain a competitive advantage by using know-how and new techniques generated by CDT researchers.
Companies will also gain from improved recruitment and retention of high quality staff.
Longer term economic impacts will be felt as increased turnover and profitability for companies, and perhaps other impacts such as the generation / segmentation of new markets, and companies receiving inward investment for new products.
(4) Impact on Society
Photonic imaging, sensing and related devices and analytical techniques underpin many of products and services that UK industry markets either to consumers or to other businesses. Reskilling of the workforce with an emphasis on promoting technical leadership is central to EPSRC's Productive Nation prosperity outcome, and our CDT will achieve exactly this through its development of future industrially engaged scientists, engineers and innovators. The impact that these individuals will have on society will be manifested through their contribution to the creation of new products and services that improve the quality of life in sectors like transport, dependable energy networks, security and communications.
Greater internationalisation of the cohort of CDT researchers is expected from some of the CDT activities (e.g. international summer schools), with the potential impact of greater collaboration in the future between the next generations of UK and international researchers.
Studentship Projects
| Project Reference | Relationship | Related To | Start | End | Student Name |
|---|---|---|---|---|---|
| EP/S022821/1 | 01/10/2019 | 31/03/2028 | |||
| 2270805 | Studentship | EP/S022821/1 | 01/09/2019 | 30/08/2023 | Angel Victor Juanco Muller |
| Description | Image segmentation is the task by which structures of interest are detected in images. With the advent of Deep Learning since 2015, algorithms known as Convolution Neural Networks (CNNs) can be used to find these structures in the images automatically, provided there is ground truth data with the target structures annotations available to "train" the models. However, these algorithms require Graphical Processor Units (GPUs), which are expensive and on high demand in the current time because of the lack of semi-conductor chips. The computational demands are even higher in the case of medical images, where the images are three dimensional as in Computed Tomography (CT) or Magnetic Resonance (MRI) imaging techniques. This is a major bottleneck in the deployment of automatic segmentation algorithms in hospitals that can, on the other hand, save lives, by for example, automatically detecting lumps in the lungs of people in CT scans. One outcome of this project has been developing more efficient segmentation algorithms, concretely using Graph Convolution Networks (GCNs), instead of CNNs. While CNNs predict a label for every pixel in the images, GCNs predict the labels for regions of similar pixels in the images that were obtained in a pre-processing step. Therefore, orders of magnitude fewer labels need to be predicted, with the consequent complexity reduction. By finally merging the regions with the same predicted label, the final segmentation is obtained. The computational cost of this approach is about 3 times smaller than that of CNNs based segmentation methods. We have demonstrated our approach in two contributions in the form of a short paper in the 2021 edition of Medical Image Understanding and Analysis (MIUA) UK conference (link: https://miua2021.com/abstracts/120.pdf), where the method was used to segment skin lesions in 2D colour images of the skin, and in a challenge in the 2021 Medical Image Computing and Computed Assisted Intervention (MICCAI) international conference, (https://link.springer.com/chapter/10.1007/978-3-030-98253-9_24), where the method was used to segment tumours in head and neck cancer patients in CT images. Another challenge is that algorithms optimized for general tasks may fail for specific sub-tasks. For example, in blood vessel segmentation, crucial for surgical planning, general segmentation algorithms make mistakes that are trivial for human doctors. This kind of mistakes occur because human doctors have knowledge of certain symmetries of lesions and anatomy that the AI model lacks. My research is about how to incorporate these symmetries into the AI vision systems, so they produce more reliable anatomy segmentation for clinical practice. In the context of blood vessel segmentation, we have demonstrated how symmetry information can be included into a deep neural network in a poster in SPIE Medical Imaging 2023 (https://spie.org/medical-imaging/presentation/The-Topology-Overlap-trade-off-in-Retinal-Arteriole-Venule-Segmentation/12464-98). Here we approach the task of obtaining the segmentation of arteriole and venules in retinal fundus photographs. This is a challenging task because of the presence of arteriole and venule crossings, which results in even experts being unable to classify the thinnest smaller blood vessels. This work is both a proof of concept and a useful aid for ophthalmologists who may use fundus images for diabetes and hypertension screening for which the ocular arteriole and venules are studied. Finally, I have participated for two years in the HECKTOR 2022 challenge, organised as a satellite event of the MICCAI Conference. This challenge bigger picture is towards precision cancer medicine, which aims to deliver patient personalised treatments. In essence, the main challenge goal is to predict the cancer patient's life expectancy under the usual cancer treatments consisting of radiotherapy and chemotherapy among some specific drugs, where patients identified with a poor prognosis may be offered an alternative treatment. Therefore, one particularity of this challenge is how to process a multi-modality input, consisting both of image (CT and PET scans) and tabular (electronic health records) data. My contribution in this challenge explores whether the patient tumours can be modelled as a graph and use graph learning techniques to predict patients' survival probability. I found, in parallel with other participants, that data driven approaches seem to suffer more from poor generalization when incorporating the tabular data. Under this situation, more traditional mechanisms seem to work better. This is an important research direction for the adoption of these decision support systems into the clinical practice. |
| Exploitation Route | The contribution of using GCN to classify superpixels rather than CNNs to classify pixels to segment the structures of interest in the images has been proposed in parallel by us and other authors and areas. In addition to skin lesions and tumours (https://link.springer.com/chapter/10.1007/978-3-030-70650-0_2), it can be deployed, outside the clinical area, in aerial images for example for precision agriculture purposes (https://www.mdpi.com/2072-4292/14/2/305, https://openaccess.thecvf.com/content/CVPR2021W/AgriVision/html/Dadsetan_Superpixels_and_Graph_Convolutional_Neural_Networks_for_Efficient_Detection_of_CVPRW_2021_paper.html). Nevertheless, the method is not suited for any segmentation task. For instance, noisy images like the ones obtained with Ultrasound devices, or very narrow structures such as blood vessels may fail to be captured by the superpixel aggregation step. We are interested into how to leverage the graph representation of vessels and other tubular structures to boost segmentation anatomical correctness. We incorporated orientation scores and topology loss function into a deep neural network and demonstrated how improve segmentation of arterioles and venules from fundus photographs over a baseline and some state of the art models. Orientation scores and its geometrical processing has been recently done in parallel (https://link.springer.com/article/10.1007/s10851-022-01114-x), however applied only to retinal vessel segmentation, agnostic to the arteriole/venule nature of the vessels. Regarding tumour relationship modelling with graphs, we explored this direction in our contribution to the HECKTOR challenge in 2022, where a Graph Neural Network is used to classify tumour graphs. Similar approaches can be found in https://arxiv.org/pdf/2106.01711.pdf, modelling the tumours as nodes, and in https://www.sciencedirect.com/science/article/pii/S0895611122000805, where further information about the anatomical structures where nodes lie is included. We plan to follow up on the research by including not only tumours and stations but also neighbouring organs, also referred to as Organs at Risks (OaR) in radiotherapy jargon. |
| Sectors | Aerospace, Defence and Marine,Agriculture, Food and Drink,Digital/Communication/Information Technologies (including Software),Electronics,Healthcare,Pharmaceuticals and Medical Biotechnology |
| Description | Collaboration with Canon Medical Research Europe LTD |
| Organisation | Canon |
| Department | Canon Medical Research Europe |
| Country | United Kingdom |
| Sector | Academic/University |
| PI Contribution | My contribution to this partnership is to provide experimentation and results in novel techniques and yield any arising intellectual property for the company's benefit. While not being directly involved in all the lines of work performed by the company, I carry out research in topics aligned with the interest of the company, that is, development of imaging software for radiological devices. The final aim is to incorporate any findings or advances to the products or services that the company develops. |
| Collaborator Contribution | On the other hand, my partners provide me with guidance and feedback for the experiments I carry out and the publications I aim to submit to different companies and venues. They also help me to analyse and detect IP the research outcomes. Finally, they organise 2 or 3 day events, also known as Hackathons, where we all sit together and approach problems of common interest such as volumetric segmentation or visualization of medical images using open source programming languages and libraries. |
| Impact | So far, the outcome has been the development of an efficient segmentation technique combining superpixel generation and classification using Graph Convolution Networks (GCN). This technique has been presented in two contribution, one short paper and one book chapter, respectively demonstrating the methods in 2D and 3D medical images. |
| Start Year | 2019 |
| Description | Collaboration with Golden Jubilee National Hospital |
| Organisation | NHS Greater Glasgow and Clyde (NHSGGC) |
| Department | Golden Jubilee National Hospital |
| Country | United Kingdom |
| Sector | Hospitals |
| PI Contribution | The detection of PEs and other pulmonary abnormalities is crucial as some of them are operable and thus the lives of patients can be saved. However, detecting these lesions in the CTPA scans is demanding and time consuming and relies on rather arbitrary selection of region of interest for inspection. We develop an AI system that automatically detects and segments the veins, arteries and PE candidates in the scans, specifically by incorporating geometric priors to improve over several baselines. Public datasets and evaluation code released by a recent contribution in the Medical Image Computing and Computed Assisted Intervention (MICCAI) conference are used to develop a method that is tested in the hospital inhouse medical images. |
| Collaborator Contribution | Our medical partners will contribute by assessing the predictions of our models in their CTPA scans. A first round of preliminary results was presented and provided accurate segmentations of the arteries and veins and detection of numerous PE candidates. Our medical partners noted that, while not all detected candidates actually correspond to PEs, they can be other pathologies such as blood tumours, which is promising for more general pulmonary vascular anomaly detection. They also noticed the presence of some false negatives (missed detections), which may be mitigated by a preference of sensitivity in the optimization procedure. |
| Impact | There are still not outputs however a conference paper is expected by September 2023. |
| Start Year | 2022 |
| Description | 24th International Conference on Medical Image Computing and Computer Assisted Interventions (MICCAI 2021) |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Other audiences |
| Results and Impact | This is the reference conference in the field of my research. My sponsor company, Canon, usually provides a gold sponsorship to the conference. From Canon we set up a virtual Question and Answer section where people drops and chat with us. Both in 2020 and 2021 it has taken place virtually. I enjoyed it much more in the last edition as I guess the virtual platform was much better designed and enabled a smoother interaction between conference assistants. Furthermore, this last event, I partecipated in the HECKTOR challenge, where I also submitted the paper which is going to be published in the Challenge proccedings. Although I did not actually presented the paper during the conference, the event is always a great chance to reach out with the medical imaging comunity and catch-up with the latest research. Also in this past edition the conference chair open sourced the paper review process, which disclosed the reviews of any paper being presented at the conference. Being able to read through the reviews of other colleagues' submitted papers, includying the rebuttan process, is of great help for me to prepare my own contributions for future events. |
| Year(s) Of Engagement Activity | 2020,2021 |
| URL | https://www.miccai2021.org/en/ |
| Description | 25th UK Conference on Medical Image Understanding and Analysis |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Other audiences |
| Results and Impact | This was the conference where I presented my short paper demonstrating the efficient segmentation method based on superpixel classification using Graph Convolution Networks to segment different medical images. In addition to networking and sharing views and optinions, this provided me the first chance to actually discuss my work with other acadmics outside my research team at Canon, concretely with the colleagues at University of Oxford, so it was a quite nice experience. |
| Year(s) Of Engagement Activity | 2021 |
| Description | Assistance to SPIE 2023 for a poster presentation |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Postgraduate students |
| Results and Impact | I assisted to the SPIE Medical Imaging 2023 conference, to the Image Processing track, to present my work on retinal arteriole-venule segmentation from fundus images as a poster presentation on February 20 2023, between 6 to 8 pm. This opportunity provided me with live feedback from attendees stopping by my poster, the chance of enhancing my communication skills by adapting to different audiences, and more generally the invaluable experience of assisting to a conference and networking full time during several days. I drawn relevant highlights for my own research and more broadly the field of AI applied to medical imaging and I was able to share these highlights with my project stakeholders. |
| Year(s) Of Engagement Activity | 2023 |
| URL | https://spie.org/conferences-and-exhibitions/medical-imaging |
| Description | CDT Conference and Anniversary 21st Celebration |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Postgraduate students |
| Results and Impact | In this anniversary event most current students with the Center for Doctoral Training in Applied Photonics gathered in Heriot-Watt University for a some poster sessions, keynote talks and a networking lunch. The topics ranged from photonics, lasers and precision measurements to medical imaging projects and even defense systems. I found the event truly useful to understand the research fields of my peers but also to learn how are they appliying machine learning and AI to their different problems. |
| Year(s) Of Engagement Activity | 2022 |
| URL | https://cdtphotonics.hw.ac.uk/event/annual-conference/ |
| Description | GeoMedIA Workshop 2022 |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Postgraduate students |
| Results and Impact | A workshop where progress in different applications of geometric deep learning in medical images was presented in the form of both oral and poster presentation. This event, of a one day duration, took place in the CASA Hotel in Amsterdam on 18 November 2022. It provided an excellent opportunity to network and find out about recent advances made in the field. |
| Year(s) Of Engagement Activity | 2022 |
| URL | https://geomedia-workshop.github.io/ |
| Description | Internal Company Hackathon |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Professional Practitioners |
| Results and Impact | This was an internal company meeting where we sit together and approaches a challenging volumetric image segmenation problem. It provided a great chance for in person communication with my coleagues in a time where we all are working remotely. We sit together and implemented algorithms described in papers, but more importantly, practiced team working skills and socialised. |
| Year(s) Of Engagement Activity | 2021 |
| Description | Participation in HECKTOR 2022 Workshop in MICCAI 2022 |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Postgraduate students |
| Results and Impact | I assisted virtually to this hybrid event held in conjunction with MICCAI 2022 on Singapore where the different contributions to the HECKTOR 2022 challenge were presented. I have participated in this challenge for two consecutive years. The aim of the challenge is to improve both tumour segmentation and patient risk modelling from computed tomography images and clinical health record data. The bigger picture contribution is towards precision medicine. Eventually, these systems will identify patients with poor prognosis under traditional cancer treatment (e.g. chemotherapy and radiotherpay), and offer them participate in research of new cancer therapies. The challenge organizers increased the dataset considerably from the first year to the second year, and also made the task more challenging by not providing regions of interest and including a new tumour class. Whereas my contribution in the first year was more oriented on improving the efficiency of the tumour segmentation with supervoxel graphs, in 2022 I changed focus to exploring modelling the patient tumours as a graph. A common highlight emerging from both my work and other contributions was that data-driven risk score prediction models suffer from severe drops in accuracy when tested in unseen datasets, and are surpassed by traditional machine learning models in these scenarios. Investigating how to combine both the data driven and traditional approach is an open research direction, as well as to include information from not only from tumours but other organs at risk (OaR) in the predictive systems. |
| Year(s) Of Engagement Activity | 2022 |
| URL | https://hecktor.grand-challenge.org/Overview/ |
| Description | SUSSP 76 Summer School in Photonic Imaging, Sensing & Analysis |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Other audiences |
| Results and Impact | This is a recurrent event happening every 2 years where students participating in the CDT for Applied Photonics share their research outputs. It is quite multidisciplinary as more we have presentations more on the side of lasers/microscopy and other devices, as well as, now every time more, image processing and AI projects. This was the first time I joined the event, and I presented my work of efficient segmentation of 2D skin lesions. Furthermore, it also provided me the chance to come across other people's work and found quite interesting contributions linking areas such as microscopy and super-resolution. Although the event took place remotely, the online platform provided a smooth experience. There were awards for best poster presentations and, time for social discussion, networking and everything one would expect form an online conference. EDIT 2023. This event will happen this year in person in St Andrews from 2 to 11 July 2023. I will present my work on blood vessel segmentation and PE detection there. |
| Year(s) Of Engagement Activity | 2021,2023 |
| URL | https://cdtphotonics.hw.ac.uk/event/sussp79/ |