Cross-level Convolutional Transformer and Adversarial Multi-task Learning for Medical Semantic Segmentation

This project plans to study cross-level convolutional Transformers and adversarial multi-task learning for medical semantic segmentation (MSS). The goal of MSS labels each pixel of an image with a corresponding class of what is being represented to provide the segmentation maps. Though deep learning-based methods have achieved state-of-the-art performance in MSS, they still struggle to achieve fine-grained segmentation maps in complex environments, which prohibits their implementation to real-world applications.

To handle this problem. I plan to explore some promising algorithms for MSS. First, I will focus on improving skip connections of UNet by proposing a convolutional Transformer with cross-level interaction. Second, I will aim to break the shackle of model performance caused by the small number of annotations, through a shared-private architecture with adversarial multi-task learning to use as much additional data as possible.

The potential impact of this project mainly includes two aspects: First, the methods studied in this project will improve the model performance of MSS in complex scenes and have strong potential to broaden the applicability of multi-modal/unlabeled/multi-task data. Second, this project will develop a generalized and instructive structure for MSS thanks to the shared-private mechanism and adversarial learning. It could be used for multi-tasks with heterogeneous inputs, with only a few modifications.

Aims and Objectives

This project aims to address two key challenges to improve the performance and usability of MSS in complex surgical environments.

Challenge 1: How to extract high-quality features and fuse them effectively?

The latest MSS methods based on UNet fail to explore sufficient information from full scales due to the following two aspects:

a) Not all connection pathways are effective due to the issue of semantic gaps in different layers. Those redundant and irrelevant connections increase the training difficulty of the network, even some can undermine the performance.

b) The optimal combination of skip contributions is varied among different datasets, which depends on the scales and appearance of segmentation objects.

To address the above problems, I consider replacing vanilla skip-connection pathways with Transformers to capture non-local features and perform effectively cross-level feature fusion.

Challenge 2: How to utilize multi-modal data, unlabeled data, or even data from other tasks to improve the model performance?

The scarcity of carefully-labelled datasets becomes an unavoidable limitation in DL-based MSS as both data and annotations are expensive to acquire. Previous methods pay less attention to utilizing different types of external data. Therefore, I consider using adversarial multi-task learning to build a uniform architecture for additional data, regardless of its type.

This project will propose a novel cross-level convolutional Transformer for MSS to improve the skip-connection process of UNet.

This project will propose a novel shared-private network with multiple encoders and decoders for MSS to utilize adversarial multi-task learning to handle additional data or tasks.

As surgical image data are generally characterized by multiple modalities/scales and complex scenes, the research based on it could shed light on complex MSS.

The critical mass of scientists and engineers that i4health will produce will ensure the UK's continued standing as a world-leader in medical imaging and healthcare technology research. In addition to continued academic excellence, they will further support a future culture of industry and entrepreneurship in healthcare technologies driven by highly trained engineers with deep understanding of the key factors involved in delivering effective translatable and marketable technology. They will achieve this through high quality engineering and imaging science, a broad view of other relevant technological areas, the ability to pinpoint clinical gaps and needs, consideration of clinical user requirements, and patient considerations. Our graduates will provide the drive, determination and enthusiasm to build future UK industry in this vital area via start-ups and spin-outs adding to the burgeoning community of healthcare-related SMEs in London and the rest of the UK. The training in entrepreneurship, coupled with the vibrant environment we are developing for this topic via unique linkage of Engineering and Medicine at UCL, is specifically designed to foster such outcomes. These same innovative leaders will bolster the UK's presence in medical multinationals - pharmaceutical companies, scanner manufacturers, etc. - and ensure the UK's competitiveness as a location for future R&D and medical engineering. They will also provide an invaluable source of expertise for the future NHS and other healthcare-delivery services enabling rapid translation and uptake of the latest imaging and healthcare technologies at the clinical front line. The ultimate impact will be on people and patients, both in the UK and internationally, who will benefit from the increased knowledge of health and disease, as well as better treatment and healthcare management provided by the future technologies our trainees will produce.

In addition to impact in healthcare research, development, and capability, the CDT will have major impact on the students we will attract and train. We will provide our talented cohorts of students with the skills required to lead academic research in this area, to lead industrial development and to make a significant impact as advocates of the science and engineering of their discipline. The i4health CDT's combination of the highest academic standards of research with excellent in-depth training in core skills will mean that our cohorts of students will be in great demand placing them in a powerful position to sculpt their own careers, have major impact within our discipline, while influencing the international mindset and direction. Strong evidence demonstrates this in our existing cohorts of students through high levels of conference podium talks in the most prestigious venues in our field, conference prizes, high impact publications in both engineering, clinical, and general science journals, as well as post-PhD fellowships and career progression. The content and training innovations we propose in i4health will ensure this continues and expands over the next decade.