Answering questions about medical images

Visual Question Answering (VQA) [1] is the task of answering free-text questions about the content of images. For example, given an image of a room - 'What colour is the chair standing in the corner?' - answering this question requires joint understanding of vision, language, spatial reasoning, and common sense. Thanks to advances in deep learning the state of the art in computer vision has been improving steadily over the past few years and language models experienced rapid progress recently, while spatial reasoning and common sense remain the most elusive.
Medical imaging offers plenty of interesting domain-specific challenges for VQA research [2]. For example, the images can be 3-dimensional scans, specialist vocabulary can be used in questions and answers (such as anatomical directional terms for spatial relations), but on the other hand, we have ontologies such as UMLS which can support medical common sense knowledge.
This project will focus on a range of methodological approaches in computer vision and natural language processing, which collectively support answering questions about 2D and 3D radiology images.
Also, the interpretability of VQA will be addressed here - it is both a compelling basic research direction, and a desired consideration for healthcare applications.

Complementing our Pathways to Impact document, here we state the expected real-world impact, which is of course the leading priority for our industrial partners. Their confidence that the proposed CDT will deliver valuable scientific, engineering and commercial impact is emphasized by their overwhelming financial support (£4.38M from industry in the form of cash contributions, and further in-kind support of £5.56M).

Here we summarize what will be the impacts expected from the proposed CDT.

(1) Impact on People
(a) Students
The CDT will have its major impact on the students themselves, by providing them with new understanding, skills and abilities (technical, business, professional), and by enhancing their employability.
(b) The UK public
The engagement planned in the CDT will educate and inform the general public about the high quality science and engineering being pursued by researchers in the CDT, and will also contribute to raising the profile of this mode of doctoral training -- particularly important since the public have limited awareness of the mechanisms through which research scientists are trained.

(2) Impact on Knowledge
New scientific knowledge and engineering know-how will be generated by the CDT. Theses, conference / journal papers and patents will be published to disseminate this knowledge.

(3) Impact on UK industry and economy
UK companies will gain a competitive advantage by using know-how and new techniques generated by CDT researchers.
Companies will also gain from improved recruitment and retention of high quality staff.
Longer term economic impacts will be felt as increased turnover and profitability for companies, and perhaps other impacts such as the generation / segmentation of new markets, and companies receiving inward investment for new products.

(4) Impact on Society
Photonic imaging, sensing and related devices and analytical techniques underpin many of products and services that UK industry markets either to consumers or to other businesses. Reskilling of the workforce with an emphasis on promoting technical leadership is central to EPSRC's Productive Nation prosperity outcome, and our CDT will achieve exactly this through its development of future industrially engaged scientists, engineers and innovators. The impact that these individuals will have on society will be manifested through their contribution to the creation of new products and services that improve the quality of life in sectors like transport, dependable energy networks, security and communications.

Greater internationalisation of the cohort of CDT researchers is expected from some of the CDT activities (e.g. international summer schools), with the potential impact of greater collaboration in the future between the next generations of UK and international researchers.