Multimodal analytical imaging of Old Master Paintings: addressing the challenges of registration, mosaic construction and image resolution

The cultural heritage sector is experiencing a digital revolution. It is now possible to scan entire paintings with a range of complementary imaging techniques and the National Gallery is one of the few institutions in the world with access to this equipment. It has a leading presence in the field of high-resolution digital imaging of paintings including macrophotography, imaging by X-radiography, HSI and MA-XRF scanning.

These multimodal datasets contain a wealth of information which, when properly exploited, offer unprecedented insights into the creation and history of Old Master paintings. However, the generation of huge bodies of data using different imaging techniques poses new signal processing challenges that cannot be addressed with traditional supervised approaches.

The expertise needed to adapt and apply such approaches does not exist within the heritage sector: collaboration with experts in image and signal processing is required. Through collaboration, such experts in turn would gain access to a wide range of multimodal data that presents some unique challenges and the possibility to demonstrate the potential of their approaches to a broad audience via the National Gallery's activities.

Objectives
Currently, in the cultural heritage field, images are registered and/or mosaicked using overlap regions between frames normally manually using tools like Photoshop or custom software. The process needs a lot of user input and varies considerably in how successful it is. With a wider range of imaging modalities now being used which are acquired using different instruments, the limitations of the current approaches are becoming increasingly apparent. The issue is further complicated by the increasing use of spectroscopic imaging techniques (e.g. HSI or MAXRF) generating huge datacubes. This poses new signal processing challenges and require the creation of new signal and image processing tools to facilitate the processing and the interpretation of the acquired data.

Research questions
The main research question is how to develop new registration methods that can automatically extract features which are common to different modalities and which are resilient to variation in spatial resolution as well as other form of inconsistencies and to use them for rigid or semirigid registration. The main focus will be on registration and mosaicking of MA-XRF, HSI and visible images and datacubes but the wider applicability of the methods will also be explored.

Methodology and Outcomes
The approach to be adopted will be to initially develop separate algorithms to address the challenges of automatic multimodal registration and mosaicking and of resolution enhancement for multimodal data/images. Attempts will then be made to solve the registration and resolution enhancement problem with a single approach.
Beyond the usual outcomes such as the PhD thesis, academic publications, conference presentations and seminars, other tangible outcomes of the research include:

- Development of a range of novel algorithms to facilitate the comparison and interpretation of multimodal data and images including algorithms for automatic (but human-in-the-loop based) registration and mosaicking and resolution matching of a wide range of multimodal images (and datacubes);

- Production of a range of open-source, user-friendly software tools to allow art curators and other heritage end-users to use the methods developed in the thesis and based on these algorithms;

- Development of a tool or graphical user interface to display overlaid image modalities and blend/morph from one modality to another to help visualise if features have been moved, changed or concealed;

- Development of a researcher with valuable interdisciplinary research skills, comfortable working across traditional subject boundaries and able to liaise with a range of stakeholders and end-users.