ARTICT | Art Through the ICT Lens: Big Data Processing Tools to Support the Technical Study, Preservation and Conservation of Old Master Paintings

Lead Research Organisation: University College London
Department Name: Electronic and Electrical Engineering

Abstract

The heritage science sector is experiencing a digital revolution linked to the emergence and increasing adoption of cutting-edge non-invasive analytical imaging techniques generating large volumes of multidimensional data from cultural heritage objects. These include macro X-Ray fluorescence (MA-XRF) scanning and hyperspectral imaging (HSI). Combining MA-XRF and HSI data - providing elemental and molecular information - offers huge potential for improved identification, characterization, and visualisation of the materials and features of interest in a painting, including in sub-surface layers of the painting. However, it is increasingly recognised that the wealth of data associated with these modalities cannot be fully exploited through traditional primarily manual approaches to interrogating heritage science data.

The aim of this research - bringing together ICT and Heritage Science researchers to enable the cross-pollination of ideas and expertise - is to co-create new automatic signal analysis and processing tools that are able to 'fuse' MA-XRF and HSI data to support the technical study, conservation and preservation artwork.

In particular, the proposed tools will provide the means to identify, characterize and visualize materials present within a painting, thereby leading to new insights relevant for the conservation and preservation of Old Master paintings or to new ways to engage the public with cultural heritage, science and ICT. For example, the proposed tools will also help visualise in a more integrated and accessible form the features of interest for art-historical study and conservation, such as underdrawing, pentimenti, concealed designs, losses or non-original materials.

To develop such tools, an ambitious research programme is proposed that includes development of: 1) multidimensional multimodal heritage science datasets, 2) multimodal signal processing algorithms for data correction, alignment, registration and mosaicking; and 3) new multimodal signal analysis algorithms capable of inferring material distributions in a painting from MA-XRF and HSI data. In addition, this research programme also envisions a number of significant advances in the area of signal processing - including new sparsity-driven nonlinear unmixing algorithms - that address the specific challenges arising in art investigation that do not arise in other application domains of signal processing, such as the need to identity mixtures of (aged) materials in superimposed layers using combined MA-XRF and HSI datasets.

The research programme also includes a number of case studies on National Gallery paintings that will assess the validity, potential and relevance of the proposed tools to the wider heritage sector.

The other aim of this research is to champion and sustain co-creation activities across the ICT and Heritage Science sectors. The activities planned during and beyond the project include: a) training of researchers, doctoral students, and undergraduate students; b) dedicated courses, workshops and events; c) outreach and public engagement activities; and d) a UK-wide network in the area of "ICT for Art Investigation". These aim to augment the project's co-creation and cross-disciplinary ethos and catalyse further research.

The ideas and tools conceived throughout the research will lead to impact across various levels and communities: a) the ICT sector will be exposed to new challenges stimulating developments in a new area of signal processing for art investigation; b) the Heritage Science sector will benefit from new automated, accessible, robust, user-friendly tools to aid the work of heritage scientists, art historians, and conservators. Finally, the tools and the images and insights they create will provide galleries with new innovative means to interpret and present their collections to the general public.

Planned Impact

Heritage science plays a key role in promoting access to cultural heritage, its interpretation, conservation and management, and this research will offer a range of new multimodal, multidimensional data analysis and processing tools with potential to lead to direct impact on this sector. Further, in view of culture's importance to UK life and tourism, the research will have indirect impact on the UK's economy and society.

Of particular relevance is the current interest in the cultural heritage sector - stimulated by the emergence of new non-invasive imaging techniques and a desire to minimise removal of physical samples for analysis - in the development of automated data processing/analysis tools to support the technical study, preservation and conservation of artworks. The immediate impact of this research on the activities of the National Gallery (NG) or other cultural heritage institutions in the UK and overseas will be as follows:

1 The proposed tools will enhance the ability of heritage scientists and art historians to study aspects of an artwork's creation, including the artist's palette, materials and technique, the artwork's provenance, or changing appearance over time, influencing how artworks are interpreted and presented.

2 The proposed tools will also enhance the understanding of heritage scientists and conservators of an artwork's conservation history, e.g. characterising non-original materials, areas of damage, etc., impacting on conservation approaches to unique (national) collections.

3 The availability of multimodal data analysis technology offers heritage scientists the means to readily combine, interpret, and re-interrogate the existing and expanding datasets that many cultural heritage institutions hold for their collections, maximising the potential of such bodies of data and the understanding of the associated artworks. Further, the tools also offer the potential to make complex scientific data more accessible by presenting a combined interpretation based on data from multiple modalities in a single image.

The visual outputs from the tools - and insights they will create - also provide innovative new ways to present paintings or other artworks within the cultural heritage sector (e.g. art historians&conservators) and to a wider audience. Via presentation on modern digital devices this could range from new views about an artwork's creation and conservation history to new ways to interact digitally with artwork using augmented reality technology that could be the basis for new exhibitions in museums or galleries. The NG's experience with exhibitions and web features based on technical examination of paintings suggests this highly visual area of applied science is an effective way to engage with the public and can increase public understanding of science and ICT.

The research will also promote new avenues of enquiry in the area of signal processing - notably, "Signal Processing for Art Investigation" - as well as a co-creation and cross-disciplinary culture within the wider area of "ICT for Art Investigation". It is anticipated that by demonstrating the power of ICT and signal processing research to solve real challenges in another sector, this project will influence both new research directions within the ICT community along with further co-created, cross-disciplinary ICT research in the cultural heritage sector, and within the Arts&Humanities more generally.

The reputations of the applicant organisations in their respective communities, and their wide international academic and professional networks, will ensure the impact of this research within the ICT sector, the Heritage Science sectors and with heritage end-users. Of particular relevance here is the NG's involvement in developing the European Research Infrastructure on Heritage Science (E-RIHS, www.e-rihs.eu). The experience of the applicants in public engagement activities will ensure the wider impact of the research is realised

Publications

10 25 50
 
Description The team has developed new data processing algorithms ingesting complex datasets acquired on paintings that capable of revealing concealed designs present within a paintings stratigraphy. Some of these results have been recently reported in high-profile publications and the media worldwide.
Exploitation Route (1) Art historians can use our results in order to inform art-historical investigation.

(2) Art conservators can use these results in order to inform how to better conserve and preserve paintings.
Sectors Digital/Communication/Information Technologies (including Software),Culture, Heritage, Museums and Collections

 
Description Our data processing algorithms are currently being considered by cultural heritage institutions in order to reveal key insights about paintings. We are currently in touch with the National Gallery, London, the National Gallery of Art, Washington, DC, and the Ghent Altarpiece Conservation Team.
First Year Of Impact 2019
Sector Culture, Heritage, Museums and Collections
Impact Types Cultural

 
Description Multi-Modal Signal Processing for Art Investigation
Amount £90,750 (GBP)
Funding ID NIF\R1\192656 
Organisation The Royal Society 
Sector Charity/Non Profit
Country United Kingdom
Start 03/2019 
End 02/2021
 
Description Multimodal analytical imaging of Old Master Paintings: addressing the challenges of registration, mosaic construction and image resolution
Amount £79,800 (GBP)
Funding ID AH/T002417/1 
Organisation Arts & Humanities Research Council (AHRC) 
Sector Public
Country United Kingdom
Start 10/2020 
End 03/2024
 
Description UCL / Duke Univ / National Gallery Collaboration 
Organisation Duke University
Country United States 
Sector Academic/University 
PI Contribution This research collaboration involved the development of new multi-modal data processing schemes with applications to art investigation. Our teams have contributed with the development/test of algorithms.
Collaborator Contribution This research collaboration involved the development of new multi-modal data processing schemes with applications to art investigation. Our partner has contributed with insights/datasets/
Impact This collaboration has led to the publications: -- W. Pu, B. Sober, N. Daly, C. Higgitt, I. Daubechies, M. R. D. Rodrigues. A connected auto-encoders based approach for image separation with side information: with applications to art investigation. IEEE International Conference on Acoustics, Speech and Signal Processing, Barcelona, Spain, 2020. -- Z. Sabetsarvestani, B. Sober, C. Higgitt, I. Daubechies, M. R. D. Rodrigues, Artificial intelligence for art investigation: Meeting the challenge of separating x-ray images of the Ghent Altarpiece. Sci. Adv. 5, eaaw7416 (2019).
Start Year 2018
 
Description Contribution to video produced by EPSRC (hosted on EPSRC website) to discuss cross-disciplinarity and co-creation in research 
Form Of Engagement Activity Engagement focused website, blog or social media channel
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact Contribution to video produced by EPSRC (hosted on EPSRC website) to discuss cross-disciplinarity and co-creation in research based on our experience in developing a successful grant proposal
Year(s) Of Engagement Activity 2019
URL https://epsrc.ukri.org/research/ourportfolio/themes/ict/introduction/crossictpriorities/crossdiscipl...
 
Description Imperial College Press Relerase and YouTube video 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Media (as a channel to the public)
Results and Impact The press release described research conducted by Imperial College and National Gallery teams that helped reveal under-drawing in the Leonardo da Vinci's painiting "Virgin of the Rocks". The press release is available here: https://www.imperial.ac.uk/news/195075/new-algorithm-helps-uncover-forgotten-figures/

The corresponding YouTube video with 72,000 views here:
https://www.youtube.com/watch?v=oAmVLscVKqM&feature=youtu.be
Year(s) Of Engagement Activity 2020
URL https://www.imperial.ac.uk/news/195075/new-algorithm-helps-uncover-forgotten-figures/
 
Description Presentation to the Scientific Advisory Board of E-RIHS (http://www.e-rihs.eu/) 
Form Of Engagement Activity Participation in an open day or visit at my research institution
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact A description of the activities being undertaken within the ARTICT project was included as part of a presentation to the Scientific Advisory Board of E-RIHS (http://www.e-rihs.eu/) at a meeting held at the NG (13.9.2019)
Year(s) Of Engagement Activity 2019
 
Description Tutorial on 'Signal and Image Processing for Art Investigation' at the IEEE International Conference on Acoustics, Speech and Signal Processing 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact A tutorial on 'Signal and Image Processing for Art Investigation' to be delivered at the IEEE International Conference on Acoustics, Speech, and Signal Processing Conference in Barcelona, Spain. This tutorial has been conceived/delivered by the project team, including members of University College London, Imperial College, and National Gallery.
Year(s) Of Engagement Activity 2020
URL https://2020.ieeeicassp.org/program/tutorials/signal-and-image-processing-for-art-investigation/
 
Description University College London Press Release and Media Coverage 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Media (as a channel to the public)
Results and Impact The press release described research conducted by University College London and National Gallery teams that helped separate mixed x-rays associated with double-sided paintings. The press release is available here: https://www.ucl.ac.uk/news/2019/sep/ai-uncovers-new-details-about-old-master-paintings
https://www.ucl.ac.uk/iccs/news/2019/sep/ai-uncovers-new-details-about-old-master-paintings

Media coverage includes:

https://www.telegraph.co.uk/science/2019/08/30/hidden-works-goya-van-gogh-van-eyck-could-discovered-using-artifical/

https://www.forbes.com/sites/simonchandler/2019/09/06/forget-the-future-ai-will-take-us-back-to-the-past/#1cc67ddd2d13
Year(s) Of Engagement Activity 2019
URL http://www.ucl.ac.uk/news/2019/sep/ai-uncovers-new-details-about-old-master-paintings