Bayesian model selection & calibration for computational imaging

Lead Research Organisation: Heriot-Watt University
Department Name: S of Mathematical and Computer Sciences


Modern digital images are increasingly generated by using Computer-intensive Imaging (CI) technology. Indeed, because modern imaging sensors push technology and physics to the limits, the data they produce are generally not useful in their raw form (e.g., they are corrupted by noise and only observed partially or with insufficient resolution to accelerate the acquisition process and reduce sensor energy consumption). Imaging devices address this difficulty by using CI tools to analyse the data and recover high-quality images with fine detail. The last decade has witnessed important advances in this field, with most CI technologies now adopting formal mathematical approaches to derive solutions and to study the underpinning computer algorithms. This has led to imaging devices that are faster, have greater spatial resolution and dynamic range, and are more robust to challenging conditions (e.g., night, long-rage, and underwater imaging). This has in turn produced significant social and economic benefit through impact on application areas such as medical imaging; astronomical imaging; satellite and airborne remote sensing for agriculture, earth sciences and defence; non-destructive testing; and microscopy for drug and nanotechnology development.

However, CI solutions can be very sensitive to the choice of the mathematical models used to analyse the raw sensor data (e.g., the data-fidelity term and the regularisation functions used to generate the image), and it is fundamental to carefully select and calibrate models for the specific imaging setup and type of scene considered. Presently, this requires extensive expert supervision, which is expensive and time-consuming.

The aim of this project is to develop a toolbox of new mathematical methods and computer algorithms to automatically select and calibrate CI models, directly from the observed sensor data, without using ground truth data, and with minimum expert supervision. This toolbox will be developed by combining advanced mathematical techniques stemming from Bayesian statistics, with specialised computer algorithms from the area of stochastic Monte Carlo simulation and optimisation. The expected outcome is that this toolbox will significantly simplify the development and deployment of CI technology, and amplify its adoption in science and industry as a result.

During the project, the proposed tools will be applied to two challenging CI problems related to satellite and astronomical imaging. More precisely, the methods developed in this project will be used to enhance the resolution and fine detail in hyperspectral satellite images and in radio-interferometric astronomical images. These applications will be investigated in collaboration with world-leading experts at the Mullard Space Science Laboratory, Heriot-Watt University, and University of Toulouse. These experts will provide data, training, and application-specific software. They will also help disseminate this work and amplify its impact.

To maximise the impact of the project on the economy and society, open-source code for all the proposed tools will be made publicly available on the project webpage, together with documentation and pedagogical demonstration kits.

Planned Impact

Computer-intensive imaging (CI) technologies deliver images with great detail and dynamic range, that are more robust to challenging conditions (e.g., night, long-rage, and underwater imaging), and which can be acquired using short acquisition times. This is important, for example, because it allows accelerating MRI scanners to increase the number of patients that can be imaged per day and reduce the cost of the scans. In a similar fashion, by enhancing the quality of hyperspectral images, CI technology improves precision-agriculture decisions and predictions of income from tax on cash crops and export commodities. CI technology also plays a central role in modern radio-astronomy, which is a fundamental tool for studying celestial objects such as radio galaxies, quasars, pulsars, and masers, and improving our understanding of the universe.

This project will develop a toolbox of statistical methods and computer algorithms to automatically select and calibrate CI models. The expected outcome is that this will significantly reduce the cost related to developing and deploying new CI technology, and hence amplify its adoption in science, industry and society.

During the project, the proposed new tools will be applied to two important CI problems related to enhancing hyperspectral satellite images and radio-interferometric astronomical images. These applications will be studied in collaboration with world-leading experts who will provide data, training, software, and help disseminate the work and amplify its impact to the satellite imaging and the astronomical imaging communities.

We expect that the results of this project will lead to at least 5 journal papers, which we will submit for publication to top imaging journals (e.g., SIAM Journal on Imaging Sciences and IEEE Trans. Computational Imaging) and also make freely available on the arXiv (an open access repository of preprints). The work will also be advertised at international conferences and workshops, ensuring a worldwide impact for this project. Moreover, to ensure that these results are communicated efficiently to other UK researchers, the PI will continue to regularly speak at research seminars and at national meetings.

To maximise the impact of our work, the tools developed during the project will be made publicly available in the form of open-source software, with clear documentation and easy-to-follow demonstration kits, which we will advertise on the project webpage, specialised mailing lists, and in all project-related presentations and publications. Together with publications describing the technical aspects of the work, these source codes and demonstration kits will allow companies and research groups developing and deploying CI technology to easily benefit from our proposed new tools. Moreover, the availability of a C++ implementation will allow to easily incorporate our methods into the PURIFY open-source radio-astronomy software (

Also, as part of this project, the PI will organise a 3-day workshop to bring together world-leading experts working on different aspects of CI and experts in statistics and other areas of mathematics, so that they can discuss the main challenges in CI and explore new solutions. The PI will also organise a special session at interface of CI and stochastic Bayesian computation methods at the 2021 IMA Conference on Inverse Problems (note that the PI will chair this conference, which take place in Edinburgh in Sep. 2021). This workshop, the special session, and the availability of open-source code and demonstration kits, will provide a clear opportunity for UK applied mathematicians and Bayesian computation experts to engage with the UK data science agenda, collaborate with the CI community, and deliver impact though applications to medical imaging, remote sensing, etc.
Description We have developed a new way of automatically adjusting the parameters of the mathematical equations underpinning a broad class of image enhancement algorithms, which are widely used in modern imaging technology (e.g., satellite cameras, medical imaging devices, telescopes, microscopes, etc.). We have studied the application of this new technique to a specific type of parameters that are difficult to adjust manually (the so-called regularisation parameters) and demonstrated that the new technique significantly outperforms existing techniques, both in terms of empirical performances as well as in terms of mathematical guarantees on the delivered solutions. This set the groundwork for tackling other challenging parameters, namely the parameters describing the physical aspects of the imaging instrument. In particular, we have recently modified the proposed technique to tackle blind and semi-blind image deblurring problems. Preliminary experiments suggest that the technique can accurately identify the blur degradation operator in mild and moderate natural-image deblurring problems. A more exhaustive evaluation is currently in progress.
Exploitation Route Most modern imaging algorithms are based on some mathematical equations that analyse the raw data to produce high-resolution images with sharp detail. These equations typically have some parameters that users need to set manually, this is time-consuming and requires expertise. The technique that we have developed allows automating this process. We expect that the technique will now be customised for different specific problems by applied imaging experts, and eventually incorporated into software and imaging products.
Sectors Aerospace, Defence and Marine,Agriculture, Food and Drink,Construction,Environment,Government, Democracy and Justice,Manufacturing, including Industrial Biotechology

Description The findings have led to the following pathway to impact: The French Space Agency (CNES), which is a key player in the European Space Agency (ESA), has expressed a strong interest in the Bayesian mathematical and computational methods that we are developing in this project. They are interested in using them for adjusting the parameters of the mathematical models that they use as part of their satellite imaging pipeline. In order to explore this, we had a series of regular meetings between October 2020 and February 2021, where I provided training for them and discussed specific aspects of the application they intend to develop. Following on from this, they decided to conduct a range of experiments to study the performance of Bayesian methods in the context of non-blind satellite imaging. These experiments revealed that Bayesian methods can significantly outperform their current methods, particularly in cases of high blur and poor signal-to-noise ratio. A more exhaustive evaluation with specialised Bayesian methods is currently in progress. Moreover, from June 2021 to November 2021, we had a second series of meetings where I provided training for them on blind and semi-blind Bayesian imaging methods and discussed their application to satellite image restoration problems. Preliminary experiments with artificially degraded (high-resolution) satellite image data suggest that the Bayesian methods developed in this project could be highly relevant in this setting and reduce the need for cumbersome recalibration manoeuvres. We are currently developing a Python library that will allow them to conduct a more exhaustive evaluation of the methods directly within the CNES image processing platform. This work started two months ago and could potentially lead to an impact case in the future.
First Year Of Impact 2021
Sector Aerospace, Defence and Marine
Impact Types Policy & public services

Title SOUL 
Description SOUL is a stochastic approximation proximal gradient algorithm designed for computing maximum marginal likelihood estimators in high-dimensional inverse problems with an underlying convex geometry. It is based on unadjusted Langevin Markov chain Monte Carlo samplers, such as the Moreau-Yoshida regularised unadjusted Langevin algorithm. 
Type Of Material Computer model/algorithm 
Year Produced 2020 
Provided To Others? Yes  
Impact SOUL has been successfully applied to the estimation of regularisation parameters in a range of imaging inverse problems involving synthetic data. Its application to real problems in satellite imaging is currently under investigation. 
Description Alain Durmus and Valentin De Bortoli 
Organisation École Normale Supérieure de Cachan
Country France 
Sector Academic/University 
PI Contribution We have led the development of the new technique to adjust the parameters of the mathematical equations underpinning the image enhancement algorithms, as well as the specific applications to imaging that we have studied so far in this project.
Collaborator Contribution Alain Durmus and Valentin De Bortoli have led the theoretical analysis of the new technique, particularly the convergence guarantees under easily verifiable conditions.
Impact This is a multi-disciplinary collaboration at the interface of computational imaging engineering, probability theory, computational statistics, and applied analysis.
Start Year 2017