Bayesian model selection & calibration for computational imaging
Lead Research Organisation:
Heriot-Watt University
Department Name: S of Mathematical and Computer Sciences
Abstract
Modern digital images are increasingly generated by using Computer-intensive Imaging (CI) technology. Indeed, because modern imaging sensors push technology and physics to the limits, the data they produce are generally not useful in their raw form (e.g., they are corrupted by noise and only observed partially or with insufficient resolution to accelerate the acquisition process and reduce sensor energy consumption). Imaging devices address this difficulty by using CI tools to analyse the data and recover high-quality images with fine detail. The last decade has witnessed important advances in this field, with most CI technologies now adopting formal mathematical approaches to derive solutions and to study the underpinning computer algorithms. This has led to imaging devices that are faster, have greater spatial resolution and dynamic range, and are more robust to challenging conditions (e.g., night, long-rage, and underwater imaging). This has in turn produced significant social and economic benefit through impact on application areas such as medical imaging; astronomical imaging; satellite and airborne remote sensing for agriculture, earth sciences and defence; non-destructive testing; and microscopy for drug and nanotechnology development.
However, CI solutions can be very sensitive to the choice of the mathematical models used to analyse the raw sensor data (e.g., the data-fidelity term and the regularisation functions used to generate the image), and it is fundamental to carefully select and calibrate models for the specific imaging setup and type of scene considered. Presently, this requires extensive expert supervision, which is expensive and time-consuming.
The aim of this project is to develop a toolbox of new mathematical methods and computer algorithms to automatically select and calibrate CI models, directly from the observed sensor data, without using ground truth data, and with minimum expert supervision. This toolbox will be developed by combining advanced mathematical techniques stemming from Bayesian statistics, with specialised computer algorithms from the area of stochastic Monte Carlo simulation and optimisation. The expected outcome is that this toolbox will significantly simplify the development and deployment of CI technology, and amplify its adoption in science and industry as a result.
During the project, the proposed tools will be applied to two challenging CI problems related to satellite and astronomical imaging. More precisely, the methods developed in this project will be used to enhance the resolution and fine detail in hyperspectral satellite images and in radio-interferometric astronomical images. These applications will be investigated in collaboration with world-leading experts at the Mullard Space Science Laboratory, Heriot-Watt University, and University of Toulouse. These experts will provide data, training, and application-specific software. They will also help disseminate this work and amplify its impact.
To maximise the impact of the project on the economy and society, open-source code for all the proposed tools will be made publicly available on the project webpage, together with documentation and pedagogical demonstration kits.
However, CI solutions can be very sensitive to the choice of the mathematical models used to analyse the raw sensor data (e.g., the data-fidelity term and the regularisation functions used to generate the image), and it is fundamental to carefully select and calibrate models for the specific imaging setup and type of scene considered. Presently, this requires extensive expert supervision, which is expensive and time-consuming.
The aim of this project is to develop a toolbox of new mathematical methods and computer algorithms to automatically select and calibrate CI models, directly from the observed sensor data, without using ground truth data, and with minimum expert supervision. This toolbox will be developed by combining advanced mathematical techniques stemming from Bayesian statistics, with specialised computer algorithms from the area of stochastic Monte Carlo simulation and optimisation. The expected outcome is that this toolbox will significantly simplify the development and deployment of CI technology, and amplify its adoption in science and industry as a result.
During the project, the proposed tools will be applied to two challenging CI problems related to satellite and astronomical imaging. More precisely, the methods developed in this project will be used to enhance the resolution and fine detail in hyperspectral satellite images and in radio-interferometric astronomical images. These applications will be investigated in collaboration with world-leading experts at the Mullard Space Science Laboratory, Heriot-Watt University, and University of Toulouse. These experts will provide data, training, and application-specific software. They will also help disseminate this work and amplify its impact.
To maximise the impact of the project on the economy and society, open-source code for all the proposed tools will be made publicly available on the project webpage, together with documentation and pedagogical demonstration kits.
Planned Impact
Computer-intensive imaging (CI) technologies deliver images with great detail and dynamic range, that are more robust to challenging conditions (e.g., night, long-rage, and underwater imaging), and which can be acquired using short acquisition times. This is important, for example, because it allows accelerating MRI scanners to increase the number of patients that can be imaged per day and reduce the cost of the scans. In a similar fashion, by enhancing the quality of hyperspectral images, CI technology improves precision-agriculture decisions and predictions of income from tax on cash crops and export commodities. CI technology also plays a central role in modern radio-astronomy, which is a fundamental tool for studying celestial objects such as radio galaxies, quasars, pulsars, and masers, and improving our understanding of the universe.
This project will develop a toolbox of statistical methods and computer algorithms to automatically select and calibrate CI models. The expected outcome is that this will significantly reduce the cost related to developing and deploying new CI technology, and hence amplify its adoption in science, industry and society.
During the project, the proposed new tools will be applied to two important CI problems related to enhancing hyperspectral satellite images and radio-interferometric astronomical images. These applications will be studied in collaboration with world-leading experts who will provide data, training, software, and help disseminate the work and amplify its impact to the satellite imaging and the astronomical imaging communities.
We expect that the results of this project will lead to at least 5 journal papers, which we will submit for publication to top imaging journals (e.g., SIAM Journal on Imaging Sciences and IEEE Trans. Computational Imaging) and also make freely available on the arXiv (an open access repository of preprints). The work will also be advertised at international conferences and workshops, ensuring a worldwide impact for this project. Moreover, to ensure that these results are communicated efficiently to other UK researchers, the PI will continue to regularly speak at research seminars and at national meetings.
To maximise the impact of our work, the tools developed during the project will be made publicly available in the form of open-source software, with clear documentation and easy-to-follow demonstration kits, which we will advertise on the project webpage, specialised mailing lists, and in all project-related presentations and publications. Together with publications describing the technical aspects of the work, these source codes and demonstration kits will allow companies and research groups developing and deploying CI technology to easily benefit from our proposed new tools. Moreover, the availability of a C++ implementation will allow to easily incorporate our methods into the PURIFY open-source radio-astronomy software (http://basp-group.github.io/purify/).
Also, as part of this project, the PI will organise a 3-day workshop to bring together world-leading experts working on different aspects of CI and experts in statistics and other areas of mathematics, so that they can discuss the main challenges in CI and explore new solutions. The PI will also organise a special session at interface of CI and stochastic Bayesian computation methods at the 2021 IMA Conference on Inverse Problems (note that the PI will chair this conference, which take place in Edinburgh in Sep. 2021). This workshop, the special session, and the availability of open-source code and demonstration kits, will provide a clear opportunity for UK applied mathematicians and Bayesian computation experts to engage with the UK data science agenda, collaborate with the CI community, and deliver impact though applications to medical imaging, remote sensing, etc.
This project will develop a toolbox of statistical methods and computer algorithms to automatically select and calibrate CI models. The expected outcome is that this will significantly reduce the cost related to developing and deploying new CI technology, and hence amplify its adoption in science, industry and society.
During the project, the proposed new tools will be applied to two important CI problems related to enhancing hyperspectral satellite images and radio-interferometric astronomical images. These applications will be studied in collaboration with world-leading experts who will provide data, training, software, and help disseminate the work and amplify its impact to the satellite imaging and the astronomical imaging communities.
We expect that the results of this project will lead to at least 5 journal papers, which we will submit for publication to top imaging journals (e.g., SIAM Journal on Imaging Sciences and IEEE Trans. Computational Imaging) and also make freely available on the arXiv (an open access repository of preprints). The work will also be advertised at international conferences and workshops, ensuring a worldwide impact for this project. Moreover, to ensure that these results are communicated efficiently to other UK researchers, the PI will continue to regularly speak at research seminars and at national meetings.
To maximise the impact of our work, the tools developed during the project will be made publicly available in the form of open-source software, with clear documentation and easy-to-follow demonstration kits, which we will advertise on the project webpage, specialised mailing lists, and in all project-related presentations and publications. Together with publications describing the technical aspects of the work, these source codes and demonstration kits will allow companies and research groups developing and deploying CI technology to easily benefit from our proposed new tools. Moreover, the availability of a C++ implementation will allow to easily incorporate our methods into the PURIFY open-source radio-astronomy software (http://basp-group.github.io/purify/).
Also, as part of this project, the PI will organise a 3-day workshop to bring together world-leading experts working on different aspects of CI and experts in statistics and other areas of mathematics, so that they can discuss the main challenges in CI and explore new solutions. The PI will also organise a special session at interface of CI and stochastic Bayesian computation methods at the 2021 IMA Conference on Inverse Problems (note that the PI will chair this conference, which take place in Edinburgh in Sep. 2021). This workshop, the special session, and the availability of open-source code and demonstration kits, will provide a clear opportunity for UK applied mathematicians and Bayesian computation experts to engage with the UK data science agenda, collaborate with the CI community, and deliver impact though applications to medical imaging, remote sensing, etc.
People |
ORCID iD |
Marcelo Pereyra (Principal Investigator) |
Publications
Vidal A
(2020)
Maximum Likelihood Estimation of Regularization Parameters in High-Dimensional Inverse Problems: An Empirical Bayesian Approach Part I: Methodology and Experiments
in SIAM Journal on Imaging Sciences
De Bortoli V
(2020)
Maximum Likelihood Estimation of Regularization Parameters in High-Dimensional Inverse Problems: An Empirical Bayesian Approach. Part II: Theoretical Analysis
in SIAM Journal on Imaging Sciences
Laumont R
(2023)
On Maximum a Posteriori Estimation with Plug & Play Priors and Stochastic Gradient Descent
in Journal of Mathematical Imaging and Vision
Cai X
(2022)
Proximal nested sampling for high-dimensional Bayesian model selection
in Statistics and Computing
Description | We have developed a new way of automatically adjusting the parameters of the mathematical equations underpinning a broad class of image enhancement algorithms, which are widely used in modern imaging technology (e.g., satellite cameras, medical imaging devices, telescopes, microscopes, etc.). We have studied the application of this new technique to a specific type of parameters that are difficult to adjust manually (the so-called regularisation parameters) and demonstrated that the new technique significantly outperforms existing techniques, both in terms of empirical performances as well as in terms of mathematical guarantees on the delivered solutions. This set the groundwork for tackling other challenging parameters, namely the parameters describing the physical aspects of the imaging instrument. In particular, we have recently modified the proposed technique to tackle blind and semi-blind image deblurring problems. Preliminary experiments suggest that the technique can accurately identify the blur degradation operator in mild and moderate natural-image deblurring problems. |
Exploitation Route | Most modern imaging algorithms are based on some mathematical equations that analyse the raw data to produce high-resolution images with sharp detail. These equations typically have some parameters that users need to set manually, this is time-consuming and requires expertise. The technique that we have developed allows automating this process. We expect that the technique will now be customised for different specific problems by applied imaging experts, and eventually incorporated into software and imaging products. |
Sectors | Aerospace Defence and Marine Agriculture Food and Drink Construction Environment Government Democracy and Justice Manufacturing including Industrial Biotechology |
Description | The findings have led to the following pathway to impact: The French Space Agency (CNES), which is a key player in the European Space Agency (ESA), has expressed a strong interest in the Bayesian mathematical and computational methods that we are developing in this project. They are interested in using them for adjusting the parameters of the mathematical models that they use as part of their satellite imaging pipeline. In order to explore this, we had a series of regular meetings between October 2020 and February 2021, where I provided training for them and discussed specific aspects of the application they intend to develop. Following on from this, they decided to conduct a range of experiments to study the performance of Bayesian methods in the context of non-blind satellite imaging. These experiments revealed that Bayesian methods can significantly outperform their current methods, particularly in cases of high blur and poor signal-to-noise ratio. A more exhaustive evaluation with specialised Bayesian methods is currently in progress. Moreover, from June 2021 to November 2021, we had a second series of meetings where I provided training for them on blind and semi-blind Bayesian imaging methods and discussed their application to satellite image restoration problems. Preliminary experiments with artificially degraded (high-resolution) satellite image data suggest that the Bayesian methods developed in this project could be highly relevant in this setting and reduce the need for cumbersome recalibration manoeuvres. We are currently developing a Python library that will allow them to conduct a more exhaustive evaluation of the methods directly within the CNES image processing platform. This work started two months ago and could potentially lead to an impact case in the future. |
First Year Of Impact | 2021 |
Sector | Aerospace, Defence and Marine |
Impact Types | Policy & public services |
Title | PnP-ULA |
Description | The Plug-and-Play unadjusted Langevin algorithm is a Markov chain Monte Carlo sampler designed for performing Bayesian inference with priors that are represented by an image denoising operator, which is typically encoded by a neural network that has been trained with application-specific data. |
Type Of Material | Computer model/algorithm |
Year Produced | 2022 |
Provided To Others? | Yes |
Impact | PnP-ULA has been successfully applied to problems related to image recovery and uncertainty quantification in a range of imaging inverse problems involving synthetic data. Its application to real problems in computer tomography is currently under investigation. |
URL | https://doi.org/10.1137/21M1406349 |
Title | Proximal Nested Sampling |
Description | Proximal Nested Sampling is a stochastic algorithm designed for computing the marginal likelihood of high-dimensional Bayesian models with an underlying convex geometry. It is based on the Moreau-Yoshida regularised unadjusted Langevin algorithm and it is useful for performing Bayesian model selection directly from observed data, without the need for ground truth data. |
Type Of Material | Computer model/algorithm |
Year Produced | 2022 |
Provided To Others? | Yes |
Impact | Proximal Nested Sampling has been successfully applied to Bayesian model selection in imaging inverse problems involving synthetic data. Its application to real problems in interferometric imaging is currently under investigation. |
URL | https://link.springer.com/article/10.1007/s11222-022-10152-9 |
Title | SOUL |
Description | SOUL is a stochastic approximation proximal gradient algorithm designed for computing maximum marginal likelihood estimators in high-dimensional inverse problems with an underlying convex geometry. It is based on unadjusted Langevin Markov chain Monte Carlo samplers, such as the Moreau-Yoshida regularised unadjusted Langevin algorithm. |
Type Of Material | Computer model/algorithm |
Year Produced | 2020 |
Provided To Others? | Yes |
Impact | SOUL has been successfully applied to the estimation of regularisation parameters in a range of imaging inverse problems involving synthetic data. Its application to real problems in satellite imaging is currently under investigation. |
Title | SR-SOUL |
Description | The super-resolution stochastic optimisation unadjusted Langevin algorithm is a Markov chain Monte Carlo sampler designed for performing Bayesian inference with highly realistic image priors that are represented by a generative model trained for image super-resolution. This model is typically encoded by a deep neural network that has been trained with application-specific data, for example a normalising flow. |
Type Of Material | Computer model/algorithm |
Year Produced | 2023 |
Provided To Others? | Yes |
Impact | No impact yet. |
URL | https://researchportal.hw.ac.uk/en/publications/empirical-bayesian-imaging-with-large-scale-push-for... |
Description | Alain Durmus and Valentin De Bortoli |
Organisation | École Normale Supérieure de Cachan |
Country | France |
Sector | Academic/University |
PI Contribution | We have led the development of the new technique to adjust the parameters of the mathematical equations underpinning the image enhancement algorithms, as well as the specific applications to imaging that we have studied so far in this project. |
Collaborator Contribution | Alain Durmus and Valentin De Bortoli have led the theoretical analysis of the new technique, particularly the convergence guarantees under easily verifiable conditions. |
Impact | This is a multi-disciplinary collaboration at the interface of computational imaging engineering, probability theory, computational statistics, and applied analysis. |
Start Year | 2017 |
Description | DTU CT imaging for subsea pipelines |
Organisation | Technical University of Denmark |
Country | Denmark |
Sector | Academic/University |
PI Contribution | Co-created a Bayesian model and Bayesian computation algorithm to analyse CT images of subsea pipelines to detect defects |
Collaborator Contribution | Co-created a Bayesian model and Bayesian computation algorithm to analyse CT images of subsea pipelines to detect defects, implemented algorithms in Python by using the cuqipi framework, and demonstrated the feasibility of the approach by applying it to real data provided by FORCE Technology. DTU will transfer proposed approach to FORCE Technology, who is developing an underwater drone to monitor subsea pipelines. |
Impact | We have co-authored the paper: A Bayesian approach for CT reconstruction with defect detection for subsea pipelines Silja L Christensen1, Nicolai A B Riis1,2, Marcelo Pereyra3,4 and Jakob S Jørgensen5,1 Published 22 December 2023 • © 2023 The Author(s). Inverse Problems, Volume 40, Number 2 DOI 10.1088/1361-6420/ad1348 |
Start Year | 2022 |
Description | Julie Delon and Andres Almansa |
Organisation | University of Paris - Descartes |
Country | France |
Sector | Academic/University |
PI Contribution | We have led the development of the new Bayesian AI techniques for imaging problems where the prior knowledge is available in the form of a set of training examples. |
Collaborator Contribution | Julie Delon and Andres Almansa have led the machine learning aspects of the work and the algorithmic implementation of the proposed techniques. |
Impact | This is a multi-disciplinary collaboration at the interface of imaging sciences, applied analysis, machine learning, probability theory and mathematical statistics. |
Start Year | 2019 |
Title | Code to reproduce results of article "A Bayesian approach for CT reconstruction with defect detection for subsea pipelines" |
Description | This is an permanently DOI-linked archived version of the code accompanying the article "A Bayesian approach for CT reconstruction with defect detection for subsea pipelines" by Silja L. Christensen,Nicolai A. B. Riis,Marcelo Pereyra andJakob Sauer Jørgensen The original code along with any updates is available in the GitHub repository: https://github.com/CUQI-DTU/Paper-PipeDefectSplitting |
Type Of Technology | Software |
Year Produced | 2023 |
Impact | No impact yet. |
URL | https://zenodo.org/doi/10.5281/zenodo.10201988 |
Title | MCMC for Low Photon Imaging |
Description | A PyTorch code for a Markov chain Monte Carlo methodology to perform Bayesian inference in low-photon imaging problems, with particular attention given to situations involving observation noise processes that deviate significantly from Gaussian noise, such as binomial, geometric, and low-intensity Poisson noise. The methodology is based on a reflected and regularized Langevin SDE. The code includes demos related to image deblurring, denoising, and inpainting under binomial, geometric, and Poisson noise. |
Type Of Technology | Software |
Year Produced | 2023 |
Open Source License? | Yes |
Impact | No impact, yet. |
URL | https://epubs.siam.org/doi/10.1137/22M1502240 |
Description | Invited tutorial at the IEEE International Conference on Image Processing, Bordeaux, Oct. 2022 |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | The IEEE International Conference on Image Processing is the main global conference on image processing engineering. It is attended by over 1,000 people and has a strong presence from industry. I delivered a 3-hour tutorial masterclass at the conference on the topic of this project. I was subsequently contacted by several engineering groups who are interested in adopting the tools and techniques that I presented at the conference. Two of these groups are currently working on incorporating these tools and techniques into open-source free software libraries that they develop and make available to the image-processing engineering community. |
Year(s) Of Engagement Activity | 2022 |
URL | https://2022.ieeeicip.org/ |
Description | Stakeholder 3-day workshop |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | The 3-day workshop Interfacing Bayesian Statistics, Machine Learning, Applied Analysis, and Blind and Semi-Blind Imaging Inverse Problems" focused on the Bayesian statistics, machine learning, and applied analysis frameworks for imaging inverse problems that are blind or semi-blind. The event brought together industrialists, students, world-leading experts and rising early career researchers to discuss recent developments in the fields as well as open challenges, with a focus on co-creation and on fostering synergies to pursue ideas that develop at the fertile interface where the three frameworks meet. |
Year(s) Of Engagement Activity | 2023 |
URL | https://www.icms.org.uk/workshops/2023/interfacing-bayesian-statistics-machine-learning-applied-anal... |