Bayesian computation for low-photon imaging
Lead Research Organisation:
Heriot-Watt University
Department Name: S of Mathematical and Computer Sciences
Abstract
Images are rich in data of significant economic and social value, and over the past decade, they have become fundamental sources of information in many disciplines (e.g., medicine, biology, agriculture, defence, earth sciences, and non-destructive testing). These disciplines now drive the development of sophisticated and specialised imaging devices. Such devices tightly combine two forms of innovation to deliver state-of-the-art performance: 1) sophisticated instrumentation and sensors that push technology and physics to the limits, and 2) highly advanced computational imaging (CI) methods that carefully analyse the generated raw data to produce sharp images with fine detail.
This proposal focuses on CI methodology for quantum-enhanced imaging, a new imaging paradigm that seeks to exploit the quantum nature of light to go far beyond what is possible in classical optics in terms of spatial and temporal resolution and dynamic range. This transformative approach is poised to dramatically advance imaging technologies and generate great social and economic impact. To make sure that the UK is at the forefront of this strategic technological developments, the UK government created the Quantum Enhanced Imaging Hub (QUANTIC) in 2014 as part of the UK National Quantum Technology Programme, which was renewed this year. QUANTIC has developed impressive new sensors for extreme imaging conditions. However, these advances in sensor technology have not been matched by progress in CI methodology, gravely jeopardizing the impact of these promising technologies.
The aim of this proposal is to develop CI methodology specifically designed for solving quantum-enhanced imaging problems in which very few photons are observed (i.e., low-photon and single-photon imaging problems). Our methods will be formulated in the Bayesian statistical framework, which is particularly appropriate for solving these challenging imaging problems because: 1) it enables the use of sophisticated statistical models to accurately describe the underlying physics, 2) it allows the automatic calibration of models, and 3) it provides tools to quantify the uncertainty in the solutions delivered.
At present, the benefits and superior performance of Bayesian statistical CI methods is obtained at the expense of a prohibitively high computational cost. We plan to significantly accelerate Bayesian solutions for quantum-enhanced imaging problems by developing specialised computation methods that combine and extend ideas from different areas of applied mathematics, computational statistics, and artificial intelligence.
We believe that the availability of fast Bayesian computation methods will unlock the potential of these promising quantum-enhanced imaging technologies and lead to their wide adoption in science and engineering, generating generate great social and economic benefit through an impact on medicine, biology, agriculture, defence, earth sciences, and non-destructive testing.
In order to guarantee this impact, during the project, we will apply the proposed methods to three important quantum-enhanced imaging problems (low-photon multispectral single-pixel imaging, high-resolution PGET, and single-photon 3D LIDAR with array sensors). These applications will be investigated in collaboration with world-leading experts who will provide data and training, and help disseminate the research outputs. To maximise the impact of our work, we will also develop open-source software - with documentation and demonstrations - that we will share online and use in outreach activities aimed at informing the public about STEM research and inspiring young people to pursue STEM careers. This project will also help train the next generation of top-tier talent in AI and quantum technology.
This proposal focuses on CI methodology for quantum-enhanced imaging, a new imaging paradigm that seeks to exploit the quantum nature of light to go far beyond what is possible in classical optics in terms of spatial and temporal resolution and dynamic range. This transformative approach is poised to dramatically advance imaging technologies and generate great social and economic impact. To make sure that the UK is at the forefront of this strategic technological developments, the UK government created the Quantum Enhanced Imaging Hub (QUANTIC) in 2014 as part of the UK National Quantum Technology Programme, which was renewed this year. QUANTIC has developed impressive new sensors for extreme imaging conditions. However, these advances in sensor technology have not been matched by progress in CI methodology, gravely jeopardizing the impact of these promising technologies.
The aim of this proposal is to develop CI methodology specifically designed for solving quantum-enhanced imaging problems in which very few photons are observed (i.e., low-photon and single-photon imaging problems). Our methods will be formulated in the Bayesian statistical framework, which is particularly appropriate for solving these challenging imaging problems because: 1) it enables the use of sophisticated statistical models to accurately describe the underlying physics, 2) it allows the automatic calibration of models, and 3) it provides tools to quantify the uncertainty in the solutions delivered.
At present, the benefits and superior performance of Bayesian statistical CI methods is obtained at the expense of a prohibitively high computational cost. We plan to significantly accelerate Bayesian solutions for quantum-enhanced imaging problems by developing specialised computation methods that combine and extend ideas from different areas of applied mathematics, computational statistics, and artificial intelligence.
We believe that the availability of fast Bayesian computation methods will unlock the potential of these promising quantum-enhanced imaging technologies and lead to their wide adoption in science and engineering, generating generate great social and economic benefit through an impact on medicine, biology, agriculture, defence, earth sciences, and non-destructive testing.
In order to guarantee this impact, during the project, we will apply the proposed methods to three important quantum-enhanced imaging problems (low-photon multispectral single-pixel imaging, high-resolution PGET, and single-photon 3D LIDAR with array sensors). These applications will be investigated in collaboration with world-leading experts who will provide data and training, and help disseminate the research outputs. To maximise the impact of our work, we will also develop open-source software - with documentation and demonstrations - that we will share online and use in outreach activities aimed at informing the public about STEM research and inspiring young people to pursue STEM careers. This project will also help train the next generation of top-tier talent in AI and quantum technology.
Planned Impact
Images are fundamental sources of information in many disciplines, such as medicine, biology, agriculture, defence, earth sciences, and non-destructive testing. These disciplines now drive the development of sophisticated and specialised imaging devices. Such devices tightly combine two forms of innovation to deliver state-of-the-art performance: 1) sophisticated instrumentation and sensors that push technology and physics to the limits; and 2) advanced computational imaging (CI) methods that carefully analyse the generated raw data to produce sharp images with fine detail.
This proposal focuses on CI methodology for quantum-enhanced imaging problem. Quantum-enhanced imaging is a new imaging paradigm that seeks to exploit the quantum nature of light to go far beyond what is possible in classical optics in terms of spatial and temporal resolution and dynamic range. This transformative approach is poised to dramatically advance imaging technologies and generate great social and economic impact. Maximising the impact of these technologies is an important aspect of the UK National Quantum Technology Programme.
Important recent investments have led to the development of impressive new instrumentation and sensors for quantum-enhanced imaging in extreme imaging conditions. However, these advances in sensor technology have not been matched by progress in CI methodology, gravely jeopardizing the impact of these promising technologies. In this proposal, we aim to develop the new generation of fast and powerful CI methods specifically designed for quantum-enhanced imaging problems involving photon-starved measurements. These CI methods will unlock the potential of these promising quantum-enhanced imaging technologies and lead to their wide adoption in medicine, biology, biology, agriculture, defence, earth sciences, and non-destructive testing.
In order to guarantee this impact, during the project, the proposed methods will be applied to three important quantum-enhanced imaging problems: low-photon multispectral single-pixel imaging, high-resolution passive gamma emission tomography, and single-photon 3D LIDAR with array sensors. These applications will be investigated in collaboration with world-leading experts who will provide data and training, and help disseminate the research outputs. To further amplify the impact of our work, we will also develop software tools for the considered applications, which we will design jointly with our collaborators to ensure that they meet up the expectations of the end-users. We will also make publicly available open-source implementations of our methods with documentation and demonstrations to help other mathematicians and imaging scientists engage with the UK National Quantum Technology agenda.
The team of researchers involved in this project will participate in several public engagement events aimed at inspiring and informing the public about STEM research. Examples include participating in science festivals to talk about AI&Data and visiting schools in Edinburgh to meet students in Year 9 to encourage them to pursue STEM subjects. The talks and visits will involve a mix of demonstrations, interactive activities, and competitions, all related to STEM through the prism of imaging sciences.
This proposal focuses on CI methodology for quantum-enhanced imaging problem. Quantum-enhanced imaging is a new imaging paradigm that seeks to exploit the quantum nature of light to go far beyond what is possible in classical optics in terms of spatial and temporal resolution and dynamic range. This transformative approach is poised to dramatically advance imaging technologies and generate great social and economic impact. Maximising the impact of these technologies is an important aspect of the UK National Quantum Technology Programme.
Important recent investments have led to the development of impressive new instrumentation and sensors for quantum-enhanced imaging in extreme imaging conditions. However, these advances in sensor technology have not been matched by progress in CI methodology, gravely jeopardizing the impact of these promising technologies. In this proposal, we aim to develop the new generation of fast and powerful CI methods specifically designed for quantum-enhanced imaging problems involving photon-starved measurements. These CI methods will unlock the potential of these promising quantum-enhanced imaging technologies and lead to their wide adoption in medicine, biology, biology, agriculture, defence, earth sciences, and non-destructive testing.
In order to guarantee this impact, during the project, the proposed methods will be applied to three important quantum-enhanced imaging problems: low-photon multispectral single-pixel imaging, high-resolution passive gamma emission tomography, and single-photon 3D LIDAR with array sensors. These applications will be investigated in collaboration with world-leading experts who will provide data and training, and help disseminate the research outputs. To further amplify the impact of our work, we will also develop software tools for the considered applications, which we will design jointly with our collaborators to ensure that they meet up the expectations of the end-users. We will also make publicly available open-source implementations of our methods with documentation and demonstrations to help other mathematicians and imaging scientists engage with the UK National Quantum Technology agenda.
The team of researchers involved in this project will participate in several public engagement events aimed at inspiring and informing the public about STEM research. Examples include participating in science festivals to talk about AI&Data and visiting schools in Edinburgh to meet students in Year 9 to encourage them to pursue STEM subjects. The talks and visits will involve a mix of demonstrations, interactive activities, and competitions, all related to STEM through the prism of imaging sciences.
Organisations
- Heriot-Watt University (Lead Research Organisation)
- Charles III University of Madrid (Collaboration)
- Ćcole normale supĆ©rieure de Lyon (ENS Lyon) (Collaboration)
- Alternative Energies and Atomic Energy Commission (CEA) (Collaboration)
- University of Paris - Descartes (Collaboration)
- University of Illinois (Collaboration)
- Uni of Illinois at Urbana Champaign (Project Partner)
- ENS Paris-Saclay (Normal Superior Sch) (Project Partner)
Publications
Klatzer T
(2024)
Accelerated Bayesian Imaging by Relaxed Proximal-Point Langevin Sampling
in SIAM Journal on Imaging Sciences
Melidonis S
(2023)
Efficient Bayesian Computation for Low-Photon Imaging Problems
in SIAM Journal on Imaging Sciences
Melidonis S
(2022)
Efficient Bayesian computation for low-photon imaging problems
Melidonis S
(2022)
Efficient Bayesian computation for low-photon imaging problems
Melidonis S
(2024)
Empirical Bayesian Imaging With Large-Scale Push-Forward Generative Priors
in IEEE Signal Processing Letters
Melidonis S
(2025)
Score-Based Denoising Diffusion Models for Photon-Starved Image Restoration Problems
in Transactions on Machine Learning Research
Li Z
(2022)
Sparse Linear Spectral Unmixing of Hyperspectral Images Using Expectation-Propagation
in IEEE Transactions on Geoscience and Remote Sensing
| Description | Modern imaging and computer vision systems are increasingly required to operate in extreme conditions (\eg, ultra-fast acquisition times, low illumination, long-range, unconventional environments). This has led to the development of new quantum-enhanced sensors and cameras that exploit the particle nature of light to exceed the limitations of classical imaging strategies. Unfortunately, the raw images produced by these new cameras are of very poor quality. In this project, we have developed new mathematics and algorithms to significantly enhance the quality of these images. Following some successful preliminary experiments, we are currently investigating the application of this new mathematics and algorithms to real data. We are also making progress in developing PyTorch implementations of the proposed algorithms and integrating them within the award-winning open-source Deepinverse Pythorch library for computational imaging. |
| Exploitation Route | We anticipate that future research will specialise the proposed algorithms for specific applications of quantum-enhanced imaging, and subsequently integrate them within new imaging systems and software. |
| Sectors | Aerospace Defence and Marine Agriculture Food and Drink Chemicals Energy Environment Healthcare |
| Description | Learned Quantitative Stochastic Imaging |
| Amount | £1,506,530 (GBP) |
| Funding ID | EP/Z534481/1 |
| Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
| Sector | Public |
| Country | United Kingdom |
| Start | 12/2024 |
| End | 12/2029 |
| Title | IMLA |
| Description | The implicit Langevin algorithm (IMLA) is a Markov chain Monte Carlo sampler designed for performing Bayesian inference with machine learning image priors that are log-concave. The algorithm outperforms previous algorithms for this task in terms of computational complexity and affords users with detailed theoretical accuracy guarantees. In addition, IMLA automatically takes into account constraints in the solution space, which are a key feature of low-photon imaging problems. Moreover, IMLA self-calibrates its internal algorithm parameters, and is thus significantly easier to deploy than its competitors. |
| Type Of Material | Computer model/algorithm |
| Year Produced | 2024 |
| Provided To Others? | Yes |
| Impact | We have been actively disseminating IMLA to potential end-users. A biological imaging group in Genoa (Italy), is currently considering applying IMLA to image super-resolution tasks in fluorescent microscopy of single molecule images. |
| URL | https://epubs.siam.org/doi/full/10.1137/23M1594832 |
| Title | SR-SOUL |
| Description | The super-resolution stochastic optimisation unadjusted Langevin algorithm is a Markov chain Monte Carlo sampler designed for performing Bayesian inference with highly realistic image priors that are represented by a generative model trained for image super-resolution. This model is typically encoded by a deep neural network that has been trained with application-specific data, for example a normalising flow. |
| Type Of Material | Computer model/algorithm |
| Year Produced | 2023 |
| Provided To Others? | Yes |
| Impact | No impact yet. |
| URL | https://researchportal.hw.ac.uk/en/publications/empirical-bayesian-imaging-with-large-scale-push-for... |
| Description | Dr. Angela Di Fulvio |
| Organisation | University of Illinois |
| Country | United States |
| Sector | Academic/University |
| PI Contribution | We have led the development of the new Bayesian AI techniques for a new Compton-based imager developed at the University of Illinois. |
| Collaborator Contribution | Angela Di Fulvio and her group have designed the imager and shared simulated and real data to support the development of the Bayesian AI technique, as well as provided feedback on the results delivered by the technique. |
| Impact | This is a multi-disciplinary collaboration at the interface of imaging sciences, nuclear sciences and mathematical statistics. Research paper currently under consideration for publication. |
| Start Year | 2019 |
| Description | Julian Tachella (ENS Lyon) |
| Organisation | Ćcole normale supĆ©rieure de Lyon (ENS Lyon) |
| Country | France |
| Sector | Academic/University |
| PI Contribution | We have co-created a new unsupervised statistical machine learning method that is able to quantify the uncertainty in the solution of an imaging problem directly from measurements, by leveraging symmetries and invariances in the problem, without the need for ground truth data. |
| Collaborator Contribution | Co-creation of the method and implementation of algorithms and experiments |
| Impact | https://arxiv.org/pdf/2310.11838.pdf (accepted for publication in AISTATS 2024). |
| Start Year | 2022 |
| Description | Julie Delon and Andres Almansa |
| Organisation | University of Paris - Descartes |
| Country | France |
| Sector | Academic/University |
| PI Contribution | We have led the development of the new Bayesian AI techniques for imaging problems where the prior knowledge is available in the form of a set of training examples. |
| Collaborator Contribution | Julie Delon and Andres Almansa have led the machine learning aspects of the work and the algorithmic implementation of the proposed techniques. |
| Impact | This is a multi-disciplinary collaboration at the interface of imaging sciences, applied analysis, machine learning, probability theory and mathematical statistics. |
| Start Year | 2019 |
| Description | Prof. Jesus Maria Sanz Serna |
| Organisation | Charles III University of Madrid |
| Country | Spain |
| Sector | Academic/University |
| PI Contribution | We have been analysing the convergence properties of unconstrained and constrained convex optimisation algorithms for low-photon imaging problems. |
| Collaborator Contribution | Prof. Jesus Maria Sanz Serna has contributed technical expertise related to the analysis of the convergence rate. |
| Impact | Research manuscript currently in consideration for publication. |
| Start Year | 2019 |
| Description | Tobias Liaudiat (CEA Paris-Saclay) |
| Organisation | Alternative Energies and Atomic Energy Commission (CEA) |
| Country | France |
| Sector | Public |
| PI Contribution | We are developing a Bayesian computation method to detect when state-of-the-art generative image models have become unreliable for scientific imaging because of distribution shift, which we plan to apply to low-photon imaging problems. |
| Collaborator Contribution | Dr Liaudiat is leading the implementation and experimental aspects of the project. |
| Impact | No outputs yet. |
| Start Year | 2023 |
| Title | Equivariant bootstrapping for UQ in Imaging |
| Description | The equivariant bootstrap algorithm is a scalable and intuitive resampling procedure which produces well-calibrated uncertainty regions for linear imaging problem and any reconstruction network (i.e., trained in a supervised or self-supervised manner). The method outperforms other state-of-the-art strategies in terms of accuracy and computational speed. |
| Type Of Technology | Software |
| Year Produced | 2024 |
| Open Source License? | Yes |
| Impact | No impact yet |
| URL | https://arxiv.org/pdf/2310.11838.pdf |
| Title | MCMC for Low Photon Imaging |
| Description | A PyTorch code for a Markov chain Monte Carlo methodology to perform Bayesian inference in low-photon imaging problems, with particular attention given to situations involving observation noise processes that deviate significantly from Gaussian noise, such as binomial, geometric, and low-intensity Poisson noise. The methodology is based on a reflected and regularized Langevin SDE. The code includes demos related to image deblurring, denoising, and inpainting under binomial, geometric, and Poisson noise. |
| Type Of Technology | Software |
| Year Produced | 2023 |
| Open Source License? | Yes |
| Impact | No impact, yet. |
| URL | https://epubs.siam.org/doi/10.1137/22M1502240 |
| Description | Public Lecture (ICMS) |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Public/other audiences |
| Results and Impact | I delivered a public lecture on bias in AI-based computational imaging technology, focusing on the paradox that while bias can have serious negative consequences (illustrated through examples related to discrimination of minority groups), bias it is also an essential ingredient to construct reliable computational imaging methods, as otherwise it would be impossible to separate signal from noise and deliver accurate images. |
| Year(s) Of Engagement Activity | 2024 |
| Description | Stakeholder 3-day workshop |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | The 3-day workshop Interfacing Bayesian Statistics, Machine Learning, Applied Analysis, and Blind and Semi-Blind Imaging Inverse Problems" focused on the Bayesian statistics, machine learning, and applied analysis frameworks for imaging inverse problems that are blind or semi-blind. The event brought together industrialists, students, world-leading experts and rising early career researchers to discuss recent developments in the fields as well as open challenges, with a focus on co-creation and on fostering synergies to pursue ideas that develop at the fertile interface where the three frameworks meet. |
| Year(s) Of Engagement Activity | 2023 |
| URL | https://www.icms.org.uk/workshops/2023/interfacing-bayesian-statistics-machine-learning-applied-anal... |
| Description | Training event on modern Bayesian imaging techniques for postgraduate research students and industrial researchers |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Postgraduate students |
| Results and Impact | Three-day training event, mainly aimed at postgraduate students and industry researchers, on state-of-the-art Bayesian imaging techniques with a focus on uncertainty quantification. Co-organised with Danish Technical University, at ICMS Edinburgh. |
| Year(s) Of Engagement Activity | 2024 |
