# Stochastic iterative regularization: theory, algorithms and applications

Lead Research Organisation:
University College London

Department Name: Computer Science

### Abstract

An inverse problem arises whenever one seeks the cause of observed physical phenomena or observational data, e.g., inferring the governing law from the measurements. This task essentially underlies all scientific discoveries and technological innovations. Thus, the mathematical theory and computational techniques for solving inverse problems are central, e.g., in physics, astronomy, medicine, engineering, and life sciences, and it has evolved into a highly interdisciplinary research area.

Inverse problems are usually ill-posed in the sense that the sought-for solution lacks existence, uniqueness or stability with respect to data perturbation. Since the noise is inherent in the observational data, the numerical algorithms have to employ specialized techniques, commonly known as regularization. The corresponding mathematical framework in the form of regularization theory is highly developed, since the pioneering works of A. Tikhonov in 1960s, H. Engl et al from 1980s and many other researchers. This theory has played a vital role in many research areas, and related numerical algorithms have also been intensively investigated. One versatile framework is to minimize an objective function measuring the quality of fitting between the model output and observational data, possibly plus some additional penalty term, and it covers a large class of powerful iterative inversion techniques.

Due to the unprecedented advances in data acquisition technologies, large datasets are becoming common place for many practical inverse problems. Prominent examples in medical imaging include dynamic, multispectral, multi-energy or multi-frequency data in computed tomography and optical tomography. The ever increasing volume of available data poses enormous computational challenges to image reconstruction, and traditional iterative methods can be too expensive to apply, and currently it represents one of the bottlenecks to extract useful information from the massive dataset. This is especially challenging for problems involving complex physical models, where each data set is very expensive to simulate.

The proposed research aims at addressing the aforementioned outstanding computational challenge using stochastic iterative techniques developed within the machine learning community, and providing relevant theoretical underpinnings. The central idea of stochastic iterative methods is that at each step only a (small) portion of the data set is used to steer the progression of the iterates, instead of the full data set. This allows drastically reducing the computational cost per iteration. This idea has received enormous attention within the machine learning community, and especially has achieved stunning success in deep learning in recent years. Actually stochastic gradient descent and its variants are the workhorse behind many deep learning tasks. A successful completion of this project will greatly advance modern image reconstruction by providing a systematic mathematical and computational framework, including comprehensive theoretical underpinnings, novel algorithms and detailed studies on concrete inverse problems, e.g., in medical imaging.

Inverse problems are usually ill-posed in the sense that the sought-for solution lacks existence, uniqueness or stability with respect to data perturbation. Since the noise is inherent in the observational data, the numerical algorithms have to employ specialized techniques, commonly known as regularization. The corresponding mathematical framework in the form of regularization theory is highly developed, since the pioneering works of A. Tikhonov in 1960s, H. Engl et al from 1980s and many other researchers. This theory has played a vital role in many research areas, and related numerical algorithms have also been intensively investigated. One versatile framework is to minimize an objective function measuring the quality of fitting between the model output and observational data, possibly plus some additional penalty term, and it covers a large class of powerful iterative inversion techniques.

Due to the unprecedented advances in data acquisition technologies, large datasets are becoming common place for many practical inverse problems. Prominent examples in medical imaging include dynamic, multispectral, multi-energy or multi-frequency data in computed tomography and optical tomography. The ever increasing volume of available data poses enormous computational challenges to image reconstruction, and traditional iterative methods can be too expensive to apply, and currently it represents one of the bottlenecks to extract useful information from the massive dataset. This is especially challenging for problems involving complex physical models, where each data set is very expensive to simulate.

The proposed research aims at addressing the aforementioned outstanding computational challenge using stochastic iterative techniques developed within the machine learning community, and providing relevant theoretical underpinnings. The central idea of stochastic iterative methods is that at each step only a (small) portion of the data set is used to steer the progression of the iterates, instead of the full data set. This allows drastically reducing the computational cost per iteration. This idea has received enormous attention within the machine learning community, and especially has achieved stunning success in deep learning in recent years. Actually stochastic gradient descent and its variants are the workhorse behind many deep learning tasks. A successful completion of this project will greatly advance modern image reconstruction by providing a systematic mathematical and computational framework, including comprehensive theoretical underpinnings, novel algorithms and detailed studies on concrete inverse problems, e.g., in medical imaging.

### Planned Impact

The main beneficiaries of the outcomes of this project in terms of theoretical developments and computational techniques would be research institutes or public sectors where the inverse theory and algorithms play a crucial role, e.g., various research centres on inverse problems, signal / image processing and data sciences, e.g., Centre for Inverse Problems at UCL (joint between computer science, mathematics and statistics), Centre for Image Analysis at Cambridge University, AI Centre at UCL Computer Science, and Alan Turing Institute (UCL is one of the founding partner institutes) in the UK, and Radon Institute of Applied and Computational Mathematics in Austria and Centre of Excellence for Inverse Modelling and Imaging in Finland. To disseminate the research outputs to the community, we will give presentations regularly at leading conferences, e.g., Applied Inverse Problems conference (2019, 2021), SIAM Conference on Imaging Sciences (2020, 2022), Chemnitz Symposium in Inverse Problems in Germany and Inverse Days in Finland, and the conference trips will be combined with research visits and seminar talks whenever appropriate to maximize the potential impact.

The main beneficiaries of the algorithmic outcomes and concrete inverse problems in medical imaging within the project would be research institutes or public sections where reconstruction technologies are heavily used, e.g., Centre for Medical Image Computing at UCL (jointly between computer science and medical physics at UCL), AI Centre at UCL and Alan Turing Institute in the UK. Through the connections between Department of Computer Science and these institutes, there will be many opportunities to collaborate directly, e.g., experimental evaluation of the developed technologies. These connections ensure a direct conduit of the transfer of the obtained technologies to medical imaging and machine learning. This transfer will be greatly facilitated through the software package to be developed, as well as the proposed research on novel applications in optical tomography etc.

The wider public will benefit in many ways from a successful completion of this project, e.g., in terms of a significant improvement in medical imaging quality and speed (hence more accurate and faster diagnosis of diseases) achieved by the new algorithms. Thus it will have significant repercussions on all economic and societal sectors involved. The outputs can potentially attract companies, e.g., on imaging device manufacturers (e.g., Phillips, GE and Siemens), and on data science and image processing companies (e.g., Microsoft Research and DeepMind). To achieve this transfer, we will regularly attend industry-sponsored events to disseminate relevant research results, e.g., the activities at Alan Turing Institute.

The main beneficiaries of the algorithmic outcomes and concrete inverse problems in medical imaging within the project would be research institutes or public sections where reconstruction technologies are heavily used, e.g., Centre for Medical Image Computing at UCL (jointly between computer science and medical physics at UCL), AI Centre at UCL and Alan Turing Institute in the UK. Through the connections between Department of Computer Science and these institutes, there will be many opportunities to collaborate directly, e.g., experimental evaluation of the developed technologies. These connections ensure a direct conduit of the transfer of the obtained technologies to medical imaging and machine learning. This transfer will be greatly facilitated through the software package to be developed, as well as the proposed research on novel applications in optical tomography etc.

The wider public will benefit in many ways from a successful completion of this project, e.g., in terms of a significant improvement in medical imaging quality and speed (hence more accurate and faster diagnosis of diseases) achieved by the new algorithms. Thus it will have significant repercussions on all economic and societal sectors involved. The outputs can potentially attract companies, e.g., on imaging device manufacturers (e.g., Phillips, GE and Siemens), and on data science and image processing companies (e.g., Microsoft Research and DeepMind). To achieve this transfer, we will regularly attend industry-sponsored events to disseminate relevant research results, e.g., the activities at Alan Turing Institute.

### Organisations

- University College London (Lead Research Organisation)
- European University Viadrina Frankfurt (Oder) (Collaboration)
- University College Hospital (Collaboration)
- UNIVERSITY OF CAMBRIDGE (Collaboration)
- Aix-Marseille University (Collaboration)
- Chinese University of Hong Kong (Project Partner)
- University of Duisburg-Essen (Project Partner)

### Publications

Abascal J
(2021)

*Material Decomposition in Spectral CT Using Deep Learning: A Sim2Real Transfer Approach*in IEEE Access
Barbano R
(2022)

*An Educated Warm Start for Deep Image Prior-Based Micro CT Reconstruction*in IEEE Transactions on Computational Imaging
Benvenuto F
(2020)

*A parameter choice rule for Tikhonov regularization based on predictive risk*in Inverse Problems
Cen S
(2023)

*Recovery of multiple parameters in subdiffusion from one lateral boundary measurement*in Inverse Problems
Duong M
(2020)

*Wasserstein gradient flow formulation of the time-fractional Fokker-Planck equation*in Communications in Mathematical Sciences
Hu T
(2023)

*Solving Elliptic Problems with Singular Sources Using Singularity Splitting Deep Ritz Method*in SIAM Journal on Scientific Computing
Huang J
(2021)

*A Unified Primal Dual Active Set Algorithm for Nonconvex Sparse Recovery*in Statistical Science
Jahn T
(2020)

*On the discrepancy principle for stochastic gradient descent*in Inverse Problems
Jin B
(2020)

*On the Convergence of Stochastic Gradient Descent for Nonlinear Ill-Posed Problems*in SIAM Journal on OptimizationDescription | (1) to develop a theory of SGD for nonlinear inverse problems; (2) to develop a discrepancy principle for linear inverse problems; (3) to develop several algorithms for PDE related inverse problems (4) to analyze rigorously inverse problems for elliptic, parabolic and integro-differential equations. (5) to develop systematically he inverse order problem for subdiffusion type models (6) to understand the saturation phenomenon for stochastic gradient method (6) to develop Bayesian neural networks for learned reconstruction techniques; (7) to develop an unsupervise knowledge transfer approach for learned image reconstruction (8) the acceleration of deep image prior for inverse problems via pretraining (9) optimal experimental design with linearised deep image prior |

Exploitation Route | The algorithmic development can be used directly by practitioners. In particular, we are currently working on project partners from UCL Hospital to put one algorithm into use in their software framework. |

Sectors | Education,Healthcare |

Description | Collaboration with Cambridge University |

Organisation | University of Cambridge |

Country | United Kingdom |

Sector | Academic/University |

PI Contribution | This project is to carried out uncertainty quantification for deep learning based approaches, especially approximate inference of a probabilistic formulation of deep image prior via linearized Laplace approximation. |

Collaborator Contribution | The development of the methodology and implementation |

Impact | This collaboration has resulted in one joint paper (preprint), current under review. |

Start Year | 2021 |

Description | Collaboration with University of Frankfurt |

Organisation | European University Viadrina Frankfurt (Oder) |

Country | Germany |

Sector | Academic/University |

PI Contribution | The PI established a collaboration with Mr. Tim Jahn on discrepancy principle for SGD. It provides a first rigorous criterion for properly stopping SGD, which is one of the major problems within the community. |

Collaborator Contribution | The collaborator made contributions to the theoretical development of the discrepancy principle. The collaboration will likely to continue. |

Impact | The collaboration resulted one joint publication "On the discrepancy principle for stochastic gradient descent" |

Start Year | 2020 |

Description | Collaboration with University of Marseille |

Organisation | Aix-Marseille University |

Country | France |

Sector | Academic/University |

PI Contribution | This collaboration is to conduct joint research on inverse problems for fractional diffusion, especially concerning the determination of the order of fractional derivatives. My research mainly contributes to the mathematical and numerical analysis of relevant inverse problems. |

Collaborator Contribution | The collaborator mainly contributes to the theoretical aspects of the inverse problem, especially the uniqueness. |

Impact | This collaboration is with Dr. Yavar Kian from Aix Marseille University, to conduct joint projects on anomalous diffusion processes, one class of inverse problems associated with PDEs. The collaboration has resulted in three joint publications, and several others are in the pipeline. It requires substantial time-commitment of the collaborator (or in-kind contribution). |

Start Year | 2021 |

Description | UCL PET team |

Organisation | University College Hospital |

Country | United Kingdom |

Sector | Hospitals |

PI Contribution | This collaboration is to apply randomized algorithms to PET reconstruction. Our focus is the methodological development. |

Collaborator Contribution | The project partner's contribution is to provide relevant experimental data within their software framework, and to integrate the algorithm into their framework. |

Impact | One short conference abstract has been submitted, and one full-length paper is in preparation. |

Start Year | 2020 |