# Stochastic iterative regularization: theory, algorithms and applications

Lead Research Organisation:
University College London

Department Name: Computer Science

### Abstract

An inverse problem arises whenever one seeks the cause of observed physical phenomena or observational data, e.g., inferring the governing law from the measurements. This task essentially underlies all scientific discoveries and technological innovations. Thus, the mathematical theory and computational techniques for solving inverse problems are central, e.g., in physics, astronomy, medicine, engineering, and life sciences, and it has evolved into a highly interdisciplinary research area.

Inverse problems are usually ill-posed in the sense that the sought-for solution lacks existence, uniqueness or stability with respect to data perturbation. Since the noise is inherent in the observational data, the numerical algorithms have to employ specialized techniques, commonly known as regularization. The corresponding mathematical framework in the form of regularization theory is highly developed, since the pioneering works of A. Tikhonov in 1960s, H. Engl et al from 1980s and many other researchers. This theory has played a vital role in many research areas, and related numerical algorithms have also been intensively investigated. One versatile framework is to minimize an objective function measuring the quality of fitting between the model output and observational data, possibly plus some additional penalty term, and it covers a large class of powerful iterative inversion techniques.

Due to the unprecedented advances in data acquisition technologies, large datasets are becoming common place for many practical inverse problems. Prominent examples in medical imaging include dynamic, multispectral, multi-energy or multi-frequency data in computed tomography and optical tomography. The ever increasing volume of available data poses enormous computational challenges to image reconstruction, and traditional iterative methods can be too expensive to apply, and currently it represents one of the bottlenecks to extract useful information from the massive dataset. This is especially challenging for problems involving complex physical models, where each data set is very expensive to simulate.

The proposed research aims at addressing the aforementioned outstanding computational challenge using stochastic iterative techniques developed within the machine learning community, and providing relevant theoretical underpinnings. The central idea of stochastic iterative methods is that at each step only a (small) portion of the data set is used to steer the progression of the iterates, instead of the full data set. This allows drastically reducing the computational cost per iteration. This idea has received enormous attention within the machine learning community, and especially has achieved stunning success in deep learning in recent years. Actually stochastic gradient descent and its variants are the workhorse behind many deep learning tasks. A successful completion of this project will greatly advance modern image reconstruction by providing a systematic mathematical and computational framework, including comprehensive theoretical underpinnings, novel algorithms and detailed studies on concrete inverse problems, e.g., in medical imaging.

Inverse problems are usually ill-posed in the sense that the sought-for solution lacks existence, uniqueness or stability with respect to data perturbation. Since the noise is inherent in the observational data, the numerical algorithms have to employ specialized techniques, commonly known as regularization. The corresponding mathematical framework in the form of regularization theory is highly developed, since the pioneering works of A. Tikhonov in 1960s, H. Engl et al from 1980s and many other researchers. This theory has played a vital role in many research areas, and related numerical algorithms have also been intensively investigated. One versatile framework is to minimize an objective function measuring the quality of fitting between the model output and observational data, possibly plus some additional penalty term, and it covers a large class of powerful iterative inversion techniques.

Due to the unprecedented advances in data acquisition technologies, large datasets are becoming common place for many practical inverse problems. Prominent examples in medical imaging include dynamic, multispectral, multi-energy or multi-frequency data in computed tomography and optical tomography. The ever increasing volume of available data poses enormous computational challenges to image reconstruction, and traditional iterative methods can be too expensive to apply, and currently it represents one of the bottlenecks to extract useful information from the massive dataset. This is especially challenging for problems involving complex physical models, where each data set is very expensive to simulate.

The proposed research aims at addressing the aforementioned outstanding computational challenge using stochastic iterative techniques developed within the machine learning community, and providing relevant theoretical underpinnings. The central idea of stochastic iterative methods is that at each step only a (small) portion of the data set is used to steer the progression of the iterates, instead of the full data set. This allows drastically reducing the computational cost per iteration. This idea has received enormous attention within the machine learning community, and especially has achieved stunning success in deep learning in recent years. Actually stochastic gradient descent and its variants are the workhorse behind many deep learning tasks. A successful completion of this project will greatly advance modern image reconstruction by providing a systematic mathematical and computational framework, including comprehensive theoretical underpinnings, novel algorithms and detailed studies on concrete inverse problems, e.g., in medical imaging.

### Planned Impact

The main beneficiaries of the outcomes of this project in terms of theoretical developments and computational techniques would be research institutes or public sectors where the inverse theory and algorithms play a crucial role, e.g., various research centres on inverse problems, signal / image processing and data sciences, e.g., Centre for Inverse Problems at UCL (joint between computer science, mathematics and statistics), Centre for Image Analysis at Cambridge University, AI Centre at UCL Computer Science, and Alan Turing Institute (UCL is one of the founding partner institutes) in the UK, and Radon Institute of Applied and Computational Mathematics in Austria and Centre of Excellence for Inverse Modelling and Imaging in Finland. To disseminate the research outputs to the community, we will give presentations regularly at leading conferences, e.g., Applied Inverse Problems conference (2019, 2021), SIAM Conference on Imaging Sciences (2020, 2022), Chemnitz Symposium in Inverse Problems in Germany and Inverse Days in Finland, and the conference trips will be combined with research visits and seminar talks whenever appropriate to maximize the potential impact.

The main beneficiaries of the algorithmic outcomes and concrete inverse problems in medical imaging within the project would be research institutes or public sections where reconstruction technologies are heavily used, e.g., Centre for Medical Image Computing at UCL (jointly between computer science and medical physics at UCL), AI Centre at UCL and Alan Turing Institute in the UK. Through the connections between Department of Computer Science and these institutes, there will be many opportunities to collaborate directly, e.g., experimental evaluation of the developed technologies. These connections ensure a direct conduit of the transfer of the obtained technologies to medical imaging and machine learning. This transfer will be greatly facilitated through the software package to be developed, as well as the proposed research on novel applications in optical tomography etc.

The wider public will benefit in many ways from a successful completion of this project, e.g., in terms of a significant improvement in medical imaging quality and speed (hence more accurate and faster diagnosis of diseases) achieved by the new algorithms. Thus it will have significant repercussions on all economic and societal sectors involved. The outputs can potentially attract companies, e.g., on imaging device manufacturers (e.g., Phillips, GE and Siemens), and on data science and image processing companies (e.g., Microsoft Research and DeepMind). To achieve this transfer, we will regularly attend industry-sponsored events to disseminate relevant research results, e.g., the activities at Alan Turing Institute.

The main beneficiaries of the algorithmic outcomes and concrete inverse problems in medical imaging within the project would be research institutes or public sections where reconstruction technologies are heavily used, e.g., Centre for Medical Image Computing at UCL (jointly between computer science and medical physics at UCL), AI Centre at UCL and Alan Turing Institute in the UK. Through the connections between Department of Computer Science and these institutes, there will be many opportunities to collaborate directly, e.g., experimental evaluation of the developed technologies. These connections ensure a direct conduit of the transfer of the obtained technologies to medical imaging and machine learning. This transfer will be greatly facilitated through the software package to be developed, as well as the proposed research on novel applications in optical tomography etc.

The wider public will benefit in many ways from a successful completion of this project, e.g., in terms of a significant improvement in medical imaging quality and speed (hence more accurate and faster diagnosis of diseases) achieved by the new algorithms. Thus it will have significant repercussions on all economic and societal sectors involved. The outputs can potentially attract companies, e.g., on imaging device manufacturers (e.g., Phillips, GE and Siemens), and on data science and image processing companies (e.g., Microsoft Research and DeepMind). To achieve this transfer, we will regularly attend industry-sponsored events to disseminate relevant research results, e.g., the activities at Alan Turing Institute.

### Organisations

### Publications

Benvenuto F
(2020)

*A parameter choice rule for Tikhonov regularization based on predictive risk*in Inverse Problems
Duong M
(2020)

*Wasserstein gradient flow formulation of the time-fractional Fokker-Planck equation*in Communications in Mathematical Sciences
Jahn T
(2020)

*On the discrepancy principle for stochastic gradient descent*in Inverse Problems
Jin B
(2020)

*Incomplete iterative solution of subdiffusion*in Numerische Mathematik
Jin B
(2020)

*On the Convergence of Stochastic Gradient Descent for Nonlinear Ill-Posed Problems*in SIAM Journal on Optimization
Jin B
(2020)

*Subdiffusion with time-dependent coefficients: improved regularity and second-order time stepping*in Numerische Mathematik
Jin B
(2020)

*An inverse potential problem for subdiffusion: stability and reconstruction**in Inverse Problems
Jin B
(2021)

*Error Analysis of Finite Element Approximations of Diffusion Coefficient Identification for Elliptic and Parabolic Problems*in SIAM Journal on Numerical Analysis
Lunz S
(2021)

*On Learned Operator Correction in Inverse Problems*in SIAM Journal on Imaging SciencesDescription | (1) to develop a theory of SGD for nonlinear inverse problems; (2) to develop a discrepancy principle for linear inverse problems; (3) to develop several algorithms for PDE related inverse problems (4) to analyze rigorously inverse problems for elliptic, parabolic and integro-differential equations. |

Exploitation Route | The algorithmic development can be used directly by practitioners. In particular, we are currently working on project partners from UCL Hospital to put one algorithm into use in their software framework. |

Sectors | Education,Healthcare |

Description | Collaboration with University of Frankfurt |

Organisation | European University Viadrina Frankfurt (Oder) |

Country | Germany |

Sector | Academic/University |

PI Contribution | The PI established a collaboration with Mr. Tim Jahn on discrepancy principle for SGD. It provides a first rigorous criterion for properly stopping SGD, which is one of the major problems within the community. |

Collaborator Contribution | The collaborator made contributions to the theoretical development of the discrepancy principle. The collaboration will likely to continue. |

Impact | The collaboration resulted one joint publication "On the discrepancy principle for stochastic gradient descent" |

Start Year | 2020 |

Description | UCL PET team |

Organisation | University College Hospital |

Country | United Kingdom |

Sector | Hospitals |

PI Contribution | This collaboration is to apply randomized algorithms to PET reconstruction. Our focus is the methodological development. |

Collaborator Contribution | The project partner's contribution is to provide relevant experimental data within their software framework, and to integrate the algorithm into their framework. |

Impact | One short conference abstract has been submitted, and one full-length paper is in preparation. |

Start Year | 2020 |