This research project falls under the EPSRC Artificial Intelligence Technologies category and specifically focuses on Bayesian Optimization

This research project falls under the EPSRC Artificial Intelligence Technologies category and specifically focuses on Bayesian Optimization [1], a subfield of AI. Bayesian Optimization is employed for optimizing unknown "black-box" functions, where the relationship between input and output is not known. For instance, such a "black-box" function could represent the connection between the parameters of an electronic circuit and its speed [2] or a neural network architecture and its performance on a given task [3]. These types of problems often arise across computer science and engineering, and Bayesian Optimization provides a suitable framework for addressing them.
In Bayesian Optimization, the first step involves constructing a model of the input-output relationship, which is later used to find the most promising input value to try next. However, this model relies on hyperparameters [4], which significantly influence its performance. Typically, these hyperparameters are estimated from data. However, in engineering problems, obtaining new samples can be costly. As such the data at our disposal is usually limited, making these estimates potentially unreliable. Up to this point there has not been a systematic method proposed for dealing with this issue.
Recognizing this existing gap in the literature, our project is driven by the goal of developing a principled approach for selecting hyperparameters in "black-box" function models. Given the constraint of limited data, we envision two potential strategies for addressing this challenge. These two research directions are independent, and our project is designed to explore both comprehensively.
One approach involves quantifying the uncertainty associated with current hyperparameter estimates. This entails establishing a theoretically sound method for estimating uncertainty and incorporating these estimates into the optimization process to make it more robust. Alternatively, in the face of limited data, we could leverage information from similar tasks that we have previously solved. However, before this could be achieved, we need to automatically identify which past tasks bear similarity to the one we are tackling.
A majority of existing literature assumes that hyperparameters are given or can be easily estimated from data. Therefore, the novelty of our methodology lies in lifting this assumption and posing a fundamental question about how to conduct Bayesian Optimization in a more general case. Throughout this project, our goal is to develop more robust algorithms capable of efficiently solving engineering problems and handling the uncertainty in hyperparameters in a safe manner.
[1] Srinivas, Niranjan, et al. "Gaussian process optimization in the bandit setting: no regret and experimental design." Proceedings of the 27th International Conference on International Conference on Machine Learning. 2010.
[2] Grosnit, Antoine, et al. "BOiLS: Bayesian optimisation for logic synthesis." 2022 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 2022.
[3] Nguyen, Vu, et al. "Optimal transport kernels for sequential and parallel neural architecture search." International Conference on Machine Learning. PMLR, 2021.
[4] Williams, Christopher KI, and Carl Edward Rasmussen. Gaussian processes for machine learning. Vol. 2. No. 3. Cambridge, MA: MIT press, 2006.