Hierarchy in Bayesian Optimization

Bayesian optimization (BO) is a growing field for finding optimal inputs to black-box functions. These functions are often expensive to evaluate, and do not provide gradient information. A typical setting involves `experiments', where a BO algorithm will sequentially propose experiments that have a natural trade-off between exploring the problem domain, and exploiting high-performing past experiments. BO has proved useful for a wide number of scientific challenges, from chemical engineering (such as battery design) to machine learning (neural architecture search).

Many tasks in the BO literature exhibit a hierarchical nature - that is, for a defined domain, some parameters of an experiment will determine the possible decisions made for other parameters. In my work, we consider a range of problems that have hierarchy, and introduce novel methods to address them.

First, we investigate problems where the domain is non-stationary and implicit hierarchy can be discovered and exploited. Tree-based methods (such as gradient boosting and random forests) are widely used across many domains, providing a simple-to-use and general framework for fitting arbitrary functions. These methods use decision trees that can model a mixed feature space (continuous, integer, and categorical inputs), using an ensemble of weak learners to provide reliable predictions. However, such methods have not seen significant adoption in the BO literature, primarily due to a lack of closed-form predictive posterior. We introduce the BART-Kernel (BARK), a fully Bayesian tree-kernel Gaussian process for BO. This approach gives access to a closed-form acquisition function that can be optimized efficiently (and to global optimality), whilst also providing uncertainty over tree structure unlike previous work. This method effortlessly extends to mixed domains, providing interpretable uncertainty across all features. These domains are prevalent in neural architecture search, where a practitioner aims to optimize the design of a neural network using BO. The method explores the space of tree structures, and is therefore also able to learn dependencies between features, ie. where the impact of one input depends on the value of another.

The second project considers the problem of optimizing the geometry of catalyst pellets to be used inside of reactors, maximizing the rate of reaction. The geometry can belong to one of many classes, each one with different set of parameters, resulting in a complex decision space. Specifically, we observe hierarchies in the features that we optimize over - the "hole radius" of a catalyst is only meaningful if the chosen catalyst class has a hole. We therefore use expert-informed hierarchical structures to reflect this, and enable transfer learning between optimization tasks. We combine this with a multi-fidelity approach to further expand our ability to share information between optimization tasks.

Beyond these two projects, future work in the PhD will consider other domains where hierarchy may arise in BO, and evaluate novel approaches for optimizing over non-standard domains. Whilst developing new methods, I will also focus on making my code reproducible and widely accessible by working closely with the developers of the BO software BoFire.