A Natural Treatment of Generative Models

Modern generative models have achieved impressive performance in a wide range of applications from language modelling, image generation and drug discovery tasks. However,
as the complexity of these systems grow, there has been a growing literature on the increasing gulf between the traditional measure-theoretic notation driving these past successes, and how these systems are implemented and intuitively reasoned about. This has led to the proposal of using Markov categories to analyse and design future generative models. This is a recently introduced framework for probability theory, based in category theory (as opposed to measure theory) constructed from the generalised notion and axioms of Markov kernels. One advantage of this framework is that it allows for a more conceptually scalable way to reason about probabilistic systems, and which stays close to the underlying computational implementation, through abstracting away irrelevant measure-theoretic details. For my thesis, I am interested in using Markov categories for the methodological improvement of modern generative models. For a concrete direction, I am interested in using Markov categories for the design of equivariant diffusion models, and equivariance for neural networks more generally. Additionally, I am interested in using Markov categories for studying and inducing compositional generalisation in language modelling.