Topics in deep stochastic control

Many real-world decision-making problems can be understood in the sense of stochastic control, especially problems in mathematical finance. Finding or approximating the optimal control is often challenging and classic dynamic programming algorithms suffer from the curse of dimensionality.

Recently, the following approach for approximating solutions to stochastic control problems has become popular [1]: parameterize the control with a neural network and optimize its parameters to minimize the problem's loss function. Neural networks can express sophisticated controls and parameters can be optimized with gradient-based methods when the problem's transition function is pathwise differentiable. This is a natural assumption for many stochastic control problems and distinguishes those from problems in reinforcement learning, where the environment is treated as a 'black box'.

This approach of "deep stochastic control" has been particularly successful in the context of hedging problems [2]. The resulting method (deep hedging) can find hedging strategies that are optimal with respect to any chosen hedging criterion and market environment.

Deep stochastic control is an attractive solution method due to its methodological simplicity and broad applicability. While its effectiveness has been proven on a range of problems, much of the previous research has been aimed at demonstrating the applicability on easier problems. On real-world problems, the underlying optimization can be difficult as loss signals are often sparse and noisy, and long time horizons induce a high backpropagation depth at which accurate credit assignment becomes difficult. While plenty of research efforts have been aimed at similar challenges in reinforcement learning and the training of recurrent neural networks, few have addressed the stochastic control context. This project aims to address challenges of deep stochastic control and deep hedging. We aim to gain a better understanding of the methodology and its difficulties and find algorithm improvements to solve stochastic control problems more reliably and faster.