An investigation into the use of machine learning techniques - specifically reinforcement learning (RL)

This PhD will be an investigation into the use of machine learning techniques - specifically reinforcement learning (RL) - to solve problems within the trading domain, such as: market making; optimal liquidation; and multi-venue statistical arbitrage; covering financial and prediction markets. Historically, approaches to solving these problems have been grounded in mathematical optimisation techniques, most notably stochastic optimal control and dynamic programming [1, 2]. However, these suffer from a fundamental limitation: to obtain closed form solutions one must specify the dynamics of the price process (the model) which can only be assumed. As a result, the derived algorithms may be impractical or intractable to solve. The aim is thus to develop algorithms that learn directly from historical data via simulation and limit order book reconstruction. The problem of high-frequency market making in limit order book markets will be central to this study. In this context an agent must learn to manage inventory and balance risk and reward from tick-to-tick where individual actions occupy very short time-scales ( ms and less). Though there are sparingly few papers that apply reinforcement learning to this practice [3], there have been some great successes in solving similar problems [5, 6] from which to draw inspiration. One envisions a reinforcement learning agent - that is responsive to changes in the market and derived signals - that could replace human traders, enabling a single person to manage a large basket of securities simultaneously. The research will also have a significant component addressing the techniques for simulating an order book given partial information as in [4], a crucial component of training these agents. The results should thus have real practical applications, especially in the financial services industry. There are many problems that must be addressed to solve these trading problems using reinforcement learning. For example: 1. Overcoming stochasticity: It would be naive to suggest that the markets are Markovian or fully observable, as such how does one learn efficiently in this environment. 2. State representation: Alternative formulations of the learning problem, such as predictive state representations, may be better suited to trading problems. 3. Temporal abstraction: A trading agent should have knowledge of the impact of a decision over multiple time scales in order to better capture the balance between short- and long-term payoffs. 4. Encoding risk and reward: There exists some interesting work on safe reinforcement learning which could be adapted for the trading domain. 5. Function approximation: An important question to address is whether the dynamics of the markets can be captured sufficiently by linear methods (such as tile coding) versus non-linear approaches (like neural networks). 6. A priori knowledge: How does one direct the agent into areas of the policy space that are known to be profitable beforehand, say, using potential-based reward shaping. Further, is it possible to transfer knowledge across trading problems?
References
[1] Avellaneda, M., and Stoikov, S. High-frequency trading in a limit order book. Quantitative Finance 8, 3 (2008), 217-224.
[2] Cartea, Á., Jaimungal, S., and Penalva, J. Algorithmic and High-Frequency Trading. Cambridge University Press, 2015.
[3] Chan, N. T., and Shelton, C. An Electronic Market-Maker. Seventh International Conference of the Society for Computational Economics (2001), 43.
[4] Christensen, H. L., Turner, R. E., Hill, S. I., and Godsill, S. J. Rebuilding the limit order book: sequential Bayesian inference on hidden states. Quantitative Finance 13, 11 (nov 2013), 1779-1799.
[5] Kim, A. J., Shelton, C. R., and Poggio, T. Modeling Stock Order Flows and Learning Market-Making from Data. SSRN Electronic Journal (2002), 8.
[6] Nevmyvaka, Y., Feng, Y., and Kearns, M. Reinforcement learning for optimized trade execution. In Proceedings of the 23rd in