Reinforcement Learning in Large Complex Partially Observable Environments

This project falls into the EPSRC Research Area: Artificial Intelligence Technologies EPSRC Research Theme: Information and Communication Technologies

This research project is an exploration of the use of Reinforcement Learning for achieving a sophisticated level of control in large partially observable environments which exhibit complex dynamics and long-term dependencies.

Reinforcement Learning (RL) is a branch of Machine Learning that deals with how to act in an environment in order to maximise some notion of cumulative reward. In order to accomplish this, RL agents must carefully balance their exploration and exploitation of said environment, which is a difficult task in large complex environments. In recent years there has been much progress made on applying model-free approaches to such environments with much success. Most notably, approaches involving Deep Q Networks have been able to play a range of Atari games with superhuman performance.

We wish to continue this line of research and further investigate the use of Deep Q Networks and their many extensions to environments which require long-term planning. Specifically, we aim to produce an agent that can learn how to play a real time strategy game. In order to be able to accomplish such a goal, an agent must be adapt at many complex tasks. In addition to learning the consequences of its actions, an agent must learn to formulate a long-term goal to build towards, and also learn how to react to changes in its environment. Even humans struggle to play Real Time Strategy games without some prior training or guidance, which highlights the complexity of the problem. It is our belief that pursuing a complex problem such as this would lead to the development of useful ideas and techniques that would be applicable in a multitude of other areas.

In order to tackle this problem we will make use of ideas from Hierarchical Reinforcement Learning. We strongly believe that decomposing a problem into simpler sub-problems is a crucial part of being able to tackle complex environments, since the larger problem is often intractable whereas the simpler sub-problems are significantly easier to solve. In addition we will make use of recent advances in Machine Learning, specifically Deep Learning, in order to further refine our internal representation of the environment. An accurate representation of the environment is crucial in order to be able to intelligently act in partially observable domains, especially in the case of Real Time Strategy games where we must also learn to predict our opponent's behaviour.