Causal Foundations of Reinforcement Learning

Reinforcement learning is essentially a causal inference task, although much of traditional RL is done with fully controllable agents in which core causal techniques can only play a trivial role and hence have been mostly ignored. With the recognition that many multistage decision processes are hard to learn without a large number of trials, there is an increased need to use data collected under observational or known off-policy regimes. This project will focus on learning techniques to make the most of tapping on uncontrolled logged trajectories, assessing the fundamental questions of uncertainty and extrapolation problems when data is not generated by direct exploration of the environment by the learner, and how sensitivity analysis techniques can highlight the brittleness of our knowledge in regions of the trajectory space where new interventions need to be done. If AI research wants to engage with real-world problems of sequential planning, it needs to go deeper on methodology that addresses how to combine multiple data sources and structural assumptions, as data-intensive reinforcement learning is not feasible in many problems.