Novel gaming experience through natural language interaction

I aim to research ideas around zero-shot skill transfer/learning and fast/few-shot language grounding within the context of, and in order to bring to life, a novel kind of game. It would revolve around the concept of guiding agents, through natural language interaction, to play/solve the game instead of using some controllers. Having an agent that follows the player's instructions, or co- operate with the player, would obviously bring about a novel gaming experience and it might attract a new kind of public too, hopefully. This novel kind of game could contribute to the whole hardware virtualization trend that is starting to become more and more real.

Leveraging OpenAI's Universe module/ideas, it should be made easy to replace any common controllers with natural language interactions. Thus, it makes the concept applicable to a great variety of games out-of-the-box.

This project tackles very hot topics in the AI scene at the moment, to wit, language grounding, zero/few-shot transfer/learning and the ever so hot human-machine interface topic. Happily, it does not stop there. With regards to robotics, assuming that a very fine level of language grounding can be achieved so as to enable the control of multiple degrees of freedom robots, it could lead to new ways of teaching robots to do tasks and, furthermore, it could enable anybody to teach a robot anything (https://robonlp2017.github.) given the fact that it would only require natural language interactions, instead of extensive knowledge in mathematics/robotics or VR tools like the current trend shows.

Zero/few-shot transfer/learning problems could be tackled through the lens of the DARLA architecture by Irina Higgins et al. (https://arxiv.org/pdf/1707.), augmented with ideas of the Hindsight Experience Replay (https://arxiv.org/abs/1707.) and a human preferences-based framework (https://arxiv.org/pdf/1706.).

With regards to language grounding, the main issue might lie in its pace, as far as players would be concerned. Offline/few-shot learning approaches can be considered to fill the gap, or real innovations could be sought in order to make it achievable in an online manner using human feedback/preferences and a curiosity-driven behaviour of the agent.