Implementing human innate knowledge into an algorithm for self-driving vehicles

The successful development of automated vehicles has been, in recent years, the object of research by several organisations, and is thought to provide a solution for minimising accidents and energy expenditure (Gwak et al., 2019). State-of-the-art self-driving algorithms utilise reinforcement-based learning (RL) which, much like biological systems, are model-free (i.e. they do not require supervision or a great amount of training data) and rely on trial and error for performance optimisation. For example, (Manuelli and Florence, 2015) successfully implemented an algorithm into a self-driving car simulation capable of avoiding obstacles detected through a LIDAR sensor; (Duan et al., 2020) created a model capable of driving safely and at optimal speed in a highway environment. These models are based on the computation and choice of the action of maximum reward value, which is time-consuming, does not provide safety guarantees, and is therefore impractical for implementation in real vehicles.
More recently, implementation of a model-checker into RL algorithms has been proposed, which segregates a scenario and the consequent action by the ego vehicle into a unit and evaluates the plan for that action based on its outcomes, i.e. does not require computation of maximum reward (Bouton et al., 2019). While this model outperformed classic RL, its training environments, like the ones for the aforementioned algorithns, are still too simple and not stochastic or noisy enough to mirror real-life settings requiring adequate planning and/or prioritisation, which may therefore let algorithms get away with not being 'intelligent' enough to react to unpredictable events, e.g. a cat suddenly crossing the road, or prioritise saving human lives (e.g. do not avoid a branch in the current lane by invading the adjacent lane).
The concept of 'core', or innate knowledge in humans has been proposed based on the fact that infants and even newborns have the capability to quickly understand an agent's intent, discriminate between animate and inanimate objects, and group themselves with other individuals in an 'us vs. them' fashion (Spelke and Kinzler, 2007). The translation of these properties into code has proved feasible in similar (Chinchali et al., 2017) and less so (Pospisil, Pasupathy and Bair, 2018) lines of work; their implementation into a self-driving RL algorithm may pose a solution to the previously outlined problems.
With the present project we propose to integrate the aforementioned works by developing a highly stochastic and realistic city road simulation environment (building upon e.g. Li et al. (2018)) and a novel belief-updating RL decision-making algorithm for self-driving with in-built agency attribution, biological motion recognition with preference for humans. Fundamental to these goals will be the determination of decision-making criteria and the range of decisions possible in the simulation environment. Moreover, given the increased complexity of the task, we will additionally attempt to improve the efficiency of the model checker and the storage of the updated motor plans into memory.


Bouton, M. et al. (2019) 'Safe Reinforcement Learning with Scene Decomposition for Navigating Complex Urban Environments', 2019 30TH IEEE INTELLIGENT VEHICLES SYMPOSIUM, pp. 1469-1476. Available at: https://github.com/sisl/ (Accessed: 24 January 2021).
Chinchali, S. P. et al. (2017) 'Multi-objective optimal control for proactive decision making with temporal logic models', The International Journal of Robotics Research, 38, pp. 12-13. doi: 10.1177/0278364919868290.
Duan, J. et al. (2020) 'Hierarchical reinforcement learning for self-driving decision-making without reliance on labelled driving data', IET Intelligent Transport Systems. Institution of Engineering and Technology, 14(5), pp. 297-305. doi: 10.1049/iet-its.2019.0317.
Gwak, J. et al. (2019) 'A review of intelligent self-driving vehicle software research', KSII Transactions on Inter