COMMOTIONS: Computational Models of Traffic Interactions for Testing of Automated Vehicles

Lead Research Organisation: University of Leeds
Department Name: Institute for Transport Studies

Abstract

As automated vehicles (AVs) are being developed for driving in increasingly complex and diverse traffic environments, it becomes increasingly difficult to comprehensively test that the AVs always behave in ways that are safe and acceptable to human road users. There is wide consensus that a key part of the solution to this problem will be the use of virtual traffic simulations, where simulated versions of an AV under development can meet simulated surrounding traffic. Such simulations could in theory cover vast ranges of possible scenarios, including both routine and more safety-critical interactions. However, the current understanding and models of human road user behaviour is not good enough to permit realistic simulations of traffic interactions at the level of detail needed for such testing to be meaningful. This fellowship aims to develop the missing simulation models of human behaviour, to ensure that development of the future automated transport system can be carried out in a responsible, human-centric way.

Behaviour of car drivers and pedestrians will be observed both in real traffic as well as in controlled studies in driving and pedestrian simulators, in some cases complementing behavioural data with neurophysiological (EEG) data, since several candidate component models make specific predictions about brain activity. The fellowship will then build on existing models of driver and pedestrian behaviour in routine and safety-critical situations, and extend these with state of the art neuroscientific models of specific phenomena like perceptual judgments, beliefs about others' intentions, and communication, to create an integrated cognitive modelling framework allowing simulations of traffic interactions across a variety of targeted scenarios.

Such cognitive interaction models, based on well-understood underlying mechanisms, will be one main contribution from the fellowship. Some researchers have suggested the use of another type of model altogether, instead obtained directly by applying machine learning (ML) methods to large data sets of human road user behaviour, i.e., without an ambition to correctly model underlying mechanisms. This fellowship hypothesises that to achieve reliable virtual testing of AVs, both types of modelling approaches will be needed, and methods for combining them will be researched. Not least, due to their "black box" nature, ML models need to be investigated and benchmarked, to for example determine their ability to generalise to rare, safety-critical events.

The multi-disciplinary research, building on and extending on the fellow's past experience in vehicle engineering, cognitive neuroscience, and machine learning, will be carried out at the Institute for Transport Studies, University of Leeds, with support also from the Schools of Psychology and Computing. The fellowship has direct support from industry, both in advisory capacities and as project partners actively sharing data and methods as well as providing first proof-of-concept uptake of the developed models into industrial environments for simulated testing.

Planned Impact

The primary, long-term impact that the project envisions, and which it will actively work towards, is the development and successful deployment of safe and acceptable automated vehicles (AVs). There are large hoped-for economical and societal benefits from transport automation. Long-term, the global economy for AVs and AV-enabled services is projected to be worth trillions of pounds per year, and the UK government is targeting a leading role for the UK in this economy, with an estimated £51 billion annual benefit for the UK economy and 300,000 new jobs by 2030, as well as reductions in road traffic injuries and death, improved inclusive mobility, reduced congestion, and increased productivity.

However, a primarily technology-driven approach to automation, without proper consideration of human behaviour, risks resulting in AVs that behave in ways that are unappreciated by, and potentially unsafe to human road users. If AVs for example cause traffic jams because they are overly cautious, or misinterpret human road user behaviour in ways that lead to crashes, public acceptance and market penetration will suffer, which could in turn severely limit the abovementioned potential benefits. The human behaviour models and virtual testing simulations developed by this project will help mitigate against these risks, by providing a direct means of supporting human-centred, responsible innovation on vehicle automation, to develop AV technology that puts human behaviour, capabilities and well-being first. The project therefore holds promise of impact both at the level of the individual UK citizen, in terms of a safer and more desirable urban road traffic environment, as well as on the national level, giving the UK industry (vehicle manufacturers, suppliers, simulation tool developers, ...) and economy an edge over competitors, and as a result a greater share for the UK of the global market for automation.

Early impacts are expected already while the fellowship is active (2019-2023), in terms of first proofs of concept of the developed models in industrial simulation tools, as well as a raised awareness of the need for proper consideration of human road user behaviours in testing of AVs, among industry, general public, and policy makers. In the first years after the fellowship, this can in turn help drive policy-making on AV testing requirements, while in parallel the models should start to see actual use as part of industrial development processes. This will in turn support larger-scale deployment of safe and acceptable AVs in urban traffic in the UK and elsewhere, by current estimates circa 2025-2035.

Given that the project touches on a wide range of applied disciplines beyond just road vehicle automation, there are many more potential industrial and societal impacts. For example, transport planners can make use of improved traffic simulation tools to make better decisions on public spending on road traffic infrastructure. Furthermore, better models of human interactive locomotion and human situational awareness can be useful also outside the road traffic context, for example in the design of robots locomoting among humans, and of safety-critical environments like aircraft or nuclear power plants.

Publications

10 25 50
publication icon
Goodridge C (2022) Steering is initiated based on error accumulation. in Journal of Experimental Psychology: Human Perception and Performance

publication icon
Pekkanen J (2021) Variable-Drift Diffusion Models of Pedestrian Road-Crossing Decisions in Computational Brain & Behavior

 
Description The main achievement so far has been the development of a computational framework for mathematical modelling of human road user interactions, integrating a range of previously separate computational theories from psychology and cognitive neuroscience. We have demonstrated that models based on this framework can explain a number of previously unexplained interaction phenomena in driver-pedestrian interactions, such as hesitation before committing to decisions, and implicit communication to convey intent.

Other notable achievements include:

The definition and publication of a conceptual framework -- a structured way of talking about -- interactions betweens humans, and between humans and automated vehicles, in road traffic. This framework has seen substantial adoption by researchers in this field, and has already been useful for this specific project's goal of developing mathematical models of human road user behaviour.

The finding that current approaches for training machine-learned models of human road user interaction do not put enough emphasis on details of interaction that are important to humans, which can lead to these models missing some important phenomena.

Demonstrations that combination of the abovementioned integrative computational framework with deep reinforcement learning may provide "the best of both worlds" from (1) cognitive/mechanistic models and (2) machine-learned models.

The finding is that human behaviour and brain responses during detection and judgment of collision threats (such as when interacting with other road users in traffic) can be described as being determined by evidence accumulation - a type of decision-making mechanism that has previously been studied mainly for more abstract laboratory-based tasks. This provides a further solid foundation for the models being developed in this project, and opens many new research directions both within road traffic research, and within human decision-making research in general.

We have also found that evidence accumulation models work well for describing human decision-making also in a number of specific interactive situations in road traffic, including (1) pedestrian road crossing, (2) driver gap acceptance, and (3) near-crash driver braking.
Exploitation Route The cognitive modelling framework can be used by others in academia and industry (and is already to some extent) for simulating interactions, and developing further models for scientific or applied purposes. The mentioned conceptual framework provides theoretical structure for (1) academic or industry investigations into how humans interact with automated vehicles, and for (2) formulation of clear and appropriate requirements on the interactive capabilities of automated vehicles. The mentioned models of collision threat detection are useful to traffic safety research and development in industry and academia, as well as to traffic accident litigation (our findings refute a previously dominant account, which has been leveraged by expert witnesses and similar). The developed mathematical models of interactions can be adopted by industry and academia, for example to generate simulated testing environments for automated vehicles. We are pursuing such collaborations with several partners.
Sectors Digital/Communication/Information Technologies (including Software)

Manufacturing

including Industrial Biotechology

Transport

Other

 
Description As described under "Influence on policy, practice, ...", we have been able to leverage the project to gain access to various standardisation fora where discussions are currently happening that are important for how automated vehicles will be tested going forward, and the project's results have also attracted the interest of additional industry and local government stakeholders. In these various contexts, we have been able to raise awareness of why there is a need for the type of human behaviour models that the project is targeting, as well as of the project's specific findings and solutions. We have provided concrete input into standardisation documents in development. We are actively collaborating with and advising a range of stakeholders in the automotive industry, and are working on integrations and evaluations of our models in their toolchains - and we are aware of several cases where such integration has already happened to some extent.
First Year Of Impact 2020
Sector Digital/Communication/Information Technologies (including Software),Manufacturing, including Industrial Biotechology,Transport,Other
Impact Types Societal

Economic

 
Description Scientific advisor to automotive industry (Volvo, Nissan, Waymo) in recurring meetings on road user behaviour modelling
Geographic Reach Multiple continents/international 
Policy Influence Type Participation in a guidance/advisory committee
 
Description Shaping ISO Technical Specification on simulation-based safety testing
Geographic Reach Multiple continents/international 
Policy Influence Type Participation in a guidance/advisory committee
 
Description Society of Automotive Engineers Automated Driving Simulation Task Force
Geographic Reach Multiple continents/international 
Policy Influence Type Participation in a guidance/advisory committee
 
Description Automated vehicles testing with human-like traffic (ADVENTURE)
Amount £44,614 (GBP)
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 03/2024 
End 02/2025
 
Description COVID 19 Grant Extension Allocation
Amount £23,821 (GBP)
Organisation United Kingdom Research and Innovation 
Sector Public
Country United Kingdom
Start 01/2021 
End 09/2021
 
Description Two PhD studentships sponsored by Nissan
Amount £170,000 (GBP)
Organisation Nissan Motor Manufacturing Ltd 
Sector Private
Country United Kingdom
Start 01/2020 
End 12/2023
 
Title Collision threat detection dataset 
Description Primary research data (behavioural responses and electroencephalography), from the collision threat detection study described in this paper: Markkula G, Uludag Z, Wilkie R M, Billington J. 2020. Accumulation of continuously time-varying sensory evidence constrains neural and behavioral responses in human collision threat detection. PsyArXiv preprint: https://doi.org/10.31234/osf.io/ca3h9 
Type Of Material Database/Collection of data 
Year Produced 2020 
Provided To Others? Yes  
Impact No impacts yet, but the findings have applied and societal impacts in terms of refuting a widely used assumption about human collision threat detection, which has been used in traffic safety research and development, as well as in traffic accident litigation. 
URL https://osf.io/ku3h4/
 
Title Framework for modelling human road user interaction 
Description A Python implementation of a cognitive modelling framework for modelling road user interaction, including code used for testing a large number of model variants based on this framework, in straight-crossing interactions between drivers and pedestrians. 
Type Of Material Computer model/algorithm 
Year Produced 2022 
Provided To Others? Yes  
Impact Follow-on research activities with collaborators. 
URL https://osf.io/zmk9t/
 
Title Model of human collision threat detection 
Description The collision threat detection model described in this paper: Markkula G, Uludag Z, Wilkie R M, Billington J. 2020. Accumulation of continuously time-varying sensory evidence constrains neural and behavioral responses in human collision threat detection. PsyArXiv preprint: https://doi.org/10.31234/osf.io/ca3h9 
Type Of Material Computer model/algorithm 
Year Produced 2020 
Provided To Others? Yes  
Impact No known impacts yet, but the model has applied and societal impacts in terms of replacing a previously widely used assumption about human collision threat detection, which has been used in traffic safety research and development, as well as in traffic accident litigation. 
URL https://github.com/gmarkkula/LoomingDetectionStudy
 
Title The COMMOTIONS Urban Interactions Driving Simulator Study Dataset 
Description Accurate modelling of road user interaction has received lot of attention in recent years due to the advent of increasingly automated vehicles. To support such modelling, there is a need to complement naturalistic datasets of road user interaction with targeted, controlled study data. This project describes a dataset collected in a simulator study conducted in the project COMMOTIONS, addressing urban driving interactions, in a state of the art moving base driving simulator. The study focused on two types of near-crash situations that can arise in urban driving interactions, and also collected data on human driver gap acceptance across a range of controlled gap sequences. 
Type Of Material Database/Collection of data 
Year Produced 2023 
Provided To Others? Yes  
URL https://osf.io/eazg5/
 
Description Collaboration with Aalto University 
Organisation Aalto University
Country Finland 
Sector Academic/University 
PI Contribution Contributions toward jointly authored publications.
Collaborator Contribution Hosting a Leeds postdoc and a Leeds PhD student for one month research visit each, at separate times. Contributions toward jointly authored publications.
Impact See Secondments: ARS and YW, and Publications: (Wang et al, 2023)
Start Year 2022
 
Description Collaboration with Leeds City Council 
Organisation Leeds City Council
Country United Kingdom 
Sector Public 
PI Contribution Defining a collection of pedestrian-vehicle interaction at selected locations in Leeds.
Collaborator Contribution Providing guidance for the data collection and research from a perspective of the interests of the Council in terms of traffic safety improvement. Supporting the identification of suitable locations. Facilitating the administrative and practical preparations for sensor installation.
Impact Data collection still in progress.
Start Year 2021
 
Description Collaboration with TU Delft AiTech 
Organisation Delft University of Technology (TU Delft)
Country Netherlands 
Sector Academic/University 
PI Contribution Researchers in the TU Delft project AiTech are pursuing similar objectives to COMMOTIONS. We have organised several workshops, mutual meetings, and authored joint publications.
Collaborator Contribution Input to discussion and collaboration as mentioned above.
Impact See (1) Engagement Activities: "A talk or presentation - Presentation at TU Delft AiTech Agora...", and "A talk or presentation - Invited talk at TU Delft...", (2) Further funding: "TAILOR European AI Network of Excellence...", (3) Secondments: JS, and (4) Publications: (Zgonnikov et al, 2022) and (Schumann et al, 2023)
Start Year 2019
 
Description Collaboration with US projects on driver modelling 
Organisation Texas A&M University
Country United States 
Sector Academic/University 
PI Contribution Advising so far two different projects involving this consortium (in two different constellations) on driver behaviour modelling.
Collaborator Contribution Wider application of the types of models researched in COMMOTIONS, thus increasing our understanding of these models.
Impact See Publications: Sarkar et al (2021) Accident Analysis & Prevention; Wei et al (2022) 3rd International Workshop on Active Inference. See also the project report: https://trid.trb.org/view/1765408
Start Year 2019
 
Description Collaboration with US projects on driver modelling 
Organisation University of Wisconsin-Madison
Country United States 
Sector Academic/University 
PI Contribution Advising so far two different projects involving this consortium (in two different constellations) on driver behaviour modelling.
Collaborator Contribution Wider application of the types of models researched in COMMOTIONS, thus increasing our understanding of these models.
Impact See Publications: Sarkar et al (2021) Accident Analysis & Prevention; Wei et al (2022) 3rd International Workshop on Active Inference. See also the project report: https://trid.trb.org/view/1765408
Start Year 2019
 
Description Collaboration with US projects on driver modelling 
Organisation Virginia Tech
Department Transportation Institution
Country United States 
Sector Academic/University 
PI Contribution Advising so far two different projects involving this consortium (in two different constellations) on driver behaviour modelling.
Collaborator Contribution Wider application of the types of models researched in COMMOTIONS, thus increasing our understanding of these models.
Impact See Publications: Sarkar et al (2021) Accident Analysis & Prevention; Wei et al (2022) 3rd International Workshop on Active Inference. See also the project report: https://trid.trb.org/view/1765408
Start Year 2019
 
Description Collaboration with the German Aerospace Centre (DLR) 
Organisation German Aerospace Centre (DLR)
Country Germany 
Sector Public 
PI Contribution Advising two visiting researchers working in the same area as this grant, working toward joint publications.
Collaborator Contribution Research work toward a joint publication.
Impact See Publications: (Theisen et al, 2024)
Start Year 2021
 
Description Circulation and discussion of project-produced "green paper" 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Industry/Business
Results and Impact The "green paper" produced by the project (see Publications) was circulated via e-mail lists, Twitter, and at conferences, to raise awareness of the project and elicit feedback on the outlined project approach. These objectives were both achieved.
Year(s) Of Engagement Activity 2019
URL https://osf.io/vbcaz
 
Description Invited talk at Einride AB: Modelling interactions between human road users and automated vehicles 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Industry/Business
Results and Impact An online presentation for R&D engineers at a Swedish company developing automated vehicles.
Year(s) Of Engagement Activity 2022
 
Description Invited talk at TU Delft: Do we need to understand the human brain to make robots and automated vehicles that can coexist with humans? 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact An invited talk for a mix of researchers and students, many of which are outside my normal peer group.
Year(s) Of Engagement Activity 2023
 
Description Invited talks at two workshops at IEEE Intelligent Vehicles 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Two invited talks at "the 2023 Workshop on Scenario Generation for Testing Autonomous Vehicles" (talk title "The role of human behavioural phenomena in scenario-based testing of automatee vehicles") and at "the 1st International Workshop on Socially Interactive Autonomous Mobility" (talk title "Adopting knowledge and models from cognitive neuroscience to enable socially interactive automation"). Led to follow-up meetings for a potential collaboration with one of the attendees, and another invitation for a talk in 2024.
Year(s) Of Engagement Activity 2023
 
Description Online news item "Understanding the 'traffic brain'" 
Form Of Engagement Activity Engagement focused website, blog or social media channel
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact An online news item on the University web pages with a summary for the general public of our paper "Accumulation of continuously time-varying sensory evidence constrains neural and behavioral responses in human collision threat detection".
Year(s) Of Engagement Activity 2021
URL https://environment.leeds.ac.uk/transport/news/article/5423/understanding-the-traffic-brain
 
Description Participation in "I'm a scientist, get me out of here!" 
Form Of Engagement Activity Engagement focused website, blog or social media channel
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Schools
Results and Impact I'm a Scientist is "an online, student-led STEM enrichment activity. It connects school students with scientists through energetic real-time text based chats." Gustav Markkula participated in a two-week "Psychology Zone" event discussing his research and Psychology/STEM careers in general with secondary school children.
Year(s) Of Engagement Activity 2021
URL https://imascientist.ie/
 
Description Participation in Dagstuhl Seminar on "Computational Models of Human-Automated Vehicle Interaction" 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Interactive research seminar on a topic that is central to this research project, for information exchange between researchers at all career levels as well as industry representatives (Bosch, Microsoft, and others). Several collaborations identified.
Year(s) Of Engagement Activity 2022
URL http://dagstuhl.de/en/program/calendar/semhp/?semnr=22102
 
Description Presentation at MathPsych 2019 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Presentation at the annual meeting of the Mathematical Psychology Society, Montreal, Canada: "Mathematical psychology in the wild - why and how?Insights from applying basic modelling concepts to applied problems in traffic safety and self-driving cars"
Year(s) Of Engagement Activity 2019
 
Description Presentation at SHIFT Mobility: "Self-Driving priorities: Building robots or understanding humans?" 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Industry/Business
Results and Impact Hybrid/online event as part of IFA Berlin.
Year(s) Of Engagement Activity 2020
URL https://xtended.ifa-berlin.com/eventgrid/stage/7/110
 
Description Presentation at SIMUSAFE workshop 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Presentation at March 31 2021 workshop SIMULATORS APPLIED TO ROAD SAFETY FOR BEHAVIOURAL ANALYSIS AND TRAINING: "Data for road user behaviour models: Naturalistic or simulator studies?"
Year(s) Of Engagement Activity 2021
URL https://simusafe.eu/workshop_details/simulators-applied-to-road-safety-for-behavioural-analysis-and-...
 
Description Presentation at TU Delft AiTech Agora: "Modeling human-AV interactions for safety and acceptance of automated vehicles" 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Presentation and discussion at the TU Delft AiTech Agora (https://www.tudelft.nl/aitech/agora). Led on to a couple of follow-on discussions with researchers, currently developing into more substantial collaborations.
Year(s) Of Engagement Activity 2020
URL https://www.youtube.com/watch?v=nRCbKFK2b2A
 
Description Press release and press coverage "Lack of simulations hampering driverless vehicle revolution" 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact A University of Leeds press release alongside the publication of our 2023 paper in PNAS Nexus, generating stories in many online publications worldwide (including ten stories linked to the paper by Altmetric).
Year(s) Of Engagement Activity 2023
URL https://www.leeds.ac.uk/news-environment/news/article/5322/lack-of-simulations-hampering-driverless-...
 
Description Press release and press coverage "Making self-driving cars human-friendly" 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Media (as a channel to the public)
Results and Impact A University press release describing the findings in our paper "Variable-drift diffusion models of pedestrian road-crossing decisions", written about in at least 67 outlets internationally, with a combined monthly reach of over 20 million readers, in at least six different languages.
Year(s) Of Engagement Activity 2021
URL https://www.leeds.ac.uk/news-technology/news/article/4931/making-self-driving-cars-human-friendly
 
Description Special Session at the International Conference on Traffic and Transport Psychology: Modelling road user interaction 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact A session with four speakers from academia and industry, and a panel discussion at the end.
Year(s) Of Engagement Activity 2022
 
Description Talk at IEEE Intelligent Vehicles Workshop: Simulation of Driver Behavior for the Assessment of Automated Driving 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact About 15 attendees from academia and industry.
Year(s) Of Engagement Activity 2022
 
Description Tutorial on cognitive modelling at the ACM CHI conference 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact Supported our Aalto University collaborators on delivering a tutorial, including a programming exercise making use of some of the COMMOTIONS modelling results.
Year(s) Of Engagement Activity 2023
URL https://github.com/howesa/CHI22-CogMod-Tutorial/