Enabling a Novel Evaluation Continuum for Connected & Autonomous Vehicles (CAV)
Lead Research Organisation:
University of Warwick
Department Name: WMG
Abstract
The global Connected & Autonomous Vehicles (CAV) industry is estimated to be worth over £50billion (by 2035), with the UK CAV industry being projected over £3billion. Additionally, the UK Government's Industrial Strategy aims to bring fully autonomous cars without a human operator on the UK roads by 2021, one of the first countries in the world to achieve this. However, in order to realise this vision and the market potential, safe introduction of CAV is necessary, requiring significant research to overcome diverse barriers (technological, legislative and societal) associated with public deployment of CAV.
While prototype CAV technologies have existed for some time now, ensuring the safety level of these technologies has been proving to be a hindrance to the commercialization of CAV technologies. The vision for CAV is coupled with the challenge of testing and safety analysis as it needs complex solutions to include interactions between a large number of variables and the environment. It is suggested that in order to prove that CAV are safer than human drivers, they will need to be driven for more than 11 billion miles.
The vision for this fellowship is to support positioning the UK as the world leader in CAV research and innovation for a long lasting societal and economic benefit. This fellowship will develop pioneering testing methodologies and standards to enable robust and safe use of CAV with a focus on creating both fundamental knowledge and applied research methods and tools. At WMG, University of Warwick, UK, we have created a concept of the "evaluation continuum" for CAV, which involves using various environment like digital world, simulated environment, test track testing and real-world for testing.
There are two aspects which are common to each of the evaluation continuum environments and also the focus areas of the fellowship research 1) Test Scenarios (input to the environment) 2) Safety Evidence (output of the environment). On the scenarios theme, while the 11-billion-miles requirement has garnered a lot of publicity, the focus needs to be on what happens in those miles (i.e., smart miles which expose failures in CAV) and not on the number of miles themselves. As a part of this fellowship, three approaches will be explored to identify these smart miles. These include 1) using Machine Learning (ML) based methods including Bayesian Optimisation to create test cases for test scenarios, 2) Safety Of The Intended Functionality (SOTIF) (Innovative safety analysis of CAV) based test scenarios and 3) translating real-world data into executable test scenarios for a simulation tool.
All these approaches will together contribute to the creation of a UK's National CAV Test Scenario Database, which will help coordinate the research work in various CAV projects part-funded by the UK Government and will prevent "reinventing of the wheel" in each of the projects with respect to test scenario identification.
Industry trends in CAV suggest the widespread adoption of machine learning (ML) in the autonomous control systems. ML-systems by their structure are non-deterministic in nature, making the CAV system highly opaque in nature. Therefore, it is difficult to identify the reason of a failure in such ML-based systems and take the corrective measures. Thus, on the safety strand, fundamental research will be conducted as a part of this fellowship to explore how to make ML-based systems interpretable enabling us to explain the results. This is an essential requirement for safety of CAV due to the critical nature of their deployment and the mitigation of risk.
In addition, the fellowship will also benefit from the fellow's first-hand experience as the UK's technical representative on the ISO standards committees, providing further insight and a clear route to deliver impact from the proposed research through the development of international standards, while also ensuring that the UK becomes a world leader in this area.
While prototype CAV technologies have existed for some time now, ensuring the safety level of these technologies has been proving to be a hindrance to the commercialization of CAV technologies. The vision for CAV is coupled with the challenge of testing and safety analysis as it needs complex solutions to include interactions between a large number of variables and the environment. It is suggested that in order to prove that CAV are safer than human drivers, they will need to be driven for more than 11 billion miles.
The vision for this fellowship is to support positioning the UK as the world leader in CAV research and innovation for a long lasting societal and economic benefit. This fellowship will develop pioneering testing methodologies and standards to enable robust and safe use of CAV with a focus on creating both fundamental knowledge and applied research methods and tools. At WMG, University of Warwick, UK, we have created a concept of the "evaluation continuum" for CAV, which involves using various environment like digital world, simulated environment, test track testing and real-world for testing.
There are two aspects which are common to each of the evaluation continuum environments and also the focus areas of the fellowship research 1) Test Scenarios (input to the environment) 2) Safety Evidence (output of the environment). On the scenarios theme, while the 11-billion-miles requirement has garnered a lot of publicity, the focus needs to be on what happens in those miles (i.e., smart miles which expose failures in CAV) and not on the number of miles themselves. As a part of this fellowship, three approaches will be explored to identify these smart miles. These include 1) using Machine Learning (ML) based methods including Bayesian Optimisation to create test cases for test scenarios, 2) Safety Of The Intended Functionality (SOTIF) (Innovative safety analysis of CAV) based test scenarios and 3) translating real-world data into executable test scenarios for a simulation tool.
All these approaches will together contribute to the creation of a UK's National CAV Test Scenario Database, which will help coordinate the research work in various CAV projects part-funded by the UK Government and will prevent "reinventing of the wheel" in each of the projects with respect to test scenario identification.
Industry trends in CAV suggest the widespread adoption of machine learning (ML) in the autonomous control systems. ML-systems by their structure are non-deterministic in nature, making the CAV system highly opaque in nature. Therefore, it is difficult to identify the reason of a failure in such ML-based systems and take the corrective measures. Thus, on the safety strand, fundamental research will be conducted as a part of this fellowship to explore how to make ML-based systems interpretable enabling us to explain the results. This is an essential requirement for safety of CAV due to the critical nature of their deployment and the mitigation of risk.
In addition, the fellowship will also benefit from the fellow's first-hand experience as the UK's technical representative on the ISO standards committees, providing further insight and a clear route to deliver impact from the proposed research through the development of international standards, while also ensuring that the UK becomes a world leader in this area.
Planned Impact
The fellowship research outcomes will impact a broad range of beneficiaries:
Societal beneficiaries
- UK and wider Public. Connected and Autonomous Vehicles (CAV) will offer convenience and comfort to passengers and drivers and the ability to operate anytime anywhere. Cars will communicate with other cars and road infrastructure to enhance traffic and travel efficiency and in particular, road safety with the potential to reduce the number of accidents. CAV will also potentially provide mobility to the disabled
- UK and wider public. Another very important societal impact of this research would be to cut through the media hype around CAVs and inform public perception accurately via various outreach activities (TEDx talks, public lectures at Warwick Family Days, British Science Festivals etc.).
- WMG Talks, a new initiative started by the fellow at WMG, University of Warwick, involves early career researchers to give a TEDx style 10min talk on their research in an engaging and fun to listen style. Talks will be stored online and can be viewed via the WMG webpage. Not only will it enhance my team's and wider WMG researchers' science and engineering communication skills, the online repository will have far reaching society impact in shaping the image of research and engineering. The WMG website receives over 114,000 hits every week.
- Young People: The fellowship will also involve outreach activities to school pupils, enabling young people to engage with emerging UK developed technology. Current WMG and JLR education activities host 20,000 individuals each year and over 1,500 schools e.g. pupils at the new WMG Academies for Young Engineers (Coventry and Solihull).
Commercial Beneficiaries
- CAV related companies. This fellowship research outcomes will enable CAV manufacturers, technology providers (Tier-1s) to progress and commercialise their technology quickly as the research focus areas are of direct relevance to the industry (demonstrated via the industrial support received for this fellowship).
- International and national standards. This fellowship's research outcome will help inform and lead the creation of national and international standards on test scenarios for CAV and scenario description language. This will enable a common understanding of scenarios in the CAV industry, enhancing effectiveness of the product development cycle involving multiple stakeholders (OEMs, Tier-1 suppliers, Consultancies)
- Other industry sectors. There will be exploitation opportunity in sectors including aerospace, rail, commercial and off-road, ICT, marine and energy. The project will address Innovate UK's strategic aim to 'accelerate the journey from concept to commercialisation'. Exploiting the technology and data from future ITS will benefit, in particular, the large number of SMEs in niche applications. WMG centre HVM Catapult's REACH SME programme will help facilitate this.
Economic Beneficiaries
- UK and wider Economy. The global CAV industry is said to be worth over £50billion (by 2035), with the UK CAV industry being projected over £3billion. Additionally, the UK Government's Industrial Strategy intends to deploy "fully self-driving cars without a human operator on the UK roads by 2021. Research outcomes from this fellowship will enable the safe deployment of CAV technologies on UK and international roads helping realise the vision and the market potential.
- National and local government and policy makers. Project outputs will lead to greater confidence in decisions affecting transport and transport infrastructure, and will contribute to discussions on new standards affecting CAV.
- Engineers at all levels will benefit from enhanced training. In particular, young engineers (from summer interns, to post-graduates to PDRA) will be targeted, equipping them with the new skills necessary for advanced engineering of CAV. Learning will be passed on through WMG's many post-experience programmes.
Societal beneficiaries
- UK and wider Public. Connected and Autonomous Vehicles (CAV) will offer convenience and comfort to passengers and drivers and the ability to operate anytime anywhere. Cars will communicate with other cars and road infrastructure to enhance traffic and travel efficiency and in particular, road safety with the potential to reduce the number of accidents. CAV will also potentially provide mobility to the disabled
- UK and wider public. Another very important societal impact of this research would be to cut through the media hype around CAVs and inform public perception accurately via various outreach activities (TEDx talks, public lectures at Warwick Family Days, British Science Festivals etc.).
- WMG Talks, a new initiative started by the fellow at WMG, University of Warwick, involves early career researchers to give a TEDx style 10min talk on their research in an engaging and fun to listen style. Talks will be stored online and can be viewed via the WMG webpage. Not only will it enhance my team's and wider WMG researchers' science and engineering communication skills, the online repository will have far reaching society impact in shaping the image of research and engineering. The WMG website receives over 114,000 hits every week.
- Young People: The fellowship will also involve outreach activities to school pupils, enabling young people to engage with emerging UK developed technology. Current WMG and JLR education activities host 20,000 individuals each year and over 1,500 schools e.g. pupils at the new WMG Academies for Young Engineers (Coventry and Solihull).
Commercial Beneficiaries
- CAV related companies. This fellowship research outcomes will enable CAV manufacturers, technology providers (Tier-1s) to progress and commercialise their technology quickly as the research focus areas are of direct relevance to the industry (demonstrated via the industrial support received for this fellowship).
- International and national standards. This fellowship's research outcome will help inform and lead the creation of national and international standards on test scenarios for CAV and scenario description language. This will enable a common understanding of scenarios in the CAV industry, enhancing effectiveness of the product development cycle involving multiple stakeholders (OEMs, Tier-1 suppliers, Consultancies)
- Other industry sectors. There will be exploitation opportunity in sectors including aerospace, rail, commercial and off-road, ICT, marine and energy. The project will address Innovate UK's strategic aim to 'accelerate the journey from concept to commercialisation'. Exploiting the technology and data from future ITS will benefit, in particular, the large number of SMEs in niche applications. WMG centre HVM Catapult's REACH SME programme will help facilitate this.
Economic Beneficiaries
- UK and wider Economy. The global CAV industry is said to be worth over £50billion (by 2035), with the UK CAV industry being projected over £3billion. Additionally, the UK Government's Industrial Strategy intends to deploy "fully self-driving cars without a human operator on the UK roads by 2021. Research outcomes from this fellowship will enable the safe deployment of CAV technologies on UK and international roads helping realise the vision and the market potential.
- National and local government and policy makers. Project outputs will lead to greater confidence in decisions affecting transport and transport infrastructure, and will contribute to discussions on new standards affecting CAV.
- Engineers at all levels will benefit from enhanced training. In particular, young engineers (from summer interns, to post-graduates to PDRA) will be targeted, equipping them with the new skills necessary for advanced engineering of CAV. Learning will be passed on through WMG's many post-experience programmes.
Organisations
- University of Warwick (Fellow, Lead Research Organisation)
- Department of Transport (Collaboration)
- National Institute of Standards & Technology (NIST) (Collaboration)
- Transport Canada (Collaboration, Project Partner)
- World Economic Forum (Collaboration)
- JSAE (Project Partner)
- British Standards Institution (Project Partner)
- RDM Group (United Kingdom) (Project Partner)
- Anstalt für Verbrennungskraftmaschinen List (Project Partner)
- Five AI Limited (Project Partner)
Publications
Brackstone M
(2020)
OmniCAV: A Simulation and Modelling System that enables "CAVs for All"
Bruto Da Costa A
(2022)
Writing Accessible and Correct Test Scenarios for Automated Driving Systems
Chen S
(2021)
Analyzing National Responses to COVID-19 Pandemic using STPA.
in Safety science
Esenturk E
(2022)
Identification of Traffic Accident Patterns via Cluster Analysis and Test Scenario Development for Autonomous Vehicles
in IEEE Access
Esenturk E
(2023)
A data mining approach for traffic accidents, pattern extraction and test scenario generation for autonomous vehicles
in International Journal of Transportation Science and Technology
Irvine P
(2021)
A Two-Level Abstraction ODD Definition Language: Part I
Description | A safety assurance framework for automated mobility which helps ensure new (automated) technology is safe for society |
Exploitation Route | Extending the methodologies, knowledge and tools to new domains of mobility and technology, along with advancing them within the CAM domain. |
Sectors | Digital/Communication/Information Technologies (including Software),Transport |
Description | Influencing national and international standards and policy at ISO, SAE, ASAM and UNECE (United Nations Economic Commission for Europe) and the UK Government. |
First Year Of Impact | 2021 |
Sector | Digital/Communication/Information Technologies (including Software),Transport |
Impact Types | Societal,Economic,Policy & public services |
Description | ALKS - Call for Evidence |
Geographic Reach | Europe |
Policy Influence Type | Contribution to a national consultation/review |
Impact | I submitted evidence to the Automated Lane-Keeping System (ALKS) call for evidence which has been subsequently cited by the Law Commission in their third consultation paper. The evidence submitted by me focusses on the safety of ALKS and the role of the Operational Design Domain. This is now been used for the formulation of the UK regulation. |
URL | https://www.lawcom.gov.uk/document/automated-vehicles-cp-3-a-regulatory-framework-for-automated-vehi... |
Description | ASAM OpenODD - Project Lead |
Geographic Reach | Multiple continents/international |
Policy Influence Type | Implementation circular/rapid advice/letter to e.g. Ministry of Health |
Impact | I am the project lead for ASAM OpenODD standard - creating a language for defining Operational Design Domain enabling manufacturers to exchange ODD information more efficiently reducing costs. |
Description | ASAM OpenX standards - project member |
Geographic Reach | Multiple continents/international |
Policy Influence Type | Membership of a guideline committee |
Impact | Creating standards for simulation based testing of automated driving to enable manufacturers to exchange information more efficiently reducing costs. |
Description | BSI PAS 1881, 1882, 1884 |
Geographic Reach | National |
Policy Influence Type | Participation in a guidance/advisory committee |
Impact | I am the steering group member of BSI PAS 1881, 1882, 1884. |
Description | BSI PAS 1883 - Technical Author |
Geographic Reach | Multiple continents/international |
Policy Influence Type | Implementation circular/rapid advice/letter to e.g. Ministry of Health |
Impact | I was the technical author of the new British standard - BSI PAS 1883 which provides a taxonomy for Operational Design Domain. I led this activity on behalf of Centre for Connected and Autonomous Vehicles (CCAV) and BSI, to enable manufacturers to exchange ODD information more efficiently and thus reducing costs. |
URL | https://www.bsigroup.com/en-GB/CAV/pas-1883/ |
Description | Future of Mobility Consultaion from Department for Transport |
Geographic Reach | National |
Policy Influence Type | Contribution to a national consultation/review |
Description | ISO 22737 - LSAD - Lead Author |
Geographic Reach | Multiple continents/international |
Policy Influence Type | Implementation circular/rapid advice/letter to e.g. Ministry of Health |
Impact | I am the lead author of the first ever International (ISO) standard for SAE Level 4 automated system - ISO 22737 which successfully completed Draft International Standard (DIS) ballot in Oct 2020. This standard enabled driverless shuttle manufacturers to sell products by opening up markets worth £250M. |
Description | SAE ORAD Committees |
Geographic Reach | North America |
Policy Influence Type | Membership of a guideline committee |
Impact | I am the member of various SAE On-Road Automated Driving (ORAD) committees on Verification and Validation and am tasked for the creation of SAE standards for simulation based testing of automated driving to enable manufacturers to exchange information more efficiently reducing costs. |
Description | UK Government Transport Committee on the Subject of Self-driving vehicles |
Geographic Reach | National |
Policy Influence Type | Participation in a guidance/advisory committee |
Description | UNECE VMAD SG2 |
Geographic Reach | Multiple continents/international |
Policy Influence Type | Membership of a guideline committee |
Impact | I am the UK representative on United Nations Economic Commission for Europe VMAD SG2 committee on creating regulations for simulation requirements. |
Description | Integrated 4D driver modelling under uncertainty (i4Driving) |
Amount | £6,766,958 (GBP) |
Funding ID | UKRI 10038791 / EC 101076165 |
Organisation | United Kingdom Research and Innovation |
Sector | Public |
Country | United Kingdom |
Start | 10/2022 |
End | 09/2025 |
Description | ReliablE in-Vehicle pErception and decisioN-making in complex environmenTal conditionS (EVENTS) |
Amount | £6,920,598 (GBP) |
Funding ID | UKRI 10044773/ EC 101069614 |
Organisation | United Kingdom Research and Innovation |
Sector | Public |
Country | United Kingdom |
Start | 09/2022 |
End | 08/2025 |
Description | Safety assUraNce fRamework for connected, automated mobIlity SystEms (SUNRISE) |
Amount | £13,455,866 (GBP) |
Funding ID | UKRI 10044530/ EC 101069573 |
Organisation | United Kingdom Research and Innovation |
Sector | Public |
Country | United Kingdom |
Start | 09/2022 |
End | 08/2025 |
Description | National Institute of Standards and Technology (NIST), USA |
Organisation | National Institute of Standards & Technology (NIST) |
Country | United States |
Sector | Public |
PI Contribution | Collaboration on Automated Driving System Safety Measurement and role of Operational Design Domain. |
Collaborator Contribution | Introducing the concept of Operational Envelope Specification (OES). |
Impact | ADS Safety Measurement and Operational Design Domain Workshop presentation |
Start Year | 2020 |
Description | Transport Canada |
Organisation | Transport Canada |
Country | Canada |
Sector | Public |
PI Contribution | Undertaken research in two areas: - Scenario based testing in the Canadian context - Qualification of Virtual Testing Tools for Automated Driving Systems |
Collaborator Contribution | Influencing research to maintain policy relevance and using the outcomes of the research in policy development |
Impact | Internal reports and collaborative publications |
Start Year | 2021 |
Description | UK Department for Transport CAVPASS programme |
Organisation | Department of Transport |
Department | DfT, DVLA and VOSA |
Country | United Kingdom |
Sector | Public |
PI Contribution | Two work packages of research to influence the UK's policy work on Connected and automated vehicles: process for assuring safety and security (CAVPASS). The two work packages are: - Defining a Codified Behavioural Ruleset - Scenario Generation, Selection and Coverage |
Collaborator Contribution | Linking and influencing our research to wider UK policy |
Impact | Two reports that will be published |
Start Year | 2021 |
Description | World Economic Forum |
Organisation | World Economic Forum |
Country | Switzerland |
Sector | Charity/Non Profit |
PI Contribution | Member of Steering group and the technical working group of World Economic Forum's Safe Drive Initiative. |
Collaborator Contribution | Safe Drive Initiative |
Impact | Safe Drive Initiative: SafeDI Scenario-Based AV Policy Framework - an overview for policymakers |
Start Year | 2020 |
Description | Blogs |
Form Of Engagement Activity | Engagement focused website, blog or social media channel |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | I published four blogs: 1. Phive (5) Ps of a PhD 2. The Curious Case of Operational Design Domain: What it is and is not? 3. Why is it wrong to 'just' talk about diversity? 4. Failure is the only option. |
Year(s) Of Engagement Activity | 2020 |
Description | CertiCAV - Dept. for Transport project |
Form Of Engagement Activity | A formal working group, expert panel or dialogue |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | I am part of the Dept. for Transport CertiCAV project responsible for providing a framework for Certification of Automated Driving Systems. As a result of this project, I was involved in 2 workshops. |
Year(s) Of Engagement Activity | 2020 |
Description | ISO 22737 standard |
Form Of Engagement Activity | A formal working group, expert panel or dialogue |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Industry/Business |
Results and Impact | ISO 22737 international standard on low-speed automated driving (LSAD) systems. World's first SAE level 4 standard. I served as the technical author of the standard. |
Year(s) Of Engagement Activity | 2021 |
URL | https://www.iso.org/standard/73767.html |
Description | Member, Department of Transport Scientific Advisory Council (SAC) |
Form Of Engagement Activity | A formal working group, expert panel or dialogue |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Policymakers/politicians |
Results and Impact | Along with the other members of the Scientific Advisory Council (SAC), this roles ensures UK Policy is informed by research. Specific contributions made in Safety Assurance of Automated Mobility, including a series of events and workshops on cross-domain safety assurance for automated mobility, which will inform the work of the SAC |
Year(s) Of Engagement Activity | 2022,2023 |
URL | https://www.gov.uk/government/groups/dft-science-advisory-council |