Testing Autonomous Vehicle Software using Situation Generation

Autonomous vehicles (AVs) must be controlled by software, and such software thus has responsibility for safe vehicle behaviour. It is therefore essential that we rigorously test such software. This is difficult to do for AVs, as they have to respond appropriately to a great diversity of external situations as they go about their missions.

It is possible to find faults in an AV software specification by testing its behaviour in a variety of external situations, either in reality or in computer simulation. Such testing may reveal that the specification ignores certain situations (e.g. negotiating a motorway contraflow lane) or defines behaviour that is unsafe in a subset of situations (e.g. its policy for adapting to icy surfaces leads to unsafe speed control in crowded urban environments).

This project will test the hypothesis that testing based on coverage of possible external situations ("situation coverage") is an effective means of finding AV specification faults. We will test the hypothesis by creating a tool that generates situations for simulated AVs, both randomly and using heuristic search, and assessing whether higher situation coverage correlates with greater success at revealing seeded specification faults. (For the search, the fitness function will be based on the situation coverage achieved)

The project will draw on previous work on test coverage measures, on search-based testing, and on automated scenario generation in training simulations. To assess the effectiveness of the approach, we will use a small but practically-motivated case study of an autonomous ground vehicle, informed by the advice of an advisory panel set up for this project.

The short-term result of the project will be a technique, supported by proof-of-concept tools, for AV testing by means of situation generation. Developers of AVs and AV algorithms will be able to use developments of this technique to get better understanding of the limitations, weaknesses and risks in their technology.

In the medium term, development of this and similar techniques will increase industry's ability to justify the safety of their AV systems, and thus convince UK regulators to allow more adventurous (inherently risky) technologies and systems, thus maintaining our competitive edge. This will have benefits for potential users of AVs, including the military, law enforcement, and many industries.

These techniques will be adaptable to non-vehicle domains, such as service robotics, which will benefit the potential users of such robots - industry, retail and people with physical disabilities.

A broader scientific benefit will be a general insight into the kind of situations that give rise to problems for AV controllers, and also the kinds of specification faults in them that are difficult to find. This will stimulate researchers in AV technologies to resolve these problems, thus helping to advance what is possible.

In the long term, situation coverage could eventually form part of certification criteria for autonomous vehicle control systems (as MC/DC test coverage does for software under the DO-178B standard for avionics). This would allow its use across many industries and domains.