RoboTest: : Systematic Model-Based Testing and Simulation of Mobile Autonomous Robots

Lead Research Organisation: University of York
Department Name: Computer Science

Abstract

Mobile and autonomous robots have an increasingly important role in industry and the wider society; from driverless vehicles to home assistance, potential applications are numerous. The UK government identified robotics as a key technology that will lead us to future economic growth (tinyurl.com/q8bhcy7). They have recognised, however, that autonomous robots are complex and typically operate in ever-changing environments (tinyurl.com/o2u2ts7). How can we be confident that they perform useful functions, as required, but are safe?

It is standard practice to use testing to check correctness and safety. The software-development practice for robotics typically includes testing within simulations, before robots are built, and then testing of the actual robots. Simulations have several benefits: we can test early, and test execution is cheaper and faster. For example, simulation does not require a robot to move physically. Testing with the real robots is, however, still needed, since we cannot be sure that a simulation captures all the important aspects of the hardware and environment.

In the current scenario, test generation is typically manual; this makes testing expensive and unreliable, and introduces delays. Manual test generation is error-prone and can lead to tests that produce the wrong verdict. If a test incorrectly states that the robot has a failure, then developers have to investigate, with extra cost and time. If a test incorrectly states that the robot behaves as expected, then a faulty system may be released. Without a systematic approach, tests may also identify infeasible environments; such tests cannot be used with the real robot. To make matters worse, manual test generation limits the number of tests produced.

All this affects the cost and quality of robot software, and is in contrast with current practice in other safety-critical areas, like the transport industry, which is highly regulated. Translation of technology, however, is not trivial. For example, lack of a driver to correct mistakes or respond to unforeseen circumstances leads to a much larger set of working conditions for an autonomous vehicle. Another example is provided by probabilistic algorithms, which make the robot behaviour nondeterministic, and so, difficult to repeat in testing and more difficult to characterise as correct or not.

We will address all these issues with novel automated test-generation techniques for mobile and autonomous robots. To use our techniques, a RoboTest tester constructs a model of the robot using a familiar notation already employed in the design of simulations and implementations. After that, instead of spending time designing simulation scenarios, the RoboTest tester, with the push of a button, generates tests. With RoboTest, testing is cheaper, since it takes less time, and is more effective, because the RoboTest tester can use many more tests, especially when using a simulation.

To execute the tests, the RoboTest tester can choose from a few simulators employing a variety of approaches to programming. Execution of the tests also follows the push of a button. Yet another button translates simulation to deployment tests. So, the RoboTest tester can trace back the results from the deployment tests to the simulation and the original model. So, the RoboTest tester is in a strong position to understand the reality gap between the simulation and the real world.

The RoboTest tester knows that the verdicts for the tests are correct, and understands what the testing achieves; for example, it can be guaranteed to find faults of an identified class. So, the RoboTest tester can answer the very difficult question: have we tested enough?

In conclusion, RoboTest will move the testing of mobile and autonomous robots onto a sound footing. RoboTest will make testing more efficient and effective in terms of person effort, and so, achieve longer term reduced costs.
 
Description This is the same as EP/R025134/1.
Exploitation Route This is the same as EP/R025134/1.
Sectors Other

 
Description Chair in Emerging Technologies
Amount £1,300,000 (GBP)
Organisation Royal Academy of Engineering 
Sector Learned Society
Country United Kingdom
Start 03/2018 
End 02/2028
 
Description Collaboration on RoboSim language: semantics, verification, examples, tools. 
Organisation Federal University of Pernambuco
Country Brazil 
Sector Academic/University 
PI Contribution Training on robotics
Collaborator Contribution Partnership in the design and implementation of RoboSim
Impact Joint papers.
Start Year 2017
 
Description Sound simulation of robotic applications 
Organisation Federal University of Pernambuco
Country Brazil 
Sector Academic/University 
PI Contribution We have brought to them expertise on the results and approaches to modelling and verifying robotic applications, and developing simulations.
Collaborator Contribution They have brought in experience in model-based software engineering, compositional verification, testing, and probabilistic model checking. They have employed a research assistant to to work on the project and collaborate with our group. Together, we are developing a domain-specific language for modelling simulations. The language has a formal semantics under development. They are in charge of developing tool support for this language, and developing tool support for probabilistic reasoning.
Impact Tools are under development and papers are submitted.
Start Year 2017
 
Description Transport Systems Catapult - Autonomous Vehicle 
Organisation Transport Systems Catapult
Country United Kingdom 
Sector Private 
PI Contribution Modelling, analysis and simulation of the control system of an autonomous vehicle. Using the tools and techniques developed in RoboCalc, we will explore the limits of what is possible to model, analyse and simulate with respect to a complex realistic controller where safety concerns are of utmost importance.
Collaborator Contribution They are experts in the development of transport systems. As well as their expertise, they will provides us with documentation and source code regarding the specific case study.
Impact No outputs yet.
Start Year 2017
 
Description bangalore-2018 
Organisation Indian Institute of Science
Country India 
Sector Academic/University 
PI Contribution Newton grant from RAEng held by IIIT-B. Contributions on refinement of simulink models.
Collaborator Contribution Research in refinement of simulink models.
Impact Simon Foster, Kangfeng Ye, Ana Cavalcanti, Jim Woodcock: Calculational Verification of Reactive Programs with Reactive Relations and Kleene Algebra. RAMiCS 2018: 205-224 Simon Foster, Kangfeng Ye, Ana Cavalcanti, Jim Woodcock: Calculational Verification of Reactive Programs with Reactive Relations and Kleene Algebra. CoRR abs/1806.02101 (2018)
Start Year 2018
 
Description butterfield-tcd-march-2018 
Organisation Trinity College Dublin
Department Trinity Biomedical Sciences Institute
Country Ireland 
Sector Hospitals 
PI Contribution Research discussions at TCD.
Collaborator Contribution Research discussions.
Impact Discussions on future research collaborations.
Start Year 2018
 
Description chongqing-2-2018 
Organisation Southwest University
Country China 
Sector Academic/University 
PI Contribution Royal Society China Cost Share Programme
Collaborator Contribution Research contributions.
Impact None yet.
Start Year 2018
 
Description chongqing2018 
Organisation Southwest University
Country China 
Sector Academic/University 
PI Contribution Royal Society China Cost Share Programme.
Collaborator Contribution Rsearch contributions.
Impact No outputs yet.
Start Year 2018
 
Description digit-2018 
Organisation Aarhus University
Department Aarhus Institute of Advanced Studies
PI Contribution Planning for future collaboration.
Collaborator Contribution Research exchanges.
Impact Proposal to Grundfos Foundation.
Start Year 2018
 
Description peleska-testing-2018 
Organisation University of Bremen
Country Germany 
Sector Academic/University 
PI Contribution Rsearch discussions.
Collaborator Contribution Rsearch discussions.
Impact Research proposals.
Start Year 2018
 
Description Invited lecture University of Leicester 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Postgraduate students
Results and Impact Present the research at a level accessible to young researchers.
Year(s) Of Engagement Activity 2018
URL https://www2.le.ac.uk/departments/informatics/news/researchevents/external
 
Description Presentation at IFIP WG 2.3 Meeting 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Our work on modelling for robotics was presented to the members of the Working Group on Programming Methodology, as well as to the invited observers. The goal of the presentation was to disseminate our work and receive feedback on current research.
Year(s) Of Engagement Activity 2019
 
Description Presentation at IFIP WG 2.3 Meeting 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Recent work on priorities, related to the formal semantics of RoboChart and RoboSim, was presented to the Working Group on Programming Methodology, as well as to the invited speakers. The goal of the presentation was to disseminate our work and receive feedback on ongoing research.
Year(s) Of Engagement Activity 2019
 
Description Presentation at Pint of Science Festival 2018 in York 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact The Pint of Science festival is run every year bringing researchers to local pubs to present their work to the general public. In this talk, we motivated our work using robotic demonstrators and presented our main results. The talk was followed by a number of interesting questions from the public.
Year(s) Of Engagement Activity 2018
URL https://pintofscience.co.uk/event/are-you-sure-you-are-safe
 
Description Presentation at meeting with BSI 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact I presented an overview of the RoboCalc and RoboTest projects at a meeting with British Standards Institution and the Assuring Autonomy International Programme at York, which discussed standards in AI, robotics and autonomy.
Year(s) Of Engagement Activity 2018
 
Description Presentation of project to Thales 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Industry/Business
Results and Impact Presentation of the project to consider possible collaboration. Six colleagues from Thales attended.
Year(s) Of Engagement Activity 2019
 
Description Workshop for Advisory Board and additional representatives from academia and industry 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Industry/Business
Results and Impact Meeting of the Advisory Board to report on our results from RoboCalc and kick-off RoboTest. The joint meeting of the projects was very positive. There was a lot of discussion and good feedback. Further colleagues from industry asked to join the Advisory Board afterwards.
Year(s) Of Engagement Activity 2018