RoboTest: : Systematic Model-Based Testing and Simulation of Mobile Autonomous Robots

Lead Research Organisation: University of York
Department Name: Computer Science

Abstract

Mobile and autonomous robots have an increasingly important role in industry and the wider society; from driverless vehicles to home assistance, potential applications are numerous. The UK government identified robotics as a key technology that will lead us to future economic growth (tinyurl.com/q8bhcy7). They have recognised, however, that autonomous robots are complex and typically operate in ever-changing environments (tinyurl.com/o2u2ts7). How can we be confident that they perform useful functions, as required, but are safe?

It is standard practice to use testing to check correctness and safety. The software-development practice for robotics typically includes testing within simulations, before robots are built, and then testing of the actual robots. Simulations have several benefits: we can test early, and test execution is cheaper and faster. For example, simulation does not require a robot to move physically. Testing with the real robots is, however, still needed, since we cannot be sure that a simulation captures all the important aspects of the hardware and environment.

In the current scenario, test generation is typically manual; this makes testing expensive and unreliable, and introduces delays. Manual test generation is error-prone and can lead to tests that produce the wrong verdict. If a test incorrectly states that the robot has a failure, then developers have to investigate, with extra cost and time. If a test incorrectly states that the robot behaves as expected, then a faulty system may be released. Without a systematic approach, tests may also identify infeasible environments; such tests cannot be used with the real robot. To make matters worse, manual test generation limits the number of tests produced.

All this affects the cost and quality of robot software, and is in contrast with current practice in other safety-critical areas, like the transport industry, which is highly regulated. Translation of technology, however, is not trivial. For example, lack of a driver to correct mistakes or respond to unforeseen circumstances leads to a much larger set of working conditions for an autonomous vehicle. Another example is provided by probabilistic algorithms, which make the robot behaviour nondeterministic, and so, difficult to repeat in testing and more difficult to characterise as correct or not.

We will address all these issues with novel automated test-generation techniques for mobile and autonomous robots. To use our techniques, a RoboTest tester constructs a model of the robot using a familiar notation already employed in the design of simulations and implementations. After that, instead of spending time designing simulation scenarios, the RoboTest tester, with the push of a button, generates tests. With RoboTest, testing is cheaper, since it takes less time, and is more effective, because the RoboTest tester can use many more tests, especially when using a simulation.

To execute the tests, the RoboTest tester can choose from a few simulators employing a variety of approaches to programming. Execution of the tests also follows the push of a button. Yet another button translates simulation to deployment tests. So, the RoboTest tester can trace back the results from the deployment tests to the simulation and the original model. So, the RoboTest tester is in a strong position to understand the reality gap between the simulation and the real world.

The RoboTest tester knows that the verdicts for the tests are correct, and understands what the testing achieves; for example, it can be guaranteed to find faults of an identified class. So, the RoboTest tester can answer the very difficult question: have we tested enough?

In conclusion, RoboTest will move the testing of mobile and autonomous robots onto a sound footing. RoboTest will make testing more efficient and effective in terms of person effort, and so, achieve longer term reduced costs.
 
Description This is the same as EP/R025134/1.
Exploitation Route This is the same as EP/R025134/1.
Sectors Other

 
Description 1000 Talents Programme
Amount ¥1,950,000 (CNY)
Organisation National Science Foundation China 
Sector Public
Country China
Start 11/2019 
End 11/2024
 
Description Aarhus University Centre for Digital Twins
Amount kr 970,392 (DKK)
Funding ID 2018-019 
Organisation Aarhus University 
Sector Academic/University
Country Denmark
Start 11/2019 
End 11/2024
 
Description Chair in Emerging Technologies
Amount £1,300,000 (GBP)
Organisation Royal Academy of Engineering 
Sector Charity/Non Profit
Country United Kingdom
Start 03/2018 
End 02/2028
 
Description Aarhus University Centre for Digital Twins 
Organisation Aarhus University
Department School of Engineering
Country Denmark 
Sector Academic/University 
PI Contribution Research collaboration with Professor Peter Gorm Larsen at Aarhus University's Centre for Digital Twins. Appointment as Adjunct Professor with support for activities, including regular visits. Appointment of Larsen as Visiting Professor at York.
Collaborator Contribution Collaboration on research papers and research proposals. Joint supervision of students at Aarhus.
Impact Ongoing collaboration, outputs in progress,
Start Year 2019
 
Description Adjunct Professorship, Southwest University, Chongqing, China 
Organisation Southwest University
Country China 
Sector Academic/University 
PI Contribution Collaboration on a joint research project on human-cyber-physical systems. Appointment as Adjunct professor at Southwest University. Supervision of joint research students at Southwest University. Personal research fund for JCPW at Southwest University.
Collaborator Contribution Joint research papers and research proposals. Proposed joint research laboratory. Joint direction of FM4AI research programme.
Impact Collaboration has just started.
Start Year 2019
 
Description CSP-based testing 
Organisation University of Bremen
Country Germany 
Sector Academic/University 
PI Contribution Expertise in CSP and its semantics
Collaborator Contribution Expertise on testing and automation
Impact J. Peleska, W. l. Huang, and A. L. C. Cavalcanti. Finite complete suites for CSP refinement testing. Science of Computer Programming, 179:1 -- 23, 2019. A. L. C. Cavalcanti, W.-L. Huang, J. Peleska, and J. C. P. Woodcock. CSP and Kripke Structures. In M. Leucker, C. Rueda, and F. D. Valencia, editors, Theoretical Aspects of Computing, pages 505--523. Springer, 2015.
Start Year 2015
 
Description Collaboration on RoboSim language: semantics, verification, examples, tools. 
Organisation Federal University of Pernambuco
Country Brazil 
Sector Academic/University 
PI Contribution Training on robotics
Collaborator Contribution Partnership in the design and implementation of RoboSim
Impact Joint papers.
Start Year 2017
 
Description Simomics Robotics 
Organisation Simomics Ltd
Country United Kingdom 
Sector Private 
PI Contribution Expertise on modelling and verification of robotics systems.
Collaborator Contribution Tools for modelling auditing and assurance cases.
Impact A tool is under development.
Start Year 2020
 
Description Sound simulation of robotic applications 
Organisation Federal University of Pernambuco
Country Brazil 
Sector Academic/University 
PI Contribution We have brought to them expertise on the results and approaches to modelling and verifying robotic applications, and developing simulations.
Collaborator Contribution They have brought in experience in model-based software engineering, compositional verification, testing, and probabilistic model checking. They have employed a research assistant to to work on the project and collaborate with our group. Together, we are developing a domain-specific language for modelling simulations. The language has a formal semantics under development. They are in charge of developing tool support for this language, and developing tool support for probabilistic reasoning.
Impact Tools are under development and papers are submitted.
Start Year 2017
 
Description Transport Systems Catapult - Autonomous Vehicle 
Organisation Transport Systems Catapult
Country United Kingdom 
Sector Private 
PI Contribution Modelling, analysis and simulation of the control system of an autonomous vehicle. Using the tools and techniques developed in RoboCalc, we will explore the limits of what is possible to model, analyse and simulate with respect to a complex realistic controller where safety concerns are of utmost importance.
Collaborator Contribution They are experts in the development of transport systems. As well as their expertise, they will provides us with documentation and source code regarding the specific case study.
Impact No outputs yet.
Start Year 2017
 
Description bangalore-2018 
Organisation Indian Institute of Science
Country India 
Sector Academic/University 
PI Contribution Newton grant from RAEng held by IIIT-B. Contributions on refinement of simulink models.
Collaborator Contribution Research in refinement of simulink models.
Impact Simon Foster, Kangfeng Ye, Ana Cavalcanti, Jim Woodcock: Calculational Verification of Reactive Programs with Reactive Relations and Kleene Algebra. RAMiCS 2018: 205-224 Simon Foster, Kangfeng Ye, Ana Cavalcanti, Jim Woodcock: Calculational Verification of Reactive Programs with Reactive Relations and Kleene Algebra. CoRR abs/1806.02101 (2018)
Start Year 2018
 
Description butterfield-tcd-march-2018 
Organisation Trinity College Dublin
Department Trinity Biomedical Sciences Institute
Country Ireland 
Sector Hospitals 
PI Contribution Research discussions at TCD.
Collaborator Contribution Research discussions.
Impact Discussions on future research collaborations.
Start Year 2018
 
Description chongqing-2-2018 
Organisation Southwest University
Country China 
Sector Academic/University 
PI Contribution Royal Society China Cost Share Programme
Collaborator Contribution Research contributions.
Impact None yet.
Start Year 2018
 
Description chongqing2018 
Organisation Southwest University
Country China 
Sector Academic/University 
PI Contribution Royal Society China Cost Share Programme.
Collaborator Contribution Rsearch contributions.
Impact No outputs yet.
Start Year 2018
 
Description digit-2018 
Organisation Aarhus University
Department Aarhus Institute of Advanced Studies
Country Denmark 
Sector Academic/University 
PI Contribution Planning for future collaboration.
Collaborator Contribution Research exchanges.
Impact Proposal to Grundfos Foundation.
Start Year 2018
 
Description peleska-testing-2018 
Organisation University of Bremen
Country Germany 
Sector Academic/University 
PI Contribution Rsearch discussions.
Collaborator Contribution Rsearch discussions.
Impact Research proposals.
Start Year 2018
 
Description CyPhyAssure Spring School 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Study participants or study members
Results and Impact Participation in Spring School on the assurance of cyber-physical systems.
Year(s) Of Engagement Activity 2019
URL https://www.cs.york.ac.uk/circus/CyPhyAssure/school/
 
Description DIGIT Advisory Board 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Study participants or study members
Results and Impact Participation in the advisory board of the center for digital twins at Aarhus University,
Year(s) Of Engagement Activity 2018
 
Description FM 2019 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Study participants or study members
Results and Impact Talks and discussions with international colleagues.
Year(s) Of Engagement Activity 2019
 
Description Huawei Research Forum 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Industry/Business
Results and Impact Presentation on formal methods for assurance of security-related software in Huawei 5G products.
Year(s) Of Engagement Activity 2019
 
Description Huawei Research Symposium 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Industry/Business
Results and Impact Presentation of my research agenda to Huawei engineers. I gave advice on future research policy for their company.
Year(s) Of Engagement Activity 2019
 
Description IFIP WG 2.3 Programming Methodology 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Study participants or study members
Results and Impact Hosted meeting of IFIP WG 2.3 on Programming Methodology. Organised scientific and social agenda, chaired the meeting, and gave an extended scientific talk. I involved my doctoral students and research assistants in the meeting.
Year(s) Of Engagement Activity 2019
URL https://ifip-tc2-wg23.paluno.uni-due.de/?page_id=455
 
Description Invited lecture University of Leicester 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Postgraduate students
Results and Impact Present the research at a level accessible to young researchers.
Year(s) Of Engagement Activity 2018
URL https://www2.le.ac.uk/departments/informatics/news/researchevents/external
 
Description Invited talk at HIS (High- Integrity Software) 2019 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Industry/Business
Results and Impact Now in its sixth year, the mission of the High Integrity Software conference is to share challenges, best practice and experience between software engineering practitioners. The conference features talks from industrial and academic specialists which disseminate experience and knowledge of important techniques and methods that are applicable across industry sectors. I have presented the technology developed in our researcher. It sparked questions and discussion, afterwards, and later contacts with a view to exchange ideas and collaborate.
Year(s) Of Engagement Activity 2019
URL https://www.his-2017.co.uk/
 
Description Newton Institue Workshop 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Study participants or study members
Results and Impact I gave a talk about the impact of verification technology on practical application.
Year(s) Of Engagement Activity 2019
 
Description Opening of Aarhus Centre for Digital Twins 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Industry/Business
Results and Impact Keynote talk on research direction at the opening of the Centre for Digital Twins at Aarhus University.
Year(s) Of Engagement Activity 2019
URL https://digit.au.dk/research-projects/centre-for-digital-twins/
 
Description Presentation at IFIP WG 2.3 Meeting 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Recent work on priorities, related to the formal semantics of RoboChart and RoboSim, was presented to the Working Group on Programming Methodology, as well as to the invited speakers. The goal of the presentation was to disseminate our work and receive feedback on ongoing research.
Year(s) Of Engagement Activity 2019
 
Description Presentation at IFIP WG 2.3 Meeting 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Our work on modelling for robotics was presented to the members of the Working Group on Programming Methodology, as well as to the invited observers. The goal of the presentation was to disseminate our work and receive feedback on current research.
Year(s) Of Engagement Activity 2019
 
Description Presentation at Pint of Science Festival 2018 in York 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact The Pint of Science festival is run every year bringing researchers to local pubs to present their work to the general public. In this talk, we motivated our work using robotic demonstrators and presented our main results. The talk was followed by a number of interesting questions from the public.
Year(s) Of Engagement Activity 2018
URL https://pintofscience.co.uk/event/are-you-sure-you-are-safe
 
Description Presentation at Pint of Science Festival 2019 in York 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact The Pint of Science festival is run every year bringing researchers to local pubs to present their work to the general public. In this talk, we presented our work on modelling and verifying the robot software, with particular focus on application of our approach to robot swarms. The talk was followed by a number of interesting questions from the public.
Year(s) Of Engagement Activity 2019
URL https://pintofscience.co.uk/event/-blurred-lines-boundaries-between-man-and-machine
 
Description Presentation at meeting with BSI 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact I presented an overview of the RoboCalc and RoboTest projects at a meeting with British Standards Institution and the Assuring Autonomy International Programme at York, which discussed standards in AI, robotics and autonomy.
Year(s) Of Engagement Activity 2018
 
Description Presentation of project to Thales 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Industry/Business
Results and Impact Presentation of the project to consider possible collaboration. Six colleagues from Thales attended.
Year(s) Of Engagement Activity 2019
 
Description Public talk as part of YorkTalks 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact About 300 people attended an event in the university The talks sparked questions and discussion afterwards, with the public, and with other colleagues involved in the event. The aim of the overall event is to showcase the very best of the University's research across seven research themes in order to exemplify our commitment to research excellence which forms a key component of the University Strategy and the Research Strategy.
Year(s) Of Engagement Activity 2020
URL https://www.york.ac.uk/research/events/yorktalks/
 
Description RoboSoft: software engineering for robotics 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Professional Practitioners
Results and Impact RoboSoft was a two-day event that will brought together researchers working on software engineering for robotics, developers working on the next generation of robotic systems in various areas of application, and regulation authorities interested in safety of robotic systems. RoboSoft provides a platform to establish software engineering for robotics as an important discipline in computer science and electronic engineering. The objective was to establish a common understanding of the challenges that need to be addressed to ensure that results are relevant to industrial practice and regulation. The group organised the whole event and gave three presentations.
Year(s) Of Engagement Activity 2019
URL https://www.raeng.org.uk/events/events-programme/2019/november/robosoft-software-engineering-for-rob...
 
Description Royal Academy of Engineering Research Forum 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Other audiences
Results and Impact The Research Forum is a highlight of the Academy's activities. It showcases the world-class engineering research funded through the Academy's wide range of programmes. The event consists of a mix of keynote talks from current awardees and alumni. Awardees from across the portfolio of Academy's research schemes will also exhibit posters and demos throughout the day. The project was presented in a keynote talk.
Year(s) Of Engagement Activity 2019
URL https://www.raeng.org.uk/events/events-programme/2019/november/research-forum
 
Description Symposium for Marie-Claude Gaudel 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Study participants or study members
Results and Impact I gave a talk on software engineering for robotic control.
Year(s) Of Engagement Activity 2019
 
Description Talk at Verification Future Meeting 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Industry/Business
Results and Impact I gave an invited talk at this industry meeting.
Year(s) Of Engagement Activity 2019
URL https://www.testandverification.com/conferences/verification-futures/vf2019
 
Description Visit by Jan Pekeska 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Discussions on research collaboration.
Year(s) Of Engagement Activity 2018
 
Description Visit by Ziming Liiu 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Study participants or study members
Results and Impact Visit to discuss long-term research collaboration.
Year(s) Of Engagement Activity 2018
 
Description Visit from Southwest University 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Study participants or study members
Results and Impact Visit from a delegation from Southwest University, Chongqing, China.
Year(s) Of Engagement Activity 2019
 
Description Visit from TCD Ireland 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Visit from colleagues from Trinity College Dublin. Contributions on probabilistic semantic for robotics.
Year(s) Of Engagement Activity 2018,2019
 
Description Visit ti Aarhus University 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Study participants or study members
Results and Impact Visit to exchange research ideas.
Year(s) Of Engagement Activity 2018
 
Description Visit to Chongqing 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact Workshop on formal methods for human-cyber-=physical systems. I gave talks on software engineering for robotics.
Year(s) Of Engagement Activity 2019
 
Description Visit to Chongqing 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Study participants or study members
Results and Impact Visit Southwest University to give lectures and direct doctoral students.
Year(s) Of Engagement Activity 2019
 
Description Visit to IISc 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Study participants or study members
Results and Impact Presentation of research work. Exchange of research ideas for future research proposals.
Year(s) Of Engagement Activity 2018
 
Description Visit to IISc Bangalore 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Study participants or study members
Results and Impact Workshop at IISc funded by Royal Academy of Engineering. Discussed work on formal verification of operating system real-time kernels.
Year(s) Of Engagement Activity 2019
 
Description Visit to Malardalen University 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact External examining of a doctoral student. Interaction with academic colleagues on research directions. Interaction with industry colleagues on research directions.
Year(s) Of Engagement Activity 2019
 
Description Visit to Southwest University 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Study participants or study members
Results and Impact Visit to exchange ideas.
Year(s) Of Engagement Activity 2018
 
Description White Rose Retreat on Robotics 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Study participants or study members
Results and Impact Discussed future collaborative in robotics.
Year(s) Of Engagement Activity 2019
 
Description Workshop for Advisory Board and additional representatives from academia and industry 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Industry/Business
Results and Impact Meeting of the Advisory Board to report on our results from RoboCalc and kick-off RoboTest. The joint meeting of the projects was very positive. There was a lot of discussion and good feedback. Further colleagues from industry asked to join the Advisory Board afterwards.
Year(s) Of Engagement Activity 2018
 
Description YorRobots Exhibition 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact More than 100 researchers and industrialists working in the area of robotics attended a one-day exhibition involving talks, posters, and demonstrations of work carried out in the University of York and in industry around the UK. There was plenty of opportunity for discussion and engagement. I have presented the results and the vision for RoboCalc and RoboTest, and beyond.
Year(s) Of Engagement Activity 2020
URL https://www.york.ac.uk/yorrobots/news-events/yorrobots-events/2020/exhibition-2020/
 
Description YorRobots launch event 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Other audiences
Results and Impact This was an afternoon event organised for scientists across nine departments of the university: Computer Science, Electronic Engineering, Physics, Chemistry, Environment, Music, Law, Philosophy, and Mathematics. We presented our work, and as a result created a network of scientists in robotics at York.
Year(s) Of Engagement Activity 2019
URL https://www.york.ac.uk/yorrobots/