StaMInA: A Novel Competition to Drive the Comparative Evaluation of State Machine Inference Approaches

Software systems pervade modern life; they control everything from fly-by-wire aircraft and financial transfer systems to ABS breaking systems in cars and cooking modes in microwaves. The ability to understand these complex systems, and to make sure that they behave as expected, is crucial. State machines are a formal, diagrammatic notation that can be used to visualise behaviour of these systems in an accessible way. They can also be used as a basis for several rigorous and automated testing and verification techniques.Currently, state machines have to be designed and maintained by hand. This is an expensive and error-prone task, particularly when the system in question is constantly subject to change. Faced with this challenge, a substantial amount of research has been devoted to solving this problem with automated techniques; to automatically infer state machines of software systems, usually from samples of their behaviour. This has resulted in a multitude of proposed solutions from groups around the world.Although these advances are welcome, they have given rise to an important problem: There is no accepted process by which these techniques can be evaluated and compared against each other. There is no evidence to indicate which technique is better than the others, and why certain techniques excel. This in turn hampers further research in the area.With this project, we will address the above problems by organising an international competition to thoroughly compare and evaluate a diverse range of state machine inference techniques. The competition is especially novel because it will employ a range of techniques to compare the results of different techniques against each other. This will identify (a) which techniques are the most effective ones and (b) shed light on the possible reasons for their effectiveness. It is envisaged that this competition will become a regular event, driving research in the area.