The Birth, Life and Death of Semantic Mutants

Lead Research Organisation: University of York
Department Name: Computer Science

Abstract

Abstracts are not currently available in GtR for all funded research. This is normally because the abstract was not required at the time of proposal submission, but may be because it included sensitive information such as personal details.
 
Description Our work has concerned mutation testing. Suppose someone asked you to test a software system with the aim of discovering errors. You would supply it with inputs and judge whether the system behaved as expected, i.e. gave the 'right' outputs. But how would they know you had made a good fist of testing it - that the testing was sufficiently thorough? One way is to inject (one at a time) different faults into a system (i.e. make small syntactic code changes) and see whether the system with a fault injected into it behaves differently to the original system when subjected to your suite of tests.

If your test suite has at least one test that gives different behaviours then it has 'killed' the mutant. If an injected fault goes undetected by all tests, then it is said to be'live'. Live mutants either indicate that the injected fault is actually semantically the same as the original, or else your test suite just isn't thorough enough to discover it. If the latter you might usefully consider generating an additional test that can distinguish the original and mutant systems.

However, all mutant fault injection to date is syntactic - it aims by and large to discover programmer 'slips'. However, faults may be more subtle. In particular faults in a system may arise when two users simply have different interpretations of what the code or design actually represents. This may arise simply because the underpinning programmign languages are semantically ambiguous.

We have developed a prototype fault injection engine that aims to target those parts of a program or design that might prove problematic. Test can be generated to target those.

We have also developed test data generation technqiues based ion operations research based optimisation texchnqiues, that allows efficient mutant killing test data generation startegies to be discovered. Using a form of evolutionary computation we can 'home in' on highly effective test data generations strategies.

Thus, in collaboration with our project partner at Brunel, we are able to generate new forms of mutant to give a radically different yardstick for test thoroughness and we have potentially very strong methods of generatign test data to kill mutants (whether syntactic or semantic).
Exploitation Route Yes. The principles of our underpinning work leading to the journal paper Semantic Mutation Testing; can be extended to other systems and other notations. Although published in 2013, the paper has already attracted a fair amount of academic interest and citations. It represents both a novel contribution to mutation testing itself, but also clearly has practical applications. (In fact, the grant proposal was actually motivated by significantly practical concerns.)

We would hope that the ideas of mutation testing, semantic mutation testing and automated test data generation for mutation testign in teh context of multi-core systems will prove a promising route to impact. Multi-core systems will find application everywhere but we believe there are significant issues faced by the defence sector that makes that sector in real need of assistance. However, any modern software is likely to have to run on a multi-core chip (and so we identify IT as an exploitation sector too).
Sectors Aerospace, Defence and Marine,Digital/Communication/Information Technologies (including Software)
 
Description The work led to further recognition in mutate testing and a further industrially sponsored collaboration. Essentially we have shown that mutation is practical on real systems. A publication with the nuclear industry has resulted. Evaluation of Mutation Testing in a Nuclear Industry Case Study P Delgado-Pérez, I Habli, S Gregory, R Alexander, J Clark, I Medina-Bulo IEEE Transactions on Reliability 67 (4), 1406-1419
First Year Of Impact 2018
Sector Aerospace, Defence and Marine,Energy
Impact Types Policy & public services
 
Description CINIF Awards
Amount £15,000 (GBP)
Organisation Government of India 
Sector Public
Country India
Start 03/2016 
End 12/2016