The Birth, Life and Death of Semantic Mutants

Lead Research Organisation: Brunel University
Department Name: Information Systems Computing and Maths

Abstract

Traditional Mutation Testing produces test cases that distinguish between some description N and variants of it. Each variant is produced by applying a mutation operator to N. A test set that is good at distinguishing N from variants of N is likely to be good at finding faults that are similar to applications of the mutation operators. Mutation testing was originally applied to programs but more recently it has been applied to other forms of descriptions such as specifications. Mutants are produced through the application of mutation operators, each of which may be applied to a relevant point in a program in order to produce a mutant. The mutation operators carry out small syntactic changes. For example, + might be replaced by -, > might be replaced by >=, a variable in an expression may be replaced by a constant, or part of an expression may be deleted. The mutation operators are designed to represent syntactically small errors. Typically, mutants are used to either judge the adequacy of a test set (does it distinguish between N and its mutants?) and also to drive test generation (we want a test set that distinguishes between N and its mutants).Traditional mutation testing produces mutants that represent small slips or mistakes in programming and thus represent a class of faults. A mutant program differs from the program under test by a small syntactic change (e.g. a / replaces a * ). However, real developers will also suffer from misunderstandings, especially when moving between description notations. They misapprehend the semantics of the description before them. They may, for example, import their understanding from a previously used programming language, or else from an understanding of how a particular tool interprets the notation. We believe that a semantically oriented mutation testing approach may assist in the discovery of such problems. We seek to show that a semantically oriented mutation testing approach is feasible and can find faults not found by traditional syntactic mutation (and likely, by other popular testing strategies).Misunderstanding the semantics of descriptive notations is a common source of problems in software development. We believe that these misunderstandings can be represented as semantic mutants over descriptions and that test data produced to kill semantic mutants is effective at finding faults caused by such misunderstandings: It will often find faults that are typically missed by test sets produced by extant testing strategies (and in particular, by test sets that are produced to kill traditional syntactic mutants). We also believe that he production of semantic mutants and the generation of test data to kill them can be automated.

Publications

10 25 50
publication icon
Clark J (2013) Semantic mutation testing in Science of Computer Programming

publication icon
Dan H (2011) Controllability Problems in MSC-Based Testing in The Computer Journal

publication icon
Dan H (2012) A framework for pathologies of message sequence charts in Information and Software Technology

publication icon
Dan H (2013) The Oracle Problem When Testing from MSCs in The Computer Journal

 
Description In mutation testing, the code of the system under test (SUT) S is mutated/changed to produce a set M of mutants that are similar to S. The quality of a set T of test cases, to be used to test S, is then assessed by determining how good T is at distinguishing between S and the mutants in M. The idea is that if T is good at showing that the mutants in M (which are similar to S) are not the same as S then it will also be good at finding faults in S.

Traditionally, mutation testing has used mutation operators that make small changes to the syntax of the code (i.e. how the code is written). For example, a statement such as x = y + z might be changed to x = y - z. These types of changes capture a class of mistakes that a programmer can make - those that lead to small syntactic mistakes. However, it does not capture semantic mistakes, in which the programmer misunderstands a feature of the programming language used. This project explored how mutation testing can instead be based on such mistakes in the understand of semantics, an approach we called semantic mutation testing.

One of the main outcomes of the project was a more detailed framework for semantic mutation testing along with a software tool that implements this for one of the most important programming languages (C). Semantic mutation testing was evaluate in the context of an important class of semantic mistakes (those associated with floating point computation). We also investigated semantic mistakes for a particular type of model, regarding how the SUT should behave (a message sequence chart or a sequence diagram), and how this can be used as the basis for software testing.
Exploitation Route First, the outcomes provide a framework and it is possible to explore other types of semantic mutants. We showed that the framework can be applied at the level of code (C) and also models (MSCs). For work that considers C code, there is also the potential to implement new mutation operators in our tool SMT-C or to apply a similar approach.
Sectors Digital/Communication/Information Technologies (including Software)