The Birth, Life and Death of Semantic Mutants

Traditional Mutation Testing produces test cases that distinguish between some description N and variants of it. Each variant is produced by applying a mutation operator to N. A test set that is good at distinguishing N from variants of N is likely to be good at finding faults that are similar to applications of the mutation operators. Mutation testing was originally applied to programs but more recently it has been applied to other forms of descriptions such as specifications. Mutants are produced through the application of mutation operators, each of which may be applied to a relevant point in a program in order to produce a mutant. The mutation operators carry out small syntactic changes. For example, + might be replaced by -, > might be replaced by >=, a variable in an expression may be replaced by a constant, or part of an expression may be deleted. The mutation operators are designed to represent syntactically small errors. Typically, mutants are used to either judge the adequacy of a test set (does it distinguish between N and its mutants?) and also to drive test generation (we want a test set that distinguishes between N and its mutants).Traditional mutation testing produces mutants that represent small slips or mistakes in programming and thus represent a class of faults. A mutant program differs from the program under test by a small syntactic change (e.g. a / replaces a * ). However, real developers will also suffer from misunderstandings, especially when moving between description notations. They misapprehend the semantics of the description before them. They may, for example, import their understanding from a previously used programming language, or else from an understanding of how a particular tool interprets the notation. We believe that a semantically oriented mutation testing approach may assist in the discovery of such problems. We seek to show that a semantically oriented mutation testing approach is feasible and can find faults not found by traditional syntactic mutation (and likely, by other popular testing strategies).Misunderstanding the semantics of descriptive notations is a common source of problems in software development. We believe that these misunderstandings can be represented as semantic mutants over descriptions and that test data produced to kill semantic mutants is effective at finding faults caused by such misunderstandings: It will often find faults that are typically missed by test sets produced by extant testing strategies (and in particular, by test sets that are produced to kill traditional syntactic mutants). We also believe that he production of semantic mutants and the generation of test data to kill them can be automated.