Comparing Classifiers Correctly

Finding an effective way to predict defect-prone parts of software systems has been a longstanding challenge for software engineering researchers. This would assist both the allocation of scarce software testing resources and also guide decision-making concerning when to release software systems. Thus it's no surprise that defect prediction has attracted a great deal of research attention with many hundreds of published studies.

In order to combine these studies, researchers need a sound means of comparing predictive performance and currently this is surprisingly problematic. Currently, the popular methods are biased (and therefore unreliable) and also difficult to interpret.

By addressing this need, this proposed research will help unlock the potential value of the large number of studies, primarily using machine learning methods, on defect prediction. This will yield more trustworthy and therefore actionable results meaning that practitioners can be better guided as how to find and therefore fix defects.

This travel grant will enable close collaboration between Profs Shepperd and MacDonell to work intensively on this problem.

Software is now ubiquitous and impinges upon almost every aspect of our lives. So naturally the quality of software is hugely important, hence the ability to predict potentially defect-prone parts of such systems is vital. Accurate costs of software defects are hard to come by, however NIST estimated in 2012 that defects cost the US economy of the order of $59bn annually. Elsewhere it is estimated that North American businesses lose $26.5bn in revenue each year due to IT downtime (Information Week, 24th May 2011). This of course ignores the even more serious non-monetary costs of harm to life and health of software failure.

So it's no surprise that software engineering researchers are harnessing machine learning and data science developments to attack this challenge of pinpointing software defects. To do this many suggestions have been made but we must compare techniques experimentally. However, such work is undermined if we can't adequately compare techniques or assess likely predictive performance.

There are two areas of impact. In the short term the impact will be upon academic practice but the intention, in the medium term, is for industrial impact once we can make more reliable recommendations as to which techniques are most suitable for defect prediction. One vehicle for this is via the existing EPSRC-funded network "Fault analyses in industry and academic research" at Brunel which is provides a forum for researchers AND practitioners.