RE-COST: REducing the Cost of Oracles for Software Testing

Testing involves examining the behaviour of a system in order to discover potential faults. The problem of determining the desired correct behaviour for a given input is called the Oracle Problem. Since manual testing is expensive and time consuming there has been a great deal of work on automation and part automation of Software Testing. Unfortunately, it is often impossible to fully automate the process of determining whether the system behaves correctly. This must be performed by a human, and the cost of the effort expended is referred to as the Human Oracle Cost.RE-COST will develop Search-Based Optimisation techniques to attack the Human Oracle Cost problem quantitatively and qualitatively. The quantitative approach will develop methods and algorithms to both reduce the number of test cases and the evaluation effort per test case. The qualitative approach will develop methods and algorithms that will reduce test case cognition time.The RE-COST project seeks to transform the way that researchers and practitioners think about the problem of Software Test Data Generation. This has the potential to provide a breakthrough in Software Testing, dramatically increasing real world industrial uptake of automated techniques for Software Test Data Generation.

The RECOST approach to reducing human effort targets the oracle effort. In current testing practice, the oracle is typically a team of engineers who have the burdensome task of checking copious software output for errors. This is tedious, costly and error-prone. Reducing the oracle cost will not only make software testing more reliable, it will alleviate some of this burden. It will facilitate a migration to the more rewarding and intellectually stimulating activities associated with software testing. RECOST will, therefore, directly impact on that part of society employed in software testing careers (which is often estimated to be about half of the software industry workforce). It will also have a longer term effect on society as a whole. Society depends critically on well-tested software, not merely for the familiar desktop applications and systems with which the public are well aware, but also the less visible software running in embedded controllers and devices. The 2009 EU ARTEMIS programme estimated that by 2016 there will be 40 billion such systems worldwide. These systems control transport, healthcare and military sectors and the Critical National Infrastructure. They pervade both the home and the workplace and so their successful operation is a critical concern. The RECOST project partners have an ambitious but realistic plan for ensuring that the project achieves maximum possible impact. To address the immediate academic and research communities, the project team will publish in the world's leading journals and conferences in Software Testing and Software Engineering. As the Case For Support demonstrates, the team has a strong track record of regular publication in all of these publication outlets and so this goal is demanding, but realistic. In order to reach out to the potential industrial users of the technologies that the project will develop, the team will partner both with the project's direct industrial collaborators and also the wider set of existing industrial collaborators of the CREST centre, UCL, and Sheffield departments of Computer Science. Both PIs, Harman and McMinn have many existing collaborations (and funding) from industry on which they will build during the the course of the RECOST project. In order to further widen this set of collaborators and initial technology adopters, the project team will seek to publish in trade press journals, and to present results at software developer events. Once again, the proposers have a strong track record of doing this. For example Prof. Harman has given several invited keynote talks at industrial Software Testing conferences, while Dr. McMinn is a key player in the development of Sheffield's Genesys initiative. Genesys is a stepping stone into industry for final year undergraduates and masters students, who will be among the first to use the techniques developed on the project. As the Case For Support demonstrates in more detail, the RECOST project team complement their outstanding record of publishing in the top international journals and conferences with an equally strong track record of industrial involvement in their research. For example, Prof. Harman's work on industrial impact was recognised by the EPSRC when it invited him to give the keynote talk on working with industry at the EPSRC Early Career Workshop in Information Communication Technology in November 2008. Finally, in order to reach out to the public at large, the RECOST proposers will seek to publish their work in popular science journals and will seek to participate in activities that offer opportunities to explain software testing issues to the general public. For example, the team will follow up on news stories relating to software problems, with short press releases that briefly explain the issues and point the interested reader to the RECOST website and relevant literature.