ASSESSING THE QUALITY OF TEST SUITES IN INDUSTRIAL CODE ("AQUATIC")

Most computer-based systems deteriorate in quality as they age. This places an increasing burden on the developers of the system to ensure that a certain level of service is maintained. While the running system represents code that is constantly monitored, an aspect of systems that has received very little attention is its test code. This is code that ensures a level of correctness of the running system and is therefore an important aspect of a system. According to US-based NIST, more than a third of the cost of defects in the US IT industry could be avoided if better software testing was performed i.e., approximately $8 billion. In a large number of organisations, test code is at least as large as the code which it tests. Financially, it is of great importance that an organisation's test code is maintained well - test code has a strong mapping to the 'in production' code and changes to one usually require changes to the other.

The purpose of the AQUATIC project is to explore how test code decays, how it evolves and similarities and differences between test code and production code. The project adopts an empirical methodology and uses test and production code from two industrial partners as a basis. Data mining techniques are used to analyse the large volumes of data involved and to extract knowledge from the data. The project is supported by a set of deliverables and associated dissemination activities including a set of project progress Workshops.

The following 'Pathways to Impact' lists the impact through knowledge advancement and the initiatives the AQUATIC project will use, the people aspects of the project, i.e., who is likely to benefit from the research and, finally, the economic and societal pathways through which the research impact will be felt.

PEOPLE PERSPECTIVES

From a people perspective, the project promotes a set of skills by the application of empirical analysis to large-scale software engineering artifacts. Through the initiatives described in this paragraph, these skills can be adopted by practitioners who wish to apply the same principles to their software code/fault bases. A range of people will also benefit from the research. Software architects, managers and staff at the industrial partners will benefit directly through results generated from the use and analysis of their Java code; the benefits from the research are demonstrable, quantifiable and validated with developers in each industrial partner in a fashion that few studies have been able to employ before. Academic researchers in the areas of software evolution will benefit from demonstration of the value that the transferable techniques hold and researchers in other aspects of software engineering will also benefit. Fields of reverse engineering, re-engineering and formal testing techniques are just a few of the relevant, related fields we anticipate to be informed by the results of the research. We see the project as an example of how collaboration between academia and industry can have an enduring and positive effect on both.

ECONOMIC PERSPECTIVES

From an economic viewpoint, faults in software will always occur and there will always be some cost associated with faults and fixing them when they occur. However, there are significant potential cost savings to be gained by UK IT industry if research can produce demonstrably successful testing techniques and practices which can be adopted to prevent faults. Firstly, resources can be pre-allocated and directed where they are most needed - to increase effort in refactoring on a proactive, rather than on a reactive basis which can cause disruption to other projects. Secondly, explanation of causes of code decay can improve and streamline the processes through which software is developed. For example, if the research reveals that one form of coupling is more prevalent than other coupling forms, then we can show how refactoring (or not doing refactoring) which addresses this aspect of system complexity can be introduced to help. There are compelling economic reasons for re-engineering test code so that it is less complex. Finally, the ability to understand trends in the evolution of a system (from a test and production code perspective) in a more timely fashion ultimately frees up time for developers to work on improving/perfecting the quality of their code. Put another way, there is a high opportunity cost to the time spent dealing with and fixing issues which might be prevented with greater knowledge of test structures; the proposed research has huge potential for unlocking the 'opportunity' that a more timely awareness and understanding of system pressures in test code create.

SOCIETAL PERSPECTIVES

From a societal viewpoint, the research is in alignment with a key aim of any company of developing robust code and high quality products. Customers of the industrial partners will benefit in terms of the software that they 'deliver'. The value of the research also appeals to the wider societal community. In fact, the research affects anyone who uses software since improvement in the quality of code and a better understanding of test processes has a direct influence on us, its users, who increasingly use software of different types (whether embedded, real-time, web-based) in everyday products and services.