ASSESSING THE QUALITY OF TEST SUITES IN INDUSTRIAL CODE ("AQUATIC")

Lead Research Organisation: Brunel University
Department Name: Computer Science

Abstract

Most computer-based systems deteriorate in quality as they age. This places an increasing burden on the developers of the system to ensure that a certain level of service is maintained. While the running system represents code that is constantly monitored, an aspect of systems that has received very little attention is its test code. This is code that ensures a level of correctness of the running system and is therefore an important aspect of a system. According to US-based NIST, more than a third of the cost of defects in the US IT industry could be avoided if better software testing was performed i.e., approximately $8 billion. In a large number of organisations, test code is at least as large as the code which it tests. Financially, it is of great importance that an organisation's test code is maintained well - test code has a strong mapping to the 'in production' code and changes to one usually require changes to the other.

The purpose of the AQUATIC project is to explore how test code decays, how it evolves and similarities and differences between test code and production code. The project adopts an empirical methodology and uses test and production code from two industrial partners as a basis. Data mining techniques are used to analyse the large volumes of data involved and to extract knowledge from the data. The project is supported by a set of deliverables and associated dissemination activities including a set of project progress Workshops.

Planned Impact

The following 'Pathways to Impact' lists the impact through knowledge advancement and the initiatives the AQUATIC project will use, the people aspects of the project, i.e., who is likely to benefit from the research and, finally, the economic and societal pathways through which the research impact will be felt.

PEOPLE PERSPECTIVES

From a people perspective, the project promotes a set of skills by the application of empirical analysis to large-scale software engineering artifacts. Through the initiatives described in this paragraph, these skills can be adopted by practitioners who wish to apply the same principles to their software code/fault bases. A range of people will also benefit from the research. Software architects, managers and staff at the industrial partners will benefit directly through results generated from the use and analysis of their Java code; the benefits from the research are demonstrable, quantifiable and validated with developers in each industrial partner in a fashion that few studies have been able to employ before. Academic researchers in the areas of software evolution will benefit from demonstration of the value that the transferable techniques hold and researchers in other aspects of software engineering will also benefit. Fields of reverse engineering, re-engineering and formal testing techniques are just a few of the relevant, related fields we anticipate to be informed by the results of the research. We see the project as an example of how collaboration between academia and industry can have an enduring and positive effect on both.

ECONOMIC PERSPECTIVES

From an economic viewpoint, faults in software will always occur and there will always be some cost associated with faults and fixing them when they occur. However, there are significant potential cost savings to be gained by UK IT industry if research can produce demonstrably successful testing techniques and practices which can be adopted to prevent faults. Firstly, resources can be pre-allocated and directed where they are most needed - to increase effort in refactoring on a proactive, rather than on a reactive basis which can cause disruption to other projects. Secondly, explanation of causes of code decay can improve and streamline the processes through which software is developed. For example, if the research reveals that one form of coupling is more prevalent than other coupling forms, then we can show how refactoring (or not doing refactoring) which addresses this aspect of system complexity can be introduced to help. There are compelling economic reasons for re-engineering test code so that it is less complex. Finally, the ability to understand trends in the evolution of a system (from a test and production code perspective) in a more timely fashion ultimately frees up time for developers to work on improving/perfecting the quality of their code. Put another way, there is a high opportunity cost to the time spent dealing with and fixing issues which might be prevented with greater knowledge of test structures; the proposed research has huge potential for unlocking the 'opportunity' that a more timely awareness and understanding of system pressures in test code create.

SOCIETAL PERSPECTIVES

From a societal viewpoint, the research is in alignment with a key aim of any company of developing robust code and high quality products. Customers of the industrial partners will benefit in terms of the software that they 'deliver'. The value of the research also appeals to the wider societal community. In fact, the research affects anyone who uses software since improvement in the quality of code and a better understanding of test processes has a direct influence on us, its users, who increasingly use software of different types (whether embedded, real-time, web-based) in everyday products and services.
 
Description That asserts in code are problematic and that test code suffers from the same issues as regular code
There are big differences in the characteristics of each.
Industry considers test code as a second rate type of code.
Exploitation Route More studies of industry code.
More analysis of the sources of defects need to be undertaken,
A taxonomy of defects is elusive and the area is highly under-researched
Sectors Digital/Communication/Information Technologies (including Software)

 
Description The findings have fed into another grant where a company has used a tool to find faults. This was cross-company.
First Year Of Impact 2017
Sector Creative Economy,Digital/Communication/Information Technologies (including Software)
Impact Types Economic

 
Description Brunel WAT scheme (Incentive Scheme)
Amount £2,000 (GBP)
Organisation Brunel University London 
Sector Academic/University
Country United Kingdom
Start 06/2017 
End 09/2018
 
Description Brunel WAT scheme (Incentive Scheme)
Amount £2,000 (GBP)
Organisation Brunel University London 
Sector Academic/University
Country United Kingdom
Start 07/2015 
End 07/2018
 
Description Exploiting Defect Prediction for Automatic Software Repair (Fixie)
Amount £367,237 (GBP)
Funding ID EP/S005730/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Academic/University
Country United Kingdom
Start 11/2018 
End 10/2021
 
Description Concrete collaborations with BaeSystems and outcome of Visiting Industry Research Fellow 
Organisation BAE Systems
Country United Kingdom 
Sector Private 
PI Contribution The FIAR-NET project has allowed a more formal footing with BAE Systems to be established and will result in a Fellowship submission to the EPSRC. We are in the process of inviting Dr Ken Wallace to become a Visiting Industry Fellow at Brunel.
Collaborator Contribution Invited talks and contributions to the University's Made In Brunel.
Impact Visiting Industry Fellowship in process.
Start Year 2014
 
Description University of Cagliari 
Organisation University of Cagliari
Country Italy 
Sector Academic/University 
PI Contribution I was Visiting Professor at the University of Cagliari - and from which a number of papers have emerged. I have also just submitted a grant application with the FlossLab which is a spin-off company from the University. I have also hosted a number of RFs at Brunel from the scheme.
Collaborator Contribution University of Cagliari hosted my trip and provided the mechanisms for collaboration.
Impact Please see publications list.
Start Year 2010