ASSESSING THE QUALITY OF TEST SUITES IN INDUSTRIAL CODE ("AQUATIC")
Lead Research Organisation:
Brunel University London
Department Name: Computer Science
Abstract
Most computer-based systems deteriorate in quality as they age. This places an increasing burden on the developers of the system to ensure that a certain level of service is maintained. While the running system represents code that is constantly monitored, an aspect of systems that has received very little attention is its test code. This is code that ensures a level of correctness of the running system and is therefore an important aspect of a system. According to US-based NIST, more than a third of the cost of defects in the US IT industry could be avoided if better software testing was performed i.e., approximately $8 billion. In a large number of organisations, test code is at least as large as the code which it tests. Financially, it is of great importance that an organisation's test code is maintained well - test code has a strong mapping to the 'in production' code and changes to one usually require changes to the other.
The purpose of the AQUATIC project is to explore how test code decays, how it evolves and similarities and differences between test code and production code. The project adopts an empirical methodology and uses test and production code from two industrial partners as a basis. Data mining techniques are used to analyse the large volumes of data involved and to extract knowledge from the data. The project is supported by a set of deliverables and associated dissemination activities including a set of project progress Workshops.
The purpose of the AQUATIC project is to explore how test code decays, how it evolves and similarities and differences between test code and production code. The project adopts an empirical methodology and uses test and production code from two industrial partners as a basis. Data mining techniques are used to analyse the large volumes of data involved and to extract knowledge from the data. The project is supported by a set of deliverables and associated dissemination activities including a set of project progress Workshops.
Planned Impact
The following 'Pathways to Impact' lists the impact through knowledge advancement and the initiatives the AQUATIC project will use, the people aspects of the project, i.e., who is likely to benefit from the research and, finally, the economic and societal pathways through which the research impact will be felt.
PEOPLE PERSPECTIVES
From a people perspective, the project promotes a set of skills by the application of empirical analysis to large-scale software engineering artifacts. Through the initiatives described in this paragraph, these skills can be adopted by practitioners who wish to apply the same principles to their software code/fault bases. A range of people will also benefit from the research. Software architects, managers and staff at the industrial partners will benefit directly through results generated from the use and analysis of their Java code; the benefits from the research are demonstrable, quantifiable and validated with developers in each industrial partner in a fashion that few studies have been able to employ before. Academic researchers in the areas of software evolution will benefit from demonstration of the value that the transferable techniques hold and researchers in other aspects of software engineering will also benefit. Fields of reverse engineering, re-engineering and formal testing techniques are just a few of the relevant, related fields we anticipate to be informed by the results of the research. We see the project as an example of how collaboration between academia and industry can have an enduring and positive effect on both.
ECONOMIC PERSPECTIVES
From an economic viewpoint, faults in software will always occur and there will always be some cost associated with faults and fixing them when they occur. However, there are significant potential cost savings to be gained by UK IT industry if research can produce demonstrably successful testing techniques and practices which can be adopted to prevent faults. Firstly, resources can be pre-allocated and directed where they are most needed - to increase effort in refactoring on a proactive, rather than on a reactive basis which can cause disruption to other projects. Secondly, explanation of causes of code decay can improve and streamline the processes through which software is developed. For example, if the research reveals that one form of coupling is more prevalent than other coupling forms, then we can show how refactoring (or not doing refactoring) which addresses this aspect of system complexity can be introduced to help. There are compelling economic reasons for re-engineering test code so that it is less complex. Finally, the ability to understand trends in the evolution of a system (from a test and production code perspective) in a more timely fashion ultimately frees up time for developers to work on improving/perfecting the quality of their code. Put another way, there is a high opportunity cost to the time spent dealing with and fixing issues which might be prevented with greater knowledge of test structures; the proposed research has huge potential for unlocking the 'opportunity' that a more timely awareness and understanding of system pressures in test code create.
SOCIETAL PERSPECTIVES
From a societal viewpoint, the research is in alignment with a key aim of any company of developing robust code and high quality products. Customers of the industrial partners will benefit in terms of the software that they 'deliver'. The value of the research also appeals to the wider societal community. In fact, the research affects anyone who uses software since improvement in the quality of code and a better understanding of test processes has a direct influence on us, its users, who increasingly use software of different types (whether embedded, real-time, web-based) in everyday products and services.
PEOPLE PERSPECTIVES
From a people perspective, the project promotes a set of skills by the application of empirical analysis to large-scale software engineering artifacts. Through the initiatives described in this paragraph, these skills can be adopted by practitioners who wish to apply the same principles to their software code/fault bases. A range of people will also benefit from the research. Software architects, managers and staff at the industrial partners will benefit directly through results generated from the use and analysis of their Java code; the benefits from the research are demonstrable, quantifiable and validated with developers in each industrial partner in a fashion that few studies have been able to employ before. Academic researchers in the areas of software evolution will benefit from demonstration of the value that the transferable techniques hold and researchers in other aspects of software engineering will also benefit. Fields of reverse engineering, re-engineering and formal testing techniques are just a few of the relevant, related fields we anticipate to be informed by the results of the research. We see the project as an example of how collaboration between academia and industry can have an enduring and positive effect on both.
ECONOMIC PERSPECTIVES
From an economic viewpoint, faults in software will always occur and there will always be some cost associated with faults and fixing them when they occur. However, there are significant potential cost savings to be gained by UK IT industry if research can produce demonstrably successful testing techniques and practices which can be adopted to prevent faults. Firstly, resources can be pre-allocated and directed where they are most needed - to increase effort in refactoring on a proactive, rather than on a reactive basis which can cause disruption to other projects. Secondly, explanation of causes of code decay can improve and streamline the processes through which software is developed. For example, if the research reveals that one form of coupling is more prevalent than other coupling forms, then we can show how refactoring (or not doing refactoring) which addresses this aspect of system complexity can be introduced to help. There are compelling economic reasons for re-engineering test code so that it is less complex. Finally, the ability to understand trends in the evolution of a system (from a test and production code perspective) in a more timely fashion ultimately frees up time for developers to work on improving/perfecting the quality of their code. Put another way, there is a high opportunity cost to the time spent dealing with and fixing issues which might be prevented with greater knowledge of test structures; the proposed research has huge potential for unlocking the 'opportunity' that a more timely awareness and understanding of system pressures in test code create.
SOCIETAL PERSPECTIVES
From a societal viewpoint, the research is in alignment with a key aim of any company of developing robust code and high quality products. Customers of the industrial partners will benefit in terms of the software that they 'deliver'. The value of the research also appeals to the wider societal community. In fact, the research affects anyone who uses software since improvement in the quality of code and a better understanding of test processes has a direct influence on us, its users, who increasingly use software of different types (whether embedded, real-time, web-based) in everyday products and services.
Publications
Ortu M.
(2020)
Describing software developers affectiveness through Markov chain models
in Electronic Journal of Applied Statistical Analysis
Ajienka N
(2017)
An empirical study on the interplay between semantic coupling and co-change of software classes
in Empirical Software Engineering
Shepperd M
(2018)
The role and value of replication in empirical software engineering results
in Information and Software Technology
Ortu M
(2017)
How diverse is your team? Investigating gender and nationality diversity in GitHub teams
in Journal of Software Engineering Research and Development
Child M
(2019)
A comparison and evaluation of variants in the coupling between objects metric
in Journal of Systems and Software
Destefanis G
(2016)
Software development: do good manners matter?
in PeerJ Computer Science
Counsell S
(2018)
Re-visiting a Test Taxonomy with Refactoring and Defect-fix Data
Destefanis G
(2017)
On the randomness and seasonality of affective metrics for software development
Counsell S
(2017)
Assert Use and Defectiveness in Industrial Code
Amjed Tahir
(2016)
Conference Paper
Description | That asserts in code are problematic and that test code suffers from the same issues as regular code There are big differences in the characteristics of each. Industry considers test code as a second rate type of code. |
Exploitation Route | More studies of industry code. More analysis of the sources of defects need to be undertaken, A taxonomy of defects is elusive and the area is highly under-researched |
Sectors | Digital/Communication/Information Technologies (including Software) |
Description | The findings have fed into another grant where a company has used a tool to find faults, trialled at one company. This was also a cross-company project. Testing is a fundamental issue associated with refactoring and this thread of my research has led to connections with a large computer games company in London. A grant is being prepared on Debugging at the moment, with that company and two other companies. The grant has made us aware of the problem in many application domains and for future grant applications that will target those areas. Working on test-based research for some time has inspired a deeper look at debugging as a very under-estimated and under-researched topic. The work in AQUATIC has also led to other work on test and production code for the presence of "clones", a very topical area and of wide concern to industry, since good practice would suggest avoiding 'copy and pasting' as poor practice. See https://dzone.com/articles/copy-and-paste-programming The work also feeds into the current work on Automatic Program Repair, the topic of the Fixie project (EPSRC-funded) that I am currently CI on and the subject of a large grant being prepared. |
First Year Of Impact | 2015 |
Sector | Creative Economy,Digital/Communication/Information Technologies (including Software) |
Impact Types | Economic |
Description | Brunel WAT scheme (Incentive Scheme) |
Amount | £2,000 (GBP) |
Organisation | Brunel University London |
Sector | Academic/University |
Country | United Kingdom |
Start | 06/2015 |
End | 07/2018 |
Description | Brunel WAT scheme (Incentive Scheme) |
Amount | £2,000 (GBP) |
Organisation | Brunel University London |
Sector | Academic/University |
Country | United Kingdom |
Start | 05/2017 |
End | 09/2018 |
Description | Exploiting Defect Prediction for Automatic Software Repair (Fixie) |
Amount | £367,237 (GBP) |
Funding ID | EP/S005730/1 |
Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
Sector | Public |
Country | United Kingdom |
Start | 11/2018 |
End | 10/2022 |
Description | Collaboration with University of Otago |
Organisation | University of Otago |
Country | New Zealand |
Sector | Academic/University |
PI Contribution | We have ongoing research with a NZ based University on code smells. This was generated directly from the work in the AQUATIC project and the identification of code smells and how that influences unit test cases also. |
Collaborator Contribution | Paper writing |
Impact | ICPC paper - won Best Paper and an EMSE Journal output. |
Start Year | 2021 |
Description | Concrete collaborations with BaeSystems and outcome of Visiting Industry Research Fellow |
Organisation | BAE Systems |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | The FIAR-NET project has allowed a more formal footing with BAE Systems to be established and will result in a Fellowship submission to the EPSRC. We are in the process of inviting Dr Ken Wallace to become a Visiting Industry Fellow at Brunel. |
Collaborator Contribution | Invited talks and contributions to the University's Made In Brunel. |
Impact | Visiting Industry Fellowship in process. |
Start Year | 2014 |
Description | University of Cagliari |
Organisation | University of Cagliari |
Country | Italy |
Sector | Academic/University |
PI Contribution | I was Visiting Professor at the University of Cagliari - and from which a number of papers have emerged. I have also just submitted a grant application with the FlossLab which is a spin-off company from the University. I have also hosted a number of RFs at Brunel from the scheme. |
Collaborator Contribution | University of Cagliari hosted my trip and provided the mechanisms for collaboration. |
Impact | Please see publications list. |
Start Year | 2010 |