Lead Research Organisation: Brunel University London
Department Name: Computer Science


Most computer-based systems deteriorate in quality as they age. This places an increasing burden on the developers of the system to ensure that a certain level of service is maintained. While the running system represents code that is constantly monitored, an aspect of systems that has received very little attention is its test code. This is code that ensures a level of correctness of the running system and is therefore an important aspect of a system. According to US-based NIST, more than a third of the cost of defects in the US IT industry could be avoided if better software testing was performed i.e., approximately $8 billion. In a large number of organisations, test code is at least as large as the code which it tests. Financially, it is of great importance that an organisation's test code is maintained well - test code has a strong mapping to the 'in production' code and changes to one usually require changes to the other.

The purpose of the AQUATIC project is to explore how test code decays, how it evolves and similarities and differences between test code and production code. The project adopts an empirical methodology and uses test and production code from two industrial partners as a basis. Data mining techniques are used to analyse the large volumes of data involved and to extract knowledge from the data. The project is supported by a set of deliverables and associated dissemination activities including a set of project progress Workshops.

Planned Impact

The following 'Pathways to Impact' lists the impact through knowledge advancement and the initiatives the AQUATIC project will use, the people aspects of the project, i.e., who is likely to benefit from the research and, finally, the economic and societal pathways through which the research impact will be felt.


From a people perspective, the project promotes a set of skills by the application of empirical analysis to large-scale software engineering artifacts. Through the initiatives described in this paragraph, these skills can be adopted by practitioners who wish to apply the same principles to their software code/fault bases. A range of people will also benefit from the research. Software architects, managers and staff at the industrial partners will benefit directly through results generated from the use and analysis of their Java code; the benefits from the research are demonstrable, quantifiable and validated with developers in each industrial partner in a fashion that few studies have been able to employ before. Academic researchers in the areas of software evolution will benefit from demonstration of the value that the transferable techniques hold and researchers in other aspects of software engineering will also benefit. Fields of reverse engineering, re-engineering and formal testing techniques are just a few of the relevant, related fields we anticipate to be informed by the results of the research. We see the project as an example of how collaboration between academia and industry can have an enduring and positive effect on both.


From an economic viewpoint, faults in software will always occur and there will always be some cost associated with faults and fixing them when they occur. However, there are significant potential cost savings to be gained by UK IT industry if research can produce demonstrably successful testing techniques and practices which can be adopted to prevent faults. Firstly, resources can be pre-allocated and directed where they are most needed - to increase effort in refactoring on a proactive, rather than on a reactive basis which can cause disruption to other projects. Secondly, explanation of causes of code decay can improve and streamline the processes through which software is developed. For example, if the research reveals that one form of coupling is more prevalent than other coupling forms, then we can show how refactoring (or not doing refactoring) which addresses this aspect of system complexity can be introduced to help. There are compelling economic reasons for re-engineering test code so that it is less complex. Finally, the ability to understand trends in the evolution of a system (from a test and production code perspective) in a more timely fashion ultimately frees up time for developers to work on improving/perfecting the quality of their code. Put another way, there is a high opportunity cost to the time spent dealing with and fixing issues which might be prevented with greater knowledge of test structures; the proposed research has huge potential for unlocking the 'opportunity' that a more timely awareness and understanding of system pressures in test code create.


From a societal viewpoint, the research is in alignment with a key aim of any company of developing robust code and high quality products. Customers of the industrial partners will benefit in terms of the software that they 'deliver'. The value of the research also appeals to the wider societal community. In fact, the research affects anyone who uses software since improvement in the quality of code and a better understanding of test processes has a direct influence on us, its users, who increasingly use software of different types (whether embedded, real-time, web-based) in everyday products and services.


10 25 50

publication icon
Amjed Tahir (2016) Conference Paper

publication icon
Child M (2019) A comparison and evaluation of variants in the coupling between objects metric in Journal of Systems and Software

publication icon
Destefanis G (2016) Software development: do good manners matter? in PeerJ Computer Science

publication icon
Ortu M (2017) How diverse is your team? Investigating gender and nationality diversity in GitHub teams in Journal of Software Engineering Research and Development

publication icon
Ortu M. (2020) Describing software developers affectiveness through Markov chain models in Electronic Journal of Applied Statistical Analysis

publication icon
Shepperd M (2018) The role and value of replication in empirical software engineering results in Information and Software Technology

Description That asserts in code are problematic and that test code suffers from the same issues as regular code
There are big differences in the characteristics of each.
Industry considers test code as a second rate type of code.
Exploitation Route More studies of industry code.
More analysis of the sources of defects need to be undertaken,
A taxonomy of defects is elusive and the area is highly under-researched
Sectors Digital/Communication/Information Technologies (including Software)

Description The findings have fed into another grant where a company has used a tool to find faults, trialled at one company. This was also a cross-company project. Testing is a fundamental issue associated with refactoring and this thread of my research has led to connections with a large computer games company in London. A grant is being prepared on Debugging at the moment, with that company and two other companies. The grant has made us aware of the problem in many application domains and for future grant applications that will target those areas. Working on test-based research for some time has inspired a deeper look at debugging as a very under-estimated and under-researched topic. The work in AQUATIC has also led to other work on test and production code for the presence of "clones", a very topical area and of wide concern to industry, since good practice would suggest avoiding 'copy and pasting' as poor practice. See The work also feeds into the current work on Automatic Program Repair, the topic of the Fixie project (EPSRC-funded) that I am currently CI on and the subject of a large grant being prepared.
First Year Of Impact 2015
Sector Creative Economy,Digital/Communication/Information Technologies (including Software)
Impact Types Economic

Description Brunel WAT scheme (Incentive Scheme)
Amount £2,000 (GBP)
Organisation Brunel University London 
Sector Academic/University
Country United Kingdom
Start 06/2015 
End 07/2018
Description Brunel WAT scheme (Incentive Scheme)
Amount £2,000 (GBP)
Organisation Brunel University London 
Sector Academic/University
Country United Kingdom
Start 05/2017 
End 09/2018
Description Exploiting Defect Prediction for Automatic Software Repair (Fixie)
Amount £367,237 (GBP)
Funding ID EP/S005730/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 11/2018 
End 10/2022
Description Collaboration with University of Otago 
Organisation University of Otago
Country New Zealand 
Sector Academic/University 
PI Contribution We have ongoing research with a NZ based University on code smells. This was generated directly from the work in the AQUATIC project and the identification of code smells and how that influences unit test cases also.
Collaborator Contribution Paper writing
Impact ICPC paper - won Best Paper and an EMSE Journal output.
Start Year 2021
Description Concrete collaborations with BaeSystems and outcome of Visiting Industry Research Fellow 
Organisation BAE Systems
Country United Kingdom 
Sector Academic/University 
PI Contribution The FIAR-NET project has allowed a more formal footing with BAE Systems to be established and will result in a Fellowship submission to the EPSRC. We are in the process of inviting Dr Ken Wallace to become a Visiting Industry Fellow at Brunel.
Collaborator Contribution Invited talks and contributions to the University's Made In Brunel.
Impact Visiting Industry Fellowship in process.
Start Year 2014
Description University of Cagliari 
Organisation University of Cagliari
Country Italy 
Sector Academic/University 
PI Contribution I was Visiting Professor at the University of Cagliari - and from which a number of papers have emerged. I have also just submitted a grant application with the FlossLab which is a spin-off company from the University. I have also hosted a number of RFs at Brunel from the scheme.
Collaborator Contribution University of Cagliari hosted my trip and provided the mechanisms for collaboration.
Impact Please see publications list.
Start Year 2010