Verifying Changes: How Much Will it Cost?
Lead Research Organisation:
University of Edinburgh
Department Name: Sch of Informatics
Abstract
Most software systems evolve over time to adapt to changing environment, needs, new
concepts, new technologies and standards. Modification to the software is
inevitable in software evolution and may be through growth in
the number of functions, components and interfaces, or alteration of existing modules.
Examples commonly seen are changes or upgrades in operating systems, browsers, communication software.
Software maintenance to understand, implement and validate changes is a very expensive activity and costs billions of dollars each year in the software industry.
In this proposal, we focus on the problem of validating changes and their
effects. This is typically carried out using regression testing,
defined as ``selective retesting of a system or component to verify that
modifications have not caused unintended effects and that the system or
component still complies with its specified requirements''
Regression testing is an expensive and time consuming process and
accounts for majority of the maintenance costs.
Balancing confidence in the software correctness (gained from extensive and
frequent regression testing) against cost of regression test effort is one
of the major challenges in software maintenance.
Estimates of regression test cost will help developers and managers in
achieving this balance. Accurate regression test cost estimates are crucial
for planning project schedules, allocating resources, software reliability, monitoring and control.
Cost models estimating maintenance effort and development effort have been developed in the past.
Nevertheless, these models cannot be used since crucial elements for predicting test effort
such as testing requirements, test cases, test quality metrics are ignored for the most part in these models.
Our goal in this proposal is to define a model that accurately estimates regression test cost for evolving
software taking into account effects of changes on software behaviour, testing requirements, industry and process specific information, and quality metrics to be satisfied.
At the end of this proposal, we will have tools that when given the software changes and quality requirements as inputs can predict the cost of validating the changes.
We will evaluate the test effort estimates on large open-source software evolution repositories
such as DEBIAN GNU/Linux distribution releases and OpenSSH.
We will also collaborate with IBM and evaluate our estimates on their software.
The prototype analysis tools that we develop during the course of the project
will be made freely available in open-source form. We will package and ship our analysis tools as part of
well-established distributions such as Debian.
This will allow numerous developers of large-scale software (including our
project partner) to directly apply our techniques to their problems,
improving their development and maintenance processes.
concepts, new technologies and standards. Modification to the software is
inevitable in software evolution and may be through growth in
the number of functions, components and interfaces, or alteration of existing modules.
Examples commonly seen are changes or upgrades in operating systems, browsers, communication software.
Software maintenance to understand, implement and validate changes is a very expensive activity and costs billions of dollars each year in the software industry.
In this proposal, we focus on the problem of validating changes and their
effects. This is typically carried out using regression testing,
defined as ``selective retesting of a system or component to verify that
modifications have not caused unintended effects and that the system or
component still complies with its specified requirements''
Regression testing is an expensive and time consuming process and
accounts for majority of the maintenance costs.
Balancing confidence in the software correctness (gained from extensive and
frequent regression testing) against cost of regression test effort is one
of the major challenges in software maintenance.
Estimates of regression test cost will help developers and managers in
achieving this balance. Accurate regression test cost estimates are crucial
for planning project schedules, allocating resources, software reliability, monitoring and control.
Cost models estimating maintenance effort and development effort have been developed in the past.
Nevertheless, these models cannot be used since crucial elements for predicting test effort
such as testing requirements, test cases, test quality metrics are ignored for the most part in these models.
Our goal in this proposal is to define a model that accurately estimates regression test cost for evolving
software taking into account effects of changes on software behaviour, testing requirements, industry and process specific information, and quality metrics to be satisfied.
At the end of this proposal, we will have tools that when given the software changes and quality requirements as inputs can predict the cost of validating the changes.
We will evaluate the test effort estimates on large open-source software evolution repositories
such as DEBIAN GNU/Linux distribution releases and OpenSSH.
We will also collaborate with IBM and evaluate our estimates on their software.
The prototype analysis tools that we develop during the course of the project
will be made freely available in open-source form. We will package and ship our analysis tools as part of
well-established distributions such as Debian.
This will allow numerous developers of large-scale software (including our
project partner) to directly apply our techniques to their problems,
improving their development and maintenance processes.
Planned Impact
The main direct beneficiaries of the project include
1) Software developers and project managers in the software industry.
The analysis techniques developed in our proposal will help developers of large scale software
analyse the impact of changes and the test cases required to ensure software quality.
Project managers can use the cost model we develop to plan and allocate resources for validating changes and reliability control.
The cost model will also help managers in creating realistic project schedules.
2) Any industry maintaining evolving software.
Accurate estimates of testing costs in evolving software will help organizations better plan the software maintenance process
resulting in more effective use of the resources to produce better quality software. This will result in
considerable economic gains for organizations maintaining large scale software
by reducing cost of corrective effort, reducing redundancy in resource estimates,
preventing job losses from overbidding, and improving reliability of
software.
3) Software customers.
With the prevalence of software in today's world, most of us interact on a daily basis with software that is evolving.
Our cost models promise better project and resource planning, and improved reliability which will lead to better quality software at cheaper rates.
1) Software developers and project managers in the software industry.
The analysis techniques developed in our proposal will help developers of large scale software
analyse the impact of changes and the test cases required to ensure software quality.
Project managers can use the cost model we develop to plan and allocate resources for validating changes and reliability control.
The cost model will also help managers in creating realistic project schedules.
2) Any industry maintaining evolving software.
Accurate estimates of testing costs in evolving software will help organizations better plan the software maintenance process
resulting in more effective use of the resources to produce better quality software. This will result in
considerable economic gains for organizations maintaining large scale software
by reducing cost of corrective effort, reducing redundancy in resource estimates,
preventing job losses from overbidding, and improving reliability of
software.
3) Software customers.
With the prevalence of software in today's world, most of us interact on a daily basis with software that is evolving.
Our cost models promise better project and resource planning, and improved reliability which will lead to better quality software at cheaper rates.
People |
ORCID iD |
| Ajitha Rajan (Principal Investigator) |
Publications
Noureddine A
(2015)
Optimising Energy Consumption of Design Patterns
Rajan A
(2014)
Accelerated test execution using GPUs
| Description | For changes proposed to a software, we have developed a methodology and tool to analyse the size of change and its impact on the software. We have also analysed the test effort required to verify these changes. We measured the execution time and energy consumed by the tests for changes on several open source benchmarks. We used a statistical machine learning technique to estimate the cost of test suite execution based on the application and hardware characteristics. We have recently submitted our findings to an international peer reviewed conference on empirical software engineering. While analysing changes made during software maintenance, we discovered we could optimise the software design for improved energy efficiency. We proposed simple compiler transformations for popular design patterns in software. Our findings were published in a top software engineering conference. |
| Exploitation Route | Energy efficient software development, Improved software design, Improved testing practices |
| Sectors | Aerospace, Defence and Marine,Digital/Communication/Information Technologies (including Software),Electronics,Energy,Financial Services, and Management Consultancy |
| Description | We have presented and discussed our approach and findings with respect to improving test suite execution time at companies with software and tests in the embedded systems domain and compilers for parallel accelerators. We are still in discussion about using the techniques and their feasibility on company software and tests. |
| First Year Of Impact | 2016 |
| Sector | Digital/Communication/Information Technologies (including Software) |
| Impact Types | Economic |