Metacognitive Instruction, Confidence and Prediction Accuracy in Software Engineering (MICaPASE)
Lead Research Organisation:
Brunel University London
Department Name: Information Systems Computing and Maths
Abstract
Effective prediction, for example of project cost, is an essential aspect of software engineering. Although considerable research has been devoted to this topic, the role of human experts has been under-emphasised.
Our aim is to investigate the impact of enhanced metacognitive awareness on prediction and confidence (uncertainty assessment) to improve the prediction practices of software professionals. This will be accomplished by developing metacognitive awareness during a series of experiments with software professionals as participants. The major outcomes will be a better understanding of (i) the factors that influence prediction and uncertainty assessment skills and (ii) how industry practice can be enhanced. The findings will impact the software industry, its clients and other sectors where accurate predictions are required in uncertain environments.
Our aim is to investigate the impact of enhanced metacognitive awareness on prediction and confidence (uncertainty assessment) to improve the prediction practices of software professionals. This will be accomplished by developing metacognitive awareness during a series of experiments with software professionals as participants. The major outcomes will be a better understanding of (i) the factors that influence prediction and uncertainty assessment skills and (ii) how industry practice can be enhanced. The findings will impact the software industry, its clients and other sectors where accurate predictions are required in uncertain environments.
Planned Impact
The IT industry, both government and private sector are plagued with problems of poor software project cost prediction. Project cost underestimation and overrun is a global phenomenon which has not diminished over the last 70 years - there has been little evidence of improvement.
The goal of this research is to help IT project managers improve their prediction practices through metacognitive training. The aims are to improve prediction accuracy and assessment of confidence in the prediction (e.g. 80% confident the value will fall in the range 100-120 days) as over-confidence is known to lead to underestimation and bias.
We see four groups of beneficiary: researchers, the IT industry (including our collaborators HP and Lloyds TSB), government and the general public.
This proposal, based upon rigorous experimentation, will be valuable for empirical software engineering (ESE) researchers since it pioneers an interdisciplinary approach between computer science and cognitive psychology and the use of professional participants in realistic experiments. Such work will augment the world-leading position of the UK ESE research community. It will also foster strong relationships with other world class centres such as the Simula Labs (Norway).
Improved prediction will enable the IT industry (estimated in 2008 as forming 5% of the total UK economy) to reach better business decisions such as tendering for business and cost-benefit analyses. It will also facilitate more effective project management and tracking. The opportunity to improve prediction practices is seen as so important by our two collaborators (HP and Lloyds TSB) who have committed time and resources to this project with the expectation that lessons and principles derived from the research can be fed directly back to their IT functions.
The UK Government is a major consumer and procurer of IT. The recent coalition spending cuts have highlighted the need for better management of government IT projects (which of necessity, are predicated upon better prediction practices). This urgent need for improved project management was mentioned by the Prime Minister on 8 November, 2010 and it is also a major feature of the Cabinet Office Business Plan 2011-2015. Thus the proposed project is important and timely. In parallel, improved prediction will underpin more rational procurement and cost-benefit analysis (i.e. whether the project is worth undertaking in the first place).
Lastly the general public, as tax payers and consumers of IT, will benefit from more cost effective choices. Worldwide the total state spending on IT is estimated to exceed $420bn (Gartner, 2010) and in the UK the 2009 expenditure was £7.6bn (Office of Government Commerce). Clearly these are very substantial sums; we all stand to gain from fewer cost overruns, cancelled projects and wasted resources due to ill-informed procurement processes.
The goal of this research is to help IT project managers improve their prediction practices through metacognitive training. The aims are to improve prediction accuracy and assessment of confidence in the prediction (e.g. 80% confident the value will fall in the range 100-120 days) as over-confidence is known to lead to underestimation and bias.
We see four groups of beneficiary: researchers, the IT industry (including our collaborators HP and Lloyds TSB), government and the general public.
This proposal, based upon rigorous experimentation, will be valuable for empirical software engineering (ESE) researchers since it pioneers an interdisciplinary approach between computer science and cognitive psychology and the use of professional participants in realistic experiments. Such work will augment the world-leading position of the UK ESE research community. It will also foster strong relationships with other world class centres such as the Simula Labs (Norway).
Improved prediction will enable the IT industry (estimated in 2008 as forming 5% of the total UK economy) to reach better business decisions such as tendering for business and cost-benefit analyses. It will also facilitate more effective project management and tracking. The opportunity to improve prediction practices is seen as so important by our two collaborators (HP and Lloyds TSB) who have committed time and resources to this project with the expectation that lessons and principles derived from the research can be fed directly back to their IT functions.
The UK Government is a major consumer and procurer of IT. The recent coalition spending cuts have highlighted the need for better management of government IT projects (which of necessity, are predicated upon better prediction practices). This urgent need for improved project management was mentioned by the Prime Minister on 8 November, 2010 and it is also a major feature of the Cabinet Office Business Plan 2011-2015. Thus the proposed project is important and timely. In parallel, improved prediction will underpin more rational procurement and cost-benefit analysis (i.e. whether the project is worth undertaking in the first place).
Lastly the general public, as tax payers and consumers of IT, will benefit from more cost effective choices. Worldwide the total state spending on IT is estimated to exceed $420bn (Gartner, 2010) and in the UK the 2009 expenditure was £7.6bn (Office of Government Commerce). Clearly these are very substantial sums; we all stand to gain from fewer cost overruns, cancelled projects and wasted resources due to ill-informed procurement processes.
People |
ORCID iD |
Martin Shepperd (Principal Investigator) |
Publications
Carolyn Mair (Author)
(2014)
Cognitive bias remains after debiasing information
Carolyn Mair (Author)
(2013)
Cognitive bias remains after debiasing information
Magne Jorgensen (Author)
(2013)
Cognitive bias remains after debiasing information
Martin Shepperd (Author)
(2013)
Cognitive bias remains after debiasing information
Description | We showed that suitable training to raise awareness could help reduce, but not eliminate, bias in making predictions amongst software professionals. In other words debiasing strategies are possible. However the problems of cognitive bias (e.g., anchoring to irrelevant information) are hard to completely overcome. |
Exploitation Route | The idea that various cognitive biases impact the decision making and predictions of profefssionals is well known. Exploring remedies or debiasing strategies is less well explored and this is something that might be investigated further. Also the application domain might be broadened beyond software engineering. |
Sectors | Digital/Communication/Information Technologies (including Software) |
Description | We ran a total of 5 workshops for software professionals in the UK, Poland, Ukraine and NZ to help them improve their estiumation skills and become more aware of various sources of cognitive bias. This helps better software project governance and reduce cost and schedule overruns. |
First Year Of Impact | 2012 |
Sector | Digital/Communication/Information Technologies (including Software) |
Impact Types | Economic |
Description | AUT collaboration |
Organisation | Auckland University of Technology (AUT) |
Country | New Zealand |
Sector | Academic/University |
PI Contribution | This is with Prof. Stephen MacDonell (Auckland University of Technology, NZ). As a consequence of the workshops and the experimental work conducted during the main project we (Dr Mair and Prof. Shepperd) have been invited to New Zealand to repeat the work with the Auckland University of Technology acting as host (April 29 - May 3, 2013). This will take the form of a software estimation workshop for practitioners in conjunction with a repeat of our experiment which will form the basis of a research collaboration with Prof. MacDonell. |
Start Year | 2001 |
Description | AUT collaboration |
Organisation | Auckland University of Technology (AUT) |
Country | New Zealand |
Sector | Academic/University |
PI Contribution | This is with Prof. Stephen MacDonell (Auckland University of Technology, NZ). As a consequence of the workshops and the experimental work conducted during the main project we (Dr Mair and Prof. Shepperd) have been invited to New Zealand to repeat the work with the Auckland University of Technology acting as host (April 29 - May 3, 2013). This will take the form of a software estimation workshop for practitioners in conjunction with a repeat of our experiment which will form the basis of a research collaboration with Prof. MacDonell. |
Start Year | 2001 |
Description | Collaboration with Auckland University of Technology |
Organisation | Auckland University of Technology (AUT) |
Country | New Zealand |
Sector | Academic/University |
PI Contribution | A second collaboration is with Prof. Stephen MacDonell (Auckland University of Technology, NZ). As a consequence of the workshops and the experimental work conducted during the main project we (Dr Mair and Prof. Shepperd) have been invited to New Zealand to repeat the work with the Auckland University of Technology acting as host (April 29 - May 3, 2013). This will take the form of a software estimation workshop for practitioners in conjunction with a repeat of our experiment which will form the basis of a research collaboration with Prof. MacDonell. |
Start Year | 2005 |
Description | Collaboration with Simula Labs |
Organisation | Simula Research Laboratory |
Country | Norway |
Sector | Academic/University |
PI Contribution | As a consequence of this project we have had the opportunity of working closely, in particular over the design and conduct of the main experiments, with Prof Magne Jørgensen (Simula Labs, Norway). Prof. Jørgensen is a world leading authority on empirical software engineering and has been consistently ranked as the most published scholar in the field of systems and software engineering according to the Journal of Systems and Software ranking system. The data generated from this experiment will underpin continued collaborative research for some time to come and we hope for other informal arrangements such as exchange visits of PhD students. |
Start Year | 2005 |
Description | Simula Labs collaboration |
Organisation | Simula Research Laboratory |
Country | Norway |
Sector | Academic/University |
PI Contribution | We have had the opportunity of working closely, in particular over the design and conduct of the main experiments, with Prof Magne Jørgensen (Simula Labs, Norway). Prof. Jørgensen is a world leading authority on empirical software engineering and has been consistently ranked as the most published scholar in the field of systems and software engineering according to the Journal of Systems and Software ranking system. The data generated from this experiment will underpin continued collaborative research for some time to come and we hope for other informal arrangements such as exchange visits of PhD students. |
Start Year | 2005 |
Description | Simula Labs collaboration |
Organisation | Simula Research Laboratory |
Country | Norway |
Sector | Academic/University |
PI Contribution | We have had the opportunity of working closely, in particular over the design and conduct of the main experiments, with Prof Magne Jørgensen (Simula Labs, Norway). Prof. Jørgensen is a world leading authority on empirical software engineering and has been consistently ranked as the most published scholar in the field of systems and software engineering according to the Journal of Systems and Software ranking system. The data generated from this experiment will underpin continued collaborative research for some time to come and we hope for other informal arrangements such as exchange visits of PhD students. |
Start Year | 2005 |
Description | BCS Project Management Specialist Group Invited Talk |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Other academic audiences (collaborators, peers etc.) |
Results and Impact | The research has attracted industry sufficient attention that we have now been invited to present the work to the British Computer Society - Project Management Specialist Group on May 14, 2013. The audience will be predominantly from industry. |
Year(s) Of Engagement Activity | 2013 |
Description | Eclipse estimation workshops |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | We have given 2 half day workshops to over 40 software professionals on estimation for the software compay Eclipse. The workshop was derived from our research on how bias impacts the estimates of individuals, how deep rooted these biases are and different de-biasing strategies. |
Year(s) Of Engagement Activity | 2012 |
Description | Estimating: it's all in the mind! |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | Yes |
Primary Audience | Professional Practitioners |
Results and Impact | BCS Software Management Group seminar - May 14, 2013. Talk to software practitioners about our research project results. |
Year(s) Of Engagement Activity | 2013 |
Description | UKSMA keynote |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | The initial ideas underpinning the project were presented as part of joint keynote by Mair and Shepperd at the annual UK Software Metrics Association conference (Oct 27-28, 2011). This is a non-academic event organised by practitioners with a particular focus on software measurement, costing and productivity analysis. Consequently, this was an ideal opportunity to explain how this research can apply to the needs of those responsible for estimation within the software industry |
Year(s) Of Engagement Activity | 2011 |