Metacognitive Instruction, Confidence and Prediction Accuracy in Software Engineering (MICaPASE)

Lead Research Organisation: Brunel University
Department Name: Information Systems Computing and Maths

Abstract

Effective prediction, for example of project cost, is an essential aspect of software engineering. Although considerable research has been devoted to this topic, the role of human experts has been under-emphasised.

Our aim is to investigate the impact of enhanced metacognitive awareness on prediction and confidence (uncertainty assessment) to improve the prediction practices of software professionals. This will be accomplished by developing metacognitive awareness during a series of experiments with software professionals as participants. The major outcomes will be a better understanding of (i) the factors that influence prediction and uncertainty assessment skills and (ii) how industry practice can be enhanced. The findings will impact the software industry, its clients and other sectors where accurate predictions are required in uncertain environments.

Planned Impact

The IT industry, both government and private sector are plagued with problems of poor software project cost prediction. Project cost underestimation and overrun is a global phenomenon which has not diminished over the last 70 years - there has been little evidence of improvement.

The goal of this research is to help IT project managers improve their prediction practices through metacognitive training. The aims are to improve prediction accuracy and assessment of confidence in the prediction (e.g. 80% confident the value will fall in the range 100-120 days) as over-confidence is known to lead to underestimation and bias.

We see four groups of beneficiary: researchers, the IT industry (including our collaborators HP and Lloyds TSB), government and the general public.

This proposal, based upon rigorous experimentation, will be valuable for empirical software engineering (ESE) researchers since it pioneers an interdisciplinary approach between computer science and cognitive psychology and the use of professional participants in realistic experiments. Such work will augment the world-leading position of the UK ESE research community. It will also foster strong relationships with other world class centres such as the Simula Labs (Norway).

Improved prediction will enable the IT industry (estimated in 2008 as forming 5% of the total UK economy) to reach better business decisions such as tendering for business and cost-benefit analyses. It will also facilitate more effective project management and tracking. The opportunity to improve prediction practices is seen as so important by our two collaborators (HP and Lloyds TSB) who have committed time and resources to this project with the expectation that lessons and principles derived from the research can be fed directly back to their IT functions.

The UK Government is a major consumer and procurer of IT. The recent coalition spending cuts have highlighted the need for better management of government IT projects (which of necessity, are predicated upon better prediction practices). This urgent need for improved project management was mentioned by the Prime Minister on 8 November, 2010 and it is also a major feature of the Cabinet Office Business Plan 2011-2015. Thus the proposed project is important and timely. In parallel, improved prediction will underpin more rational procurement and cost-benefit analysis (i.e. whether the project is worth undertaking in the first place).

Lastly the general public, as tax payers and consumers of IT, will benefit from more cost effective choices. Worldwide the total state spending on IT is estimated to exceed $420bn (Gartner, 2010) and in the UK the 2009 expenditure was £7.6bn (Office of Government Commerce). Clearly these are very substantial sums; we all stand to gain from fewer cost overruns, cancelled projects and wasted resources due to ill-informed procurement processes.
 
Description We showed that suitable training to raise awareness could help reduce, but not eliminate, bias in making predictions amongst software professionals. In other words debiasing strategies are possible. However the problems of cognitive bias (e.g., anchoring to irrelevant information) are hard to completely overcome.
Exploitation Route The idea that various cognitive biases impact the decision making and predictions of profefssionals is well known. Exploring remedies or debiasing strategies is less well explored and this is something that might be investigated further. Also the application domain might be broadened beyond software engineering.
Sectors Digital/Communication/Information Technologies (including Software)

 
Description We ran a total of 5 workshops for software professionals in the UK, Poland, Ukraine and NZ to help them improve their estiumation skills and become more aware of various sources of cognitive bias. This helps better software project governance and reduce cost and schedule overruns.
First Year Of Impact 2012
Sector Digital/Communication/Information Technologies (including Software)
Impact Types Economic

 
Description AUT collaboration 
Organisation Auckland University of Technology (AUT)
Country New Zealand 
Sector Academic/University 
PI Contribution This is with Prof. Stephen MacDonell (Auckland University of Technology, NZ). As a consequence of the workshops and the experimental work conducted during the main project we (Dr Mair and Prof. Shepperd) have been invited to New Zealand to repeat the work with the Auckland University of Technology acting as host (April 29 - May 3, 2013). This will take the form of a software estimation workshop for practitioners in conjunction with a repeat of our experiment which will form the basis of a research collaboration with Prof. MacDonell.
Start Year 2001
 
Description AUT collaboration 
Organisation Auckland University of Technology (AUT)
Country New Zealand 
Sector Academic/University 
PI Contribution This is with Prof. Stephen MacDonell (Auckland University of Technology, NZ). As a consequence of the workshops and the experimental work conducted during the main project we (Dr Mair and Prof. Shepperd) have been invited to New Zealand to repeat the work with the Auckland University of Technology acting as host (April 29 - May 3, 2013). This will take the form of a software estimation workshop for practitioners in conjunction with a repeat of our experiment which will form the basis of a research collaboration with Prof. MacDonell.
Start Year 2001
 
Description Collaboration with Auckland University of Technology 
Organisation Auckland University of Technology (AUT)
Country New Zealand 
Sector Academic/University 
PI Contribution A second collaboration is with Prof. Stephen MacDonell (Auckland University of Technology, NZ). As a consequence of the workshops and the experimental work conducted during the main project we (Dr Mair and Prof. Shepperd) have been invited to New Zealand to repeat the work with the Auckland University of Technology acting as host (April 29 - May 3, 2013). This will take the form of a software estimation workshop for practitioners in conjunction with a repeat of our experiment which will form the basis of a research collaboration with Prof. MacDonell.
Start Year 2005
 
Description Collaboration with Simula Labs 
Organisation Simula Research Laboratory
Country Norway 
Sector Academic/University 
PI Contribution As a consequence of this project we have had the opportunity of working closely, in particular over the design and conduct of the main experiments, with Prof Magne Jørgensen (Simula Labs, Norway). Prof. Jørgensen is a world leading authority on empirical software engineering and has been consistently ranked as the most published scholar in the field of systems and software engineering according to the Journal of Systems and Software ranking system. The data generated from this experiment will underpin continued collaborative research for some time to come and we hope for other informal arrangements such as exchange visits of PhD students.
Start Year 2005
 
Description Simula Labs collaboration 
Organisation Simula Research Laboratory
Country Norway 
Sector Academic/University 
PI Contribution We have had the opportunity of working closely, in particular over the design and conduct of the main experiments, with Prof Magne Jørgensen (Simula Labs, Norway). Prof. Jørgensen is a world leading authority on empirical software engineering and has been consistently ranked as the most published scholar in the field of systems and software engineering according to the Journal of Systems and Software ranking system. The data generated from this experiment will underpin continued collaborative research for some time to come and we hope for other informal arrangements such as exchange visits of PhD students.
Start Year 2005
 
Description Simula Labs collaboration 
Organisation Simula Research Laboratory
Country Norway 
Sector Academic/University 
PI Contribution We have had the opportunity of working closely, in particular over the design and conduct of the main experiments, with Prof Magne Jørgensen (Simula Labs, Norway). Prof. Jørgensen is a world leading authority on empirical software engineering and has been consistently ranked as the most published scholar in the field of systems and software engineering according to the Journal of Systems and Software ranking system. The data generated from this experiment will underpin continued collaborative research for some time to come and we hope for other informal arrangements such as exchange visits of PhD students.
Start Year 2005
 
Description BCS Project Management Specialist Group Invited Talk 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Other academic audiences (collaborators, peers etc.)
Results and Impact The research has attracted industry sufficient attention that we have now been invited to present the work to the British Computer Society - Project Management Specialist Group on May 14, 2013. The audience will be predominantly from industry.
Year(s) Of Engagement Activity 2013
 
Description Eclipse estimation workshops 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact We have given 2 half day workshops to over 40 software professionals on estimation for the software compay Eclipse. The workshop was derived from our research on how bias impacts the estimates of individuals, how deep rooted these biases are and different de-biasing strategies.
Year(s) Of Engagement Activity 2012
 
Description Estimating: it's all in the mind! 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? Yes
Primary Audience Professional Practitioners
Results and Impact BCS Software Management Group seminar - May 14, 2013. Talk to software practitioners about our research project results.
Year(s) Of Engagement Activity 2013
 
Description UKSMA keynote 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact The initial ideas underpinning the project were presented as part of joint keynote by Mair and Shepperd at the annual UK Software Metrics Association conference (Oct 27-28, 2011). This is a non-academic event organised by practitioners with a particular focus on software measurement, costing and productivity analysis. Consequently, this was an ideal opportunity to explain how this research can apply to the needs of those responsible for estimation within the software industry
Year(s) Of Engagement Activity 2011