Metacognitive Instruction, Confidence and Prediction Accuracy in Software Engineering (MICaPASE)

Effective prediction, for example of project cost, is an essential aspect of software engineering. Although considerable research has been devoted to this topic, the role of human experts has been under-emphasised.

Our aim is to investigate the impact of enhanced metacognitive awareness on prediction and confidence (uncertainty assessment) to improve the prediction practices of software professionals. This will be accomplished by developing metacognitive awareness during a series of experiments with software professionals as participants. The major outcomes will be a better understanding of (i) the factors that influence prediction and uncertainty assessment skills and (ii) how industry practice can be enhanced. The findings will impact the software industry, its clients and other sectors where accurate predictions are required in uncertain environments.

The IT industry, both government and private sector are plagued with problems of poor software project cost prediction. Project cost underestimation and overrun is a global phenomenon which has not diminished over the last 70 years - there has been little evidence of improvement.

The goal of this research is to help IT project managers improve their prediction practices through metacognitive training. The aims are to improve prediction accuracy and assessment of confidence in the prediction (e.g. 80% confident the value will fall in the range 100-120 days) as over-confidence is known to lead to underestimation and bias.

We see four groups of beneficiary: researchers, the IT industry (including our collaborators HP and Lloyds TSB), government and the general public.

This proposal, based upon rigorous experimentation, will be valuable for empirical software engineering (ESE) researchers since it pioneers an interdisciplinary approach between computer science and cognitive psychology and the use of professional participants in realistic experiments. Such work will augment the world-leading position of the UK ESE research community. It will also foster strong relationships with other world class centres such as the Simula Labs (Norway).

Improved prediction will enable the IT industry (estimated in 2008 as forming 5% of the total UK economy) to reach better business decisions such as tendering for business and cost-benefit analyses. It will also facilitate more effective project management and tracking. The opportunity to improve prediction practices is seen as so important by our two collaborators (HP and Lloyds TSB) who have committed time and resources to this project with the expectation that lessons and principles derived from the research can be fed directly back to their IT functions.

The UK Government is a major consumer and procurer of IT. The recent coalition spending cuts have highlighted the need for better management of government IT projects (which of necessity, are predicated upon better prediction practices). This urgent need for improved project management was mentioned by the Prime Minister on 8 November, 2010 and it is also a major feature of the Cabinet Office Business Plan 2011-2015. Thus the proposed project is important and timely. In parallel, improved prediction will underpin more rational procurement and cost-benefit analysis (i.e. whether the project is worth undertaking in the first place).

Lastly the general public, as tax payers and consumers of IT, will benefit from more cost effective choices. Worldwide the total state spending on IT is estimated to exceed $420bn (Gartner, 2010) and in the UK the 2009 expenditure was £7.6bn (Office of Government Commerce). Clearly these are very substantial sums; we all stand to gain from fewer cost overruns, cancelled projects and wasted resources due to ill-informed procurement processes.