Understanding the benefits and burdens of funding processes, from idea to award.
Lead Research Organisation:
King's College London
Department Name: The Policy Institute
Abstract
The cost of developing, writing, reviewing and deciding on research proposals is an under-researched but critical prerequisite to understanding the efficiency of funding decisions. This is increasingly a priority issue, with the UK Prime Minister stating last year his wish to reduce funding red-tape "to ensure brilliant scientists are able to spend as much time as possible creating new ideas, not filling in unnecessary forms." This commitment has recently been backed up with the publication of a policy paper by the Government on 'Reducing bureaucratic burden in research, innovation and higher education', which included a commitment to work "with external advisers to provide additional, independent challenge and to calculate the total costs of bureaucracy".
Whilst this may seem fanciful to some, it is also worth recalling estimates that 85% of medical research funding is wasted. This waste occurs throughout the research pipeline, but two steps - asking the right research question and using the right methods to address the research question - are key to deciding what research to fund. However, 'peer review' - as these decision-making processes are known in research funding - plays an important role in the allocation of funding and may bring particular benefits. It may be the case that the peer review process strengthens a grant, allowing the honing of both research questions and methods, and as such reduces waste and brings value to the scientific process. However, our ability to assess these benefits and burdens is hampered by our limited understanding - and lack of empirical, comparative evidence - around how peer review processes actually work.
Despite these concerns about the burden, cost and cost-effectiveness of peer review, there has been surprisingly little research on the topic. The most significant study in the UK is a 2006 government report on the efficiency and effectiveness of peer review. This study looked at the cost of preparing proposals, the cost of peer review, the administrative cost to the Research Councils and the cost of preparing end of grant reports, and concluded that this was equivalent to £196m per year (in 2006 money), equivalent to 13% of total funding awarded. Another key study comes from Australia where it was estimated that the applicant burden for the Australian National Health and Medical Research Council (NHMRC) was equivalent to 14% of total funding.
The primary objective of the project is to contribute to and update this limited body of work and to estimate the monetised costs (burden) of the peer review process for a sample of grant schemes for UK Research and Innovation (UKRI), and to balance that against the benefits that are derived from the process. A secondary objective is to develop a toolkit that will allow UKRI, and other research funders, to assess the benefits and burdens of other peer review processes in the future. To deliver on these two key objectives the project will map out the peer review process for four to six schemes and survey all those individuals involved in developing, writing, reviewing and deciding on research proposals. The surveys will capture the benefits and burdens, by source and stage, of the decision making process and use that to develop policy relevant recommendations on 'reducing bureaucratic burden in research' whilst maintaining the benefits that are derived from such processes.
Whilst this may seem fanciful to some, it is also worth recalling estimates that 85% of medical research funding is wasted. This waste occurs throughout the research pipeline, but two steps - asking the right research question and using the right methods to address the research question - are key to deciding what research to fund. However, 'peer review' - as these decision-making processes are known in research funding - plays an important role in the allocation of funding and may bring particular benefits. It may be the case that the peer review process strengthens a grant, allowing the honing of both research questions and methods, and as such reduces waste and brings value to the scientific process. However, our ability to assess these benefits and burdens is hampered by our limited understanding - and lack of empirical, comparative evidence - around how peer review processes actually work.
Despite these concerns about the burden, cost and cost-effectiveness of peer review, there has been surprisingly little research on the topic. The most significant study in the UK is a 2006 government report on the efficiency and effectiveness of peer review. This study looked at the cost of preparing proposals, the cost of peer review, the administrative cost to the Research Councils and the cost of preparing end of grant reports, and concluded that this was equivalent to £196m per year (in 2006 money), equivalent to 13% of total funding awarded. Another key study comes from Australia where it was estimated that the applicant burden for the Australian National Health and Medical Research Council (NHMRC) was equivalent to 14% of total funding.
The primary objective of the project is to contribute to and update this limited body of work and to estimate the monetised costs (burden) of the peer review process for a sample of grant schemes for UK Research and Innovation (UKRI), and to balance that against the benefits that are derived from the process. A secondary objective is to develop a toolkit that will allow UKRI, and other research funders, to assess the benefits and burdens of other peer review processes in the future. To deliver on these two key objectives the project will map out the peer review process for four to six schemes and survey all those individuals involved in developing, writing, reviewing and deciding on research proposals. The surveys will capture the benefits and burdens, by source and stage, of the decision making process and use that to develop policy relevant recommendations on 'reducing bureaucratic burden in research' whilst maintaining the benefits that are derived from such processes.
Technical Summary
The cost of developing, writing, reviewing and deciding on research proposals is an under-researched but critical prerequisite to understanding the efficiency of funding decisions. This has recently become a priority issue for the UK Government, who recently published a policy paper on 'Reducing bureaucratic burden in research, innovation and higher education', which included a commitment to work "with external advisers to provide additional, independent challenge and to calculate the total costs of bureaucracy".
The primary aim of the project is to estimate the monetised costs of the peer review process for a sample of grant schemes, and to balance that against the value that is derived from the process, for the applicants, reviewers, panel members and Research Council staff. A secondary aim is to develop a toolkit that will allow UKRI, and other research funders, to assess the cost and value of other peer review processes in the future. To deliver on these two key aims a mixed method approach of data collection and analysis will be employed including process mapping, surveys, interviews, cost modelling and policy analysis. Data on burdens and benefits will be collected through a series of surveys administered to applicants, peer reviewers and panel members and through key informant interviews with UKRI staff and university research office staff. The data will be analysed through rigorous and appropriate methods using SPSS and NVivo (or similar packages). The results of the quantitative and qualitative analysis will be synthesised to draw out estimates of the benefits and burdens of peer review processes, and examined by source of benefit/burden and each stage of decision making, as well as assessing the degree of commonality or difference across funding schemes. The results will be written up for publication in peer reviewed journals along with message led policy briefs and a practical cost modelling toolkit for the funder community.
The primary aim of the project is to estimate the monetised costs of the peer review process for a sample of grant schemes, and to balance that against the value that is derived from the process, for the applicants, reviewers, panel members and Research Council staff. A secondary aim is to develop a toolkit that will allow UKRI, and other research funders, to assess the cost and value of other peer review processes in the future. To deliver on these two key aims a mixed method approach of data collection and analysis will be employed including process mapping, surveys, interviews, cost modelling and policy analysis. Data on burdens and benefits will be collected through a series of surveys administered to applicants, peer reviewers and panel members and through key informant interviews with UKRI staff and university research office staff. The data will be analysed through rigorous and appropriate methods using SPSS and NVivo (or similar packages). The results of the quantitative and qualitative analysis will be synthesised to draw out estimates of the benefits and burdens of peer review processes, and examined by source of benefit/burden and each stage of decision making, as well as assessing the degree of commonality or difference across funding schemes. The results will be written up for publication in peer reviewed journals along with message led policy briefs and a practical cost modelling toolkit for the funder community.