Reliable and Efficient Estimation of the Economic Value of medical Research (REEEVR)
Lead Research Organisation:
University of Bristol
Department Name: Bristol Medical School
Abstract
In the UK, taxpayers and charities pay for most healthcare research. National bodies like the National Institute for Health Research (NIHR) allocate public funds. Charities like Cancer Research UK (CRUK) rely on donations. To spend money wisely, they have to decide which areas of research matter most and which studies to fund.
Value of Information (VoI) analysis estimates how much good a research study might do. Methods for estimating VoI are well established. But the people who decide what research gets funded seldom have access to the information it provides. This means we run the risk of wasting research funds. It also means that patients face unnecessary risks if they take part in research that is not justified from an NHS perspective.
VoI is based on formal cost-effectiveness analysis, which NHS decision-makers already rely on. Mostly, this comes through the National Institute for Health and Care Excellence (NICE), one of our project partners. NICE uses cost-effectiveness analysis to decide which drugs, tests and procedures the NHS should provide. To get their drugs approved, manufacturers submit cost-effectiveness analyses to NICE. These are often conducted by specialist consultancies like our partner Source Health Economics. NICE also makes suggestions about what future research might be useful. But it does not conduct or require VoI, so it cannot provide formal guidance to research funders.
The main reason VoI is not currently used more widely is that it is hard to do. Analysts need advanced technical skills. Even modern computers can struggle with all the calculations it needs. Online tools (including one developed by our collaborator) can approximate VoI, making it more accessible for analysts and easier for computers. Unfortunately, the approximations are only reliable for simple analyses. In a previous MRC grant, we showed how to estimate VoI accurately with much lower computational demands. We used a mathematical technique called multilevel Monte Carlo (MLMC). However, MLMC relies on complicated maths that few people developing cost-effectiveness analyses understand.
We want to improve the use of VoI by creating an easy to use online tool that can quickly and reliably estimate it, even for complex analyses. We will first test existing approximation methods to understand the circumstances in which they are good enough. For more complex and realistic analyses, we will use MLMC. To do this, we will develop software to convert cost-effectiveness models from Microsoft Excel to a programming language. Microsoft Excel is the most commonly used software for cost-effectiveness analysis, but it is too slow to run MLMC itself. We will develop reusable and efficient MLMC code that users can apply to the converted models.
Our partners are NICE Centre for Guidelines (NICE CfG) and Source Health Economics. They both develop large numbers of Excel-based cost-effectiveness analyses. They will provide guidance on the types of models on which to test the approximations. They will also provide example cost-effectiveness models on which we can test our software. People who work for NICE CfG and Source Health Economics will pilot our tool. We will use their feedback to make sure it is easy to use.
We will also work with our research partners and NIHR to ensure people are aware of our online tool and use it in future practice. We will work with NICE CfG to include VoI in new national guidelines, underpinning recommendations for future research. Source Health Economics will recommend using our new tools to manufacturers so they can include VoI in NICE submissions. At each stage of our project, we will engage directly with NIHR, who have expressed interest in our work. We will also invite them to online workshops at the beginning and end of our project. Our ultimate ambition is to provide NIHR with tools and resources to prioritise healthcare research formally. This will reduce wasted research and benefit patients across the UK.
Value of Information (VoI) analysis estimates how much good a research study might do. Methods for estimating VoI are well established. But the people who decide what research gets funded seldom have access to the information it provides. This means we run the risk of wasting research funds. It also means that patients face unnecessary risks if they take part in research that is not justified from an NHS perspective.
VoI is based on formal cost-effectiveness analysis, which NHS decision-makers already rely on. Mostly, this comes through the National Institute for Health and Care Excellence (NICE), one of our project partners. NICE uses cost-effectiveness analysis to decide which drugs, tests and procedures the NHS should provide. To get their drugs approved, manufacturers submit cost-effectiveness analyses to NICE. These are often conducted by specialist consultancies like our partner Source Health Economics. NICE also makes suggestions about what future research might be useful. But it does not conduct or require VoI, so it cannot provide formal guidance to research funders.
The main reason VoI is not currently used more widely is that it is hard to do. Analysts need advanced technical skills. Even modern computers can struggle with all the calculations it needs. Online tools (including one developed by our collaborator) can approximate VoI, making it more accessible for analysts and easier for computers. Unfortunately, the approximations are only reliable for simple analyses. In a previous MRC grant, we showed how to estimate VoI accurately with much lower computational demands. We used a mathematical technique called multilevel Monte Carlo (MLMC). However, MLMC relies on complicated maths that few people developing cost-effectiveness analyses understand.
We want to improve the use of VoI by creating an easy to use online tool that can quickly and reliably estimate it, even for complex analyses. We will first test existing approximation methods to understand the circumstances in which they are good enough. For more complex and realistic analyses, we will use MLMC. To do this, we will develop software to convert cost-effectiveness models from Microsoft Excel to a programming language. Microsoft Excel is the most commonly used software for cost-effectiveness analysis, but it is too slow to run MLMC itself. We will develop reusable and efficient MLMC code that users can apply to the converted models.
Our partners are NICE Centre for Guidelines (NICE CfG) and Source Health Economics. They both develop large numbers of Excel-based cost-effectiveness analyses. They will provide guidance on the types of models on which to test the approximations. They will also provide example cost-effectiveness models on which we can test our software. People who work for NICE CfG and Source Health Economics will pilot our tool. We will use their feedback to make sure it is easy to use.
We will also work with our research partners and NIHR to ensure people are aware of our online tool and use it in future practice. We will work with NICE CfG to include VoI in new national guidelines, underpinning recommendations for future research. Source Health Economics will recommend using our new tools to manufacturers so they can include VoI in NICE submissions. At each stage of our project, we will engage directly with NIHR, who have expressed interest in our work. We will also invite them to online workshops at the beginning and end of our project. Our ultimate ambition is to provide NIHR with tools and resources to prioritise healthcare research formally. This will reduce wasted research and benefit patients across the UK.
Technical Summary
Value of Information (VoI) analysis is not regularly used by healthcare research funders or the National Institute for Health and Care Excellence (NICE) due to high skill requirements and computational demands. Regression/approximation to estimate Expected Value of Partial Perfect Information (EVPPI) have reduced skill requirements but are only reliable for simple cost-effectiveness models (CEMs). We developed multilevel Monte Carlo (MLMC) to reliably estimate EVPPI, but must be implemented in programming languages such as R or C++, making it inaccessible to Microsoft Excel users.
We will develop an online tool for reliable and efficient EVPPI estimation in four workstreams. Workstream 1 will assess the limits of approximation/regression for EVPPI by comparing with gold standard Monte Carlo estimation. We will explore decision tree and Markov CEMs with increasing structural complexity, parameter dimension/correlation, and number of decision options. Workstream 2 will develop software to convert Excel CEMs to R/C++, which works by 'crawling' over the eXtended Markup Language (XML) and Visual Basic scrips underlying Excel CEMs to re-express the net benefit function in R/C++. Example CEMs will come be provided by our partners at NICE and Source Health Economics. We will encourage community extension by making all code publicly available and providing thorough documentation. Workstream 3 will develop R/C++ code for running MLMC on generic R/C++ CEMs. Workstream 4 will package the conversion and MLMC software as an easy-to-use online tool. Based on user-supplied information on the CEM and the findings of workstream 1 our tool will recommend if MLMC is required or if regression/approximation are sufficient.
These workstreams will be supplemented by a workstream collaborating with NICE and Source Health Economics to increase the use of our online EVPPI estimation tool, and a final workstream involving outreach to research funders to use VoI for research prioritisation.
We will develop an online tool for reliable and efficient EVPPI estimation in four workstreams. Workstream 1 will assess the limits of approximation/regression for EVPPI by comparing with gold standard Monte Carlo estimation. We will explore decision tree and Markov CEMs with increasing structural complexity, parameter dimension/correlation, and number of decision options. Workstream 2 will develop software to convert Excel CEMs to R/C++, which works by 'crawling' over the eXtended Markup Language (XML) and Visual Basic scrips underlying Excel CEMs to re-express the net benefit function in R/C++. Example CEMs will come be provided by our partners at NICE and Source Health Economics. We will encourage community extension by making all code publicly available and providing thorough documentation. Workstream 3 will develop R/C++ code for running MLMC on generic R/C++ CEMs. Workstream 4 will package the conversion and MLMC software as an easy-to-use online tool. Based on user-supplied information on the CEM and the findings of workstream 1 our tool will recommend if MLMC is required or if regression/approximation are sufficient.
These workstreams will be supplemented by a workstream collaborating with NICE and Source Health Economics to increase the use of our online EVPPI estimation tool, and a final workstream involving outreach to research funders to use VoI for research prioritisation.
