What is the value of adaptive designs? Estimating expected value of sample information for adaptive trial designs.

Lead Research Organisation: University of Bristol
Department Name: Bristol Medical School


Aim: To develop expected value of sample information (EVSI) methods that allow efficient comparison of economic value of conventional fixed and adaptive approaches for a trial design.
Background: Healthcare decision makers, such as the National Institute for Health and Care Excellence (NICE) in the UK, are guided in their decisions by cost-effectiveness analysis. These are trial- or model-based comparisons of the costs and effects of interventions for diseases. The value of eliminating the uncertainty in these decision recommendations is the expected value of perfect information (EVPI). The value of eliminating only a subset of the uncertainties is the expected value of partial perfect information (EVPPI). Finally, the EVSI is the economic value of reducing, rather than eliminating, uncertainty via a particular trial design.
Adaptive designs are those that adapt in response to results as they accumulate. Examples include: multi-arm multi-stage (MAMS) trials that drop or add treatment arms; trials that change randomisation ratios; and population enrichment trials that focus recruitment on patients most likely to benefit, or in whom benefit is most uncertain. Such designs can achieve the same level of accuracy and precision but with fewer patients assigned to non-performing or harmful treatments, smaller overall sample sizes, shorter durations, or lower trial costs. Adaptive designs can also test multiple hypotheses, potentially offering a better return on investment. Adaptive designs have been growing in popularity but remain the exception rather than the rule. EVSI has only been applied to a limited range of adaptive designs, with computational burden being a primary barrier to wider usage. I will extend and develop computationally efficient EVSI methods to adaptive designs to assess their economic value compared with conventional designs.
Work stream 1. Extending efficient methods for estimating EVSI to adaptive designs.
When cost-effectiveness analysis relies on complex models, such as Markov multi-state models, estimating EVSI for any trial can represent a substantial computational challenge. This is especially true for adaptive designs. Recent work has focussed on methods to approximate the cost-effectiveness model, including regression via flexible Gaussian processes or generalised additive models, approximation via splines or Taylor series, and moment matching of cost and effect distributions. These have been explored for valuing fixed but not adaptive designs. I will extend these methods to valuing the economic benefits of different adaptive designs.
Work stream 2. Develop advanced Monte Carlo sampling schemes to estimate the EVSI of adaptive designs.
The second approach I will investigate is that of efficient sampling schemes. Estimation of EVSI relies on nested loops of random number generation, called Monte-Carlo simulation. Brute force Monte-Carlo sampling is usually too slow to be practical for EVSI. Advanced Monte-Carlo schemes that reduce the number of necessary simulations are available. These include Multilevel Monte-Carlo, which replaces the estimation target with a lower variance alternative, and Quasi Monte-Carlo, which uses quasi-random rather than random numbers to reduce variance. These have been applied to EVPPI but not EVSI; I will extend them to EVSI and furthermore to adaptive designs.
Work stream 3. Application to real world cost-effectiveness models and adaptive trial designs.
Practical applications include the evaluation of treatments for depression, anticoagulants for prevention of stroke in atrial fibrillation, and the comparison of prosthetics for hip replacement. MAMS trials may be considered for hip replacement due to the large number of available prosthetics and population enrichment may be applied to depression treatment due to uncertain treatment effects in low severity depression patients. Funding applications will be submitted for the most promising of these trial designs.

Technical Summary

Adaptive trials, as opposed to conventional trials, are those that change aspects of their design in response to interim analyses. Expected value of sample information (EVSI) estimates the economic value of trial designs to national health service decision makers. Our project aims to apply EVSI to the design and prioritisation of real-world adaptive trials exploring clinical and cost-effectiveness of competing interventions.
The cost-effectiveness models underlying the EVSI will be decision trees and Markov models of varying complexity drawn from our own research. These include comparisons of implant combinations for total hip replacement, directly acting oral anticoagulants for prevention of stroke in atrial fibrillation, depression severity thresholds above which to prescribe antidepressants, and the Ross procedure with mechanical/biological aortic valve replacement. Adaptive designs will include trials with group sequential, multi-arm multi-stage, sample size re-estimation, adaptive randomisation, and population enrichment designs. EVSI will indicate the optimal design in terms of sample size, follow-up, number of interim analyses, number of treatment arms etc. Funding will be sought for the trials with the greatest EVSI compared with total trial cost.
As EVSI estimation requires nested Monte Carlo simulation and repeated running of the cost-effectiveness model, estimating the EVSI for adaptive designs can carry a substantial computational burden. Efficient approaches to EVSI will therefore be explored and developed. We will use the recently published moment matching, importance sampling and Gaussian approximation techniques for EVSI estimation. We will also develop novel efficient estimation techniques for EVSI using quasi Monte Carlo, multilevel Monte Carlo, and multi-index Monte Carlo. The efficiency of all methods will be compared. These developments and comparisons will also be of benefit when estimating EVSI for conventional designs.

Planned Impact

The overall goal of our project is to improve the uptake, where appropriate, of adaptive designs in clinical and cost-effectiveness research. Such designs can be more cost-effective than conventional designs as they can use fewer patients, shorter follow-ups or fewer treatments to generate the same quality of evidence. There is a diverse range of people who will benefit from this goal, and we explain in our Pathways to Impact section how our dissemination and impact strategy will achieve these benefits.
Our research methods will be of direct benefit to those undertaking trial-based clinical and cost-effectiveness research. The adaptive designs for cost-effectiveness trials we develop will provide an efficient alternative to conventional designs to testing their hypotheses. Furthermore, our methods for valuing and comparing trial designs will allow them to choose the optimal design to test their hypotheses. Those interested in the methodology of value of information analysis of adaptive trial design will also be stimulated by our new ideas and challenges to undertake further methodology research.
Trial funders, such as the National Institute for Health Research and Medical Research Council Efficacy and Mechanism Evaluation and Developmental Pathway Funding Schemes, will benefit from improved methods for comparing and assessing the cost-effectiveness of research. This will have a benefit to taxpayers as funding will be spent more efficiently. This improve efficiency of resource allocation will also indirectly benefit researchers conducting trials.
Decision makers, including the National Institute of Health and Care Excellence (NICE) for the National Health Service (NHS) in the UK, will benefit from improved and more timely evidence. As adaptive designs can stop early, drop unsuccessful treatment arms, or use smaller overall sample sizes, evidence for decision making can be generated more quickly and efficiently. The improved efficiency also means a greater number of hypotheses can be tested, leading to a wider overall evidence base for decision making.
Those who use Monte Carlo sampling schemes to solve problems will benefit from our research. Applying advanced sampling schemes, such as multilevel Monte Carlo or multi-index Monte Carlo, to the estimation of the value of adaptive trial designs will generate new challenges and insights to these methods. Researchers purely interested in methodology will be stimulated by these challenges and our ideas in solving them. Researchers applying these methods in areas other than medical decision making, for example in financial derivative pricing or oil exploration, will benefit new and improved methods.
A further beneficiary of our adaptive trial designs for cost-effectiveness research and methods for estimating their economic value will be patients. The reduced sample sizes, removal of redundant treatment arms, and better targeting of patients who will benefit exposes fewer patients to unnecessary risk or treatments that have no benefit. A related beneficiary are doctors, nurses and other research staff involved in trials, as they are less likely to spend time investigating suboptimal treatments or strategies. The improved evidence and decision making, noted earlier, will have an obvious benefit for patients and healthcare workers as the optimal treatment or management strategy for conditions can be identified sooner while consuming fewer resources.
These varied beneficiaries of our research, and the multifaceted ways they can benefit, demonstrate the tremendous impact of our proposed project.


10 25 50
publication icon
Fang W (2022) Multilevel and Quasi Monte Carlo Methods for the Calculation of the Expected Value of Partial Perfect Information. in Medical decision making : an international journal of the Society for Medical Decision Making

publication icon
Hironaka T (2020) Multilevel Monte Carlo Estimation of the Expected Value of Sample Information in SIAM/ASA Journal on Uncertainty Quantification

publication icon
Keeney E (2022) Systematic Review of Cost-Effectiveness Models in Prostate Cancer: Exploring New Developments in Testing and Diagnosis. in Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research

Description Cost-effectiveness of the UK covid-19 response
Amount £928 (GBP)
Funding ID Elizabeth Blackwell Institute Rapid Response Call (COVID-19) 
Organisation University of Bristol 
Department Elizabeth Blackwell Institute for Health Research
Sector Academic/University
Country United Kingdom
Start 05/2020 
End 08/2020
Description Production of Technology Assessment Reviews (TARs) for the National Institute for Health Research (NIHR). NIHR Evidence Synthesis Programme
Amount £4,325,000 (GBP)
Organisation National Institute for Health Research 
Sector Public
Country United Kingdom
Start 03/2022 
End 03/2027
Description Reliable and Efficient Estimation of the Economic Value of medical Research (REEEVR)
Amount £342,034 (GBP)
Funding ID MR/W029855/1 
Organisation Medical Research Council (MRC) 
Sector Public
Country United Kingdom
Start 12/2022 
End 05/2025
Description How can we improve the cost-effectiveness of medical trials? 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Schools
Results and Impact Created an article for the website and print version of Futurum Careers (https://futurumcareers.com/). Futurum Careers is a free online resource and magazine aimed at introducing 14-19-year-olds worldwide to the world of work in STEM (science, tech, engineering, maths, medicine) and SHAPE (social sciences, humanities and the arts for people and the economy). They collaborate with academics all over the world and translate their research into free education resources that can be used in the classroom, at home and in STEM and SHAPE clubs
Year(s) Of Engagement Activity 2022
URL https://futurumcareers.com/how-can-we-improve-the-cost-effectiveness-of-medical-trials
Description I organised this "Workshop on R for trial and model-based cost-effectiveness analysis" 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact I organised this workshop on R for health technology assessment (HTA). Started running them annually in 2018 but continued in 2020, 2021, 2022 (and 2023) with support of this grant. The overall goal is to present interesting and enlightening presentations on the use of R that will engage an audience of those working in the field of health technology assessment and related analysis. Audience included academics, government, industry, and consulting. The 2022 event was a hybrid with 2 days online and one day in person at University of Oxford.
Year(s) Of Engagement Activity 2022
URL https://r-hta.org/events/workshop/2022/
Description Presentation at R for HTA Workshop 2022 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Hacking hesim: Using the continuous-time semi-Markov package 'hesim' to run general state-transition microsimulations evaluating cost-effectiveness of implants for knee replacement surgery
Year(s) Of Engagement Activity 2022
URL https://r-hta.org/events/workshop/2022/
Description What is the value of adaptive designs? Estimating expected value of sample information for adaptive trial designs 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Seminar given to Precision Value and Health, an international health economic consulting company.
Year(s) Of Engagement Activity 2021