What is the value of adaptive designs? Estimating expected value of sample information for adaptive trial designs.

Aim: To develop expected value of sample information (EVSI) methods that allow efficient comparison of economic value of conventional fixed and adaptive approaches for a trial design.
Background: Healthcare decision makers, such as the National Institute for Health and Care Excellence (NICE) in the UK, are guided in their decisions by cost-effectiveness analysis. These are trial- or model-based comparisons of the costs and effects of interventions for diseases. The value of eliminating the uncertainty in these decision recommendations is the expected value of perfect information (EVPI). The value of eliminating only a subset of the uncertainties is the expected value of partial perfect information (EVPPI). Finally, the EVSI is the economic value of reducing, rather than eliminating, uncertainty via a particular trial design.
Adaptive designs are those that adapt in response to results as they accumulate. Examples include: multi-arm multi-stage (MAMS) trials that drop or add treatment arms; trials that change randomisation ratios; and population enrichment trials that focus recruitment on patients most likely to benefit, or in whom benefit is most uncertain. Such designs can achieve the same level of accuracy and precision but with fewer patients assigned to non-performing or harmful treatments, smaller overall sample sizes, shorter durations, or lower trial costs. Adaptive designs can also test multiple hypotheses, potentially offering a better return on investment. Adaptive designs have been growing in popularity but remain the exception rather than the rule. EVSI has only been applied to a limited range of adaptive designs, with computational burden being a primary barrier to wider usage. I will extend and develop computationally efficient EVSI methods to adaptive designs to assess their economic value compared with conventional designs.
Work stream 1. Extending efficient methods for estimating EVSI to adaptive designs.
When cost-effectiveness analysis relies on complex models, such as Markov multi-state models, estimating EVSI for any trial can represent a substantial computational challenge. This is especially true for adaptive designs. Recent work has focussed on methods to approximate the cost-effectiveness model, including regression via flexible Gaussian processes or generalised additive models, approximation via splines or Taylor series, and moment matching of cost and effect distributions. These have been explored for valuing fixed but not adaptive designs. I will extend these methods to valuing the economic benefits of different adaptive designs.
Work stream 2. Develop advanced Monte Carlo sampling schemes to estimate the EVSI of adaptive designs.
The second approach I will investigate is that of efficient sampling schemes. Estimation of EVSI relies on nested loops of random number generation, called Monte-Carlo simulation. Brute force Monte-Carlo sampling is usually too slow to be practical for EVSI. Advanced Monte-Carlo schemes that reduce the number of necessary simulations are available. These include Multilevel Monte-Carlo, which replaces the estimation target with a lower variance alternative, and Quasi Monte-Carlo, which uses quasi-random rather than random numbers to reduce variance. These have been applied to EVPPI but not EVSI; I will extend them to EVSI and furthermore to adaptive designs.
Work stream 3. Application to real world cost-effectiveness models and adaptive trial designs.
Practical applications include the evaluation of treatments for depression, anticoagulants for prevention of stroke in atrial fibrillation, and the comparison of prosthetics for hip replacement. MAMS trials may be considered for hip replacement due to the large number of available prosthetics and population enrichment may be applied to depression treatment due to uncertain treatment effects in low severity depression patients. Funding applications will be submitted for the most promising of these trial designs.

Adaptive trials, as opposed to conventional trials, are those that change aspects of their design in response to interim analyses. Expected value of sample information (EVSI) estimates the economic value of trial designs to national health service decision makers. Our project aims to apply EVSI to the design and prioritisation of real-world adaptive trials exploring clinical and cost-effectiveness of competing interventions.
The cost-effectiveness models underlying the EVSI will be decision trees and Markov models of varying complexity drawn from our own research. These include comparisons of implant combinations for total hip replacement, directly acting oral anticoagulants for prevention of stroke in atrial fibrillation, depression severity thresholds above which to prescribe antidepressants, and the Ross procedure with mechanical/biological aortic valve replacement. Adaptive designs will include trials with group sequential, multi-arm multi-stage, sample size re-estimation, adaptive randomisation, and population enrichment designs. EVSI will indicate the optimal design in terms of sample size, follow-up, number of interim analyses, number of treatment arms etc. Funding will be sought for the trials with the greatest EVSI compared with total trial cost.
As EVSI estimation requires nested Monte Carlo simulation and repeated running of the cost-effectiveness model, estimating the EVSI for adaptive designs can carry a substantial computational burden. Efficient approaches to EVSI will therefore be explored and developed. We will use the recently published moment matching, importance sampling and Gaussian approximation techniques for EVSI estimation. We will also develop novel efficient estimation techniques for EVSI using quasi Monte Carlo, multilevel Monte Carlo, and multi-index Monte Carlo. The efficiency of all methods will be compared. These developments and comparisons will also be of benefit when estimating EVSI for conventional designs.

The overall goal of our project is to improve the uptake, where appropriate, of adaptive designs in clinical and cost-effectiveness research. Such designs can be more cost-effective than conventional designs as they can use fewer patients, shorter follow-ups or fewer treatments to generate the same quality of evidence. There is a diverse range of people who will benefit from this goal, and we explain in our Pathways to Impact section how our dissemination and impact strategy will achieve these benefits.
Our research methods will be of direct benefit to those undertaking trial-based clinical and cost-effectiveness research. The adaptive designs for cost-effectiveness trials we develop will provide an efficient alternative to conventional designs to testing their hypotheses. Furthermore, our methods for valuing and comparing trial designs will allow them to choose the optimal design to test their hypotheses. Those interested in the methodology of value of information analysis of adaptive trial design will also be stimulated by our new ideas and challenges to undertake further methodology research.
Trial funders, such as the National Institute for Health Research and Medical Research Council Efficacy and Mechanism Evaluation and Developmental Pathway Funding Schemes, will benefit from improved methods for comparing and assessing the cost-effectiveness of research. This will have a benefit to taxpayers as funding will be spent more efficiently. This improve efficiency of resource allocation will also indirectly benefit researchers conducting trials.
Decision makers, including the National Institute of Health and Care Excellence (NICE) for the National Health Service (NHS) in the UK, will benefit from improved and more timely evidence. As adaptive designs can stop early, drop unsuccessful treatment arms, or use smaller overall sample sizes, evidence for decision making can be generated more quickly and efficiently. The improved efficiency also means a greater number of hypotheses can be tested, leading to a wider overall evidence base for decision making.
Those who use Monte Carlo sampling schemes to solve problems will benefit from our research. Applying advanced sampling schemes, such as multilevel Monte Carlo or multi-index Monte Carlo, to the estimation of the value of adaptive trial designs will generate new challenges and insights to these methods. Researchers purely interested in methodology will be stimulated by these challenges and our ideas in solving them. Researchers applying these methods in areas other than medical decision making, for example in financial derivative pricing or oil exploration, will benefit new and improved methods.
A further beneficiary of our adaptive trial designs for cost-effectiveness research and methods for estimating their economic value will be patients. The reduced sample sizes, removal of redundant treatment arms, and better targeting of patients who will benefit exposes fewer patients to unnecessary risk or treatments that have no benefit. A related beneficiary are doctors, nurses and other research staff involved in trials, as they are less likely to spend time investigating suboptimal treatments or strategies. The improved evidence and decision making, noted earlier, will have an obvious benefit for patients and healthcare workers as the optimal treatment or management strategy for conditions can be identified sooner while consuming fewer resources.
These varied beneficiaries of our research, and the multifaceted ways they can benefit, demonstrate the tremendous impact of our proposed project.