Calibration of multiple treatment comparisons using individual patient data

There are well established methods for comparing multiple treatments based on a synthesis of all the trials that compare any two or more of them. These "network meta-analysis" methods are routinely used in NICE appraisals, NICE Clinical Guidelines, health technology assessment in general, and with growing frequency in Cochrane Collaboration reviews. These methods do, however, assume that the distribution of effect modifiers is the same in the included trials. Such assumptions are difficult to verify, but if they are not met results are likely to be biased.

As a result, there has been recent interest in methods that appear to address these concerns by using IPD from one trial to adjust the between-treatment comparisons. These methods are being seen with increasing frequency in the applied health technology assessment literature, and also in submissions to NICE. They are particularly attractive to manufacturers, who have access to IPD from their own trials but not to trials on competitor products.

The methods that have been published have limited scope, and their reliability has not been assessed. Of still greater concern is that they are also being applied to non-randomised (one-arm) studies: this is a radical departure from standard practice because it makes assumptions that have always been regarded as implausible.

Using simulation studies, and real examples from industry partners, we propose to develop and evaluate new methods that can use IPD from one or more trials to adjust treatment comparisons across an entire network of comparisons, and to provide improved comparative effectiveness estimates that are relevant to any specified target population. We will assess and compare the reliability of or the methods we plan to develop with standard methods, and with the methods now appearing in the literature, and we will check the validity of the various assumptions made by the different methods, using real data.

The integrity of conclusions drawn from indirect comparisons and network meta-analysis depend on the assumption that there is no imbalance in effect modifiers across the evidence on the different contrasts. Recently, new methods have been proposed which use IPD from a single (AB) trial (typically a manufacturer's trial) to generate an comparison that is supposed to be "adjusted" for imbalance in "baseline variables" between the AB and competitor AC trials

These methods are appearing with increasing frequency in the literature and in submissions to NICE, although their reliability remains untested. Of some concern, they are also used to derive treatment effects from one arm studies, which implausibly assumes all prognostic factors affecting absolute outcomes are accounted for.

We have developed new calibration-based methods which can be used to adjust for imbalances in effect modifiers in an entire connected network of treatment comparison, based on whatever IPD trial evidence is available, to generate a coherent set of treatment comparisons in any population with a known distribution of covariates. The methods make strong assumptions (specifically that all effect modifiers are included in the IPD data), but these are still far weaker than the assumptions made by standard network meta-analysis.

We will compare the proposed calibration methods with other methods in a series of simulation studies. We will conduct empirical studies using data provided by project partners to: study the performance of adjustment and calibration methods applied to one-arm studies, to assess the assumption that all effect modifiers are accounted for, and to study the differences between methods in a number of real life examples.

1. The methodology to be developed will have commercial value for the pharmaceutical industry generally, because it will allow for a more robust and accurate comparison of multiple treatments.
2. The new methodology will be of benefit to the national re-imbursement agencies, health care systems such as the NHS, health insurance companies, and through them to patients, for the same reason.
3. The proposed work will have benefit to regulatory agencies such as the US Food and Drug Administration and the European Medicines Agency, as it facilitates the comparison of active treatments against placebo in situations where the there is no direct evidence making this comparison.
4. The project will potentially have impact on the way trial results are reported, and specifically on the reporting the distribution of demographic and baseline factors.
5. The work will have impact on the academic study of methods for adjusting for imbalances in effect modifiers in network meta-analysis and indirect comparisons, and on applied work using such methods in health technology assessment. It is likely to have an impact on wider methods for causal inference