Investigation of discrepancies between direct and indirect comparison of competing healthcare interventions

Lead Research Organisation: University of East Anglia
Department Name: Rehabilitation Sciences


To choose the best treatment from several available treatments to an illness, the best evidence is from randomised trials (RCTs) where different treatments have been directly tested against each other (direct comparison).
A statistical method known as adjusted indirect comparison may be helpful when available treatments have not been directly compared in RCTs. However, this indirect comparison method may produce results that are different from a direct comparison of different treatments.
We aim to investigate factors associated with the difference between direct and indirect approaches. We will see if indirect comparison gives the same answer as trials where the different treatments have been directly compared. If the two methods give different answers, we will look through the studies in detail, and try to work out what may have caused the difference.
The research we plan to do is very important because government bodies like the National Institute of Clinical Excellence may have to use indirect comparison methods in choosing treatments, when no direct comparison study is available. Our research findings should help us recognize situations where the indirect comparison may be a good, reliable choice, and also when it can give us wrong answers.

Technical Summary

BACKGROUND: Adjusted indirect comparison (AIC) has been increasingly used in the evaluation of healthcare interventions. Although the basic assumption for a valid AIC is theoretically clear, there is only limited empirical evidence on factors associated with the validity of AIC. Practically useful methods for assessing the appropriateness of AIC and mixed treatment comparison (MTC) have not been systematically developed and tested. Consequently, the use of AIC and MTC is still controversial and over-shadowed by concern of their misuse. We propose a methodological research to address these limitations and concerns.
OBJECTIVES: The proposed research aims (1) to greatly expand empirical evidence on the validity of AIC, (2) to develop and test methods for assessing clinical similarities in AIC, and for assessing clinical consistency between direct comparison (DC) and indirect comparison (AIC), and (3) to investigate factors associated with the discrepancy between direct and indirect estimates.
RESEARCH PLANS: We will search the Cochrane Library to identify Cochrane Systematic Reviews (CSRs) that provide sufficient data to compare competing interventions by both DC and AIC (using Bucher et al s method). The discrepancy between the DC and AIC will be measured by the difference between the two estimates. We will extract data on trial participants, interventions compared, outcome measures, results and other study characteristics from trials in relevant CSRs. Based on data compiled above, we will assess similarities of the two sets of trials involved in AIC, and assess clinical consistency between DC and AIC. The results of similarity and consistency assessment will be conveyed to a clinical similarity score and a clinical consistency score for each meta-analysis. Reliability and validity of the similarity and consistency score will be investigated. Using the observed discrepancy between DC and AIC estimates as the dependent variable, meta-regression analyses will be conducted to investigate factors (including similarity and consistency scores, types of interventions, number of trials, heterogeneity, etc) that may be associated with bias in AIC. Exploratory sensitivity analyses will be conducted when possible to improve trials comparability by excluding trials that cause poor clinical similarity or inconsistency, and by using meta-regression to adjust for imbalance in factors which modify treatment effects. We will investigate whether the improved clinical similarity and/or consistency is associated with reduced discrepancy.
APPLICATION OF THE RESEARCH RESULTS: Findings from the proposed research will enable us to provide practical recommendations for valid indirect and/or mixed treatment comparison in the evaluation of competing healthcare interventions.
Description Citation in JAMA guidance for reading papers of network meta-analysis
Geographic Reach Multiple continents/international 
Policy Influence Type Influenced training of practitioners or researchers
Description Citation in systematic reviews (multiple)
Geographic Reach Multiple continents/international 
Policy Influence Type Citation in systematic reviews
Description Presentation at NICE's Technical Forum (London 21-03-2012)
Geographic Reach National 
Policy Influence Type Influenced training of practitioners or researchers
Impact NICE issued a total of 42 Technology Assessment Guidance from January 2011 to June 2012. Indirect or mixed treatment comparison methods have been used in 26 of the 42 Technology Assessment Reviews (62%). Results of these Technology Assessment Reviews provided the important research evidence for NICE to develop national guidance to the NHS on the clinical and cost-effectiveness of a wide range of health technologies. The implementation of NICE guidance in practice can improve quality and cost-effectiveness of healthcare services by promoting cost-effective health technologies and avoiding the use ineffective treatments.
Description Presentation at Tongji Medical College (Wuhan 09-05-12)
Geographic Reach Asia 
Policy Influence Type Influenced training of practitioners or researchers
Description Presentation of preliminary findings at Cochrane Colloquium in 2010
Geographic Reach Multiple continents/international 
Policy Influence Type Influenced training of practitioners or researchers
Impact Findings from this work will help the development of guidance for indirect comparison of healthcare interventions.
Description Training of staff in the centre for drug evaluation in Taiwan
Geographic Reach Asia 
Policy Influence Type Influenced training of practitioners or researchers
Impact Invited to provide training on indirect comparison of different pharmacological intervention in Centre for Drug Evaluation in Taiwan
Title Trial similarity & evidence consistency assessment method 
Description We have revised the methods for assessing trial similarity and evidence consistency in indirect comparison of healthcare interventions. We are testing this new method in the ongoing project. 
Type Of Material Data analysis technique 
Provided To Others? No  
Impact The method is crucial for valid indirect comparison of healthcare interventions 
Description Cochrane Collaboration Multiple Treatment Meta-analysis Method Group 
Organisation The Cochrane Collaboration
Country Global 
Sector Charity/Non Profit 
PI Contribution We have contributed to the establishment of the Cochrane Collaboration Multiple treatment meta-analysis method group.
Collaborator Contribution Authors of Cochrane systematic reviewers participated a survey of using indirect and multiple treatment comparison methods in systematic reviews
Impact Establishment of the MTM group which included researchers and systematic reviewers with statistics, research synthesis, and clinical research backgrounds.
Start Year 2010