Development of a method for adjusting trial results for biases in meta-analysis: combining generic evidence on bias with detailed trial assessment

Health-care decisions are increasingly made on the basis of the results of systematic reviews and meta-analyses. These are scientific reviews and analyses that aim to identify and integrate all reliable research studies pertinent to the question at hand. The Cochrane Collaboration is an international organisation undertaking systematic reviews and meta-analyses to compare interventions in all areas of health care. Its database currently includes over 4,500 systematic reviews, many of which contain meta-analyses to combine key results across multiple clinical trials of the same intervention.

The validity of a meta-analysis depends on the validity of the individual studies contained in it. Since January 2008, Cochrane authors have been encouraged to use a specific 'Risk of Bias' tool for assessing potential biases in the individual trials prior to conducting the meta-analysis. However, it is unclear how these assessments should be incorporated into the meta-analyses. Two recently proposed methods are: (A) to adjust the results using off-the-shelf adjustments based on what is known about previous trials with similar flaws (a data-based method); and (B) to adjust the results using expert opinion, elicited specifically for each trial in the meta-analysis (an opinion-based method).

In our research, we will make use of a newly constructed data set consisting of 300-400 recent meta-analyses with completed Risk of Bias assessments, from The Cochrane Collaboration. First we will use the research database to examine the agreement between opinion-based and data-based approaches to evaluating biases. On the basis of these investigations, we will develop an integrated approach to adjusting for bias in meta-analysis, which makes use of both data and opinion. We will ask experts to draw on the Risk of Bias table for each particular trial as well as data we have about biases from trials with the same types of flaws as the trial at hand. Three different approaches for integrating methods A and B will be considered.

To assess the practicalities of the proposed new integrated method, we will select two case study meta-analyses from the research database, one including fewer than 10 trials and one including 30-50 trials, and compare methods A, B and the newly developed integrated methods for bias adjustment in these meta-analyses.

On the basis of our experiences of using Risk of Bias tables, we will consider how Cochrane review authors could most usefully discuss and interpret the information presented.

Finally, we will develop guidelines for bias adjustment in meta-analysis, targeted at researchers carrying out Cochrane reviews or reviews commissioned by NICE or the HTA programme.

A meta-analysis of the results from relevant randomised trials is often regarded as the best evidence concerning the effectiveness of a healthcare intervention. In practice, however, randomised trials vary in methodological quality, and flaws in trial conduct can lead to biased estimation of the intervention effect. Two different methods of adjusting for bias in meta-analysis have been proposed recently: method A (from Welton et al.) constructs distributions for biases in the new meta-analysis using generic evidence from an external collection of meta-analyses; method B (from Turner et al.) relies on detailed assessment of the methodological quality of each trial in the new meta-analysis and elicitation of opinion about the degree of bias likely to result, which is used to make trial-specific adjustments for biases. Both approaches have advantages and disadvantages.

Our overall aim is to investigate and integrate these two approaches in order to produce a third approach that gains the advantages of both. We will work entirely within a Bayesian framework. We will first examine agreement in predicting the impact of a single methodological flaw (e.g. inadequate/unclear allocation concealment) between methods A and B, and then examine agreement in predicting the impact of multiple flaws. Building on the findings of these investigations, we will develop an integrated approach to addressing within-study bias in meta-analysis, combining generic evidence and detailed study information. Three different possible approaches to integrating the methods will be considered: data and opinion will be formally combined in a Bayesian analysis; or experts will be asked to choose ranges for biases on the basis of both the trial's Risk of Bias table and the generic bias distribution for all studies within the same bias profile, either qualitatively or quantitatively. We will provide two full case studies to compare methods A and B and one or more of the integrated approaches.

Hundreds of meta-analyses are carried out every year, covering every area of health care. Meta-analyses produce results that are more precise and therefore more influential than individual trials. Results from meta-analyses inform health care decisions made by individual doctors and health authorities and also wider public health policy decisions made by bodies such as the National Institute for Health and Clinical Excellence (NICE). Numerous meta-analyses are carried out directly to inform technology appraisals commissioned by NICE.

If a meta-analysis makes no allowance for biases caused by methodological flaws, there is a danger that the results could be biased and over-precise, which could lead to inappropriate healthcare decisions. Currently, there is no consensus in the meta-analysis and systematic reviews community over how best to make allowance for biases in meta-analysis. The two methods recently proposed by Welton et al. (method A) and Turner et al. (method B) have been well received and much discussed when presented at conferences. The papers have attracted early citations and also early adoption of the methods by others.

The new integrated method proposed would draw on the strengths of both existing methods and make use of the information now available in the 'Risk of Bias' tables completed by most Cochrane review authors. As a practical method for bias adjustment in meta-analysis, this method will have the potential to improve the results of numerous meta-analyses affected by biases, and thus lead to better-informed healthcare decisions.