Understanding the process and impact of within-study selective reporting bias for harm outcomes (ORBIT II - Outcome Reporting Bias in Trials).

The old medical tenet of "Do no harm" has revolutionised interest in health care safety in modern day practice. Balanced decision-making in health care requires that both the intended beneficial effects and the unwanted adverse effects of interventions are considered. Measures such as the benefit-harm ratio have been used in order to quantify how much the amount of benefit of a treatment clearly outweighs the amount of harm. Such measures are aimed at clinicians and patients as a medical resource for informing treatment decisions and improving patient care. In some cases, only if there is a favourable harm-benefit ratio, will a treatment be considered ethical to use.

Systematic reviews aim to include all relevant studies conducted on a particular topic and to provide an unbiased summary of their results taking into account the best evidence on the benefits and harms of medical treatments. Recommendations made by the National Institute for Health and Clinical Excellence (NICE), who issue guidance on the use of treatments and procedures within the NHS, are based largely on systematic reviews. Results from systematic reviews are also the basis for many benefit-harm ratios.

However, benefit-harm assessments could be misleading if the available evidence from these systematic reviews are prone to potential sources of bias that are unknown to the researchers. A recent study has shown that outcome reporting bias (ORB) for efficacy outcomes has the potential to overturn statistically significant treatment effects as a result of the lack of reporting of non-significant results, incomplete reporting or complete suppression of outcome data. Empirical evidence suggests that the incomplete reporting of harms data is likely to be higher than that of efficacy measures and the mechanisms for the missingness may well differ as the decision to suppress harms data may be related to the undesirable nature of the harms data to be disseminated in the researchers' view. To our knowledge no data exists on the impact of ORB in harms in systematic reviews.

The proposed programme aims to improve decision-making in the NHS in the area of benefit-harm analysis. We aim to do this through two related projects on methods: (A) to estimate the prevalence and assess the impact of selective outcome reporting in trials within a cohort of published meta-analyses whose outcome is a harm and (B) to further understand the data collection methods of harms data in clinical trials.

Overall, the proposed programme aims to raise the awareness of the problem of outcome reporting bias in harms and to ultimately improve the ability of decision makers to make informed decisions that consider both benefits and harms of an intervention, ultimately improving patient safety.

Research Objectives:

(a) To estimate the prevalence and assess the impact of selective outcome reporting in trials within a cohort of published meta-analyses where the outcome is a harm;
(b) Estimate the sensitivity and specificity of a method for assessing the likelihood of outcome reporting bias within trial reports in relation to harms;
(c) To compute the benefit-harm ratio in a selection of reviews that contain a meta-analysis of the primary harm outcome where ORB is suspected and look at the effect of ORB on this ratio;
(d) To understand the mechanisms that may lead to incomplete reporting of harms data, in particular the selective or incomplete reporting of either significant or non-significant findings.

To estimate the prevalence and assess the impact of selective reporting in trials within a cohort of published meta-analyses on the review primary harm outcome. To develop a methodology for assessing the impact of harms taking into account the mechanism for missing outcome data. This will include an investigation into the maximum bias bound method. For reviews that have demonstrated evidence of ORB, the benefit-harm ratio will be calculated to quantify whether the amount of benefit of a treatment clearly outweighs the amount of risk. Benefit-harm ratios will be calculated based on (i) the original analyses reported in the review (ii) the adjusted estimates once ORB has been accounted for. This comparison between (i) and (ii) will provide a clearer picture of the impact of ORB on the decision making process.

An email survey of trialists will also be carried out. The purpose of the survey is to understand the process behind a trialists approach when developing methods for collecting, analysing and reporting data on harms.

The main beneficiary of knowledge arising from this research is anticipated to be the consumers of the Cochrane Collaboration. Through this innovative research our aim is to raise the awareness of outcome reporting bias to researchers and demonstrate that it is essential that unbiased estimates of both benefits and harms are needed in the decision-making process. Through our training initiatives systematic reviewers will firstly learn how to identify outcome reporting bias (ORB) in their reviews and secondly learn how to adjust for it should the missing outcome data not be obtainable from the trialists. All our training material will be made publically available on our website. During the grant tenure, we will engage with the editors of the Cochrane Handbook such that guidance on assessing ORB work can be included in the next edition of the handbook. We anticipate that this will be a big step forward for putting our methods for assessment into policy and practice for all future reviews.

We will also send out an email survey to trialists, to gain an understanding of the methods for collecting, analysing harms data. Whilst raising the awareness of ORB, we will also encourage trialists to adopt methods described in the CONSORT extension (Consolidated Standards of Reporting Trials) for harm reporting. This incentive will not only help reduce the prevalence of ORB in harm outcomes in the future but also alleviate most other problems arising from inadequate reporting of randomised controlled trials. This approach will offer a standard way for reporting harms in all trials.