Understanding the process and impact of within-study selective reporting bias for harm outcomes (ORBIT II - Outcome Reporting Bias in Trials).

Lead Research Organisation: University of Liverpool
Department Name: Institute of Translational Medicine

Abstract

The old medical tenet of "Do no harm" has revolutionised interest in health care safety in modern day practice. Balanced decision-making in health care requires that both the intended beneficial effects and the unwanted adverse effects of interventions are considered. Measures such as the benefit-harm ratio have been used in order to quantify how much the amount of benefit of a treatment clearly outweighs the amount of harm. Such measures are aimed at clinicians and patients as a medical resource for informing treatment decisions and improving patient care. In some cases, only if there is a favourable harm-benefit ratio, will a treatment be considered ethical to use.

Systematic reviews aim to include all relevant studies conducted on a particular topic and to provide an unbiased summary of their results taking into account the best evidence on the benefits and harms of medical treatments. Recommendations made by the National Institute for Health and Clinical Excellence (NICE), who issue guidance on the use of treatments and procedures within the NHS, are based largely on systematic reviews. Results from systematic reviews are also the basis for many benefit-harm ratios.

However, benefit-harm assessments could be misleading if the available evidence from these systematic reviews are prone to potential sources of bias that are unknown to the researchers. A recent study has shown that outcome reporting bias (ORB) for efficacy outcomes has the potential to overturn statistically significant treatment effects as a result of the lack of reporting of non-significant results, incomplete reporting or complete suppression of outcome data. Empirical evidence suggests that the incomplete reporting of harms data is likely to be higher than that of efficacy measures and the mechanisms for the missingness may well differ as the decision to suppress harms data may be related to the undesirable nature of the harms data to be disseminated in the researchers' view. To our knowledge no data exists on the impact of ORB in harms in systematic reviews.

The proposed programme aims to improve decision-making in the NHS in the area of benefit-harm analysis. We aim to do this through two related projects on methods: (A) to estimate the prevalence and assess the impact of selective outcome reporting in trials within a cohort of published meta-analyses whose outcome is a harm and (B) to further understand the data collection methods of harms data in clinical trials.

Overall, the proposed programme aims to raise the awareness of the problem of outcome reporting bias in harms and to ultimately improve the ability of decision makers to make informed decisions that consider both benefits and harms of an intervention, ultimately improving patient safety.

Technical Summary

Research Objectives:

(a) To estimate the prevalence and assess the impact of selective outcome reporting in trials within a cohort of published meta-analyses where the outcome is a harm;
(b) Estimate the sensitivity and specificity of a method for assessing the likelihood of outcome reporting bias within trial reports in relation to harms;
(c) To compute the benefit-harm ratio in a selection of reviews that contain a meta-analysis of the primary harm outcome where ORB is suspected and look at the effect of ORB on this ratio;
(d) To understand the mechanisms that may lead to incomplete reporting of harms data, in particular the selective or incomplete reporting of either significant or non-significant findings.

To estimate the prevalence and assess the impact of selective reporting in trials within a cohort of published meta-analyses on the review primary harm outcome. To develop a methodology for assessing the impact of harms taking into account the mechanism for missing outcome data. This will include an investigation into the maximum bias bound method. For reviews that have demonstrated evidence of ORB, the benefit-harm ratio will be calculated to quantify whether the amount of benefit of a treatment clearly outweighs the amount of risk. Benefit-harm ratios will be calculated based on (i) the original analyses reported in the review (ii) the adjusted estimates once ORB has been accounted for. This comparison between (i) and (ii) will provide a clearer picture of the impact of ORB on the decision making process.

An email survey of trialists will also be carried out. The purpose of the survey is to understand the process behind a trialists approach when developing methods for collecting, analysing and reporting data on harms.

Planned Impact

The main beneficiary of knowledge arising from this research is anticipated to be the consumers of the Cochrane Collaboration. Through this innovative research our aim is to raise the awareness of outcome reporting bias to researchers and demonstrate that it is essential that unbiased estimates of both benefits and harms are needed in the decision-making process. Through our training initiatives systematic reviewers will firstly learn how to identify outcome reporting bias (ORB) in their reviews and secondly learn how to adjust for it should the missing outcome data not be obtainable from the trialists. All our training material will be made publically available on our website. During the grant tenure, we will engage with the editors of the Cochrane Handbook such that guidance on assessing ORB work can be included in the next edition of the handbook. We anticipate that this will be a big step forward for putting our methods for assessment into policy and practice for all future reviews.

We will also send out an email survey to trialists, to gain an understanding of the methods for collecting, analysing harms data. Whilst raising the awareness of ORB, we will also encourage trialists to adopt methods described in the CONSORT extension (Consolidated Standards of Reporting Trials) for harm reporting. This incentive will not only help reduce the prevalence of ORB in harm outcomes in the future but also alleviate most other problems arising from inadequate reporting of randomised controlled trials. This approach will offer a standard way for reporting harms in all trials.

Publications

10 25 50
 
Description Consolidating guidelines for the prevention, detection and appraisal of reporting biases in clinical trials
Amount £46,220 (GBP)
Organisation Medical Research Council (MRC) 
Department Network of Hubs for Trials Methodology Research (HTMR)
Sector Academic/University
Country United Kingdom
Start 05/2012 
End 04/2013
 
Description Development of an interactive website for outcome reporting bias
Amount £7,461 (GBP)
Organisation Medical Research Council (MRC) 
Department MRC Hubs for Trial Methodology Research
Sector Public
Country United Kingdom
Start 07/2015 
End 06/2016
 
Description Multivariate meta-analysis of multiple correlated outcomes: development and application of methods, with empirical investigation of clinical impact.
Amount £422,069 (GBP)
Funding ID MR/J013595/1 
Organisation Medical Research Council (MRC) 
Sector Academic/University
Country United Kingdom
Start 01/2013 
End 10/2015
 
Description Refinement of and extension to the Cochrane Risk of Bias tool for Randomised trials
Amount £49,133 (GBP)
Organisation Medical Research Council (MRC) 
Department MRC Hubs for Trial Methodology Research
Sector Public
Country United Kingdom
Start 05/2015 
End 04/2016
 
Description Selective reporting in clinical trials: a feasibility study for examining discrepancies between trial protocols and trial reports submitted to journals.
Amount £45,231 (GBP)
Funding ID R47 
Organisation Medical Research Council (MRC) 
Sector Academic/University
Country United Kingdom
Start 06/2014 
End 05/2015
 
Description Workshop and collaborative exchange on outcome reporting bias in clinical trials
Amount £3,000 (GBP)
Organisation The Great Britain Sasakawa Foundation 
Sector Charity/Non Profit
Country United Kingdom
Start 06/2014 
End 05/2015
 
Description Cochrane Handbook - Update of harms chapter 
Organisation Cochrane Adverse Effects Methods Group
Country United Kingdom 
Sector Charity/Non Profit 
PI Contribution Discussion on updating the Adverse Events section of the Cochrane Handbook taking into account the selective reporting of outcomes, how to assess this and how to implement the research tools in Cocranre reviews
Collaborator Contribution Handbook Chapter is currently under revision
Impact None avaliable yet.
Start Year 2016
 
Description Cochrane Risk of Bias Tool for Non-Randomised Studies 
Organisation University of Bristol
Department School of Social and Community Medicine
Country United Kingdom 
Sector Academic/University 
PI Contribution As part of the development of the new Cochrane Risk of Bias Tool for Non-Randomised studies, members of the research team have contributed to the development of the selective reporting element of the tool
Collaborator Contribution Provided input / examples through teleconferences, face-to-face meetings. Develop signalling questions for the new tool
Impact Tool to be launched in 2014.
Start Year 2012
 
Description Copas adjustment method for harms 
Organisation University of Warwick
Department Department of Statistics
Country United Kingdom 
Sector Academic/University 
PI Contribution Development of a new method to adjust for outcome reporting bias in harm outcomes
Collaborator Contribution Provided the methodological input surrounding the problem
Impact Publication (in-preparation). Development of method into a website for non-methodologists.
Start Year 2015
 
Description Identification of Reviews 
Organisation The Cochrane Collaboration
Country Global 
Sector Charity/Non Profit 
PI Contribution Highlight the importance of this work including aims and objectives. Engage in future collaboration.
Collaborator Contribution The collaboration has identified all systematic reviews for use in the project where the primary outcome of interest is a harm.
Impact None to date
Start Year 2012
 
Title ORBIT Toolkit 
Description The ORBIT toolkit via an interactive website will enable non-methodologists to apply our tools and adjustment methods for this form of bias. 
Type Of Technology Webtool/Application 
Year Produced 2016 
Impact Website only recently launched, although web-diagnostics are being monitored. 
URL http://www.outcome-reporting-bias.org/
 
Description Cochrane Colloquium 2015 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact To demonstrate a prototype for our new ORBIT website and the ORBIT Toolkit. To engage with the 40+ potential users of the website to gain feedback and suggested inprovements.
Year(s) Of Engagement Activity 2015
URL https://community.cochrane.org/
 
Description HTMR Meeting (Oxford) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Type Of Presentation Paper Presentation
Geographic Reach National
Primary Audience Health professionals
Results and Impact In February 2013 the Hub for Trials Methodolgy Research Network hosted a research day in Oxford to showcase some of the ongoing work within its working groups. Current activities from the ORBIT II project was presented within the 'outcomes' working group breakaway session in which about 50-75 people attended.

Members of the audience asked a series of questions about the research and many wanted to be kept up-to-date on progress.
Year(s) Of Engagement Activity 2012,2013
 
Description Knowledge Exchange Activity - Japan 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact The ORBIT work was presented at two research institutes in Japan: University of Tokyo and Kyoto. About 50 Japanese professionals attended the workshops. The purpose was to raise awareness of the problem of outcome reporting bias and to engage particpants in workshop activities.

Knowledge exchange between UK/Japan: future collaborations to help improve the reporting of trials conducted in Japan.
Year(s) Of Engagement Activity 2014
 
Description Methods in Meta-Analysis Meeting 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Type Of Presentation Paper Presentation
Geographic Reach National
Primary Audience Other academic audiences (collaborators, peers etc.)
Results and Impact The is a regular bi-annual meeting (held a the Royal Statistical Society in London) where experts in the field of meta-analysis present their current research to others in the field. It is usually attended by about 30 researchers. During the December 2012 meeting we presented the aims/objectives and planned methodolgies for this research project.

Following the presentations, we received positive feedback about ths importance of this research as well as receiving guidance on extra items we might consider. The most notable which we did consider was investigating the problem in non-Cochrane reviews of harm outcomes.
Year(s) Of Engagement Activity 2012
 
Description UK/Ireland Cochrane Meeting (Manchester) - ORBIT II workshop 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact About 20 healthcare professionals/statisiticians and systematic reviewers attended the ORBIT II workshop. The purpose was to raise awareness of the ORBIT study II findings and to engage particpants into thinking about improving harm reporting.

Improved specification of harm outcomes in reviews and improved reporting of harms in both primary studies and reviews.
Year(s) Of Engagement Activity 2014