Understanding the process and impact of within-study selective reporting bias

Lead Research Organisation: University of Liverpool
Department Name: Medical Statistics and Health Evaluation

Abstract

Published research will form a biased subset of all research undertaken if the decision on whether to publish, and which of several outcomes to report, is influenced by the results of the research. In the medical research field, empirical evidence suggests that studies achieving statistical significance are more likely to be submitted and accepted for publication. This bias may lead to incorrect decisions concerning effective healthcare. We intend to investigate how much of a threat such selective reporting is to decisions about patient care. We will gain an understanding of how this bias arises through interviews with medical researchers. The likely benefits of this work will be improvements in the methods for assessing how robust research results are to this source of bias and an increased awareness of the problem which may lead to its reduction.

Technical Summary

Systematic reviews can lead to important inferences, providing evidence to support treatments not widely used or showing that evidence is lacking to support treatments that are in wide current use, and thus it is essential to identify any potential bias in the approach to the analysis. Publication bias has previously been identified as a threat to the validity of a meta-analysis and there is now widespread acceptance of the need to search for unpublished studies. Recently new evidence has documented an additional threat to validity, the selective reporting of trial outcomes within published studies. It is vital that we establish how much of a threat within-study selective reporting is to the validity of meta-analysis. We will assess prevalence and impact in cohorts of aggregate data and individual patient data reviews. We will gain an understanding of how this bias arises both through interviews with trialists and a review of institutional guidelines on research conduct. Expected outputs from this work include the development of a valid method for assessing the likelihood of within-study selection in trial reports, recommendations for assessing whether inferences are robust to this potential bias, and recommendations for medical research training modules and research conduct guidelines.
 
Description An assessment of the impact of outcome reporting bias in randomised controlled trials within systematic trials
Geographic Reach Multiple continents/international 
Policy Influence Type Influenced training of practitioners or researchers
 
Description Impact of outcome reporting bias
Geographic Reach Multiple continents/international 
Policy Influence Type Influenced training of practitioners or researchers
 
Description Methods referred to in Cochrane handbook
Geographic Reach Multiple continents/international 
Policy Influence Type Influenced training of practitioners or researchers
 
Description MRC Network of Hubs for Trials Methodology Research N67
Amount £7,461 (GBP)
Funding ID N67 
Organisation Medical Research Council (MRC) 
Sector Academic/University
Country United Kingdom
Start 07/2015 
End 01/2016
 
Description Methodology Research Programme
Amount £422,069 (GBP)
Organisation Medical Research Council (MRC) 
Sector Academic/University
Country United Kingdom
Start 01/2013 
End 10/2015
 
Description Network of Hubs for Trials Methodology Research R18
Amount £46,220 (GBP)
Funding ID R18 
Organisation Medical Research Council (MRC) 
Sector Academic/University
Country United Kingdom
Start 05/2012 
End 04/2013
 
Description New Investigator Research Grant
Amount £139,014 (GBP)
Organisation Medical Research Council (MRC) 
Sector Academic/University
Country United Kingdom
Start 06/2012 
End 12/2013
 
Title Outcome matrix 
Description Outcome reporting bias is an under-recognised problem which can affect the inference in a systematic review. Reviewers should address explicitly the issue of missing outcome data in order for reviews to be considered a reliable source of evidence about healthcare practice. We have proposed an outcome matrix be used for this purpose (Figure 1 of BMJ publication) as a way of detecting potential outcome reporting bias in both included and excluded studies. 
Type Of Material Improvements to research infrastructure 
Provided To Others? No  
Impact We aim to change the behaviour of systematic reviewers such that they routinely consider the potential impact of outcome reporting bias on their review. This is part of the future dissemination plan for this work (see knowledge and future potential section). As part of this plan, we will encourage all reviewers to include an outcome matrix and text justifications for reporting bias classifications in their next review update. We will also contact the Cochrane Handbook Advisory Group, in collaboration with Professor Mike Clarke (Director, UK Cochrane Centre), to discuss the inclusion of outcome matrices in reviews. 
 
Description ORBIT workshop / presentations 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? Yes
Geographic Reach International
Primary Audience Health professionals
Results and Impact Members of the research team have presented the findings at several national and international research meetings. During the project, the ORBIT workshop was developed and delivered, usually by three of the team, at several Cochrane meetings (Dublin, Freiburg, Edinburgh, Halifax, Singapore). The workshop includes a small group exercise, in which participants read trial reports and use our classification system to determine the likelihood of outcome reporting bias. The examples we use are chosen to illustrate a variety of classifications and are taken from systematic reviews where we have obtained the information from the trialist themselves. The workshop also demonstrates how our classification of the risk of outcome reporting bias helps to assess the related dimension in the new 'Risk of bias' tool being used by the Cochrane Collaboration. The presentations have focussed on both the prevalence and impact of outcome reporting bias in a large unselected cohort of Cochrane systematic reviews (see BMJ publication) and interviews with trialists to understand the reasons for discrepancies between outcomes specified in the trial protocol and those reported in the study publication (submitted to BMJ).

The feedback evaluations received from the most recent workshop have been extremely positive with all participants finding the workshop useful, and 60-75% finding it very useful. Recently, the Cochrane Bias Methods Group designated this a 'core' workshop which enabled one of our team to attend the Cochrane Colloquium in Singapore for free.
Year(s) Of Engagement Activity 2006,2007,2008,2009