Understanding the process and impact of within-study selective reporting bias

Published research will form a biased subset of all research undertaken if the decision on whether to publish, and which of several outcomes to report, is influenced by the results of the research. In the medical research field, empirical evidence suggests that studies achieving statistical significance are more likely to be submitted and accepted for publication. This bias may lead to incorrect decisions concerning effective healthcare. We intend to investigate how much of a threat such selective reporting is to decisions about patient care. We will gain an understanding of how this bias arises through interviews with medical researchers. The likely benefits of this work will be improvements in the methods for assessing how robust research results are to this source of bias and an increased awareness of the problem which may lead to its reduction.

Systematic reviews can lead to important inferences, providing evidence to support treatments not widely used or showing that evidence is lacking to support treatments that are in wide current use, and thus it is essential to identify any potential bias in the approach to the analysis. Publication bias has previously been identified as a threat to the validity of a meta-analysis and there is now widespread acceptance of the need to search for unpublished studies. Recently new evidence has documented an additional threat to validity, the selective reporting of trial outcomes within published studies. It is vital that we establish how much of a threat within-study selective reporting is to the validity of meta-analysis. We will assess prevalence and impact in cohorts of aggregate data and individual patient data reviews. We will gain an understanding of how this bias arises both through interviews with trialists and a review of institutional guidelines on research conduct. Expected outputs from this work include the development of a valid method for assessing the likelihood of within-study selection in trial reports, recommendations for assessing whether inferences are robust to this potential bias, and recommendations for medical research training modules and research conduct guidelines.