Understanding the process and impact of within-study selective reporting bias for harm outcomes (ORBIT II - Outcome Reporting Bias in Trials).
Lead Research Organisation:
University of Liverpool
Department Name: Institute of Translational Medicine
Abstract
The old medical tenet of "Do no harm" has revolutionised interest in health care safety in modern day practice. Balanced decision-making in health care requires that both the intended beneficial effects and the unwanted adverse effects of interventions are considered. Measures such as the benefit-harm ratio have been used in order to quantify how much the amount of benefit of a treatment clearly outweighs the amount of harm. Such measures are aimed at clinicians and patients as a medical resource for informing treatment decisions and improving patient care. In some cases, only if there is a favourable harm-benefit ratio, will a treatment be considered ethical to use.
Systematic reviews aim to include all relevant studies conducted on a particular topic and to provide an unbiased summary of their results taking into account the best evidence on the benefits and harms of medical treatments. Recommendations made by the National Institute for Health and Clinical Excellence (NICE), who issue guidance on the use of treatments and procedures within the NHS, are based largely on systematic reviews. Results from systematic reviews are also the basis for many benefit-harm ratios.
However, benefit-harm assessments could be misleading if the available evidence from these systematic reviews are prone to potential sources of bias that are unknown to the researchers. A recent study has shown that outcome reporting bias (ORB) for efficacy outcomes has the potential to overturn statistically significant treatment effects as a result of the lack of reporting of non-significant results, incomplete reporting or complete suppression of outcome data. Empirical evidence suggests that the incomplete reporting of harms data is likely to be higher than that of efficacy measures and the mechanisms for the missingness may well differ as the decision to suppress harms data may be related to the undesirable nature of the harms data to be disseminated in the researchers' view. To our knowledge no data exists on the impact of ORB in harms in systematic reviews.
The proposed programme aims to improve decision-making in the NHS in the area of benefit-harm analysis. We aim to do this through two related projects on methods: (A) to estimate the prevalence and assess the impact of selective outcome reporting in trials within a cohort of published meta-analyses whose outcome is a harm and (B) to further understand the data collection methods of harms data in clinical trials.
Overall, the proposed programme aims to raise the awareness of the problem of outcome reporting bias in harms and to ultimately improve the ability of decision makers to make informed decisions that consider both benefits and harms of an intervention, ultimately improving patient safety.
Systematic reviews aim to include all relevant studies conducted on a particular topic and to provide an unbiased summary of their results taking into account the best evidence on the benefits and harms of medical treatments. Recommendations made by the National Institute for Health and Clinical Excellence (NICE), who issue guidance on the use of treatments and procedures within the NHS, are based largely on systematic reviews. Results from systematic reviews are also the basis for many benefit-harm ratios.
However, benefit-harm assessments could be misleading if the available evidence from these systematic reviews are prone to potential sources of bias that are unknown to the researchers. A recent study has shown that outcome reporting bias (ORB) for efficacy outcomes has the potential to overturn statistically significant treatment effects as a result of the lack of reporting of non-significant results, incomplete reporting or complete suppression of outcome data. Empirical evidence suggests that the incomplete reporting of harms data is likely to be higher than that of efficacy measures and the mechanisms for the missingness may well differ as the decision to suppress harms data may be related to the undesirable nature of the harms data to be disseminated in the researchers' view. To our knowledge no data exists on the impact of ORB in harms in systematic reviews.
The proposed programme aims to improve decision-making in the NHS in the area of benefit-harm analysis. We aim to do this through two related projects on methods: (A) to estimate the prevalence and assess the impact of selective outcome reporting in trials within a cohort of published meta-analyses whose outcome is a harm and (B) to further understand the data collection methods of harms data in clinical trials.
Overall, the proposed programme aims to raise the awareness of the problem of outcome reporting bias in harms and to ultimately improve the ability of decision makers to make informed decisions that consider both benefits and harms of an intervention, ultimately improving patient safety.
Technical Summary
Research Objectives:
(a) To estimate the prevalence and assess the impact of selective outcome reporting in trials within a cohort of published meta-analyses where the outcome is a harm;
(b) Estimate the sensitivity and specificity of a method for assessing the likelihood of outcome reporting bias within trial reports in relation to harms;
(c) To compute the benefit-harm ratio in a selection of reviews that contain a meta-analysis of the primary harm outcome where ORB is suspected and look at the effect of ORB on this ratio;
(d) To understand the mechanisms that may lead to incomplete reporting of harms data, in particular the selective or incomplete reporting of either significant or non-significant findings.
To estimate the prevalence and assess the impact of selective reporting in trials within a cohort of published meta-analyses on the review primary harm outcome. To develop a methodology for assessing the impact of harms taking into account the mechanism for missing outcome data. This will include an investigation into the maximum bias bound method. For reviews that have demonstrated evidence of ORB, the benefit-harm ratio will be calculated to quantify whether the amount of benefit of a treatment clearly outweighs the amount of risk. Benefit-harm ratios will be calculated based on (i) the original analyses reported in the review (ii) the adjusted estimates once ORB has been accounted for. This comparison between (i) and (ii) will provide a clearer picture of the impact of ORB on the decision making process.
An email survey of trialists will also be carried out. The purpose of the survey is to understand the process behind a trialists approach when developing methods for collecting, analysing and reporting data on harms.
(a) To estimate the prevalence and assess the impact of selective outcome reporting in trials within a cohort of published meta-analyses where the outcome is a harm;
(b) Estimate the sensitivity and specificity of a method for assessing the likelihood of outcome reporting bias within trial reports in relation to harms;
(c) To compute the benefit-harm ratio in a selection of reviews that contain a meta-analysis of the primary harm outcome where ORB is suspected and look at the effect of ORB on this ratio;
(d) To understand the mechanisms that may lead to incomplete reporting of harms data, in particular the selective or incomplete reporting of either significant or non-significant findings.
To estimate the prevalence and assess the impact of selective reporting in trials within a cohort of published meta-analyses on the review primary harm outcome. To develop a methodology for assessing the impact of harms taking into account the mechanism for missing outcome data. This will include an investigation into the maximum bias bound method. For reviews that have demonstrated evidence of ORB, the benefit-harm ratio will be calculated to quantify whether the amount of benefit of a treatment clearly outweighs the amount of risk. Benefit-harm ratios will be calculated based on (i) the original analyses reported in the review (ii) the adjusted estimates once ORB has been accounted for. This comparison between (i) and (ii) will provide a clearer picture of the impact of ORB on the decision making process.
An email survey of trialists will also be carried out. The purpose of the survey is to understand the process behind a trialists approach when developing methods for collecting, analysing and reporting data on harms.
Planned Impact
The main beneficiary of knowledge arising from this research is anticipated to be the consumers of the Cochrane Collaboration. Through this innovative research our aim is to raise the awareness of outcome reporting bias to researchers and demonstrate that it is essential that unbiased estimates of both benefits and harms are needed in the decision-making process. Through our training initiatives systematic reviewers will firstly learn how to identify outcome reporting bias (ORB) in their reviews and secondly learn how to adjust for it should the missing outcome data not be obtainable from the trialists. All our training material will be made publically available on our website. During the grant tenure, we will engage with the editors of the Cochrane Handbook such that guidance on assessing ORB work can be included in the next edition of the handbook. We anticipate that this will be a big step forward for putting our methods for assessment into policy and practice for all future reviews.
We will also send out an email survey to trialists, to gain an understanding of the methods for collecting, analysing harms data. Whilst raising the awareness of ORB, we will also encourage trialists to adopt methods described in the CONSORT extension (Consolidated Standards of Reporting Trials) for harm reporting. This incentive will not only help reduce the prevalence of ORB in harm outcomes in the future but also alleviate most other problems arising from inadequate reporting of randomised controlled trials. This approach will offer a standard way for reporting harms in all trials.
We will also send out an email survey to trialists, to gain an understanding of the methods for collecting, analysing harms data. Whilst raising the awareness of ORB, we will also encourage trialists to adopt methods described in the CONSORT extension (Consolidated Standards of Reporting Trials) for harm reporting. This incentive will not only help reduce the prevalence of ORB in harm outcomes in the future but also alleviate most other problems arising from inadequate reporting of randomised controlled trials. This approach will offer a standard way for reporting harms in all trials.
Publications
Copas J
(2014)
A model-based correction for outcome reporting bias in meta-analysis.
in Biostatistics (Oxford, England)
Copas J
(2019)
Model-based sensitivity analysis for outcome reporting bias in the meta analysis of benefit and harm outcomes.
in Statistical methods in medical research
Kirkham JJ
(2012)
A multivariate meta-analysis approach for reducing the impact of outcome reporting bias in systematic reviews.
in Statistics in medicine
Reeves B
(2014)
Reporting of harms in systematic reviews and their primary studies
in BMJ
Saini P
(2014)
Selective reporting bias of harm outcomes within studies: findings from a cohort of systematic reviews.
in BMJ (Clinical research ed.)
Sterne JA
(2016)
ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions.
in BMJ (Clinical research ed.)
Sterne JAC
(2019)
RoB 2: a revised tool for assessing risk of bias in randomised trials.
in BMJ (Clinical research ed.)
Description | Consolidating guidelines for the prevention, detection and appraisal of reporting biases in clinical trials |
Amount | £46,220 (GBP) |
Organisation | Medical Research Council (MRC) |
Department | Network of Hubs for Trials Methodology Research (HTMR) |
Sector | Academic/University |
Country | United Kingdom |
Start | 04/2012 |
End | 04/2013 |
Description | Development of an interactive website for outcome reporting bias |
Amount | £7,461 (GBP) |
Organisation | Medical Research Council (MRC) |
Department | MRC Hubs for Trial Methodology Research |
Sector | Public |
Country | United Kingdom |
Start | 06/2015 |
End | 06/2016 |
Description | Multivariate meta-analysis of multiple correlated outcomes: development and application of methods, with empirical investigation of clinical impact. |
Amount | £422,069 (GBP) |
Funding ID | MR/J013595/1 |
Organisation | Medical Research Council (MRC) |
Sector | Public |
Country | United Kingdom |
Start | 01/2013 |
End | 10/2015 |
Description | Refinement of and extension to the Cochrane Risk of Bias tool for Randomised trials |
Amount | £49,133 (GBP) |
Organisation | Medical Research Council (MRC) |
Department | MRC Hubs for Trial Methodology Research |
Sector | Public |
Country | United Kingdom |
Start | 04/2015 |
End | 04/2016 |
Description | Selective reporting in clinical trials: a feasibility study for examining discrepancies between trial protocols and trial reports submitted to journals. |
Amount | £45,231 (GBP) |
Funding ID | R47 |
Organisation | Medical Research Council (MRC) |
Sector | Public |
Country | United Kingdom |
Start | 05/2014 |
End | 05/2015 |
Description | Workshop and collaborative exchange on outcome reporting bias in clinical trials |
Amount | £3,000 (GBP) |
Organisation | The Great Britain Sasakawa Foundation |
Sector | Charity/Non Profit |
Country | United Kingdom |
Start | 05/2014 |
End | 05/2015 |
Description | Cochrane Handbook - Update of harms chapter |
Organisation | Cochrane Adverse Effects Methods Group |
Country | United Kingdom |
Sector | Charity/Non Profit |
PI Contribution | Discussion on updating the Adverse Events section of the Cochrane Handbook taking into account the selective reporting of outcomes, how to assess this and how to implement the research tools in Cocranre reviews |
Collaborator Contribution | Handbook Chapter is currently under revision |
Impact | None avaliable yet. |
Start Year | 2016 |
Description | Cochrane Risk of Bias Tool for Non-Randomised Studies |
Organisation | University of Bristol |
Department | School of Social and Community Medicine |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | As part of the development of the new Cochrane Risk of Bias Tool for Non-Randomised studies, members of the research team have contributed to the development of the selective reporting element of the tool |
Collaborator Contribution | Provided input / examples through teleconferences, face-to-face meetings. Develop signalling questions for the new tool |
Impact | Tool to be launched in 2014. |
Start Year | 2012 |
Description | Copas adjustment method for harms |
Organisation | University of Warwick |
Department | Department of Statistics |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | Development of a new method to adjust for outcome reporting bias in harm outcomes |
Collaborator Contribution | Provided the methodological input surrounding the problem |
Impact | Publication (in-preparation). Development of method into a website for non-methodologists. |
Start Year | 2015 |
Description | Identification of Reviews |
Organisation | The Cochrane Collaboration |
Country | Global |
Sector | Charity/Non Profit |
PI Contribution | Highlight the importance of this work including aims and objectives. Engage in future collaboration. |
Collaborator Contribution | The collaboration has identified all systematic reviews for use in the project where the primary outcome of interest is a harm. |
Impact | None to date |
Start Year | 2012 |
Title | ORBIT Toolkit |
Description | The ORBIT toolkit via an interactive website will enable non-methodologists to apply our tools and adjustment methods for this form of bias. |
Type Of Technology | Webtool/Application |
Year Produced | 2016 |
Impact | Website only recently launched, although web-diagnostics are being monitored. |
URL | http://www.outcome-reporting-bias.org/ |
Description | Cochrane Colloquium 2015 |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | To demonstrate a prototype for our new ORBIT website and the ORBIT Toolkit. To engage with the 40+ potential users of the website to gain feedback and suggested inprovements. |
Year(s) Of Engagement Activity | 2015 |
URL | https://community.cochrane.org/ |
Description | HTMR Meeting (Oxford) |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Type Of Presentation | Paper Presentation |
Geographic Reach | National |
Primary Audience | Health professionals |
Results and Impact | In February 2013 the Hub for Trials Methodolgy Research Network hosted a research day in Oxford to showcase some of the ongoing work within its working groups. Current activities from the ORBIT II project was presented within the 'outcomes' working group breakaway session in which about 50-75 people attended. Members of the audience asked a series of questions about the research and many wanted to be kept up-to-date on progress. |
Year(s) Of Engagement Activity | 2012,2013 |
Description | Knowledge Exchange Activity - Japan |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | The ORBIT work was presented at two research institutes in Japan: University of Tokyo and Kyoto. About 50 Japanese professionals attended the workshops. The purpose was to raise awareness of the problem of outcome reporting bias and to engage particpants in workshop activities. Knowledge exchange between UK/Japan: future collaborations to help improve the reporting of trials conducted in Japan. |
Year(s) Of Engagement Activity | 2014 |
Description | Methods in Meta-Analysis Meeting |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Type Of Presentation | Paper Presentation |
Geographic Reach | National |
Primary Audience | Other academic audiences (collaborators, peers etc.) |
Results and Impact | The is a regular bi-annual meeting (held a the Royal Statistical Society in London) where experts in the field of meta-analysis present their current research to others in the field. It is usually attended by about 30 researchers. During the December 2012 meeting we presented the aims/objectives and planned methodolgies for this research project. Following the presentations, we received positive feedback about ths importance of this research as well as receiving guidance on extra items we might consider. The most notable which we did consider was investigating the problem in non-Cochrane reviews of harm outcomes. |
Year(s) Of Engagement Activity | 2012 |
Description | UK/Ireland Cochrane Meeting (Manchester) - ORBIT II workshop |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | About 20 healthcare professionals/statisiticians and systematic reviewers attended the ORBIT II workshop. The purpose was to raise awareness of the ORBIT study II findings and to engage particpants into thinking about improving harm reporting. Improved specification of harm outcomes in reviews and improved reporting of harms in both primary studies and reviews. |
Year(s) Of Engagement Activity | 2014 |