HOD: Assessing risk of bias in non-randomized studies of interventions

Lead Research Organisation: University of Bristol
Department Name: Social Medicine

Abstract

Research studies of the effects of interventions are critical to health technology assessment. Randomized trials are widely recognized as the most robust study design, but often they are difficult or impossible to conduct, so are not available. In such situations, we must rely on non-randomized studies. These often provide the best evidence, for example, about long-term outcomes or rare events, particularly adverse effects, and about many types of organizational or public health interventions. Systematic reviews such as Cochrane Reviews and evidence assessments informing NICE guidance include various types of non-randomized studies. It is essential that methods are available to evaluate these studies, so that decisions and recommendations are based on a full understanding of the strengths and weaknesses of the evidence. We propose a methodological programme targeted at understanding these strengths and weaknesses.

The work builds on our recent development of A Cochrane Risk Of Bias Assessment Tool: Non-Randomized Studies of Interventions (ACROBAT-NRSI). We regard the new tool as a major step forward that is likely to have a profound impact on the practice of systematic reviews, guideline development and primary research. However, initial development of ACROBAT-NRSI placed emphasis on particular types of studies in which a cohort of individuals is followed over time, and has involved only limited piloting. Furthermore, our experience from developing previous tools has shown that ongoing development and refinement is a necessary part of the tool development process.

In our proposed programme of work we aim to consolidate the first version of ACROBAT-NRSI by considering its face validity, that is, whether it adequately addresses a variety of biases as they are currently understood in the literature. We will then develop the tool further so that it appropriately covers a much larger array of types of non-randomized study and approaches to their analysis, including time series studies, before-after studies and natural experiments. This will involve development of 'signalling questions' to identify potential problems in the studies. Once we have produced a revised tool we will implement it in a large number of non-randomized studies of interventions. Besides creating a useful pool of worked examples for training purposes, this will allow us to examine the relationship between apparent problems in the studies (according to answers to the signalling questions) and the observed results of the studies, in a 'meta-epidemiology study'. Problematic studies may have systematically different results, or more variable results, than those without specific problems. Finally, we will facilitate appropriate implementation of ACROBAT-NRSI by producing an interactive, easy-to-use online version in which structure, guidance and a suite of informative examples are seamlessly integrated.

Technical Summary

We propose a methodological programme of theoretical, applied and empirical research targeted at understanding strengths and weaknesses of non-randomized studies of the effects of interventions. The project will build on an initial version of A Cochrane Risk Of Bias Assessment Tool: Non-Randomized Studies of Interventions (ACROBAT-NRSI), which we made available in September 2014. First, we will consolidate the first version of ACROBAT-NRSI by considering its face validity. This will be achieved through formal reviews of literature relevant to its theoretical and empirical basis; and through investigation of the extent to which the tool identifies potential causes of discrepant results in specific instances where non-randomized studies and corresponding randomized trials have reached different conclusions. We will then develop ACROBAT-NRSI so that it addresses a wider variety of study designs, including time series studies, before-after studies and natural experiments, and analyses such as those based on instrumental variables and regression discontinuities. This will be done by setting up a series of working groups, culminating in a meeting of co-investigators and invited experts to decide on a revised tool version through consensus. We will then implement the revised tool in a large number of non-randomized studies within a meta-epidemiological study to examine empirically the relationship between assessments made using the tool and effect estimates. Finally, we will producing an interactive, easy-to-use online version of the revised tool in which structure, guidance and a suite of informative examples are seamlessly integrated.

Planned Impact

Who might benefit from this research?
Our research will impact directly on those undertaking systematic reviews (or other types of evidence appraisal) that include non-randomized studies on the effects of interventions, and through this will have an important impact on those making evidence-based decisions about use of healthcare interventions. This includes clinicians; patients and the public; and policy and guideline developers such as NICE as well as other decision-making organizations in the UK and abroad. Other non-academic beneficiaries include funders and commissioners of reviews.

How might they benefit from this research?
Systematic reviews and other evidence appraisals will use a sophisticated new tool for assessing the risk of bias in non-randomized studies. Such studies are critical for assessing long-term outcomes or rare events, particularly adverse effects, and for evaluating many types of organizational and public health intervention. Our work will allow decision-making individuals and organizations to better understand the strengths and limitations of the available research, since the tool and findings from our empirical studies will allow them to better distinguish trustworthy evidence from untrustworthy evidence. Equipped with a framework for assessing these strengths and weaknesses, we anticipate that systematic reviewers will include more evidence about harms in their reviews. These developments will lead to more informed decisions, better health outcomes and more efficient use of resources. Furthermore, funders and commissioners will be better able to consider the methodological strengths of a funding application or protocol for either a primary non-randomized study or a systematic review.

Publications

10 25 50
 
Description Cochrane Scientific Committee
Geographic Reach Multiple continents/international 
Policy Influence Type Participation in a advisory committee
 
Description Systematic Review Fellowship
Amount £91,000 (GBP)
Organisation National Institute for Health Research 
Sector Public
Country United Kingdom
Start 10/2016 
End 09/2018
 
Title Risk of bias tool for non-randomized studies of interventions (ROBINS-I) 
Description A tool for assessing risk of bias in the result of a non-randomized study of an intervention 
Type Of Material Improvements to research infrastructure 
Year Produced 2016 
Provided To Others? Yes  
Impact BMJ paper published in 2016. Altmetric score was over 270 after four months. 
URL http://www.riskofbias.info
 
Description Risk of bias in randomized trials and non-randomized studies 
Organisation Harvard University
Country United States 
Sector Academic/University 
PI Contribution Led the development of tools to assess randomized trials and non-randomized studies of interventions; and co-led the development of a tool to assess studies of exposure.
Collaborator Contribution Participation in regular teleconferences, emails and face-to-face meetings
Impact This is a multidisciplinary collaboration. We have developed the ROBINS-I tool (published in the BMJ) and the RoB 2.0 tool (yet to be published but available in draft form from www.riskofbias.info.
Start Year 2011
 
Description Risk of bias in randomized trials and non-randomized studies 
Organisation International Initiative for Impact Evaluation
Country United Kingdom 
Sector Charity/Non Profit 
PI Contribution Led the development of tools to assess randomized trials and non-randomized studies of interventions; and co-led the development of a tool to assess studies of exposure.
Collaborator Contribution Participation in regular teleconferences, emails and face-to-face meetings
Impact This is a multidisciplinary collaboration. We have developed the ROBINS-I tool (published in the BMJ) and the RoB 2.0 tool (yet to be published but available in draft form from www.riskofbias.info.
Start Year 2011
 
Description Risk of bias in randomized trials and non-randomized studies 
Organisation London School of Hygiene and Tropical Medicine (LSHTM)
Country United Kingdom 
Sector Academic/University 
PI Contribution Led the development of tools to assess randomized trials and non-randomized studies of interventions; and co-led the development of a tool to assess studies of exposure.
Collaborator Contribution Participation in regular teleconferences, emails and face-to-face meetings
Impact This is a multidisciplinary collaboration. We have developed the ROBINS-I tool (published in the BMJ) and the RoB 2.0 tool (yet to be published but available in draft form from www.riskofbias.info.
Start Year 2011
 
Description Risk of bias in randomized trials and non-randomized studies 
Organisation McGill University
Country Canada 
Sector Academic/University 
PI Contribution Led the development of tools to assess randomized trials and non-randomized studies of interventions; and co-led the development of a tool to assess studies of exposure.
Collaborator Contribution Participation in regular teleconferences, emails and face-to-face meetings
Impact This is a multidisciplinary collaboration. We have developed the ROBINS-I tool (published in the BMJ) and the RoB 2.0 tool (yet to be published but available in draft form from www.riskofbias.info.
Start Year 2011
 
Description Risk of bias in randomized trials and non-randomized studies 
Organisation Medical Research Council (MRC)
Department MRC Clinical Trials Unit
Country United Kingdom 
Sector Public 
PI Contribution Led the development of tools to assess randomized trials and non-randomized studies of interventions; and co-led the development of a tool to assess studies of exposure.
Collaborator Contribution Participation in regular teleconferences, emails and face-to-face meetings
Impact This is a multidisciplinary collaboration. We have developed the ROBINS-I tool (published in the BMJ) and the RoB 2.0 tool (yet to be published but available in draft form from www.riskofbias.info.
Start Year 2011
 
Description Risk of bias in randomized trials and non-randomized studies 
Organisation Monash University
Country Australia 
Sector Academic/University 
PI Contribution Led the development of tools to assess randomized trials and non-randomized studies of interventions; and co-led the development of a tool to assess studies of exposure.
Collaborator Contribution Participation in regular teleconferences, emails and face-to-face meetings
Impact This is a multidisciplinary collaboration. We have developed the ROBINS-I tool (published in the BMJ) and the RoB 2.0 tool (yet to be published but available in draft form from www.riskofbias.info.
Start Year 2011
 
Description Risk of bias in randomized trials and non-randomized studies 
Organisation National Institute of Environmental Health Sciences
Country United States 
Sector Academic/University 
PI Contribution Led the development of tools to assess randomized trials and non-randomized studies of interventions; and co-led the development of a tool to assess studies of exposure.
Collaborator Contribution Participation in regular teleconferences, emails and face-to-face meetings
Impact This is a multidisciplinary collaboration. We have developed the ROBINS-I tool (published in the BMJ) and the RoB 2.0 tool (yet to be published but available in draft form from www.riskofbias.info.
Start Year 2011
 
Description Risk of bias in randomized trials and non-randomized studies 
Organisation RTI International
Country United States 
Sector Charity/Non Profit 
PI Contribution Led the development of tools to assess randomized trials and non-randomized studies of interventions; and co-led the development of a tool to assess studies of exposure.
Collaborator Contribution Participation in regular teleconferences, emails and face-to-face meetings
Impact This is a multidisciplinary collaboration. We have developed the ROBINS-I tool (published in the BMJ) and the RoB 2.0 tool (yet to be published but available in draft form from www.riskofbias.info.
Start Year 2011
 
Description Risk of bias in randomized trials and non-randomized studies 
Organisation The Cochrane Collaboration
Department UK Cochrane Centre
Country United Kingdom 
Sector Charity/Non Profit 
PI Contribution Led the development of tools to assess randomized trials and non-randomized studies of interventions; and co-led the development of a tool to assess studies of exposure.
Collaborator Contribution Participation in regular teleconferences, emails and face-to-face meetings
Impact This is a multidisciplinary collaboration. We have developed the ROBINS-I tool (published in the BMJ) and the RoB 2.0 tool (yet to be published but available in draft form from www.riskofbias.info.
Start Year 2011
 
Description Risk of bias in randomized trials and non-randomized studies 
Organisation University of Birmingham
Country United Kingdom 
Sector Academic/University 
PI Contribution Led the development of tools to assess randomized trials and non-randomized studies of interventions; and co-led the development of a tool to assess studies of exposure.
Collaborator Contribution Participation in regular teleconferences, emails and face-to-face meetings
Impact This is a multidisciplinary collaboration. We have developed the ROBINS-I tool (published in the BMJ) and the RoB 2.0 tool (yet to be published but available in draft form from www.riskofbias.info.
Start Year 2011
 
Description Risk of bias in randomized trials and non-randomized studies 
Organisation University of Ottawa
Country Canada 
Sector Academic/University 
PI Contribution Led the development of tools to assess randomized trials and non-randomized studies of interventions; and co-led the development of a tool to assess studies of exposure.
Collaborator Contribution Participation in regular teleconferences, emails and face-to-face meetings
Impact This is a multidisciplinary collaboration. We have developed the ROBINS-I tool (published in the BMJ) and the RoB 2.0 tool (yet to be published but available in draft form from www.riskofbias.info.
Start Year 2011
 
Description Risk of bias in randomized trials and non-randomized studies 
Organisation University of Oxford
Department Weatherall Institute of Molecular Medicine (WIMM)
Country United Kingdom 
Sector Public 
PI Contribution Led the development of tools to assess randomized trials and non-randomized studies of interventions; and co-led the development of a tool to assess studies of exposure.
Collaborator Contribution Participation in regular teleconferences, emails and face-to-face meetings
Impact This is a multidisciplinary collaboration. We have developed the ROBINS-I tool (published in the BMJ) and the RoB 2.0 tool (yet to be published but available in draft form from www.riskofbias.info.
Start Year 2011
 
Description Risk of bias in randomized trials and non-randomized studies 
Organisation University of Paris - Descartes
Country France 
Sector Academic/University 
PI Contribution Led the development of tools to assess randomized trials and non-randomized studies of interventions; and co-led the development of a tool to assess studies of exposure.
Collaborator Contribution Participation in regular teleconferences, emails and face-to-face meetings
Impact This is a multidisciplinary collaboration. We have developed the ROBINS-I tool (published in the BMJ) and the RoB 2.0 tool (yet to be published but available in draft form from www.riskofbias.info.
Start Year 2011
 
Description Risk of bias in randomized trials and non-randomized studies 
Organisation University of Southern Denmark
Country Denmark 
Sector Academic/University 
PI Contribution Led the development of tools to assess randomized trials and non-randomized studies of interventions; and co-led the development of a tool to assess studies of exposure.
Collaborator Contribution Participation in regular teleconferences, emails and face-to-face meetings
Impact This is a multidisciplinary collaboration. We have developed the ROBINS-I tool (published in the BMJ) and the RoB 2.0 tool (yet to be published but available in draft form from www.riskofbias.info.
Start Year 2011
 
Description ROBINS-I workshop at Cochrane conferences 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Workshops to demonstrate and train in the use of the ROBINS-I tool at the Cochrane Colloquium in Seoul (2016) and Global Evidence Summit in Cape Town (2017)
Year(s) Of Engagement Activity 2016,2017
 
Description ROBINS-I workshop at Cochrane meetings 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Workshop to demonstrate and train in the use of the ROBINS-I tool at the Cochrane editors meetings in Birmingham (2016) and Geneva (2017)
Year(s) Of Engagement Activity 2016,2017
 
Description Swiss Epidemiology Winterschool 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact A three-day interactive course on the ROBINS-I tool and other risk-of-bias assessment tools.
Year(s) Of Engagement Activity 2017,2018,2019
 
Description TV show, Brazil 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Media (as a channel to the public)
Results and Impact I was the interview subject for an edition of "Medicina Baseada em Evidências", a 30-minute TV show in Brazil hosted by Alvaro Atallah and broadcast on 28 public open channels in Brazil; first broadcast 5 May 2016; estimated audience 100,000 viewers.
Year(s) Of Engagement Activity 2016