HOD: Assessing risk of bias in non-randomized studies of interventions

Research studies of the effects of interventions are critical to health technology assessment. Randomized trials are widely recognized as the most robust study design, but often they are difficult or impossible to conduct, so are not available. In such situations, we must rely on non-randomized studies. These often provide the best evidence, for example, about long-term outcomes or rare events, particularly adverse effects, and about many types of organizational or public health interventions. Systematic reviews such as Cochrane Reviews and evidence assessments informing NICE guidance include various types of non-randomized studies. It is essential that methods are available to evaluate these studies, so that decisions and recommendations are based on a full understanding of the strengths and weaknesses of the evidence. We propose a methodological programme targeted at understanding these strengths and weaknesses.

The work builds on our recent development of A Cochrane Risk Of Bias Assessment Tool: Non-Randomized Studies of Interventions (ACROBAT-NRSI). We regard the new tool as a major step forward that is likely to have a profound impact on the practice of systematic reviews, guideline development and primary research. However, initial development of ACROBAT-NRSI placed emphasis on particular types of studies in which a cohort of individuals is followed over time, and has involved only limited piloting. Furthermore, our experience from developing previous tools has shown that ongoing development and refinement is a necessary part of the tool development process.

In our proposed programme of work we aim to consolidate the first version of ACROBAT-NRSI by considering its face validity, that is, whether it adequately addresses a variety of biases as they are currently understood in the literature. We will then develop the tool further so that it appropriately covers a much larger array of types of non-randomized study and approaches to their analysis, including time series studies, before-after studies and natural experiments. This will involve development of 'signalling questions' to identify potential problems in the studies. Once we have produced a revised tool we will implement it in a large number of non-randomized studies of interventions. Besides creating a useful pool of worked examples for training purposes, this will allow us to examine the relationship between apparent problems in the studies (according to answers to the signalling questions) and the observed results of the studies, in a 'meta-epidemiology study'. Problematic studies may have systematically different results, or more variable results, than those without specific problems. Finally, we will facilitate appropriate implementation of ACROBAT-NRSI by producing an interactive, easy-to-use online version in which structure, guidance and a suite of informative examples are seamlessly integrated.

We propose a methodological programme of theoretical, applied and empirical research targeted at understanding strengths and weaknesses of non-randomized studies of the effects of interventions. The project will build on an initial version of A Cochrane Risk Of Bias Assessment Tool: Non-Randomized Studies of Interventions (ACROBAT-NRSI), which we made available in September 2014. First, we will consolidate the first version of ACROBAT-NRSI by considering its face validity. This will be achieved through formal reviews of literature relevant to its theoretical and empirical basis; and through investigation of the extent to which the tool identifies potential causes of discrepant results in specific instances where non-randomized studies and corresponding randomized trials have reached different conclusions. We will then develop ACROBAT-NRSI so that it addresses a wider variety of study designs, including time series studies, before-after studies and natural experiments, and analyses such as those based on instrumental variables and regression discontinuities. This will be done by setting up a series of working groups, culminating in a meeting of co-investigators and invited experts to decide on a revised tool version through consensus. We will then implement the revised tool in a large number of non-randomized studies within a meta-epidemiological study to examine empirically the relationship between assessments made using the tool and effect estimates. Finally, we will producing an interactive, easy-to-use online version of the revised tool in which structure, guidance and a suite of informative examples are seamlessly integrated.

Who might benefit from this research?
Our research will impact directly on those undertaking systematic reviews (or other types of evidence appraisal) that include non-randomized studies on the effects of interventions, and through this will have an important impact on those making evidence-based decisions about use of healthcare interventions. This includes clinicians; patients and the public; and policy and guideline developers such as NICE as well as other decision-making organizations in the UK and abroad. Other non-academic beneficiaries include funders and commissioners of reviews.

How might they benefit from this research?
Systematic reviews and other evidence appraisals will use a sophisticated new tool for assessing the risk of bias in non-randomized studies. Such studies are critical for assessing long-term outcomes or rare events, particularly adverse effects, and for evaluating many types of organizational and public health intervention. Our work will allow decision-making individuals and organizations to better understand the strengths and limitations of the available research, since the tool and findings from our empirical studies will allow them to better distinguish trustworthy evidence from untrustworthy evidence. Equipped with a framework for assessing these strengths and weaknesses, we anticipate that systematic reviewers will include more evidence about harms in their reviews. These developments will lead to more informed decisions, better health outcomes and more efficient use of resources. Furthermore, funders and commissioners will be better able to consider the methodological strengths of a funding application or protocol for either a primary non-randomized study or a systematic review.