Developing ROBIS - a new tool to assess the risk of bias in systematic reviews

Lead Research Organisation: University of Bristol
Department Name: Social Medicine

Abstract

Summary (4000 characters max)
Systematic reviews synthesise evidence across primary studies and are generally considered to provide the most reliable form of evidence for informing decisions in health care. When assessing the reliability of the results of a systematic review the potential for bias in the results of the review should be considered. We are not aware of any existing tool that fulfils this role. The aim of this project is to develop a new quality assessment tool to assess the risk of bias in systematic reviews. We plan to call this tool the "ROBIS tool".

We will follow a four-staged approach, used in the development of previous critical appraisal tools, to develop the tool:
Stage 1. Defining the scope: The steering group will discuss and agree key features of the desired scope of ROBIS. We anticipate that these decision will cover aspects such as the focus of the tool, structure of the tool, and properties of the tool.

Stage 2. Reviewing the evidence base: We will conduct three reviews to inform the development of ROBIS:
Review 1: Classification of MECIR items
Cochrane have recently produced a set of "Methodological Expectations for Cochrane Reviews (MECIR)". We will review this guidance and classify each item as relating to risk of bias, variability/applicability of the review, and the reporting quality of the review.

Review 2: Review of existing quality assessment tools for systematic reviews
We will conduct a review of existing quality assessment tools for systematic reviews. We will conduct extensive literature searches to identify existing tools. We will include any tool or checklist designed to assess the quality of systematic reviews. Tools designed as guidelines for the conduct or reporting of systematic reviews will be excluded. We will extract data on the items covered by each of the tools, methods used to develop the tool, and any evaluation of the tool

Review 3: Review of studies that have used the AMSTAR tool
Although there is currently no accepted tool to assess the quality of systematic reviews, we believe that the AMSTAR tool is the most commonly used. We will carry out a review of studies that have used the AMSTAR tool to evaluate how the quality assessment of systematic reviews is conducted in practice.
The results of these reviews will be summarised in an evidence report.

Stage 3. Face-to-Face consensus meeting
We will hold a one-day meeting to develop a first draft of the ROBIS tool. We will aim to invite a group of around 25 methodological experts and reviewers working on systematic reviews to participate in this meeting. A first draft of the ROBIS tool will be produced following the meeting.

Stage 4. Piloting and refinement of the ROBIS tool
We will use multiple rounds of piloting to refine successively amended versions of the ROBIS tool. Online questionnaires will be developed to gather structured feedback for each round. In addition, we will hold workshops at relevant conferences where we will present the ROBIS tool and give participants the opportunity to pilot the tool and provide feedback. Pairs of reviewers working will pilot a draft version of the tool on a number of reviews. Once sufficient agreement has been reached, a final version of the tool will be agreed.

The ROBIS tool will be published in a peer reviewed journal and will be presented at relevant methodological conferences. We will also establish a website for the tool which will contain the tool itself and related documents providing further guidance and resources.

Technical Summary

Systematic reviews synthesise evidence across primary studies and are generally considered to provide the most reliable form of evidence for informing decisions in health care. When assessing the reliability of the results of a systematic review the potential for bias in the results of the review should be considered. The aim of this project is to develop the "ROBIS tool", a new quality assessment tool to assess the risk of bias in systematic reviews.

We will adopt the following four stage approach to develop ROBIS:

Stage 1. Defining the scope: The steering group will discuss and agree key features of the desired scope of ROBIS.

Stage 2. Reviewing the evidence base: We will conduct three reviews to inform the development of ROBIS. These include classification of MECIR (Methodological Expectations for Cochrane Reviews) items, review of existing quality assessment tools for systematic reviews, and a review of studies that have used the AMSTAR tool.

Stage 3. Face-to-Face consensus meeting
We will hold a one-day meeting of 25 methodological experts and systematic reviewers to develop a first draft of the ROBIS tool. Decisions on tool content will be made by informal consensus.

Stage 4. Piloting and refinement of the ROBIS tool
We will use multiple rounds of piloting to refine successively amended versions of the ROBIS tool. We will hold workshops at relevant conferences where we will present the ROBIS tool and give participants the opportunity to pilot the tool and provide feedback. Pairs of reviewers working will pilot a draft version of the tool on a number of reviews. Once sufficient agreement has been reached, a final version of the tool will be agreed.

The ROBIS tool will be published in a peer reviewed journal and will be presented at relevant methodological conferences. We will also establish a website for the tool which will contain the tool itself and related documents providing further guidance and resources.

Planned Impact

The development of the ROBIS tool will benefit everyone involved in systematic reviews. In particular, agencies involved in guidelines development and critical appraisal of systematic reviews, such as the National Institute for Health and Clinical Excellence (NICE), the Cochrane Collaboration, the US Agency for Healthcare Research and Quality (AHRQ) and the German Institute for Quality and Efficiency in Health Care (IQWiG). The ROBIS tool will allow researchers involved in guidelines development to assess the risk of bias in systematic reviews which can then be taken into consideration when making recommendations for treatments. This will benefit clinicians who will be provided with evidence based recommendations which reflect the reliability of the evidence and will ultimately benefit patients who should receive better treatment.

Publications

10 25 50
publication icon
Whiting P (2016) ROBIS: A new tool to assess risk of bias in systematic reviews was developed. in Journal of clinical epidemiology

publication icon
Whiting P (2017) A proposed framework for developing quality assessment tools. in Systematic reviews

 
Description Citation and use in 108 PROSPERO registrations for systematic reviews, overviews and scoping reviews directly informing policy and practice
Geographic Reach Multiple continents/international 
Policy Influence Type Citation in systematic reviews
 
Description Influencing the development of systematic review standards by researchers in the Collaboration for Environmental Evidence
Geographic Reach Multiple continents/international 
Policy Influence Type Influenced training of practitioners or researchers
Impact The influence is likely to improve the quality of systematic reviews undertaken, reported and published in the field of environmental evidence.
 
Description Developing Latitudes Network - an online Library of Assessment Tools and InsTruments Used to assess Data validity in Evidence Synthesis 
Organisation Equator Network
Country United Kingdom 
Sector Charity/Non Profit 
PI Contribution This is a collaboration at an early stage. Our core steering group approached the steering group members of the Equator Network to help us build Latitudes Network online resources in partnership with them. Latitudes Network will be complementary to the Equator Network and provide an accessible online resource for quality and bias assessment tools for health care research. We have planned the initial meeting to plan a bid to fund this initiative.
Collaborator Contribution This at an early stage. Partners will provide support for the project and will help us with their Experience with Equator Network.
Impact We had several unsuccessful attempts to obtain funding. The first paper from this collaboration was published in J Clin Epidemiol in 2017. The collaboration is still active but it is difficult to push ahead without dedicated funding.
Start Year 2013
 
Description Developing Latitudes Network - an online Library of Assessment Tools and InsTruments Used to assess Data validity in Evidence Synthesis 
Organisation Kleijnen Systematic Reviews Ltd
Country United Kingdom 
Sector Private 
PI Contribution This is a collaboration at an early stage. Our core steering group approached the steering group members of the Equator Network to help us build Latitudes Network online resources in partnership with them. Latitudes Network will be complementary to the Equator Network and provide an accessible online resource for quality and bias assessment tools for health care research. We have planned the initial meeting to plan a bid to fund this initiative.
Collaborator Contribution This at an early stage. Partners will provide support for the project and will help us with their Experience with Equator Network.
Impact We had several unsuccessful attempts to obtain funding. The first paper from this collaboration was published in J Clin Epidemiol in 2017. The collaboration is still active but it is difficult to push ahead without dedicated funding.
Start Year 2013
 
Description Developing Latitudes Network - an online Library of Assessment Tools and InsTruments Used to assess Data validity in Evidence Synthesis 
Organisation University of Birmingham
Country United Kingdom 
Sector Academic/University 
PI Contribution This is a collaboration at an early stage. Our core steering group approached the steering group members of the Equator Network to help us build Latitudes Network online resources in partnership with them. Latitudes Network will be complementary to the Equator Network and provide an accessible online resource for quality and bias assessment tools for health care research. We have planned the initial meeting to plan a bid to fund this initiative.
Collaborator Contribution This at an early stage. Partners will provide support for the project and will help us with their Experience with Equator Network.
Impact We had several unsuccessful attempts to obtain funding. The first paper from this collaboration was published in J Clin Epidemiol in 2017. The collaboration is still active but it is difficult to push ahead without dedicated funding.
Start Year 2013
 
Description Developing Latitudes Network - an online Library of Assessment Tools and InsTruments Used to assess Data validity in Evidence Synthesis 
Organisation University of Oxford
Department Nuffield Department of Primary Care Health Sciences
Country United Kingdom 
Sector Academic/University 
PI Contribution This is a collaboration at an early stage. Our core steering group approached the steering group members of the Equator Network to help us build Latitudes Network online resources in partnership with them. Latitudes Network will be complementary to the Equator Network and provide an accessible online resource for quality and bias assessment tools for health care research. We have planned the initial meeting to plan a bid to fund this initiative.
Collaborator Contribution This at an early stage. Partners will provide support for the project and will help us with their Experience with Equator Network.
Impact We had several unsuccessful attempts to obtain funding. The first paper from this collaboration was published in J Clin Epidemiol in 2017. The collaboration is still active but it is difficult to push ahead without dedicated funding.
Start Year 2013
 
Description Developing ROBIS - a new tool to assess the risk of bias in systematic reviews 
Organisation Kleijnen Systematic Reviews Ltd
Country United Kingdom 
Sector Private 
PI Contribution We designed the project and successfully applied for funding . I co-led the preliminary review work and produced relevant materials to be considered at the development meeting. I organised the international meeting of experts where we developed the first draft of the risk of bias tool. I am currently co-leading the refinement and making arrangements for piloting
Collaborator Contribution Project design, application for funding, review, participation in the development meeting, refinement. We anticipate that some of the partners will contribute to piloting.
Impact We established collaborations with many international organisations and experts who contributet to the development and piloting of the ROBIS tool, which was published in 2015 Paper: Whiting P, Savovic J, Higgins JP, Caldwell DM, Reeves BC, Shea B, Davies P, Kleijnen J, Churchill R; ROBIS group. ROBIS: A new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016 Jan;69:225-34. doi: 10.1016/j.jclinepi.2015.06.005. This paper was translated to Spanish and Italian. Presentations: We held numerous training workshops of the tool at Cochrane and other meetings Impact: We have provided training on ROBIS tool to some of the NICE guidelines development teams and to the Estonian Health Insurance Agency, who plan to use the tool for guidelines development. It is also one of the Cochrane-recommended tools for use in overviews.
Start Year 2012
 
Description Developing ROBIS - a new tool to assess the risk of bias in systematic reviews 
Organisation The Cochrane Collaboration
Department Cochrane Bias Methods Group
Country Canada 
Sector Academic/University 
PI Contribution We designed the project and successfully applied for funding . I co-led the preliminary review work and produced relevant materials to be considered at the development meeting. I organised the international meeting of experts where we developed the first draft of the risk of bias tool. I am currently co-leading the refinement and making arrangements for piloting
Collaborator Contribution Project design, application for funding, review, participation in the development meeting, refinement. We anticipate that some of the partners will contribute to piloting.
Impact We established collaborations with many international organisations and experts who contributet to the development and piloting of the ROBIS tool, which was published in 2015 Paper: Whiting P, Savovic J, Higgins JP, Caldwell DM, Reeves BC, Shea B, Davies P, Kleijnen J, Churchill R; ROBIS group. ROBIS: A new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016 Jan;69:225-34. doi: 10.1016/j.jclinepi.2015.06.005. This paper was translated to Spanish and Italian. Presentations: We held numerous training workshops of the tool at Cochrane and other meetings Impact: We have provided training on ROBIS tool to some of the NICE guidelines development teams and to the Estonian Health Insurance Agency, who plan to use the tool for guidelines development. It is also one of the Cochrane-recommended tools for use in overviews.
Start Year 2012
 
Description Developing ROBIS - a new tool to assess the risk of bias in systematic reviews 
Organisation The Cochrane Collaboration
Country Global 
Sector Charity/Non Profit 
PI Contribution We designed the project and successfully applied for funding . I co-led the preliminary review work and produced relevant materials to be considered at the development meeting. I organised the international meeting of experts where we developed the first draft of the risk of bias tool. I am currently co-leading the refinement and making arrangements for piloting
Collaborator Contribution Project design, application for funding, review, participation in the development meeting, refinement. We anticipate that some of the partners will contribute to piloting.
Impact We established collaborations with many international organisations and experts who contributet to the development and piloting of the ROBIS tool, which was published in 2015 Paper: Whiting P, Savovic J, Higgins JP, Caldwell DM, Reeves BC, Shea B, Davies P, Kleijnen J, Churchill R; ROBIS group. ROBIS: A new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016 Jan;69:225-34. doi: 10.1016/j.jclinepi.2015.06.005. This paper was translated to Spanish and Italian. Presentations: We held numerous training workshops of the tool at Cochrane and other meetings Impact: We have provided training on ROBIS tool to some of the NICE guidelines development teams and to the Estonian Health Insurance Agency, who plan to use the tool for guidelines development. It is also one of the Cochrane-recommended tools for use in overviews.
Start Year 2012
 
Description Developing ROBIS - a new tool to assess the risk of bias in systematic reviews 
Organisation University of Bristol
Department School of Clinical Sciences
Country United Kingdom 
Sector Academic/University 
PI Contribution We designed the project and successfully applied for funding . I co-led the preliminary review work and produced relevant materials to be considered at the development meeting. I organised the international meeting of experts where we developed the first draft of the risk of bias tool. I am currently co-leading the refinement and making arrangements for piloting
Collaborator Contribution Project design, application for funding, review, participation in the development meeting, refinement. We anticipate that some of the partners will contribute to piloting.
Impact We established collaborations with many international organisations and experts who contributet to the development and piloting of the ROBIS tool, which was published in 2015 Paper: Whiting P, Savovic J, Higgins JP, Caldwell DM, Reeves BC, Shea B, Davies P, Kleijnen J, Churchill R; ROBIS group. ROBIS: A new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016 Jan;69:225-34. doi: 10.1016/j.jclinepi.2015.06.005. This paper was translated to Spanish and Italian. Presentations: We held numerous training workshops of the tool at Cochrane and other meetings Impact: We have provided training on ROBIS tool to some of the NICE guidelines development teams and to the Estonian Health Insurance Agency, who plan to use the tool for guidelines development. It is also one of the Cochrane-recommended tools for use in overviews.
Start Year 2012
 
Description Developing ROBIS - a new tool to assess the risk of bias in systematic reviews 
Organisation University of Ottawa
Country Canada 
Sector Academic/University 
PI Contribution We designed the project and successfully applied for funding . I co-led the preliminary review work and produced relevant materials to be considered at the development meeting. I organised the international meeting of experts where we developed the first draft of the risk of bias tool. I am currently co-leading the refinement and making arrangements for piloting
Collaborator Contribution Project design, application for funding, review, participation in the development meeting, refinement. We anticipate that some of the partners will contribute to piloting.
Impact We established collaborations with many international organisations and experts who contributet to the development and piloting of the ROBIS tool, which was published in 2015 Paper: Whiting P, Savovic J, Higgins JP, Caldwell DM, Reeves BC, Shea B, Davies P, Kleijnen J, Churchill R; ROBIS group. ROBIS: A new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016 Jan;69:225-34. doi: 10.1016/j.jclinepi.2015.06.005. This paper was translated to Spanish and Italian. Presentations: We held numerous training workshops of the tool at Cochrane and other meetings Impact: We have provided training on ROBIS tool to some of the NICE guidelines development teams and to the Estonian Health Insurance Agency, who plan to use the tool for guidelines development. It is also one of the Cochrane-recommended tools for use in overviews.
Start Year 2012
 
Description Presentations at the First International Conference of the Collaboration for Environmental Evidence 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Two members of the team were invited to present at this conference on a themed session on How to Ensure Only High-Quality Systematic Reviews Get Published: Strategies from Medical and Chemical Risk Research. Two related presentations were provided (Assessing Risk of Bias in Systematic Reviews using the ROBIS tool; Methodological expectations in systematic review: raising the standard).
Year(s) Of Engagement Activity 2016
 
Description ROBIS Cochrane Colloquium workshop (Cape Town) 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact We ran a workshop for potential users of the ROBIS tool (largely Cochrane members, clinicians, guideline developers and policy-makers) and also collected further data on its usability.
Year(s) Of Engagement Activity 2017
 
Description ROBIS Cochrane Colloquium workshop (Vienna) 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact We ran a workshop for potential users of the ROBIS tool (largely Cochrane members, clinicians, guideline developers and policy-makers) and also collected further data on its usability.
Year(s) Of Engagement Activity 2015
 
Description ROBIS NICE workshop 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Policymakers/politicians
Results and Impact We were invited to provide training for the evidence synthesis teams at NICE to help them decide how they could use the tool as part of their guideline development (particularly for guideline updates).
Year(s) Of Engagement Activity 2015
 
Description ROBIS development workshops (York) 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Other academic audiences (collaborators, peers etc.)
Results and Impact We held two ROBIS workshops for systematic reviewers in York, UK (May and October 2014). These gave participants the opportunity to pilot the tool and provide feedback on the practical issues associated with using the tool, which were then incorporated into the guidance document.

The feedback from both activities was incorporated into the guidance document.
Year(s) Of Engagement Activity 2014
 
Description ROBIS face to face consultation meeting and user feedback, Hyderabad Oct 2014 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? Yes
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact The workshop resulted in helpful user feedback and minor iterations to the tool and the website platform. These were subsequently incorporated into the paper submitted to Annals of Internal Medicine.

Following the workshop, a number of people contacted us to ask if they could use the tool in their day to day work.
Year(s) Of Engagement Activity 2014
URL https://colloquium.cochrane.org/signup/d/2014-09-26
 
Description ROBIS face to face consultation meeting, Quebec Oct 2013 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? Yes
Geographic Reach International
Primary Audience Other academic audiences (collaborators, peers etc.)
Results and Impact Following the development of an initial draft ROBIS tool and discussion document, 30 advisory group participants attended a face to face meeting/workshop to reach provide feedback, engage in discussion and, where possible, achieve agreement on the tool. The meeting was organised to coincide with the Cochrane Colloquium in Quebec, Canada, Sept 2013. Invitees unable to attend were able to contribute comments by email.

Good engagement from multiple stakeholder organisations on the development of the tool. Continued input to multiple iterations of the tool subsequent to the meeting via Delphi exercise.
The input and contribution of participants/invitees was acknowledged in the subsequent paper submitted to Annals of Internal Medicine as follows:
The ROBIS group
Lars Beckmann (Institute for Quality and Efficiency in Health Care (IQWiG), Cologne, Germany), Patrick Bossuyt (University of Amsterdam), Deborah Caldwell (University of Bristol)*, Rachel Churchill (University of Bristol)*, Philippa Davies (University of Bristol)*, Kay Dickersin (US Cochrane Centre), Kerry Dwan (University of Liverpool), Julie Glanville (York Health Economics Consortium, University of York), Julian Higgins (University of Bristol)*, Jos Kleijnen (KSR York Ltd and University of Maastricht)*, Julia Kreis (Institute for Quality and Efficiency in Health Care (IQWiG), Cologne, Germany), Toby Lasserson (Cochrane Editorial Unit)*, Fergus Macbeth (Wales Cancer Trials Unit), Silvia Minozzi (??),Karel GM Moons (University of Utrecht), Matthew Page (Monash University), Barney Reeves (University of Bristol)*, Nancy, Santesso (McMaster University), Jelena Savovi? (University of Bristol)*, Christopher H Schmid (Brown University), Beth Shaw (NICE), Beverley Shea (University of Ottawa)*, David Tovey (Cochrane Editorial Unit)*, Peter Tugwell (University of Ottawa), Meera Viswanathan (RTI International), Jasvinder Singh (University of Alabama at Birmingham & Birmingham Veterans Affairs Medical Center), Penny Whiting (KSR York Ltd and University of Bristol)*, George Wells (University of Ottawa)*, Robert Wolff (KSR York Ltd.).
*=steering group member
Year(s) Of Engagement Activity 2013
 
Description Training of Estonian National Health Insurance Agency to use ROBIS tool as their new standard method of assessment of reviews used for developing clinical guidelines 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Policymakers/politicians
Results and Impact Two members of the project team travelled to Tallinn, Estonia in December 2016, at the invitation of the Estonian National Health Insurance Agency, to train their guideline developers to use the ROBIS tool. The agency plans to use ROBIS tool as their new standard method of assessment of reviews used for developing clinical guidelines, replacing the AMSTAR tool they had used previously.
Year(s) Of Engagement Activity 2016