Developing ROBIS - a new tool to assess the risk of bias in systematic reviews
Lead Research Organisation:
University of Bristol
Department Name: Social Medicine
Abstract
Summary (4000 characters max)
Systematic reviews synthesise evidence across primary studies and are generally considered to provide the most reliable form of evidence for informing decisions in health care. When assessing the reliability of the results of a systematic review the potential for bias in the results of the review should be considered. We are not aware of any existing tool that fulfils this role. The aim of this project is to develop a new quality assessment tool to assess the risk of bias in systematic reviews. We plan to call this tool the "ROBIS tool".
We will follow a four-staged approach, used in the development of previous critical appraisal tools, to develop the tool:
Stage 1. Defining the scope: The steering group will discuss and agree key features of the desired scope of ROBIS. We anticipate that these decision will cover aspects such as the focus of the tool, structure of the tool, and properties of the tool.
Stage 2. Reviewing the evidence base: We will conduct three reviews to inform the development of ROBIS:
Review 1: Classification of MECIR items
Cochrane have recently produced a set of "Methodological Expectations for Cochrane Reviews (MECIR)". We will review this guidance and classify each item as relating to risk of bias, variability/applicability of the review, and the reporting quality of the review.
Review 2: Review of existing quality assessment tools for systematic reviews
We will conduct a review of existing quality assessment tools for systematic reviews. We will conduct extensive literature searches to identify existing tools. We will include any tool or checklist designed to assess the quality of systematic reviews. Tools designed as guidelines for the conduct or reporting of systematic reviews will be excluded. We will extract data on the items covered by each of the tools, methods used to develop the tool, and any evaluation of the tool
Review 3: Review of studies that have used the AMSTAR tool
Although there is currently no accepted tool to assess the quality of systematic reviews, we believe that the AMSTAR tool is the most commonly used. We will carry out a review of studies that have used the AMSTAR tool to evaluate how the quality assessment of systematic reviews is conducted in practice.
The results of these reviews will be summarised in an evidence report.
Stage 3. Face-to-Face consensus meeting
We will hold a one-day meeting to develop a first draft of the ROBIS tool. We will aim to invite a group of around 25 methodological experts and reviewers working on systematic reviews to participate in this meeting. A first draft of the ROBIS tool will be produced following the meeting.
Stage 4. Piloting and refinement of the ROBIS tool
We will use multiple rounds of piloting to refine successively amended versions of the ROBIS tool. Online questionnaires will be developed to gather structured feedback for each round. In addition, we will hold workshops at relevant conferences where we will present the ROBIS tool and give participants the opportunity to pilot the tool and provide feedback. Pairs of reviewers working will pilot a draft version of the tool on a number of reviews. Once sufficient agreement has been reached, a final version of the tool will be agreed.
The ROBIS tool will be published in a peer reviewed journal and will be presented at relevant methodological conferences. We will also establish a website for the tool which will contain the tool itself and related documents providing further guidance and resources.
Systematic reviews synthesise evidence across primary studies and are generally considered to provide the most reliable form of evidence for informing decisions in health care. When assessing the reliability of the results of a systematic review the potential for bias in the results of the review should be considered. We are not aware of any existing tool that fulfils this role. The aim of this project is to develop a new quality assessment tool to assess the risk of bias in systematic reviews. We plan to call this tool the "ROBIS tool".
We will follow a four-staged approach, used in the development of previous critical appraisal tools, to develop the tool:
Stage 1. Defining the scope: The steering group will discuss and agree key features of the desired scope of ROBIS. We anticipate that these decision will cover aspects such as the focus of the tool, structure of the tool, and properties of the tool.
Stage 2. Reviewing the evidence base: We will conduct three reviews to inform the development of ROBIS:
Review 1: Classification of MECIR items
Cochrane have recently produced a set of "Methodological Expectations for Cochrane Reviews (MECIR)". We will review this guidance and classify each item as relating to risk of bias, variability/applicability of the review, and the reporting quality of the review.
Review 2: Review of existing quality assessment tools for systematic reviews
We will conduct a review of existing quality assessment tools for systematic reviews. We will conduct extensive literature searches to identify existing tools. We will include any tool or checklist designed to assess the quality of systematic reviews. Tools designed as guidelines for the conduct or reporting of systematic reviews will be excluded. We will extract data on the items covered by each of the tools, methods used to develop the tool, and any evaluation of the tool
Review 3: Review of studies that have used the AMSTAR tool
Although there is currently no accepted tool to assess the quality of systematic reviews, we believe that the AMSTAR tool is the most commonly used. We will carry out a review of studies that have used the AMSTAR tool to evaluate how the quality assessment of systematic reviews is conducted in practice.
The results of these reviews will be summarised in an evidence report.
Stage 3. Face-to-Face consensus meeting
We will hold a one-day meeting to develop a first draft of the ROBIS tool. We will aim to invite a group of around 25 methodological experts and reviewers working on systematic reviews to participate in this meeting. A first draft of the ROBIS tool will be produced following the meeting.
Stage 4. Piloting and refinement of the ROBIS tool
We will use multiple rounds of piloting to refine successively amended versions of the ROBIS tool. Online questionnaires will be developed to gather structured feedback for each round. In addition, we will hold workshops at relevant conferences where we will present the ROBIS tool and give participants the opportunity to pilot the tool and provide feedback. Pairs of reviewers working will pilot a draft version of the tool on a number of reviews. Once sufficient agreement has been reached, a final version of the tool will be agreed.
The ROBIS tool will be published in a peer reviewed journal and will be presented at relevant methodological conferences. We will also establish a website for the tool which will contain the tool itself and related documents providing further guidance and resources.
Technical Summary
Systematic reviews synthesise evidence across primary studies and are generally considered to provide the most reliable form of evidence for informing decisions in health care. When assessing the reliability of the results of a systematic review the potential for bias in the results of the review should be considered. The aim of this project is to develop the "ROBIS tool", a new quality assessment tool to assess the risk of bias in systematic reviews.
We will adopt the following four stage approach to develop ROBIS:
Stage 1. Defining the scope: The steering group will discuss and agree key features of the desired scope of ROBIS.
Stage 2. Reviewing the evidence base: We will conduct three reviews to inform the development of ROBIS. These include classification of MECIR (Methodological Expectations for Cochrane Reviews) items, review of existing quality assessment tools for systematic reviews, and a review of studies that have used the AMSTAR tool.
Stage 3. Face-to-Face consensus meeting
We will hold a one-day meeting of 25 methodological experts and systematic reviewers to develop a first draft of the ROBIS tool. Decisions on tool content will be made by informal consensus.
Stage 4. Piloting and refinement of the ROBIS tool
We will use multiple rounds of piloting to refine successively amended versions of the ROBIS tool. We will hold workshops at relevant conferences where we will present the ROBIS tool and give participants the opportunity to pilot the tool and provide feedback. Pairs of reviewers working will pilot a draft version of the tool on a number of reviews. Once sufficient agreement has been reached, a final version of the tool will be agreed.
The ROBIS tool will be published in a peer reviewed journal and will be presented at relevant methodological conferences. We will also establish a website for the tool which will contain the tool itself and related documents providing further guidance and resources.
We will adopt the following four stage approach to develop ROBIS:
Stage 1. Defining the scope: The steering group will discuss and agree key features of the desired scope of ROBIS.
Stage 2. Reviewing the evidence base: We will conduct three reviews to inform the development of ROBIS. These include classification of MECIR (Methodological Expectations for Cochrane Reviews) items, review of existing quality assessment tools for systematic reviews, and a review of studies that have used the AMSTAR tool.
Stage 3. Face-to-Face consensus meeting
We will hold a one-day meeting of 25 methodological experts and systematic reviewers to develop a first draft of the ROBIS tool. Decisions on tool content will be made by informal consensus.
Stage 4. Piloting and refinement of the ROBIS tool
We will use multiple rounds of piloting to refine successively amended versions of the ROBIS tool. We will hold workshops at relevant conferences where we will present the ROBIS tool and give participants the opportunity to pilot the tool and provide feedback. Pairs of reviewers working will pilot a draft version of the tool on a number of reviews. Once sufficient agreement has been reached, a final version of the tool will be agreed.
The ROBIS tool will be published in a peer reviewed journal and will be presented at relevant methodological conferences. We will also establish a website for the tool which will contain the tool itself and related documents providing further guidance and resources.
Planned Impact
The development of the ROBIS tool will benefit everyone involved in systematic reviews. In particular, agencies involved in guidelines development and critical appraisal of systematic reviews, such as the National Institute for Health and Clinical Excellence (NICE), the Cochrane Collaboration, the US Agency for Healthcare Research and Quality (AHRQ) and the German Institute for Quality and Efficiency in Health Care (IQWiG). The ROBIS tool will allow researchers involved in guidelines development to assess the risk of bias in systematic reviews which can then be taken into consideration when making recommendations for treatments. This will benefit clinicians who will be provided with evidence based recommendations which reflect the reliability of the evidence and will ultimately benefit patients who should receive better treatment.
Organisations
- University of Bristol (Lead Research Organisation)
- UNIVERSITY OF OXFORD (Collaboration)
- The Cochrane Collaboration (Collaboration)
- Equator Network (Collaboration)
- Kleijnen Systematic Reviews Ltd (Collaboration)
- UNIVERSITY OF BIRMINGHAM (Collaboration)
- University of Ottawa (Collaboration)
- University of Bristol (Collaboration)
Publications
Whiting P
(2016)
ROBIS: A new tool to assess risk of bias in systematic reviews was developed.
in Journal of clinical epidemiology
Whiting P
(2017)
A proposed framework for developing quality assessment tools.
in Systematic reviews
| Description | Citation and use in 108 PROSPERO registrations for systematic reviews, overviews and scoping reviews directly informing policy and practice |
| Geographic Reach | Multiple continents/international |
| Policy Influence Type | Citation in systematic reviews |
| Description | Influencing the development of systematic review standards by researchers in the Collaboration for Environmental Evidence |
| Geographic Reach | Multiple continents/international |
| Policy Influence Type | Influenced training of practitioners or researchers |
| Impact | The influence is likely to improve the quality of systematic reviews undertaken, reported and published in the field of environmental evidence. |
| Description | Developing Latitudes Network - an online Library of Assessment Tools and InsTruments Used to assess Data validity in Evidence Synthesis |
| Organisation | Equator Network |
| Country | United Kingdom |
| Sector | Charity/Non Profit |
| PI Contribution | This is a collaboration at an early stage. Our core steering group approached the steering group members of the Equator Network to help us build Latitudes Network online resources in partnership with them. Latitudes Network will be complementary to the Equator Network and provide an accessible online resource for quality and bias assessment tools for health care research. We have planned the initial meeting to plan a bid to fund this initiative. |
| Collaborator Contribution | This at an early stage. Partners will provide support for the project and will help us with their Experience with Equator Network. |
| Impact | We had several unsuccessful attempts to obtain funding. The first paper from this collaboration was published in J Clin Epidemiol in 2017. The collaboration is still active but it is difficult to push ahead without dedicated funding. After several unsuccessful attempts to secure funding we have now abandoned this project. |
| Start Year | 2013 |
| Description | Developing Latitudes Network - an online Library of Assessment Tools and InsTruments Used to assess Data validity in Evidence Synthesis |
| Organisation | Kleijnen Systematic Reviews Ltd |
| Country | United Kingdom |
| Sector | Private |
| PI Contribution | This is a collaboration at an early stage. Our core steering group approached the steering group members of the Equator Network to help us build Latitudes Network online resources in partnership with them. Latitudes Network will be complementary to the Equator Network and provide an accessible online resource for quality and bias assessment tools for health care research. We have planned the initial meeting to plan a bid to fund this initiative. |
| Collaborator Contribution | This at an early stage. Partners will provide support for the project and will help us with their Experience with Equator Network. |
| Impact | We had several unsuccessful attempts to obtain funding. The first paper from this collaboration was published in J Clin Epidemiol in 2017. The collaboration is still active but it is difficult to push ahead without dedicated funding. After several unsuccessful attempts to secure funding we have now abandoned this project. |
| Start Year | 2013 |
| Description | Developing Latitudes Network - an online Library of Assessment Tools and InsTruments Used to assess Data validity in Evidence Synthesis |
| Organisation | University of Birmingham |
| Country | United Kingdom |
| Sector | Academic/University |
| PI Contribution | This is a collaboration at an early stage. Our core steering group approached the steering group members of the Equator Network to help us build Latitudes Network online resources in partnership with them. Latitudes Network will be complementary to the Equator Network and provide an accessible online resource for quality and bias assessment tools for health care research. We have planned the initial meeting to plan a bid to fund this initiative. |
| Collaborator Contribution | This at an early stage. Partners will provide support for the project and will help us with their Experience with Equator Network. |
| Impact | We had several unsuccessful attempts to obtain funding. The first paper from this collaboration was published in J Clin Epidemiol in 2017. The collaboration is still active but it is difficult to push ahead without dedicated funding. After several unsuccessful attempts to secure funding we have now abandoned this project. |
| Start Year | 2013 |
| Description | Developing Latitudes Network - an online Library of Assessment Tools and InsTruments Used to assess Data validity in Evidence Synthesis |
| Organisation | University of Oxford |
| Department | Nuffield Department of Primary Care Health Sciences |
| Country | United Kingdom |
| Sector | Academic/University |
| PI Contribution | This is a collaboration at an early stage. Our core steering group approached the steering group members of the Equator Network to help us build Latitudes Network online resources in partnership with them. Latitudes Network will be complementary to the Equator Network and provide an accessible online resource for quality and bias assessment tools for health care research. We have planned the initial meeting to plan a bid to fund this initiative. |
| Collaborator Contribution | This at an early stage. Partners will provide support for the project and will help us with their Experience with Equator Network. |
| Impact | We had several unsuccessful attempts to obtain funding. The first paper from this collaboration was published in J Clin Epidemiol in 2017. The collaboration is still active but it is difficult to push ahead without dedicated funding. After several unsuccessful attempts to secure funding we have now abandoned this project. |
| Start Year | 2013 |
| Description | Developing ROBIS - a new tool to assess the risk of bias in systematic reviews |
| Organisation | Kleijnen Systematic Reviews Ltd |
| Country | United Kingdom |
| Sector | Private |
| PI Contribution | We designed the project and successfully applied for funding . I co-led the preliminary review work and produced relevant materials to be considered at the development meeting. I organised the international meeting of experts where we developed the first draft of the risk of bias tool. I am currently co-leading the refinement and making arrangements for piloting |
| Collaborator Contribution | Project design, application for funding, review, participation in the development meeting, refinement. We anticipate that some of the partners will contribute to piloting. |
| Impact | We established collaborations with many international organisations and experts who contributet to the development and piloting of the ROBIS tool, which was published in 2015 Paper: Whiting P, Savovic J, Higgins JP, Caldwell DM, Reeves BC, Shea B, Davies P, Kleijnen J, Churchill R; ROBIS group. ROBIS: A new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016 Jan;69:225-34. doi: 10.1016/j.jclinepi.2015.06.005. This paper was translated to Spanish and Italian. Presentations: We held numerous training workshops of the tool at Cochrane and other meetings Impact: We have provided training on ROBIS tool to some of the NICE guidelines development teams and to the Estonian Health Insurance Agency, who plan to use the tool for guidelines development. It is also one of the Cochrane-recommended tools for use in overviews. |
| Start Year | 2012 |
| Description | Developing ROBIS - a new tool to assess the risk of bias in systematic reviews |
| Organisation | The Cochrane Collaboration |
| Department | Cochrane Bias Methods Group |
| Country | Canada |
| Sector | Academic/University |
| PI Contribution | We designed the project and successfully applied for funding . I co-led the preliminary review work and produced relevant materials to be considered at the development meeting. I organised the international meeting of experts where we developed the first draft of the risk of bias tool. I am currently co-leading the refinement and making arrangements for piloting |
| Collaborator Contribution | Project design, application for funding, review, participation in the development meeting, refinement. We anticipate that some of the partners will contribute to piloting. |
| Impact | We established collaborations with many international organisations and experts who contributet to the development and piloting of the ROBIS tool, which was published in 2015 Paper: Whiting P, Savovic J, Higgins JP, Caldwell DM, Reeves BC, Shea B, Davies P, Kleijnen J, Churchill R; ROBIS group. ROBIS: A new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016 Jan;69:225-34. doi: 10.1016/j.jclinepi.2015.06.005. This paper was translated to Spanish and Italian. Presentations: We held numerous training workshops of the tool at Cochrane and other meetings Impact: We have provided training on ROBIS tool to some of the NICE guidelines development teams and to the Estonian Health Insurance Agency, who plan to use the tool for guidelines development. It is also one of the Cochrane-recommended tools for use in overviews. |
| Start Year | 2012 |
| Description | Developing ROBIS - a new tool to assess the risk of bias in systematic reviews |
| Organisation | The Cochrane Collaboration |
| Country | Global |
| Sector | Charity/Non Profit |
| PI Contribution | We designed the project and successfully applied for funding . I co-led the preliminary review work and produced relevant materials to be considered at the development meeting. I organised the international meeting of experts where we developed the first draft of the risk of bias tool. I am currently co-leading the refinement and making arrangements for piloting |
| Collaborator Contribution | Project design, application for funding, review, participation in the development meeting, refinement. We anticipate that some of the partners will contribute to piloting. |
| Impact | We established collaborations with many international organisations and experts who contributet to the development and piloting of the ROBIS tool, which was published in 2015 Paper: Whiting P, Savovic J, Higgins JP, Caldwell DM, Reeves BC, Shea B, Davies P, Kleijnen J, Churchill R; ROBIS group. ROBIS: A new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016 Jan;69:225-34. doi: 10.1016/j.jclinepi.2015.06.005. This paper was translated to Spanish and Italian. Presentations: We held numerous training workshops of the tool at Cochrane and other meetings Impact: We have provided training on ROBIS tool to some of the NICE guidelines development teams and to the Estonian Health Insurance Agency, who plan to use the tool for guidelines development. It is also one of the Cochrane-recommended tools for use in overviews. |
| Start Year | 2012 |
| Description | Developing ROBIS - a new tool to assess the risk of bias in systematic reviews |
| Organisation | University of Bristol |
| Department | School of Clinical Sciences |
| Country | United Kingdom |
| Sector | Academic/University |
| PI Contribution | We designed the project and successfully applied for funding . I co-led the preliminary review work and produced relevant materials to be considered at the development meeting. I organised the international meeting of experts where we developed the first draft of the risk of bias tool. I am currently co-leading the refinement and making arrangements for piloting |
| Collaborator Contribution | Project design, application for funding, review, participation in the development meeting, refinement. We anticipate that some of the partners will contribute to piloting. |
| Impact | We established collaborations with many international organisations and experts who contributet to the development and piloting of the ROBIS tool, which was published in 2015 Paper: Whiting P, Savovic J, Higgins JP, Caldwell DM, Reeves BC, Shea B, Davies P, Kleijnen J, Churchill R; ROBIS group. ROBIS: A new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016 Jan;69:225-34. doi: 10.1016/j.jclinepi.2015.06.005. This paper was translated to Spanish and Italian. Presentations: We held numerous training workshops of the tool at Cochrane and other meetings Impact: We have provided training on ROBIS tool to some of the NICE guidelines development teams and to the Estonian Health Insurance Agency, who plan to use the tool for guidelines development. It is also one of the Cochrane-recommended tools for use in overviews. |
| Start Year | 2012 |
| Description | Developing ROBIS - a new tool to assess the risk of bias in systematic reviews |
| Organisation | University of Ottawa |
| Country | Canada |
| Sector | Academic/University |
| PI Contribution | We designed the project and successfully applied for funding . I co-led the preliminary review work and produced relevant materials to be considered at the development meeting. I organised the international meeting of experts where we developed the first draft of the risk of bias tool. I am currently co-leading the refinement and making arrangements for piloting |
| Collaborator Contribution | Project design, application for funding, review, participation in the development meeting, refinement. We anticipate that some of the partners will contribute to piloting. |
| Impact | We established collaborations with many international organisations and experts who contributet to the development and piloting of the ROBIS tool, which was published in 2015 Paper: Whiting P, Savovic J, Higgins JP, Caldwell DM, Reeves BC, Shea B, Davies P, Kleijnen J, Churchill R; ROBIS group. ROBIS: A new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016 Jan;69:225-34. doi: 10.1016/j.jclinepi.2015.06.005. This paper was translated to Spanish and Italian. Presentations: We held numerous training workshops of the tool at Cochrane and other meetings Impact: We have provided training on ROBIS tool to some of the NICE guidelines development teams and to the Estonian Health Insurance Agency, who plan to use the tool for guidelines development. It is also one of the Cochrane-recommended tools for use in overviews. |
| Start Year | 2012 |
| Description | Presentations at the First International Conference of the Collaboration for Environmental Evidence |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Two members of the team were invited to present at this conference on a themed session on How to Ensure Only High-Quality Systematic Reviews Get Published: Strategies from Medical and Chemical Risk Research. Two related presentations were provided (Assessing Risk of Bias in Systematic Reviews using the ROBIS tool; Methodological expectations in systematic review: raising the standard). |
| Year(s) Of Engagement Activity | 2016 |
| Description | ROBIS Cochrane Colloquium workshop (Cape Town) |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | We ran a workshop for potential users of the ROBIS tool (largely Cochrane members, clinicians, guideline developers and policy-makers) and also collected further data on its usability. |
| Year(s) Of Engagement Activity | 2017 |
| Description | ROBIS Cochrane Colloquium workshop (Vienna) |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | We ran a workshop for potential users of the ROBIS tool (largely Cochrane members, clinicians, guideline developers and policy-makers) and also collected further data on its usability. |
| Year(s) Of Engagement Activity | 2015 |
| Description | ROBIS NICE workshop |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Policymakers/politicians |
| Results and Impact | We were invited to provide training for the evidence synthesis teams at NICE to help them decide how they could use the tool as part of their guideline development (particularly for guideline updates). |
| Year(s) Of Engagement Activity | 2015 |
| Description | ROBIS development workshops (York) |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Other academic audiences (collaborators, peers etc.) |
| Results and Impact | We held two ROBIS workshops for systematic reviewers in York, UK (May and October 2014). These gave participants the opportunity to pilot the tool and provide feedback on the practical issues associated with using the tool, which were then incorporated into the guidance document. The feedback from both activities was incorporated into the guidance document. |
| Year(s) Of Engagement Activity | 2014 |
| Description | ROBIS face to face consultation meeting and user feedback, Hyderabad Oct 2014 |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | Yes |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | The workshop resulted in helpful user feedback and minor iterations to the tool and the website platform. These were subsequently incorporated into the paper submitted to Annals of Internal Medicine. Following the workshop, a number of people contacted us to ask if they could use the tool in their day to day work. |
| Year(s) Of Engagement Activity | 2014 |
| URL | https://colloquium.cochrane.org/signup/d/2014-09-26 |
| Description | ROBIS face to face consultation meeting, Quebec Oct 2013 |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | Yes |
| Geographic Reach | International |
| Primary Audience | Other academic audiences (collaborators, peers etc.) |
| Results and Impact | Following the development of an initial draft ROBIS tool and discussion document, 30 advisory group participants attended a face to face meeting/workshop to reach provide feedback, engage in discussion and, where possible, achieve agreement on the tool. The meeting was organised to coincide with the Cochrane Colloquium in Quebec, Canada, Sept 2013. Invitees unable to attend were able to contribute comments by email. Good engagement from multiple stakeholder organisations on the development of the tool. Continued input to multiple iterations of the tool subsequent to the meeting via Delphi exercise. The input and contribution of participants/invitees was acknowledged in the subsequent paper submitted to Annals of Internal Medicine as follows: The ROBIS group Lars Beckmann (Institute for Quality and Efficiency in Health Care (IQWiG), Cologne, Germany), Patrick Bossuyt (University of Amsterdam), Deborah Caldwell (University of Bristol)*, Rachel Churchill (University of Bristol)*, Philippa Davies (University of Bristol)*, Kay Dickersin (US Cochrane Centre), Kerry Dwan (University of Liverpool), Julie Glanville (York Health Economics Consortium, University of York), Julian Higgins (University of Bristol)*, Jos Kleijnen (KSR York Ltd and University of Maastricht)*, Julia Kreis (Institute for Quality and Efficiency in Health Care (IQWiG), Cologne, Germany), Toby Lasserson (Cochrane Editorial Unit)*, Fergus Macbeth (Wales Cancer Trials Unit), Silvia Minozzi (??),Karel GM Moons (University of Utrecht), Matthew Page (Monash University), Barney Reeves (University of Bristol)*, Nancy, Santesso (McMaster University), Jelena Savovi? (University of Bristol)*, Christopher H Schmid (Brown University), Beth Shaw (NICE), Beverley Shea (University of Ottawa)*, David Tovey (Cochrane Editorial Unit)*, Peter Tugwell (University of Ottawa), Meera Viswanathan (RTI International), Jasvinder Singh (University of Alabama at Birmingham & Birmingham Veterans Affairs Medical Center), Penny Whiting (KSR York Ltd and University of Bristol)*, George Wells (University of Ottawa)*, Robert Wolff (KSR York Ltd.). *=steering group member |
| Year(s) Of Engagement Activity | 2013 |
| Description | Training of Estonian National Health Insurance Agency to use ROBIS tool as their new standard method of assessment of reviews used for developing clinical guidelines |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Policymakers/politicians |
| Results and Impact | Two members of the project team travelled to Tallinn, Estonia in December 2016, at the invitation of the Estonian National Health Insurance Agency, to train their guideline developers to use the ROBIS tool. The agency plans to use ROBIS tool as their new standard method of assessment of reviews used for developing clinical guidelines, replacing the AMSTAR tool they had used previously. |
| Year(s) Of Engagement Activity | 2016 |