GRADE extension for complex social interventions

Lead Research Organisation: University of Birmingham
Department Name: Institute of Applied Social Sciences

Abstract

It is now standard practice for policy makers to draw on systematic reviews as the superior source of evidence to inform decision-making regarding effective interventions. One of the crucial steps in systematic review conduct and recommendation formulation involves a synthesis and rating of the quality of research evidence to help determine its validity and relevance for practice. However, social interventions, commonly applied within the disciplines of International Development, Psychology, Education, Criminology, Public Health, Social Work and Welfare are often complex. This means they might involve a number of interacting components, multiple outcomes, and diverse delivery formats. In addition, they might be more amenable to contextual factors and intervention implementation issues. For the research synthesis endeavours to be effective in guiding policy and practice, they must utilise adequate methodology that reflects the unique features of these interventions. Without this, research conclusions may be incorrect rendering policy recommendations erroneous and interventions inefficient.
The most prominent system to guide evidence-informed decision-making has been developed by the Grading of Recommendations Assessment, Development and Evaluation Working Group (GRADE) in clinical medicine. The GRADE approach provides a systematic and transparent process for rating the "best-available evidence" to inform recommendations for practice. Having been endorsed by more than 80 organisations worldwide, including the Cochrane Collaboration and the World Health Organisation, the applicability of the GRADE approach outside of clinical practice has been questioned, in part because of its neglect of important considerations of complex social interventions. Results from our previous investigation on the application of the GRADE approach in complex interventions indicate that GRADE may have limitations for these interventions. By way of illustration, GRADE initially starts with "low" quality rating for all types of observational studies; meanwhile, many social and health policy interventions are not possible to evaluate other than through observational and quasi-experimental study designs. This may lead to misinterpretations of evidence when transferred into practice, and therefore discourage important decisions. Furthermore, GRADE does not integrate important considerations in evaluation of these interventions, such as the implementation of an intervention, and integration of different types of the evidence. This suggests the need for a new methodological framework to summarise and rate the quality of evidence to inform decision-making in social disciplines.
The proposed project will develop and disseminate a framework for rating the quality of evidence and the strength of recommendations in complex social interventions. This new framework will be an official extension to the GRADE approach. For the purposes of guidance, a preliminary Steering Committee of leading experts and evaluation specialists across social disciplines has been established. With their coordination, this project will first organise an international online panel involving multidisciplinary expertise to generate a list of evidence quality definitions and criteria that will reflect the particular aspects of social interventions. Prior to the panel, a systematic review of the existing quality assessment tools and evidence grading systems will be undertaken with a specific focus on these interventions to elucidate the main areas of target. Following the online expert panel, a consensus meeting will be hosted with a select group of participants to generate the final list of definitions and criteria for the new framework. The generated GRADE extension will be published in high-impact social science journals, disseminated online, as well as in academic and professional conferences and meetings.

Planned Impact

Apart from the impact on the academic and research sectors as described in the "Academic Beneficiaries" section, this project has a range of beneficiaries within the wider public sector and policy-making sphere both in low- and middle-income countries, and high-income countries.
Initially, the GRADE project was launched with the aim of providing a structured and transparent framework to guide healthcare management decisions and the processes of guideline development. Consequently, the GRADE extension for complex social interventions will promote transparency and consistency in the decision-making and organisation of care in a wide array of disciplines outside of clinical medicine, including International Development, Public Health, Psychology, Education, Criminology, Social Work and Welfare. By way of illustration, through the application of the generated framework agencies like Department of International Development, World Health Organisation (WHO), National Institute of Health and Care Excellence (NICE) public health and social care guidelines, Social Care Institute for Excellence (SCIE), Substance Abuse and Mental Health Services Administration (SAMHSA) among other others, that synthesise research evidence to provide guidance for effective social and healthcare interventions, will have appropriate methodology to summarise and rate the quality of evidence for these interventions and thus determine and adequately communicate their true value for practice. In addition, research organisations such as the Cochrane and Campbell Collaborations, International Initiative for Impact Evaluation (3ie), and Agency for Healthcare Research and Quality (AHRQ) will benefit from this framework in their aims of promoting high-quality research evidence syntheses regarding the effectiveness of complex psychosocial, public health, educational and development interventions. This will in turn better inform practice guidelines, policy-making and future research, and thus increase the efficiency of public services and social and health policies in a long run. In addition, this project will help optimise and enhance the cost-effectiveness of government investments when financing the design and implementation of different social, health promotion and economic development interventions to enhance social and health outcomes both in the UK, and internationally.
The GRADE extension for social interventions will allow policy-makers and practitioners to apply research evidence more consistently and explicitly in their decision-making. As already mentioned, social welfare will improve, because of enhanced rigour of empirical evidence for service providers to consult when offering various interventions. Meanwhile, consumers of interventions will have indirect benefits because of the augmented services tailored to their specific needs and circumstances. Dissemination of evidence-based practices and policy-making across core social disciplines will significantly increase health, quality of life and social wellbeing of all populations served by these interventions.

Publications

10 25 50

publication icon
Concannon TW (2019) Practical Guidance for Involving Stakeholders in Health Research. in Journal of general internal medicine

Related Projects

Project Reference Relationship Related To Start End Award Value
ES/N012267/1 01/01/2016 25/03/2017 £586,424
ES/N012267/2 Transfer ES/N012267/1 26/03/2017 31/12/2018 £381,724
 
Description Agency for Healthcare Research and Quality (AHRQ) 
Organisation Agency for Healthcare Research and Quality
Country United States 
Sector Public 
PI Contribution Prof Paul Montgomery has been asked to review the development of the Preferred Reporting Items for Systematic Reviews and Meta- Analyses extension for complex interventions (PRISMA-CI).
Collaborator Contribution The AHRQ PRISMA-CI project team provides ongoing support and contribution to developing the GRADE guidance for complex social interventions.
Impact Representatives from the AHRQ PRISMA-CI project will attend the consensus meeting for developing the GRADE guidance for complex social interventions in May 2017, in Oxford.
Start Year 2016
 
Description Cross-Whitehall Trial Advice Panel 
Organisation Cross-Government Trial Advice Panel
Country United Kingdom 
Sector Public 
PI Contribution Professor Paul Montgomery has been appointed to the Cross-Whitehall Trial Advice Panel, a group of top academics and government representatives, who will support the design and implementation of effective trials to improve the effectiveness of government policies. This is the first time the Government has brought these experts together with civil servants for such a purpose.
Collaborator Contribution Prof Montgomery will increase skills and awareness across the civil service, in understanding the value of experimental and quasi-experimental methods, and how to deliver high quality trials.
Impact https://www.gov.uk/government/publications/cross-government-trial-advice-panel-role-and-membership
Start Year 2015
 
Description GRADE Working Group 
Organisation Grading of Recommendations Assessment, Development and Evaluation Working Group
Country Global 
Sector Learned Society 
PI Contribution The Grading of Recommendations Assessment, Development and Evaluation (short GRADE) is an informal collaboration of people with an interest in addressing the shortcomings of grading systems in health care. There are several subgroup within this group, which aim to advance the GRADE methodology for specific evidence types and contexts. Prof Paul Montgomery, Dr Sean Grant and Ms Ani Movsisyan currently lead the GRADE subgroup on complex interventions. The group meets twice a year to discuss the workplace and outputs. The next meeting will take place in Rome, in April 2017.
Collaborator Contribution The GRADE Working Group provides ongoing support and advice for our research project.
Impact The research team is currently working on two manuscripts that will be official GRADE papers: GRADE guidance for complex interventions (methods, procedures, outputs); GRADE guidance for complex interventions (explanation and elaboration).
Start Year 2015
 
Description World Health Organization (WHO) 
Organisation World Health Organization (WHO)
Country Global 
Sector Public 
PI Contribution In the beginning of 2016, The World Health Organization (WHO) launched a project aiming to strengthen the process and methods for retrieval, synthesis and assessment of evidence on complex multi-disciplinary interventions. The collaboration with the WHO project began in February 2016. Prof Paul Montgomery, Dr Sean Grant and Ms Ani Movsisyan currently lead one of the working groups of this WHO project, which specifically tackles the issue of assessing and rating the certainty in the body of evidence of complex social interventions.
Collaborator Contribution The WHO project has 5 main working groups each tackling various steps in the process of evidence synthesis and guideline development for complex social interventions. This way the collaboration with the WHO project informs the keys aspects of the project, such as how to conceptualise complex interventions and what are the methods for synthesising evidence, which are important for thinking about how to develop guidance for rating the certainty in the synthesised body of evidence.
Impact In collaboration with the WHO project, Prof Paul Montgomery, Dr Sean Grant and Ms Ani Movsisyan are working on a manuscript, which will be submitted for publication in May along the outputs of other working groups of the WHO project in a special series in the Lancet Global Health.
Start Year 2016
 
Description Applying the GRADE approach to complex inventions: an empirical investigation and development of a GRADE extention 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Presentation at the World Congress on Public Health & Nutrition, Madrid, Spain - 12 March 2016
Year(s) Of Engagement Activity 2016
 
Description Developing GRADE guidance for rating the certainty of evidence of complex interventions. 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact GRADE Working Group meeting. Rome, Italy.
Year(s) Of Engagement Activity 2017
 
Description Development of a GRADE extension for complex social interventions. 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Policymakers/politicians
Results and Impact Oral presentation at Seventh EUSPR Conference and Members' Meeting, The European Society for Prevention Conference. Berlin, Germany.
Year(s) Of Engagement Activity 2017
 
Description Using GRADE in systematic reviews of complex interventions. 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Policymakers/politicians
Results and Impact A two-day workshop organised by the Department of Maternal, Newborn, Child and Adolescent Health, World Health Organization on retrieval, synthesis and assessment of evidence on complex, multidisciplinary interventions. Freising, Germany.
Year(s) Of Engagement Activity 2017
 
Description Using the GRADE approach in systematic reviews of complex interventions: the framework, challenges and ways forward 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Study participants or study members
Results and Impact Centre for Evidence-based Intervention (CEBI) Research Group Meeting, University of Oxford (20 July 2017)
Year(s) Of Engagement Activity 2017