Meta-analysis, complexity and heterogeneity (MACH): methodological review and development of guidance

Systematic reviews are commonly used to summarise large bodies of evidence, particularly on clinical trials. They are increasingly used to inform decision-making in other fields that are relevant to human health and wellbeing, including the fields of public health, social welfare, agriculture, international development, crime, education and many others. Interventions are often complex and study designs and populations in the review are often varied, and researchers may be discouraged from conducting a meta-analysis. It has been argued, however, that meta-analysis is under-used. What factors drive the decision to conduct a meta-analysis, and what factors should be considered?

There is currently a lack of clear guidance for making a decision about whether to conduct a meta-analysis or not, which can be a particularly difficult decision when synthesising complex interventions. Current guidance tends to focus on statistical heterogeneity, although there are doubts as to whether this is always a critical concern (especially if there is conceptual homogeneity and the statistical variance can be modelled), and there are arguably many other concerns (statistical, substantive, and resource-related) that could or should be considered. The lack of guidance on this point may mean that opportunities for appropriate meta-analyses are being missed, while meta-analyses are sometimes inappropriately conducted. This then raises the concern: does it matter if the 'wrong' decision is made about whether to conduct a meta-analysis or not, either in relation to the findings of different synthesis techniques or the perceived credibility/utility of different synthesis outputs?

This project aims to explore factors in the decision to conduct a meta-analysis; to compare existing guidance in health-related fields and beyond to see what can be learned; and to compare the perceived utility of different approaches to evidence synthesis by reviewer users (policymakers and practitioners) in a new trial. This will investigate the impact of meta-analytic and narrative synthesis approaches on health-related decision-making. Overall, the findings of this project will be used to develop new guidance on the meta-analysis of complex interventions. The findings will be widely disseminated with the help of experts from other disciplines, to ensure that it is of value to those conducting systematic reviews in public health, as well as in other fields where the outcomes include human health and wellbeing.

OBJECTIVES AND APPROACH. The overarching aim is to develop guidance on making a decision as to whether to conduct a meta-analysis or not when reviewing quantitative evidence. The project involves five phases.

In Phases 1 and 2, we will conduct a methodological review and three workshops and consultations with experts, to inform the development of new guidance on the appropriate conduct of meta-analysis.

In Phase 3, we will use case studies of completed meta-analyses and systematic reviews to investigate the feasibility of conducting meta-analyses; to consider what different synthesis types add to a review; and to explore researchers' reasons for the adoption of certain synthesis types. We will conduct a small number of semi-structured interviews with review authors to explore the reasoning behind their decisions.

In Phase 4, we will investigate the impact of meta-analysis on the perceived utility of reviews to users by conducting a randomised controlled trial, by comparing a wholly narrative evidence synthesis with one that includes a meta-analysis. In Phase 5, we will run two workshops to draw together the elements of the project and produce usable guidance for reviewers.

APPLICATION OF THE RESEARCH RESULTS. The project will be informed throughout by an advisory group consisting of research methodologists and review users (e.g., policymakers, practitioners) to ensure the usefulness and appropriateness of the project outputs. The guidance will be disseminated widely. We anticipate that this work will aid better methodological decision making, leading to both fewer missed opportunities and inappropriate applications of meta-analysis. It will also provide guidance about the appropriate reporting of the decision process when writing up reviews. Finally, it will facilitate knowledge translation by gaining a better understanding of the perceived ease of use, credibility, and utility by decision-makers of different types of synthesis outputs.

The broader benefits of this research are likely to be substantial, but diffuse, accruing to a range of groups over a potentially long timescale. One key benefit of the research may be the increased (as well as more appropriate) use of meta-analysis across healthcare, public health and a range of other fields which have an impact on health (see the Case for Support document). Our experimental study on the presentation of research findings (Phase 4) will provide valuable data on how the conduct and reporting of meta-analyses can promote both researchers' and research users' understanding of research, and their uptake of its findings. A key strength of the proposed research is that the guidelines which form the primary output will be based not only on abstract statistical or methodological reasoning, but on a broadly-based understanding of how research relates to the manifold contexts within which it takes place (both the micro-level of how methodological decisions are made day-to-day, and the macro-level policy and governance frameworks which shape research agendas). Thus, the dissemination and implementation of the guidelines will help to promote not only good research practice, but also to contribute to increased understanding and uptake of research findings by policy-makers and practitioners. This will, in turn, lead to the development of policy and practice which is based more firmly on the available evidence.

Pathways to Impact:
The communication and dissemination of findings obviously constitute a core part of the proposed project. Indeed, several of the core dimensions of the substantive research will effectively double as communication strategies: the workshops and Advisory Group meetings (Phases 1 and 5) will help to ensure a broad awareness of, and participation in, the project among leading specialists in meta-analysis methodology; the experimental study (Phase 4) will help to engage research users more broadly; and the trans-disciplinary review of methods (Phase 2) will facilitate dissemination of the results across broad communities of research, including a wide range of disciplinary fields outside but relevant to health, which would otherwise be difficult to reach. As such, a substantial proportion of the resources will be used in arranging and conducting the workshops and facilitating dissemination of the findings. All members of the research team will be centrally involved in communication and impact generation activities. In identifying key audiences and participants for these, we will build on our existing connections with researchers both in the field of meta-analysis and evidence synthesis, and more broadly in health and other sectors. An important benefit of the project, beyond reviewing practice and developing guidance, will be to inform further work on the presentation of research findings, both generally and to policy-makers or practitioners. As noted in the 'Case for Support', meta-analyses tend to make a greater impact on individual decision-makers than research syntheses without meta-analysis. We will build on this finding, using both experimental and qualitative process data, in phase 4; in addition, phases 2 and 3 will add greater depth in terms of understanding how the institutional settings of research impact on how it is used and understood. Thus, the increased and more appropriate use of meta-analysis has a strong potential to ensure better and more evidence-informed decisions in policy and practice.
As noted in the 'Impact summary' section, the impacts of the research will potentially bring benefits across a very wide area of research, policy and practice. While the immediate beneficiaries will mostly be academic researchers, the project will potentially help to promote evidence-based decision-making across a much wider area. This will be a particular benefit in the non-health sectors covered by the project, which have seen recent growth in systematic review and meta-analysis methods.