Investigating human influences in systematic reviews and meta-analyses to improve the reliability and validity of evidence syntheses

Billions of pounds are spent each year on health care treatments with the aim of improving and saving lives. But not all clinical research is published and what is published not be reliable. By bringing together similar studies, assessing their quality, and combined effects, more reliable findings can be used to guide future clinical and policy decisions. The gold standard of this method of reviewing studies is called systematic review. These large reports summarise all relevant studies in a clear and fair way to understand the benefits and drawbacks of new treatments. They are highly cited articles and regarded as objective and transparent research designs that are necessary to inform patient care.

There are increasing reports that systematic reviews are being published that do not live up to their high quality reputation. Systematic reviews are receiving attention for contributing to research waste, varying massively in their quality, being misleading and serving conflicted interests. When authors are asked by publishing journals to declare conflicts of interest, they relate to narrow criteria about funds received from industry within the last few years. Whilst clear guidelines for best practice in systematic reviews are available for authors, they do not provide guidance on review teams themselves. Reviews are sometimes carried out by people who have an interest in what the results are (bias). Or review teams might not take all the necessary efforts to find and present a full picture of the evidence needed for decision makers to understand (passivity). Members of the review team might have their own view on a treatment and this might shape the way the review is done. Review teams may not be sufficiently skilled or have the resources to carry out a high quality review (insufficient resources). Inflexible review methods may result in expensive review projects which do not reflect issues that matter to patients. Also, sometimes reviews are published without funding and authors' motivation for this is not questioned. The effects that human influence has on systematic reviews may be especially important when looking at health care problems that are complex.

Considering the reputation of systematic reviews as fair and objective processes, poor conduct could be more problematic than in clinical research. Health care evidence might not be evaluated fairly and patients may suffer because the health service may not provide the best treatments as a result.

Methods:
To fully explore the ways in which the researchers in the team can influence systematic reviews I plan to:
1) Search for and review studies documenting problems with systematic reviews that can occur due to team influence.
2) Work with experts in research methods to highlight risks and solutions for systematic review teams, helping to ensure that the findings of my research impact on systematic reviews that are carried out in the future.
4) Produce a dedicated website and guidance in a clear and accessible format about the risks of human influence in systematic reviews to help people assess systematic reviews for themselves.
5) Test guidance with researchers to refine and revise, ensuring that they are useful to people who do or use systematic reviews.

What impact will this research have?
This research will describe what makes an appropriate team and what types of team influence can cause problems to systematic reviews. The framework and guidance will help people who do or pay for research that involves a systematic review to understand what factors they need to consider when planning, funding or doing a systematic review. To influence the wider research agenda a user-friendly website that explains the goals, methods and results of the research will be created. This research will lead to better systematic reviews, to inform health care decision making for health services for patients and the public and to influence future health research.

Systematic reviews underpin modern evidence-based medicine. However, there is an increasing body of research showing that they are susceptible to bias and are often not conducted well, despite the availability of established guidelines. I will investigate factors related to researcher influence within systematic reviews and develop resources to reduce bias or poor conduct in future reviews, increasing the quality and rigour of future systematic reviews.

The effect that individuals may have on the conduct and conclusions of a systematic review is currently unknown. Researchers are responsible for making influential decisions at all stages of a review. Human influence over the conduct of a systematic review can occur due to author affiliation with the subject of investigation (bias or allegiance) but conversely review authors may be insufficiently skilled to take necessary efforts to perform a rigorous review. Such deviations from consistency and transparency, whether conscious or unconscious, can result in expensive outputs that fail to answer the review question. The full consequences to evidence-based medicine, of poor conduct in published reviews have not yet been explored.

Methods: Rigorous methods to examine systematic review research will be combined with input from methodological experts. The proposed research includes:
a) comprehensive systematic reviews and meta-regression team of human influences on systematic review outputs
b) consultation and consensus with international methodological experts on i) the risks of ignoring team influences in systematic review, ii) solutions for systematic reviews and users via guidance
c) highlighting the issues on the wider research agenda via a dedicated website

This work will allow commissioners, researchers, publishers and policy makers to have greater confidence that conduct of systematic reviews, relied upon to inform decisions, is being scrutinised and that methods are evolving, as in primary research.

Recent research highlights that the quality of published systematic reviews is often below what is assumed due to their reputation. The beneficiaries from this research in the short-term are commissioners and policy makers who use systematic reviews to make decisions; researchers who conduct or appraise systematic reviews; editors who publish systematic reviews and clinicians who use systematic reviews to make decisions about clinical care or treatments.

People who will benefit from this research in the longer-term include people who use the NHS, tax-payers who pay into the NHS; and other countries who use NHS/NICE decisions in developing clinical guidance. In the short-term, consumers of systematic reviews will have greater awareness of the pitfalls and biases that systematic reviews are prone to. Journal editors will have a clearer idea about which reviews to publish and which should signal red flags. Public engagement about the issues will put a spotlight on this area of evidence-based medicine and it is hoped that this will in turn raise the issues of human influences on the wider research agenda.

The longer-term impact will ensure that decisions on health and medical treatments are evaluated transparently by investigating and raising awareness of human influences such as bias and error in evidence syntheses. Systematic review products will be held to higher standards and reviews where team conduct is likely to have negatively influenced the validity of the reviews findings will lose credibility. The proposed work will lead to improvements in evidence syntheses that feed into health economic evaluations therefore tax-payers who contribute to the NHS and NHS patients will benefit by receiving the most effective and cost-effective treatments.

As a result of this work decision makers can have greater confidence that the systematic review products they rely upon are subject to continued scrutiny. Systematic reviews are highly cited articles which underpin global healthcare service efficiency such as public health evaluations, assessment of efficacy and effectiveness of medical treatments and economic evaluations. In the UK, systematic reviews have become the mainstay of evidence-based medicine for the NHS, NICE, NIHR and the Department of Health and Social Care. This collaborative project with international experts aims to develop consensus but wider public engagement of the issues will ensure that the messages from the research outputs become clear enough to ensure those who conduct, publish or appraise systematic reviews sit up and pay attention to the message that there are problems with systematic reviews.

As the debates and conversations that are relevant to the issue of the misuse of systematic reviews largely take place in the world of academic journal publications, the goal is to highlight the issues in a user-friendly, clear resource that is accessible to anyone conducting or appraising a systematic review. This resource will be a bespoke, informative website that is open to feedback and collaboration, adapting in light of further innovations in systematic review. It will not be a branded resource (i.e., Cochrane) and therefore will not speak only to a particular kind of systematic review. The aim is to ensure clear messages are available to anyone with a stake in understanding systematic reviews.

This research will describe what makes an appropriate team and what types of team influence can cause problems to systematic reviews. This framework will help people who do or pay for research that involves a systematic review to understand what factors they need to consider when planning, funding or doing a systematic review. The research in this fellowship will lead to better systematic reviews, to inform health care decision making for health services for patients and the public and to influence future health research.