Investigating human influences in systematic reviews and meta-analyses to improve the reliability and validity of evidence syntheses


Billions of pounds are spent each year on health care treatments with the aim of improving and saving lives. But not all clinical research is published and what is published not be reliable. By bringing together similar studies, assessing their quality, and combined effects, more reliable findings can be used to guide future clinical and policy decisions. The gold standard of this method of reviewing studies is called systematic review. These large reports summarise all relevant studies in a clear and fair way to understand the benefits and drawbacks of new treatments. They are highly cited articles and regarded as objective and transparent research designs that are necessary to inform patient care.

There are increasing reports that systematic reviews are being published that do not live up to their high quality reputation. Systematic reviews are receiving attention for contributing to research waste, varying massively in their quality, being misleading and serving conflicted interests. When authors are asked by publishing journals to declare conflicts of interest, they relate to narrow criteria about funds received from industry within the last few years. Whilst clear guidelines for best practice in systematic reviews are available for authors, they do not provide guidance on review teams themselves. Reviews are sometimes carried out by people who have an interest in what the results are (bias). Or review teams might not take all the necessary efforts to find and present a full picture of the evidence needed for decision makers to understand (passivity). Members of the review team might have their own view on a treatment and this might shape the way the review is done. Review teams may not be sufficiently skilled or have the resources to carry out a high quality review (insufficient resources). Inflexible review methods may result in expensive review projects which do not reflect issues that matter to patients. Also, sometimes reviews are published without funding and authors' motivation for this is not questioned. The effects that human influence has on systematic reviews may be especially important when looking at health care problems that are complex.

Considering the reputation of systematic reviews as fair and objective processes, poor conduct could be more problematic than in clinical research. Health care evidence might not be evaluated fairly and patients may suffer because the health service may not provide the best treatments as a result.

To fully explore the ways in which the researchers in the team can influence systematic reviews I plan to:
1) Search for and review studies documenting problems with systematic reviews that can occur due to team influence.
2) Work with experts in research methods to highlight risks and solutions for systematic review teams, helping to ensure that the findings of my research impact on systematic reviews that are carried out in the future.
4) Produce a dedicated website and guidance in a clear and accessible format about the risks of human influence in systematic reviews to help people assess systematic reviews for themselves.
5) Test guidance with researchers to refine and revise, ensuring that they are useful to people who do or use systematic reviews.

What impact will this research have?
This research will describe what makes an appropriate team and what types of team influence can cause problems to systematic reviews. The framework and guidance will help people who do or pay for research that involves a systematic review to understand what factors they need to consider when planning, funding or doing a systematic review. To influence the wider research agenda a user-friendly website that explains the goals, methods and results of the research will be created. This research will lead to better systematic reviews, to inform health care decision making for health services for patients and the public and to influence future health research.

Technical Summary

Systematic reviews underpin modern evidence-based medicine. However, there is an increasing body of research showing that they are susceptible to bias and are often not conducted well, despite the availability of established guidelines. I will investigate factors related to researcher influence within systematic reviews and develop resources to reduce bias or poor conduct in future reviews, increasing the quality and rigour of future systematic reviews.

The effect that individuals may have on the conduct and conclusions of a systematic review is currently unknown. Researchers are responsible for making influential decisions at all stages of a review. Human influence over the conduct of a systematic review can occur due to author affiliation with the subject of investigation (bias or allegiance) but conversely review authors may be insufficiently skilled to take necessary efforts to perform a rigorous review. Such deviations from consistency and transparency, whether conscious or unconscious, can result in expensive outputs that fail to answer the review question. The full consequences to evidence-based medicine, of poor conduct in published reviews have not yet been explored.

Methods: Rigorous methods to examine systematic review research will be combined with input from methodological experts. The proposed research includes:
a) comprehensive systematic reviews and meta-regression team of human influences on systematic review outputs
b) consultation and consensus with international methodological experts on i) the risks of ignoring team influences in systematic review, ii) solutions for systematic reviews and users via guidance
c) highlighting the issues on the wider research agenda via a dedicated website

This work will allow commissioners, researchers, publishers and policy makers to have greater confidence that conduct of systematic reviews, relied upon to inform decisions, is being scrutinised and that methods are evolving, as in primary research.

Planned Impact

Recent research highlights that the quality of published systematic reviews is often below what is assumed due to their reputation. The beneficiaries from this research in the short-term are commissioners and policy makers who use systematic reviews to make decisions; researchers who conduct or appraise systematic reviews; editors who publish systematic reviews and clinicians who use systematic reviews to make decisions about clinical care or treatments.

People who will benefit from this research in the longer-term include people who use the NHS, tax-payers who pay into the NHS; and other countries who use NHS/NICE decisions in developing clinical guidance. In the short-term, consumers of systematic reviews will have greater awareness of the pitfalls and biases that systematic reviews are prone to. Journal editors will have a clearer idea about which reviews to publish and which should signal red flags. Public engagement about the issues will put a spotlight on this area of evidence-based medicine and it is hoped that this will in turn raise the issues of human influences on the wider research agenda.

The longer-term impact will ensure that decisions on health and medical treatments are evaluated transparently by investigating and raising awareness of human influences such as bias and error in evidence syntheses. Systematic review products will be held to higher standards and reviews where team conduct is likely to have negatively influenced the validity of the reviews findings will lose credibility. The proposed work will lead to improvements in evidence syntheses that feed into health economic evaluations therefore tax-payers who contribute to the NHS and NHS patients will benefit by receiving the most effective and cost-effective treatments.

As a result of this work decision makers can have greater confidence that the systematic review products they rely upon are subject to continued scrutiny. Systematic reviews are highly cited articles which underpin global healthcare service efficiency such as public health evaluations, assessment of efficacy and effectiveness of medical treatments and economic evaluations. In the UK, systematic reviews have become the mainstay of evidence-based medicine for the NHS, NICE, NIHR and the Department of Health and Social Care. This collaborative project with international experts aims to develop consensus but wider public engagement of the issues will ensure that the messages from the research outputs become clear enough to ensure those who conduct, publish or appraise systematic reviews sit up and pay attention to the message that there are problems with systematic reviews.

As the debates and conversations that are relevant to the issue of the misuse of systematic reviews largely take place in the world of academic journal publications, the goal is to highlight the issues in a user-friendly, clear resource that is accessible to anyone conducting or appraising a systematic review. This resource will be a bespoke, informative website that is open to feedback and collaboration, adapting in light of further innovations in systematic review. It will not be a branded resource (i.e., Cochrane) and therefore will not speak only to a particular kind of systematic review. The aim is to ensure clear messages are available to anyone with a stake in understanding systematic reviews.

This research will describe what makes an appropriate team and what types of team influence can cause problems to systematic reviews. This framework will help people who do or pay for research that involves a systematic review to understand what factors they need to consider when planning, funding or doing a systematic review. The research in this fellowship will lead to better systematic reviews, to inform health care decision making for health services for patients and the public and to influence future health research.


10 25 50

publication icon
Uttley L (2023) The problems with systematic reviews: a living systematic review. in Journal of clinical epidemiology

Title Online database to support living systematic review 
Description This database, stored in the cloud (Microsoft Azure) has a customised web app to allow data entry 
Type Of Material Database/Collection of data 
Year Produced 2022 
Provided To Others? Yes  
Impact The cloud-based database and web app will allow myself and collaborators to work on data extraction and quality assessment internationally, with no version control. Extracted data will automitcally appear in the front end (the website). 
Title S Shiny app embedded in 
Description A custom app developed by University of Sheffield Bioinformatics has been developed in R Shiny to display figures for my fellowship's living systematic review. The figures are automatically updated to accommodate the most recent dataset in the living systematic review 
Type Of Material Data analysis technique 
Year Produced 2023 
Provided To Others? Yes  
Impact This development has added to the curation and dissemination of data in an accessible format from my dedicated project website 
Title Website to display living systematic review of problems with systematic reviews 
Description A living systematic review of the problems with systematic reviews. This website was built by a third party to display the overarching findings of my pre-registered living systematic review. This review has comprehensively sought studies which examine flaws, limitations an, bias and problems with published systematic reviews. The website is the front end to an online database that holds all the research data. 
Type Of Material Data handling & control 
Year Produced 2022 
Provided To Others? Yes  
Impact This website will be a resource that is publicly accessible and useful to researchers in this field. The website describes existing research which has found limitations with published systematic reviews. This living review has categorised each included paper under "problems" to help researchers understand the problems systematic reviews are prone to , and the evidence behind each problem. 
Description Advisor to the World Health Organisation Classification of Tumours Group at the International Agency for Research on Cancer in Lyon 
Organisation International Agency for Research on Cancer (IARC)
Country France 
Sector Academic/University 
PI Contribution Workshop advisory board member to the World Health Organisation Classification of Tumours Group project to develop evidence gap maps to support tumour classification.
Collaborator Contribution So far I have taken part in an workshop session followed by an online survey. I have been invited to deliver the keynote speech at the
Impact None as yet
Start Year 2023
Description In person seminar delivered to the University of Sheffield Psychology Department on the problems with systematic reviews 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Postgraduate students
Results and Impact The seminar was well attended by Psychology department staff, postgrad and undergraduate students. Following this I have made some new collaborations from relevant Psychology projects in the pipeline
Year(s) Of Engagement Activity 2022
Description Online presentation of the fellowship aims to The University of Sheffield Research on Research Initiative (RORi) 03/03/21 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Other audiences
Results and Impact This was an invited talk (10 minutes) to describe and discuss the main principles and objectives of my career development award with regards to research of research. I summarised the background to obtaining the fellowship, described the main issues and goals of the research and outlined some key challenges with the work.
Year(s) Of Engagement Activity 2021