Transition Support Award: Investigating Human Influences in the Reliability and Validity of Published Systematic Reviews

Systematic reviews are a type of literature review used by researchers to collect evidence from many different sources and summarise it in an unbiased way. However, growing research indicates that published systematic reviews may not be as reliable as they are expected to be. The purpose of this research project has been to document the growing evidence that published systematic reviews are conducted which may not be trustworthy or reliable. This has involved creating a website to help researchers to explore the many "problems" with systematic reviews and to see the variety of research articles which highlight these flaws. So far, literature searches between 2000 and 2022 identified 485 included papers which relate to 67 different problems with systematic reviews. Poorly-done systematic reviews may still be being used by clinicians and decision makers to guide patient care. Due to the vast number of papers being published about this topic, this research is currently being updated regularly to understand the emerging findings in a "living" review.

The next phase of this research is to consult with experts who do systematic reviews about what problems are the most likely to affect whether a systematic review can be trusted as reliable. This transition support award allows the process of planning and consulting with experts to be done in a rigorous and fair way. It also allows for further updates of the living review to be performed, ensuring that the emerging evidence on this topic can be understood as it evolves over time. This involves exploring how research culture or the environment in which scientists operate in can contribute to the increasing problems with systematic reviews.

This work aims to promote best practice in science including: 1. being open and sharing data; 2. being inclusive and thorough about what/who to include; 3. being rigorous and using the best practice methods that are available and; 4. conducting the research carefully, meaningfully and without any vested interests. These principles for best practice in science will be shared widely to shape the broader research agenda and influence positive changes in the academic research environment.

Systematic reviews underpin evidence-based medicine by aiming to summarise all relevant evidence in a transparent way and they are upheld as being objective, dispassionate scientific processes. However, systematic reviews can bias the evidence base when conducted carelessly or by teams who are not impartial to the results. The impact of poorly conducted and published systematic reviews are that healthcare decisions are informed on the basis of misleading evidence. These unreliable systematic reviews often pervade medical decision making for decades after they were published. The aim of this work has been to fully understand, highlight and find solutions to the increasing problems in systematic reviews including error, questionable research practices and bias. A living review and world-class research resource (http://www.systematicreviewlution.com) has been developed to disseminate this research.

Sixty seven discrete problems are documented in the conduct of published systematic reviews, identified from 485 published journal papers indicating that issues are more widespread than can be detected using existing systematic review guidance. The sheer number of unique papers from disparate research teams in different clinical areas and academic journals indicates that, until now, this conversation has not been joined up. As a result the debate has failed to move on until now and solutions have not been proposed. The next phase of this research therefore, is to work with methodological experts globally to prioritise the problems, to understand which problems pose the greatest threat to the conclusions of systematic reviews and to develop tangible solutions. This work will help peer reviewers, journal editors, researchers, decision makers and other users of systematic reviews to improve the conduct and publication of future systematic reviews. These highly cited documents are vital to the evidence ecosystem that underpins medical research and in turn, human health.