Evaluating Evidence in Medicine

Lead Research Organisation: University of Kent
Department Name: Sch of European Culture and Languages


This project seeks to improve the way that evidence is evaluated in medical research and health policy by developing a recent line of work in philosophy.

Evidence-based medicine has transformed the way in which statistical evidence is used in medicine. Hierarchies of evidence are now routinely used by medical researchers and health policy makers to assess evidence for the effectiveness of treatments and health policies: studies that simply observe patients after treatment are ranked lower than studies that randomly decide who to treat, and these in turn rank lower than studies that review the evidence obtained by a series of trials. Evidence hierarchies have become so widely endorsed that they are now being used across the social sciences and in public policy, as well as in medicine.

While there has been some debate about which sorts of trials should be placed at the top of the hierarchy, this project focuses on the bottom level, which is normally occupied by evidence that is not obtained from a statistical trial. In our view, while it is appropriate to relegate anecdotal evidence and hearsay to this lowest level, other, better quality evidence is also being ignored, simply because it is often not obtained from statistical trials.

In particular, evidence of the underlying physiological and biochemical mechanisms is often classified as inferior to statistical evidence. This is because evidence of mechanisms is normally obtained, not simply via statistical trials, but in a complex way, by integrating a mixture of laboratory experiments, basic scientific knowledge and case studies as well as past trials. Recent work suggests that it is wrong to view evidence of mechanisms as inferior. Philosophers of causality and historians of medicine have argued that evidence of mechanisms is required alongside statistical evidence in order to evaluate whether treatments or health policies are effective. This is because such evidence helps to determine whether positive results of a trial are due to genuine effectiveness or are simply a statistical blip; such evidence is also crucial when designing and interpreting a statistical trial, and when determining effectiveness in a new population or a particular patient.

In an exploratory project the project team put forward these arguments and consulted with medical researchers and health policy advisers, who verified the importance of mechanistic evidence and confirmed that it was often being tacitly used, against the explicit recommendations of the hierarchies of evidence.

But how can one formulate explicit guidelines for considering mechanistic evidence alongside statistical evidence? One reason why non-statistical evidence is relegated to the bottom of the hierarchies is that it is very hard to weigh against evidence obtained from statistical trials. In this project we seek to understand how to evaluate mechanistic evidence alongside statistical evidence in medical research and health policy.

This task needs a variety of methods. It requires work on philosophical theories of causality and causal discovery, to provide the theoretical underpinning. It requires historical case studies of causal discovery in medicine, to understand the full variety of ways in which evidence needs to be evaluated. It requires close cooperation with medical researchers and health policy advisers, to assess the practical needs and concerns of those who use the guidelines. It requires formal methods to address technical concerns.

The makeup of the project team is crucial if this task is to succeed. Our team involves the National Institute of Health and Clinical Excellence (NICE) and the WHO International Agency for Research on Cancer (IARC) as well as philosophers of causality at the Centre for Reasoning at the University of Kent, researchers in history and philosophy of science and medicine at UCL and at the University of Ferrara, and an expert in evidence-based medicine at Leiden Medical Centre.

Planned Impact

Impact Summary

This document sets out the ways in which this research project will engage with users and beneficiaries outside the history and philosophy of medicine (HPM) research community. Engaging with the needs of users in non-academic sectors, particularly those involved in the assessment of medical evidence, are paramount.

Who will benefit from this research?
This research will benefit three stakeholder groups. First, research communities outside HPM, particularly in public health and epidemiology, including private sector organisations working in evidence and healthcare. Second, the public sector, particularly in healthcare provision. Third, the wider public, particularly those interested in scientific evidence.

How will they benefit from this research?
We have identified three main pathways to impact to ensure that these communities can benefit from our research. Pathway 1 involves co-production of research with influential non-academic partners in health policy (in particular NICE and IARC). By working with practising scientists and taking part in various parts of the scientific process of evidence evaluation, project members will engage directly with issues of interest to these sectors. We plan to engage with public health scientists and policy makers via our collaboration with NICE, and to work alongside researchers in epidemiology via our partnership with IARC.

The second pathway is concerned with engaging interested stakeholder groups (particularly in the public sector) who might benefit from project research outputs. A key audience for us here is that of early-career biomedical professionals and policy makers, but we shall also target the general public, who are increasingly health-aware and increasingly responsible for making medical choices. This pathway will involve the translation of traditional research outputs (e.g. publications) into more engaging formats. We will commission 10-12 professional video clips (each of about 2 minutes) introducing core ideas from this research. These will then be made freely available via YouTube and promoted widely via social media and the web. We will collaborate with UCL's public engagement office, and the UCL department of Science and Technology Studies, both of which have substantial prior experience of producing innovative materials of this kind. A second means of ensuring impact for this audience is via a planned volunteering project. Project team members will work with members of the UCL student community to engage them with questions about evidence. In collaboration with UCL's Widening Participation office, these undergraduates will go to secondary schools (initially in UCL's local community) where they will work with Key Stage 5 students to develop projects on the role of evidence in healthcare. These student projects will aim at engaging pupils at Key Stage 4 with ideas about evidence in healthcare, and will be aligned with learning objectives at KS4 in maths and the sciences (see AQA 2012: 3.2, available online at http://filestore.aqa.org.uk/subjects/AQA-BIOL-W-SP-14.PDF).

Finally, the third pathway is to develop a substantial public engagement profile for the project. The project website - branded EBM+ - will be the hub for our public engagement. As well as linking to our video content mentioned above, it will feature a project blog, detailing events, conferences, meetings, and news. This will be complemented by the publication of interim research results on the website. Finally, we will aggregate relevant internet and social media items of interest to evidence stakeholders. We will then use social media to promote engagement between our project and other online communities interested in evidence in medicine (such as the comment sections on e.g. badscience.net; dcscience.net).

Managing this ambitious impact program will be facilitated by the recruitment of a dedicated Impact and Coordination Manager.


10 25 50
Description This project developed a theory of how evidence of mechanisms can be considered alongside evidence of associations to evaluate causal claims in medical research and health policy.
The project:
- Investigated what evidence of mechanisms is, and how we get it.
- Investigated how quality of evidence can be characterised.
- Investigated which accounts of causality best fit the programme for integrating evidence of mechanisms with evidence of correlation.

The project came up with a set of proposals for improving the practice of evidence-based medicine (EBM). We argued that EBM should a broader range of evidence into account when evaluating the effectiveness of health interventions and when evaluating proposed causes of disease. In particular, we argued that evidence of mechanisms should be explicitly and systematically considered when evaluating causal claims in medicine. Our practical finding for how best to do this are presented in our open access monograph, "Evaluating Evidence of Mechanisms in Medicine: Principles and Procedures", Springer 2018. See link.springer.com/book/10.1007/978-3-319-94610-8 . These developments led to the EBM+ programme; see ebmplus.org

More philosophical findings were published in a wide range of academic publications, available from the project website (see below). These include another monograph and two edited collections as well as numerous research papers. These findings helped to improve our understanding of: what constitutes evidence of mechanisms; what counts as high quality evidence of mechanisms; how evidence of mechanisms is relevant to the evaluation of causal claims; and which philosophical theories of causality best accommodate this role for evidence of mechanisms.
Exploitation Route Our recommendations for changes to the way evidence is evaluated in evidence-based medicine could lead to improvements in causal discovery in medicine and the health sciences. In the longer term, these recommendations could lead to improvements in evidence-based methods in other areas, such as evidence-based public policy.
Sectors Healthcare

URL https://blogs.kent.ac.uk/jonw/projects/evaluating-evidence-in-medicine/
Description We fed into a consultation to improve the National Institute of Health and Care Excellence (NICE) methods manual ("Developing NICE guidelines: the manual"). This manual is the basis for all NICE guidelines, which inform healthcare in the UK and beyond. We made some suggestions as to how to improve the way in which NICE guidelines handle evidence of mechanisms. This led to some preliminary changes to the manual, which again went out to consultation. We proposed some further changes, which informed the resulting manual of October 2018, which now offers a less rudimentary treatment of mechanistic evidence, in line with our suggestions. We fed into a consultation to improve the WHO International Agency for Research on Cancer (IARC) methods manual ("Preamble to the IARC monographs"). This methods manual is the basis for all WHO evaluations of carcinogenicity of chemicals and other exposures. We made some suggestions as to how to improve the way in which IARC handles evidence of mechanisms. In addition, PI Jon Williamson was a member of the committee responsible for revising the methods manual. This committee met in November 2018, and the new methods manual was published in January 2019. Mechanistic evidence is given a much more prominent role in the new methods manual, in line with the recommendations of our AHRC project. We also ran a schools volunteering project at UCL. Students at UCL volunteered to work in a range of schools to engage school children with the need to consider a wide range of evidence in evidence-based medicine (EBM). This ran from 2016-18 and led to a lot of interesting schoolwork on evidence in healthcare, some of which was presented at a schools conference on this topic that we hosted. Further information can be found at www.volunteering.ebmplus.org We created a website and blog for engaging with the medical community at ebmplus.org. A project webpage listing project outputs was created to engage academic researchers: blogs.kent.ac.uk/jonw/projects/evaluating-evidence-in-medicine. 45 videos were created to engage with the healthcare community and medical students: https://www.youtube.com/channel/UCf0iImd4-wdRNN_qhgpIa5Q/videos A twitter feed helped to engage both those in the healthcare community and members of the general public interested in new developments in healthcare: https://twitter.com/EBMplus All these avenues have helped to create awareness of the limitations of present-day evidence-based medicine and our finding for how to improve EBM.
First Year Of Impact 2016
Sector Education,Healthcare
Impact Types Policy & public services