Evaluating Evidence in Medicine

This project seeks to improve the way that evidence is evaluated in medical research and health policy by developing a recent line of work in philosophy.

Evidence-based medicine has transformed the way in which statistical evidence is used in medicine. Hierarchies of evidence are now routinely used by medical researchers and health policy makers to assess evidence for the effectiveness of treatments and health policies: studies that simply observe patients after treatment are ranked lower than studies that randomly decide who to treat, and these in turn rank lower than studies that review the evidence obtained by a series of trials. Evidence hierarchies have become so widely endorsed that they are now being used across the social sciences and in public policy, as well as in medicine.

While there has been some debate about which sorts of trials should be placed at the top of the hierarchy, this project focuses on the bottom level, which is normally occupied by evidence that is not obtained from a statistical trial. In our view, while it is appropriate to relegate anecdotal evidence and hearsay to this lowest level, other, better quality evidence is also being ignored, simply because it is often not obtained from statistical trials.

In particular, evidence of the underlying physiological and biochemical mechanisms is often classified as inferior to statistical evidence. This is because evidence of mechanisms is normally obtained, not simply via statistical trials, but in a complex way, by integrating a mixture of laboratory experiments, basic scientific knowledge and case studies as well as past trials. Recent work suggests that it is wrong to view evidence of mechanisms as inferior. Philosophers of causality and historians of medicine have argued that evidence of mechanisms is required alongside statistical evidence in order to evaluate whether treatments or health policies are effective. This is because such evidence helps to determine whether positive results of a trial are due to genuine effectiveness or are simply a statistical blip; such evidence is also crucial when designing and interpreting a statistical trial, and when determining effectiveness in a new population or a particular patient.

In an exploratory project the project team put forward these arguments and consulted with medical researchers and health policy advisers, who verified the importance of mechanistic evidence and confirmed that it was often being tacitly used, against the explicit recommendations of the hierarchies of evidence.

But how can one formulate explicit guidelines for considering mechanistic evidence alongside statistical evidence? One reason why non-statistical evidence is relegated to the bottom of the hierarchies is that it is very hard to weigh against evidence obtained from statistical trials. In this project we seek to understand how to evaluate mechanistic evidence alongside statistical evidence in medical research and health policy.

This task needs a variety of methods. It requires work on philosophical theories of causality and causal discovery, to provide the theoretical underpinning. It requires historical case studies of causal discovery in medicine, to understand the full variety of ways in which evidence needs to be evaluated. It requires close cooperation with medical researchers and health policy advisers, to assess the practical needs and concerns of those who use the guidelines. It requires formal methods to address technical concerns.

The makeup of the project team is crucial if this task is to succeed. Our team involves the National Institute of Health and Clinical Excellence (NICE) and the WHO International Agency for Research on Cancer (IARC) as well as philosophers of causality at the Centre for Reasoning at the University of Kent, researchers in history and philosophy of science and medicine at UCL and at the University of Ferrara, and an expert in evidence-based medicine at Leiden Medical Centre.

Impact Summary

This document sets out the ways in which this research project will engage with users and beneficiaries outside the history and philosophy of medicine (HPM) research community. Engaging with the needs of users in non-academic sectors, particularly those involved in the assessment of medical evidence, are paramount.

Who will benefit from this research?
This research will benefit three stakeholder groups. First, research communities outside HPM, particularly in public health and epidemiology, including private sector organisations working in evidence and healthcare. Second, the public sector, particularly in healthcare provision. Third, the wider public, particularly those interested in scientific evidence.

How will they benefit from this research?
We have identified three main pathways to impact to ensure that these communities can benefit from our research. Pathway 1 involves co-production of research with influential non-academic partners in health policy (in particular NICE and IARC). By working with practising scientists and taking part in various parts of the scientific process of evidence evaluation, project members will engage directly with issues of interest to these sectors. We plan to engage with public health scientists and policy makers via our collaboration with NICE, and to work alongside researchers in epidemiology via our partnership with IARC.

The second pathway is concerned with engaging interested stakeholder groups (particularly in the public sector) who might benefit from project research outputs. A key audience for us here is that of early-career biomedical professionals and policy makers, but we shall also target the general public, who are increasingly health-aware and increasingly responsible for making medical choices. This pathway will involve the translation of traditional research outputs (e.g. publications) into more engaging formats. We will commission 10-12 professional video clips (each of about 2 minutes) introducing core ideas from this research. These will then be made freely available via YouTube and promoted widely via social media and the web. We will collaborate with UCL's public engagement office, and the UCL department of Science and Technology Studies, both of which have substantial prior experience of producing innovative materials of this kind. A second means of ensuring impact for this audience is via a planned volunteering project. Project team members will work with members of the UCL student community to engage them with questions about evidence. In collaboration with UCL's Widening Participation office, these undergraduates will go to secondary schools (initially in UCL's local community) where they will work with Key Stage 5 students to develop projects on the role of evidence in healthcare. These student projects will aim at engaging pupils at Key Stage 4 with ideas about evidence in healthcare, and will be aligned with learning objectives at KS4 in maths and the sciences (see AQA 2012: 3.2, available online at http://filestore.aqa.org.uk/subjects/AQA-BIOL-W-SP-14.PDF).

Finally, the third pathway is to develop a substantial public engagement profile for the project. The project website - branded EBM+ - will be the hub for our public engagement. As well as linking to our video content mentioned above, it will feature a project blog, detailing events, conferences, meetings, and news. This will be complemented by the publication of interim research results on the website. Finally, we will aggregate relevant internet and social media items of interest to evidence stakeholders. We will then use social media to promote engagement between our project and other online communities interested in evidence in medicine (such as the comment sections on e.g. badscience.net; dcscience.net).

Managing this ambitious impact program will be facilitated by the recruitment of a dedicated Impact and Coordination Manager.