Obtaining valid and reliable estimates of treatment effectiveness from evidence synthesis of psychiatric trials.
Lead Research Organisation:
University of Bristol
Department Name: Community-Based Medicine
Abstract
Establishing the most effective treatment(s) for depression is a priority for population health. By 2020 depression will be the second highest cause of disease worldwide with the cost to the NHS currently around #9 billion. Health Technology Assessment (HTA) supports healthcare decision-making through the evaluation of the clinical and cost-effectiveness of medical treatments and the publication of treatment recommendations and guidelines for the NHS. Gold standard evidence for HTA comes from meta-analysis of randomised controlled trials (RCTs). A meta-analysis combines the data from multiple RCTs for the same condition in one analysis to produce more powerful evidence of whether a treatment is effective or not. However, empirical research suggests that RCTs in psychiatry are particularly susceptible to bias, which can lead to under- or over-estimation of treatment effectiveness. New methods for controlling for bias in meta-analysis have been proposed which generate un-biased estimates of treatment effectiveness. This project evaluates the potential benefits and reliability of these new methods compared to standard meta-analysis. The project uses these methods to evaluate the effectiveness of new-generation antidepressants, psychotherapies and older treatments, such as tricyclic antidepressants, to answer the question which is the most effective treatment for depression?
Technical Summary
Background: Establishing the most effective treatment(s) for depression is a population health priority. By 2020 depression will be the second highest cause of disease worldwide, with the cost to the NHS around #9 billion. Evidence synthesis using randomised controlled trials (RCTs) underpins Health Technology Assessment (HTA) and clinical guidelines. However, empirical evidence suggests psychiatric RCTs are susceptible to bias, raising questions about reliability and validity of effect estimates from psychiatric HTA. Two recent methodological developments make it possible to control for bias in psychiatric HTA; Bias-adjusted meta-analysis and Mixed Treatment Comparisons.
Aims & Objectives: to examine the suitability of MTC for psychiatric RCTs and assess how biases in evidence synthesis of psychiatric RCTs can be controlled and mitigated to generate more precise, valid estimates of treatment effect. I will;
1. explore the benefits of bias-adjusted MTC compared to standard meta-analysis in the presence of quality-related biases and effect modifiers,
2. assess the effect of extending an MTC network of evidence for depression to include all relevant treatments and assess the ‘risk of bias‘ for each included RCT;
3. explore and compare the effects of bias adjustment on treatment effects estimates in the extended depression network of evidence, using approaches proposed by Turner, Welton and Dias.
Design & Methodology: A review of NICE guidelines and Cochrane reviews will be conducted to identify and assemble psychiatric MTC networks of evidence. The networks inform a simulation study to examine the reliability of bias-adjustment, comparing effect estimates from meta-analysis, unadjusted MTC and bias-adjusted MTC in the presence of bias. The substantive part of the project extends an existing network of new-generation antidepressants for depression, including additional treatments and assessing each RCT for bias using the Cochrane risk of bias tool. The extended network forms the basis for an empirical study to explore and compare the effects of bias-adjusted MTC on depression effect estimates, using three recently proposed bias-adjustment methods. Analyses use a Bayesian approach implemented in WinBUGS.
Scientific Opportunities: Project outputs will
generate validated, reliable treatment effect estimates and rankings for depression;
inform a future grant proposal of bias-adjusted MTC for all depression treatments to identify the most effective treatment overall;
evaluate the benefits of bias-adjusted MTC compared to meta-analysis in psychiatric HTA
inform NICE‘s research agenda regarding the validity and reliability of MTC;
identify whether the three MTC bias-adjustment methods can be combined into a single approach.
Aims & Objectives: to examine the suitability of MTC for psychiatric RCTs and assess how biases in evidence synthesis of psychiatric RCTs can be controlled and mitigated to generate more precise, valid estimates of treatment effect. I will;
1. explore the benefits of bias-adjusted MTC compared to standard meta-analysis in the presence of quality-related biases and effect modifiers,
2. assess the effect of extending an MTC network of evidence for depression to include all relevant treatments and assess the ‘risk of bias‘ for each included RCT;
3. explore and compare the effects of bias adjustment on treatment effects estimates in the extended depression network of evidence, using approaches proposed by Turner, Welton and Dias.
Design & Methodology: A review of NICE guidelines and Cochrane reviews will be conducted to identify and assemble psychiatric MTC networks of evidence. The networks inform a simulation study to examine the reliability of bias-adjustment, comparing effect estimates from meta-analysis, unadjusted MTC and bias-adjusted MTC in the presence of bias. The substantive part of the project extends an existing network of new-generation antidepressants for depression, including additional treatments and assessing each RCT for bias using the Cochrane risk of bias tool. The extended network forms the basis for an empirical study to explore and compare the effects of bias-adjusted MTC on depression effect estimates, using three recently proposed bias-adjustment methods. Analyses use a Bayesian approach implemented in WinBUGS.
Scientific Opportunities: Project outputs will
generate validated, reliable treatment effect estimates and rankings for depression;
inform a future grant proposal of bias-adjusted MTC for all depression treatments to identify the most effective treatment overall;
evaluate the benefits of bias-adjusted MTC compared to meta-analysis in psychiatric HTA
inform NICE‘s research agenda regarding the validity and reliability of MTC;
identify whether the three MTC bias-adjustment methods can be combined into a single approach.
Publications

Ades AE
(2013)
Evidence synthesis for decision making 7: a reviewer's checklist.
in Medical decision making : an international journal of the Society for Medical Decision Making

Alfirevic Z
(2015)
Labour Induction With Prostaglandins A Systematic Review and Network Meta-Analysis
in Obstetrical & Gynecological Survey

Alfirevic Z
(2016)
Methods to induce labour: a systematic review, network meta-analysis and cost-effectiveness analysis.
in BJOG : an international journal of obstetrics and gynaecology

Alfirevic Z
(2015)
Labour induction with prostaglandins: a systematic review and network meta-analysis.
in BMJ (Clinical research ed.)

Alfirevic Z
(2016)
Which method is best for the induction of labour? A systematic review, network meta-analysis and cost-effectiveness analysis.
in Health technology assessment (Winchester, England)

Caldwell DM
(2016)
A threshold analysis assessed the credibility of conclusions from network meta-analysis.
in Journal of clinical epidemiology

Caldwell DM
(2012)
Selecting the best scale for measuring treatment effect in a network meta-analysis: a case study in childhood nocturnal enuresis.
in Research synthesis methods

Caldwell DM
(2015)
Extending Treatment Networks in Health Technology Assessment: How Far Should We Go?
in Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research

Caldwell DM
(2014)
An overview of conducting systematic reviews with network meta-analysis.
in Systematic reviews

Caldwell DM
(2016)
Approaches for synthesising complex mental health interventions in meta-analysis.
in Evidence-based mental health
Description | Use of material within a postgraduate course |
Geographic Reach | Local/Municipal/Regional |
Policy Influence Type | Influenced training of practitioners or researchers |
Description | Health Technology Assessment |
Amount | £261,107 (GBP) |
Organisation | National Institute for Health Research |
Sector | Public |
Country | United Kingdom |
Start | 09/2016 |
End | 12/2018 |
Description | NIHR Public Health Research |
Amount | £296,668 (GBP) |
Organisation | National Institute for Health Research |
Sector | Public |
Country | United Kingdom |
Start | 10/2016 |
End | 11/2018 |
Description | CIHR Drug Safety and Effectiveness network Knowledge Synthesis |
Organisation | Ottawa Hospital Research Institute |
Country | Canada |
Sector | Academic/University |
PI Contribution | Consultant for network meta-analysis projects/ steering board |
Collaborator Contribution | Submitted the research grant, and provide methodological problems similar to the fellowship project i am working on |
Impact | none yet |
Start Year | 2014 |
Description | Cochrane Collaboration Comparing Multiple Interventions Methods Group |
Organisation | Tufts Medical Center |
Country | United States |
Sector | Hospitals |
PI Contribution | As a result of the funding i receive via my fellowship I was able to form links within the Cochrane Collaboration to form a new methods group examining how to evaluate multiple interventions within systematic reviews. I am a co-convenor of this methods group. |
Collaborator Contribution | Collaboration within this group facilitates my goal to disseminate the methodology to a wider audienceCollaboration within this group facilitates my goal to disseminate the methodology to a wider audience and to explore the impact of bias within network meta-analysisCollaboration within this group facilitates my goal to disseminate the methodology to a wider audience. |
Impact | The collaboration is multi-disicplinary, involving statisticians, methodologists and clinicians. No outputs to report this year. I anticipate outcomes in the later stages of the fellowship period. |
Start Year | 2010 |
Description | Cochrane Collaboration Comparing Multiple Interventions Methods Group |
Organisation | University of Cambridge |
Department | MRC Biostatistics Unit |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | As a result of the funding i receive via my fellowship I was able to form links within the Cochrane Collaboration to form a new methods group examining how to evaluate multiple interventions within systematic reviews. I am a co-convenor of this methods group. |
Collaborator Contribution | Collaboration within this group facilitates my goal to disseminate the methodology to a wider audienceCollaboration within this group facilitates my goal to disseminate the methodology to a wider audience and to explore the impact of bias within network meta-analysisCollaboration within this group facilitates my goal to disseminate the methodology to a wider audience. |
Impact | The collaboration is multi-disicplinary, involving statisticians, methodologists and clinicians. No outputs to report this year. I anticipate outcomes in the later stages of the fellowship period. |
Start Year | 2010 |
Description | Cochrane Collaboration Comparing Multiple Interventions Methods Group |
Organisation | University of Ioannina |
Department | Department of Hygiene and Epidemiology |
Country | Greece |
Sector | Academic/University |
PI Contribution | As a result of the funding i receive via my fellowship I was able to form links within the Cochrane Collaboration to form a new methods group examining how to evaluate multiple interventions within systematic reviews. I am a co-convenor of this methods group. |
Collaborator Contribution | Collaboration within this group facilitates my goal to disseminate the methodology to a wider audienceCollaboration within this group facilitates my goal to disseminate the methodology to a wider audience and to explore the impact of bias within network meta-analysisCollaboration within this group facilitates my goal to disseminate the methodology to a wider audience. |
Impact | The collaboration is multi-disicplinary, involving statisticians, methodologists and clinicians. No outputs to report this year. I anticipate outcomes in the later stages of the fellowship period. |
Start Year | 2010 |
Description | Cochrane Depression, Anxiety and Neurosis group |
Organisation | The Cochrane Collaboration |
Department | Depression Anxiety and Neurosis Group (CCDAN) |
Country | United Kingdom |
Sector | Charity/Non Profit |
PI Contribution | I have collaborated on systematic reviews, which in turn will inform my fellowship award |
Collaborator Contribution | The data gathered from the systematic reviews will inform my fellowship project |
Impact | 1. HIRED MAP systematic review team: including Caldwell, DM. Mindfulness-based 'third wave' cognitive and behavioural therapies versus other psychological therapies for depression. Cochrane Database of Systematic Reviews 2013 2. HIRED MAP systematic review team: including Caldwell, DM. Mindfulness-based 'third wave' cognitive and behavioural therapies versus treatment as usual for depression. Cochrane Database of Systematic Reviews 2013 3. Shinohara K, Honyashiki M, Imai H, Churchill R, Hunot V, Caldwell DM, Davies P, Moore T, Furukawa TA. Behavioural therapies versus other psychological therapies for depression. Cochrane Database of Systematic Reviews 2013. In press. |
Start Year | 2008 |
Description | Approaches for synthesis of complex interventions: components-based network meta-analysis |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Regional |
Primary Audience | Professional Practitioners |
Results and Impact | Invited presentation given on approaches for synthesis of complex interventions: components-based network meta-analysis. An invited presentation on the 2016 paper published in evidence based mental health - Caldwell DM & Welton NJ. Approaches for synthesizing complex mental-health interventions in meta-analysis. Evidence Based Mental Health. 2016; 19:16-21. Audience invited from across London universities. |
Year(s) Of Engagement Activity | 2017 |
URL | http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=3471 |
Description | Caldwell, DM. Network meta-analysis: value and importance to Cochrane. Cochrane Common Mental Disorders group - 20th Anniversary meeting, York, July 2016 (Invited) |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | Caldwell, DM. "Network meta-analysis: value and importance to Cochrane". Cochrane Common Mental Disorders group - 20th Anniversary meeting, York, July 2016 (Invited presentation) |
Year(s) Of Engagement Activity | 2016 |
Description | World Health Organisation (WHO): strengthening the process and methods for retrieval, synthesis and assessment of evidence on complex, multidisciplinary interventions for WHO guidelines |
Form Of Engagement Activity | A formal working group, expert panel or dialogue |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Third sector organisations |
Results and Impact | Review of 6 background papers commissioned to strengthen WHO guideline development process to improve synthesis, assessment and application of evidence on complex and multidisciplinary interventions. There are 5 themes for the meeting. Deborah Caldwel is co-author of draft for quantitative synthesis theme (lead Julian Higgins). |
Year(s) Of Engagement Activity | 2017 |
URL | http://www.who.int/kms/scopingmeetingrsaecmi/en/ |