Multivariate meta-analysis of multiple correlated outcomes: development and application of methods, with empirical investigation of clinical impact
Lead Research Organisation:
Keele University
Department Name: Inst for Primary Care and Health Sci
Abstract
'Meta-analysis' is the statistical approach for combining results from multiple studies examining the same clinical question, such as whether a treatment is effective or not. The aim of a meta-analysis is to combine the results from all studies, to produce an overall summary of the findings (e.g. whether the treatment is effective) which helps doctors and healthcare professionals make patient-related decisions based on all the evidence available.
Reliable data to be combined in a meta-analysis are expensive and hard to come by; for example, they often come from clinical trials that have recruited hundreds of patients and followed them up for several years. It is thus important that when researchers undertake a meta-analysis they use all the available data in the most efficient way, to get the most of it. Yet, unfortunately in current practice most researchers are ignoring valuable information that is contained in the available meta-analysis data. This is because many healthcare studies have more than outcome of interest, such as time to recurrence of disease (called 'disease-free survival') and time to death (called 'overall survival'), and researchers currently meta-analyse each outcome separately. However, such multiple outcomes are often related to each other, i.e. they are correlated. For example, a patient's time to recurrence of disease is generally associated with their time of death, as when disease returns the time to death will often follow shortly after. So disease-free survival and overall survival results are correlated, but by meta-analysing each outcome independently, researcher ignores this correlation and thus lose potentially valuable information. In particular, if a study reports overall survival results, but not disease-free survival results, this study can still provide information about disease-free survival by recognising the correlation this outcome has with the available overall survival information. Yet at the moment most researchers simply 'throw away' this study when meta-analysing the disease-free survival results, and thus waste available data.
We believe that a better approach is to meta-analyse correlated outcomes together and utilise their correlated information to get the most out of the available data. The way to do this is a 'multivariate meta-analysis', which is a statistical approach for meta-analysing all the outcomes simultaneously whilst recognising and accounting for their correlation. This allows meta-analysis results for, say, outcome A to be informed by evidence from studies reporting outcome A and also from studies reporting other correlated outcomes B, C and D. As this approach uses more of the information available, it can potentially lead to more reliable results for doctors and healthcare professionals.
Our proposed research project aims to facilitate, and encourage more widespread use of, multivariate meta-analysis in current practice. We will firstly perform an empirical investigation as to how the use of multivariate meta-analysis would change existing clinical conclusions with Reviews of the Cochrane Pregnancy and Childbirth Group, to promote why the approach is important and to update clinical conclusions for pregnancy and childbirth based on more information. We will then develop statistical measures that 'flag' when multivariate meta-analysis is beneficial, so to directly inform researchers when they should use the multivariate approach. We will then develop and extend methods to estimating the correlation between outcomes, which are needed to apply the multivariate approach but are often missing. Finally, we will develop multivariate methods for meta-analysing multiple adverse outcomes. Current methods deal with more common outcomes, but adverse outcomes are rare and so more appropriate methods are needed for such situations. We will also produce a multivariate meta-analysis website, suitable statistical software, and a training course for researchers.
Reliable data to be combined in a meta-analysis are expensive and hard to come by; for example, they often come from clinical trials that have recruited hundreds of patients and followed them up for several years. It is thus important that when researchers undertake a meta-analysis they use all the available data in the most efficient way, to get the most of it. Yet, unfortunately in current practice most researchers are ignoring valuable information that is contained in the available meta-analysis data. This is because many healthcare studies have more than outcome of interest, such as time to recurrence of disease (called 'disease-free survival') and time to death (called 'overall survival'), and researchers currently meta-analyse each outcome separately. However, such multiple outcomes are often related to each other, i.e. they are correlated. For example, a patient's time to recurrence of disease is generally associated with their time of death, as when disease returns the time to death will often follow shortly after. So disease-free survival and overall survival results are correlated, but by meta-analysing each outcome independently, researcher ignores this correlation and thus lose potentially valuable information. In particular, if a study reports overall survival results, but not disease-free survival results, this study can still provide information about disease-free survival by recognising the correlation this outcome has with the available overall survival information. Yet at the moment most researchers simply 'throw away' this study when meta-analysing the disease-free survival results, and thus waste available data.
We believe that a better approach is to meta-analyse correlated outcomes together and utilise their correlated information to get the most out of the available data. The way to do this is a 'multivariate meta-analysis', which is a statistical approach for meta-analysing all the outcomes simultaneously whilst recognising and accounting for their correlation. This allows meta-analysis results for, say, outcome A to be informed by evidence from studies reporting outcome A and also from studies reporting other correlated outcomes B, C and D. As this approach uses more of the information available, it can potentially lead to more reliable results for doctors and healthcare professionals.
Our proposed research project aims to facilitate, and encourage more widespread use of, multivariate meta-analysis in current practice. We will firstly perform an empirical investigation as to how the use of multivariate meta-analysis would change existing clinical conclusions with Reviews of the Cochrane Pregnancy and Childbirth Group, to promote why the approach is important and to update clinical conclusions for pregnancy and childbirth based on more information. We will then develop statistical measures that 'flag' when multivariate meta-analysis is beneficial, so to directly inform researchers when they should use the multivariate approach. We will then develop and extend methods to estimating the correlation between outcomes, which are needed to apply the multivariate approach but are often missing. Finally, we will develop multivariate methods for meta-analysing multiple adverse outcomes. Current methods deal with more common outcomes, but adverse outcomes are rare and so more appropriate methods are needed for such situations. We will also produce a multivariate meta-analysis website, suitable statistical software, and a training course for researchers.
Technical Summary
Following a systematic review, multiple meta-analyses are often performed because multiple outcomes are of interest. For example, in meta-analyses of cancer trials the treatment effect on both overall survival and disease-free survival is important. Such multiple outcomes are not independent; e.g. a patient's time to recurrence often occurs shortly before their time of death, and so disease-free and overall survival are positively correlated. Most reviewers ignore this correlation and simply use a 'univariate' meta-analysis of each outcome independently. Alternatively, multivariate meta-analysis methods can jointly synthesise the outcomes and properly account for their correlation. This allows meta-analysis results for, say, outcome A to be informed by evidence from studies reporting outcome A and - crucially - also from studies reporting only correlated outcome B. Compared to univariate meta-analysis, this greater use of available information leads to summary results with improved statistical properties and potentially even different clinical conclusions.
This project aims to facilitate the use of multivariate meta-analysis in practice, by showing its clinical impact and overcoming methodology challenges. We will examine how multivariate meta-analyses changes existing clinical inferences within Cochrane Pregnancy and Childbirth Reviews, where correlated outcomes are common. We will derive statistical measures that reveal the impact of utilising correlation and thus 'flag' when multivariate meta-analysis is important. We will then develop and extend methods for estimating the correlation between outcomes, which are needed to apply the multivariate approach; situations involving individual patient data and only summary data will be considered for binary, continuous, and survival outcomes. Finally, we will develop methods for meta-analysing correlated adverse (rare) outcomes, for which within-study distributions other than multivariate normality are needed.
This project aims to facilitate the use of multivariate meta-analysis in practice, by showing its clinical impact and overcoming methodology challenges. We will examine how multivariate meta-analyses changes existing clinical inferences within Cochrane Pregnancy and Childbirth Reviews, where correlated outcomes are common. We will derive statistical measures that reveal the impact of utilising correlation and thus 'flag' when multivariate meta-analysis is important. We will then develop and extend methods for estimating the correlation between outcomes, which are needed to apply the multivariate approach; situations involving individual patient data and only summary data will be considered for binary, continuous, and survival outcomes. Finally, we will develop methods for meta-analysing correlated adverse (rare) outcomes, for which within-study distributions other than multivariate normality are needed.
Planned Impact
The key aim of the research project is to facilitate greater use of multivariate meta-analysis methods in practice; such methods provide evidence-based results that will impact upon clinician decision-making and patient care. The scope of potential multivariate meta-analysis applications in healthcare is broad, essentially to wherever multiple outcomes or multiple endpoints are of interest. We have observed application to therapeutic interventions [1], dentistry [2], surrogate outcomes [3, 4], predictive biomarkers [5], diagnostic tests[6], genetic epidemiology [7], and risk prediction / multivariable modelling [8], amongst others. The proposed research will provide clinical conclusions whenever it is applied in such settings, and will help clinicians and healthcare professionals make evidence-based decisions about, for example, the best treatment effects, prognostic factors, or diagnostic tests in relation to multiple outcomes that are relevant. Policy-makers both on a national and international level, such as EMA, FDA and NICE, who are making decisions on clinical practice based on all the evidence will benefit from results from the multivariate approach. Specifically, during our project the application of our methods to the Cochrane Pregnancy and Childbirth Reviews may change existing results and clinical conclusions, and this will be fed back to the original reviewers, clinicians, other healthcare professionals, and decision-makers in the field to consider in amended clinical guidelines.
The multivariate meta-analysis approach will also benefit patients. As the multivariate approach uses more information available in the data compared to currently applied meta-analysis methods, this potentially leads to more reliable (less biased) results; this means that summary estimates from a multivariate analysis should be closer to the truth, and so more reliable clinical conclusions can be drawn, that ensure the most appropriate patient care. The method will also support the endeavours of study and trials participants. Such people have agreed to participate to enhance medical research and want their data to be utilised in full; the multivariate approach will achieve this by utilising the correlation between outcomes to extract more information than is currently being used.
Our project also has large potential impact on members and guideline-makers in the Cochrane Collaboration, who need advice on when to recommend multivariate meta-analyses of therapeutic trials, and how they should be performed. Our guidelines, recommendations and methods development will be disseminated back to them for consideration in updates of the Cochrane Handbook [9] and RevMan, their review software.
References
1. Berkey CS, et al: Multiple-outcome meta-analysis of clinical trials. Stat Med 1996, 15:537-557.
2. Berkey CS, et al: Multiple-outcomes meta-analysis of treatments for periodontal disease. J Dent Res 1995, 74:1030-1039.
3. Daniels MJ, Hughes MD: Meta-analysis for the evaluation of potential surrogate markers. Stat Med 1997, 16:1965-1982.
4. Gail MH, et al: On meta-analytic assessment of surrogate outcomes. Biostatistics 2000, 1:231-246.
5. Riley RD, et al.: An evaluation of bivariate random-effects meta-analysis for the joint synthesis of two correlated outcomes. Stat Med 2007, 26:78-97.
6. Reitsma JB, et al.: Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews. J Clin Epidemiol 2005, 58:982-990.
7. Thompson JR, et al: Meta-analysis of genetic studies using Mendelian randomization--a multivariate approach. Stat Med 2005, 24:2241-2254.
8. Gasparrini A, Armstrong B: Multivariate meta-analysis: A method to summarize non-linear associations. Stat Med 2011, 30:2504-2506.
9. Higgins JPT, Green S, (editors). Cochrane Handbook for Systematic Reviews of Interventions. Chichester: John Wiley & Sons; 2008.
The multivariate meta-analysis approach will also benefit patients. As the multivariate approach uses more information available in the data compared to currently applied meta-analysis methods, this potentially leads to more reliable (less biased) results; this means that summary estimates from a multivariate analysis should be closer to the truth, and so more reliable clinical conclusions can be drawn, that ensure the most appropriate patient care. The method will also support the endeavours of study and trials participants. Such people have agreed to participate to enhance medical research and want their data to be utilised in full; the multivariate approach will achieve this by utilising the correlation between outcomes to extract more information than is currently being used.
Our project also has large potential impact on members and guideline-makers in the Cochrane Collaboration, who need advice on when to recommend multivariate meta-analyses of therapeutic trials, and how they should be performed. Our guidelines, recommendations and methods development will be disseminated back to them for consideration in updates of the Cochrane Handbook [9] and RevMan, their review software.
References
1. Berkey CS, et al: Multiple-outcome meta-analysis of clinical trials. Stat Med 1996, 15:537-557.
2. Berkey CS, et al: Multiple-outcomes meta-analysis of treatments for periodontal disease. J Dent Res 1995, 74:1030-1039.
3. Daniels MJ, Hughes MD: Meta-analysis for the evaluation of potential surrogate markers. Stat Med 1997, 16:1965-1982.
4. Gail MH, et al: On meta-analytic assessment of surrogate outcomes. Biostatistics 2000, 1:231-246.
5. Riley RD, et al.: An evaluation of bivariate random-effects meta-analysis for the joint synthesis of two correlated outcomes. Stat Med 2007, 26:78-97.
6. Reitsma JB, et al.: Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews. J Clin Epidemiol 2005, 58:982-990.
7. Thompson JR, et al: Meta-analysis of genetic studies using Mendelian randomization--a multivariate approach. Stat Med 2005, 24:2241-2254.
8. Gasparrini A, Armstrong B: Multivariate meta-analysis: A method to summarize non-linear associations. Stat Med 2011, 30:2504-2506.
9. Higgins JPT, Green S, (editors). Cochrane Handbook for Systematic Reviews of Interventions. Chichester: John Wiley & Sons; 2008.
Publications
Bujkiewicz S
(2016)
Bayesian meta-analytical methods to incorporate multiple surrogate endpoints in drug development process.
in Statistics in medicine
Burke DL
(2018)
Bayesian bivariate meta-analysis of correlated effects: Impact of the prior distributions on the between-study correlation, borrowing of strength, and joint inferences.
in Statistical methods in medical research
Burke DL
(2017)
Meta-analysis using individual participant data: one-stage and two-stage approaches, and why they may differ.
in Statistics in medicine
Chen Y
(2016)
Inference for correlated effect sizes using multiple univariate meta-analyses.
in Statistics in medicine
Copas J
(2019)
Model-based sensitivity analysis for outcome reporting bias in the meta analysis of benefit and harm outcomes.
in Statistical methods in medical research
Efthimiou O
(2015)
Joint synthesis of multiple correlated outcomes in networks of interventions.
in Biostatistics (Oxford, England)
Ensor J
(2018)
Meta-analysis of test accuracy studies using imputation for partial reporting of multiple thresholds.
in Research synthesis methods
Jackson D
(2016)
Extending DerSimonian and Laird's methodology to perform network meta-analyses with random inconsistency effects.
in Statistics in medicine
Jackson D
(2017)
Borrowing of strength and study weights in multivariate and network meta-analysis.
in Statistical methods in medical research
Riley RD
(2015)
Meta-analysis of test accuracy studies: an exploratory method for investigating the impact of missing thresholds.
in Systematic reviews
Riley RD
(2015)
Multivariate meta-analysis using individual participant data.
in Research synthesis methods
Riley RD
(2018)
Deriving percentage study weights in multi-parameter meta-analysis models: with application to meta-regression, network meta-analysis and one-stage individual participant data models.
in Statistical methods in medical research
Riley RD
(2017)
Multivariate and network meta-analysis of multiple outcomes and multiple treatments: rationale, concepts, and examples.
in BMJ (Clinical research ed.)
Snell KI
(2018)
Meta-analysis of prediction model performance across multiple studies: Which scale helps ensure between-study normality for the C-statistic and calibration measures?
in Statistical methods in medical research
Snell KI
(2016)
Multivariate meta-analysis of individual participant data helped externally validate the performance and implementation of a prediction model.
in Journal of clinical epidemiology
Takwoingi Y
(2015)
Meta-analysis of diagnostic accuracy studies in mental health.
in Evidence-based mental health
Title | Multivariate meta-analysis website |
Description | Website: www.mvmeta.org This has been registered and is being developed, with the aim to go live in April 2016. The aim of this website is to help promote multivariate meta-analysis by highlighting the rationale for the approach; methodological developments and publications; available statistical software, and upcoming training courses. |
Type Of Material | Improvements to research infrastructure |
Provided To Others? | No |
Impact | Attendees for training course |
Title | STATA module to implement sensitivity analysis approach for dealing with multiple and missing thresholds in test accuracy meta-analysis |
Description | STATA module to implement sensitivity analysis approach for dealing with multiple and missing thresholds in test accuracy meta-analysis |
Type Of Material | Improvements to research infrastructure |
Provided To Others? | No |
Impact | Under development by J Ensor |
Description | PhD supervision - Danielle Burke |
Organisation | University of Birmingham |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | PhD supervision - Danielle Burke. Entitled Use of Bayesian methods for the design, analysis and synthesis of clinical trials |
Collaborator Contribution | PhD supervision - Danielle Burke. Entitled Use of Bayesian methods for the design, analysis and synthesis of clinical trials |
Impact | N/A |
Start Year | 2011 |
Description | PhD supervision - Elena Elia |
Organisation | University of Birmingham |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | PhD supervision - Elena Elia. Entitled Meta-analysis of prognostic factor studies |
Collaborator Contribution | PhD supervision - Elena Elia. Entitled Meta-analysis of prognostic factor studies |
Impact | None as yet |
Start Year | 2012 |
Description | PhD supervision - Hairui Hua |
Organisation | University of Birmingham |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | PhD supervision - Hairui Hua. Title - Survival modelling in mathematical and medical statistics |
Collaborator Contribution | PhD supervision - Hairui Hua. Title - Survival modelling in mathematical and medical statistics |
Impact | N/A |
Start Year | 2012 |
Description | PhD supervision - Ikhlaaq Ahmed |
Organisation | University of Birmingham |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | PhD supervision - Ikhlaaq Ahmed. Title - Meta-analysis of risk prediction studies |
Collaborator Contribution | PhD supervision - Ikhlaaq Ahmed. Title - Meta-analysis of risk prediction studies |
Impact | N/A |
Start Year | 2010 |
Description | PhD supervision - Kym Snell |
Organisation | University of Birmingham |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | PhD supervision - Kym Snell. Entitled Development and application of statistical methods for prognosis research |
Collaborator Contribution | PhD supervision - Kym Snell. Entitled Development and application of statistical methods for prognosis research |
Impact | N/A |
Start Year | 2011 |
Description | Invited Presentation at York Clinical Trials Unit (January 2016) |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | Kirkham JJ. Invited Presentation at York Clinical Trials Unit (January 2016) entitled A random walk through outcomes based research for healthcare |
Year(s) Of Engagement Activity | 2016 |
Description | Invited presentation at RSS Local Group, University of Aberdeen (June 2015) |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | RD Riley - Invited presentation at RSS Local Group, University of Aberdeen (June 2015) entitled Multivariate meta-analysis: advantages, disadvantages and applications |
Year(s) Of Engagement Activity | 2015 |
Description | Invited presentation at RSS conference, Exeter (September 2015) |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | RD Riley - Invited presentation at RSS conference, Exeter entitled meta-analysis of individual participant data: opportunities and challenges |
Year(s) Of Engagement Activity | 2015 |
Description | Invited presentation at Statistics seminar, University of Lancaster (March 2016) |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | RD Riley - Invited presentation at Statistics seminar, University of Lancaster entitled multivariate meta-analysis: advantages, disadvantages and applications. |
Year(s) Of Engagement Activity | 2016 |
Description | Invited presentation at University of Leicester (June 2015) |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | Invited presentation at University of Leicester (June 2015) entitled Multivariate meta-analysis: advantages, disadvantages and applications |
Year(s) Of Engagement Activity | 2015 |
Description | Invited presentation at University of Leicester (March 2016) |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | D Burke. Invited presentation at University of Leicester entitled Bayesian bivariate meta-analysis of correlated effects |
Year(s) Of Engagement Activity | 2016 |
Description | Presentation at Biometrisches Kolloquium: Biometrics and Communication - From Statistical Theory to Perception in the Public (March 2015) |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | D Jackson. Presentation at Biometrisches Kolloquium: Biometrics and Communication - From Statistical Theory to Perception in the Public (March 2015) entitled Models for network meta-analysis with random inconsistency effects |
Year(s) Of Engagement Activity | 2015 |
Description | Presentation at ISCB annual conference, Utretcht (August 2015) |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Burke D - Presentation at ISCB annual conference, Utretcht (August 2015) entitled The choice of prior distribution for the between-study correlation in a Bayesian bivariate meta-analysis |
Year(s) Of Engagement Activity | 2015 |
Description | Presentation at ISCB annual conference, Utretcht (August 2015) |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | RD Riley - Presentation at ISCB annual conference, Utretcht (August 2015) entitled Borrowing of strength and study weights in multivariate and network meta-analysis |
Year(s) Of Engagement Activity | 2015 |
Description | Presentation at ISCB, Utrecht (August 2015) |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | White IR. Presentation at ISCB, Utrecht (August 2015) entitled A comparison of methods for meta-analysis of non-linear dose-response relationships using individual participant data |
Year(s) Of Engagement Activity | 2015 |