Multivariate meta-analysis of multiple correlated outcomes: development and application of methods, with empirical investigation of clinical impact

Lead Research Organisation: University of Birmingham
Department Name: Health and Population Sciences

Abstract

'Meta-analysis' is the statistical approach for combining results from multiple studies examining the same clinical question, such as whether a treatment is effective or not. The aim of a meta-analysis is to combine the results from all studies, to produce an overall summary of the findings (e.g. whether the treatment is effective) which helps doctors and healthcare professionals make patient-related decisions based on all the evidence available.

Reliable data to be combined in a meta-analysis are expensive and hard to come by; for example, they often come from clinical trials that have recruited hundreds of patients and followed them up for several years. It is thus important that when researchers undertake a meta-analysis they use all the available data in the most efficient way, to get the most of it. Yet, unfortunately in current practice most researchers are ignoring valuable information that is contained in the available meta-analysis data. This is because many healthcare studies have more than outcome of interest, such as time to recurrence of disease (called 'disease-free survival') and time to death (called 'overall survival'), and researchers currently meta-analyse each outcome separately. However, such multiple outcomes are often related to each other, i.e. they are correlated. For example, a patient's time to recurrence of disease is generally associated with their time of death, as when disease returns the time to death will often follow shortly after. So disease-free survival and overall survival results are correlated, but by meta-analysing each outcome independently, researcher ignores this correlation and thus lose potentially valuable information. In particular, if a study reports overall survival results, but not disease-free survival results, this study can still provide information about disease-free survival by recognising the correlation this outcome has with the available overall survival information. Yet at the moment most researchers simply 'throw away' this study when meta-analysing the disease-free survival results, and thus waste available data.

We believe that a better approach is to meta-analyse correlated outcomes together and utilise their correlated information to get the most out of the available data. The way to do this is a 'multivariate meta-analysis', which is a statistical approach for meta-analysing all the outcomes simultaneously whilst recognising and accounting for their correlation. This allows meta-analysis results for, say, outcome A to be informed by evidence from studies reporting outcome A and also from studies reporting other correlated outcomes B, C and D. As this approach uses more of the information available, it can potentially lead to more reliable results for doctors and healthcare professionals.

Our proposed research project aims to facilitate, and encourage more widespread use of, multivariate meta-analysis in current practice. We will firstly perform an empirical investigation as to how the use of multivariate meta-analysis would change existing clinical conclusions with Reviews of the Cochrane Pregnancy and Childbirth Group, to promote why the approach is important and to update clinical conclusions for pregnancy and childbirth based on more information. We will then develop statistical measures that 'flag' when multivariate meta-analysis is beneficial, so to directly inform researchers when they should use the multivariate approach. We will then develop and extend methods to estimating the correlation between outcomes, which are needed to apply the multivariate approach but are often missing. Finally, we will develop multivariate methods for meta-analysing multiple adverse outcomes. Current methods deal with more common outcomes, but adverse outcomes are rare and so more appropriate methods are needed for such situations. We will also produce a multivariate meta-analysis website, suitable statistical software, and a training course for researchers.

Technical Summary

Following a systematic review, multiple meta-analyses are often performed because multiple outcomes are of interest. For example, in meta-analyses of cancer trials the treatment effect on both overall survival and disease-free survival is important. Such multiple outcomes are not independent; e.g. a patient's time to recurrence often occurs shortly before their time of death, and so disease-free and overall survival are positively correlated. Most reviewers ignore this correlation and simply use a 'univariate' meta-analysis of each outcome independently. Alternatively, multivariate meta-analysis methods can jointly synthesise the outcomes and properly account for their correlation. This allows meta-analysis results for, say, outcome A to be informed by evidence from studies reporting outcome A and - crucially - also from studies reporting only correlated outcome B. Compared to univariate meta-analysis, this greater use of available information leads to summary results with improved statistical properties and potentially even different clinical conclusions.

This project aims to facilitate the use of multivariate meta-analysis in practice, by showing its clinical impact and overcoming methodology challenges. We will examine how multivariate meta-analyses changes existing clinical inferences within Cochrane Pregnancy and Childbirth Reviews, where correlated outcomes are common. We will derive statistical measures that reveal the impact of utilising correlation and thus 'flag' when multivariate meta-analysis is important. We will then develop and extend methods for estimating the correlation between outcomes, which are needed to apply the multivariate approach; situations involving individual patient data and only summary data will be considered for binary, continuous, and survival outcomes. Finally, we will develop methods for meta-analysing correlated adverse (rare) outcomes, for which within-study distributions other than multivariate normality are needed.

Planned Impact

The key aim of the research project is to facilitate greater use of multivariate meta-analysis methods in practice; such methods provide evidence-based results that will impact upon clinician decision-making and patient care. The scope of potential multivariate meta-analysis applications in healthcare is broad, essentially to wherever multiple outcomes or multiple endpoints are of interest. We have observed application to therapeutic interventions [1], dentistry [2], surrogate outcomes [3, 4], predictive biomarkers [5], diagnostic tests[6], genetic epidemiology [7], and risk prediction / multivariable modelling [8], amongst others. The proposed research will provide clinical conclusions whenever it is applied in such settings, and will help clinicians and healthcare professionals make evidence-based decisions about, for example, the best treatment effects, prognostic factors, or diagnostic tests in relation to multiple outcomes that are relevant. Policy-makers both on a national and international level, such as EMA, FDA and NICE, who are making decisions on clinical practice based on all the evidence will benefit from results from the multivariate approach. Specifically, during our project the application of our methods to the Cochrane Pregnancy and Childbirth Reviews may change existing results and clinical conclusions, and this will be fed back to the original reviewers, clinicians, other healthcare professionals, and decision-makers in the field to consider in amended clinical guidelines.

The multivariate meta-analysis approach will also benefit patients. As the multivariate approach uses more information available in the data compared to currently applied meta-analysis methods, this potentially leads to more reliable (less biased) results; this means that summary estimates from a multivariate analysis should be closer to the truth, and so more reliable clinical conclusions can be drawn, that ensure the most appropriate patient care. The method will also support the endeavours of study and trials participants. Such people have agreed to participate to enhance medical research and want their data to be utilised in full; the multivariate approach will achieve this by utilising the correlation between outcomes to extract more information than is currently being used.

Our project also has large potential impact on members and guideline-makers in the Cochrane Collaboration, who need advice on when to recommend multivariate meta-analyses of therapeutic trials, and how they should be performed. Our guidelines, recommendations and methods development will be disseminated back to them for consideration in updates of the Cochrane Handbook [9] and RevMan, their review software.

References
1. Berkey CS, et al: Multiple-outcome meta-analysis of clinical trials. Stat Med 1996, 15:537-557.
2. Berkey CS, et al: Multiple-outcomes meta-analysis of treatments for periodontal disease. J Dent Res 1995, 74:1030-1039.
3. Daniels MJ, Hughes MD: Meta-analysis for the evaluation of potential surrogate markers. Stat Med 1997, 16:1965-1982.
4. Gail MH, et al: On meta-analytic assessment of surrogate outcomes. Biostatistics 2000, 1:231-246.
5. Riley RD, et al.: An evaluation of bivariate random-effects meta-analysis for the joint synthesis of two correlated outcomes. Stat Med 2007, 26:78-97.
6. Reitsma JB, et al.: Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews. J Clin Epidemiol 2005, 58:982-990.
7. Thompson JR, et al: Meta-analysis of genetic studies using Mendelian randomization--a multivariate approach. Stat Med 2005, 24:2241-2254.
8. Gasparrini A, Armstrong B: Multivariate meta-analysis: A method to summarize non-linear associations. Stat Med 2011, 30:2504-2506.
9. Higgins JPT, Green S, (editors). Cochrane Handbook for Systematic Reviews of Interventions. Chichester: John Wiley & Sons; 2008.

Publications

10 25 50

publication icon
Takwoingi Y (2015) Meta-analysis of diagnostic accuracy studies in mental health. in Evidence-based mental health

publication icon
Riley RD (2015) Multivariate meta-analysis using individual participant data. in Research synthesis methods

publication icon
Efthimiou O (2015) Joint synthesis of multiple correlated outcomes in networks of interventions. in Biostatistics (Oxford, England)

publication icon
Jolly K (2016) Self-management of health care behaviors for COPD: a systematic review and meta-analysis. in International journal of chronic obstructive pulmonary disease

publication icon
Jackson D (2017) Borrowing of strength and study weights in multivariate and network meta-analysis. in Statistical methods in medical research

publication icon
Copas JB (2018) The role of secondary outcomes in multivariate meta-analysis. in Journal of the Royal Statistical Society. Series C, Applied statistics

 
Description Multivariate meta-analysis training course
Geographic Reach Asia 
Policy Influence Type Influenced training of practitioners or researchers
Impact Ability to perform multivariate meta-analysis methods when undertaking synthesis of data from multiple studies (e.g. within a systematic review)
 
Title Computer software for multivariate meta-analysis and network meta-analysis 
Description Ian White is updating the mvmeta module in the STATA software as our new methods are being developed. He has also developed a new module called 'network' to perform network meta-analysis using multivariate meta-analysis methods 
Type Of Material Improvements to research infrastructure 
Year Produced 2010 
Provided To Others? Yes  
Impact Wider and easier use of multivariate meta-analysis methods in real applications 
 
Description MSc supervision and Work experience: Helen Blake (University of Birmingham and University of Liverpool, 2013-2014) 
Organisation University of Liverpool
Country United Kingdom 
Sector Academic/University 
PI Contribution Riley, Price (Birmingham) supervised MSc project for Helen Blake entitled: Comparison of univariate and multivariate meta-analysis results within Cochrane Pregnancy and Childbirth reviews Kirkham (liverpool) provided a 6-week work experience for Helen Blake to collate multivariate meta-analysis publications
Collaborator Contribution see above
Impact MSc project (78%)
Start Year 2013
 
Description PhD supervision on multivariate meta-analysis for Giacomo Frosi 
Organisation University of Liverpool
Department Department of Biostatistics
Country United Kingdom 
Sector Academic/University 
PI Contribution PhD supervision: Multivariate Meta-Analysis Methods for Bias Reduction in Systematic Reviews
Collaborator Contribution PhD supervision: Multivariate Meta-Analysis Methods for Bias Reduction in Systematic Reviews
Impact None yet
Start Year 2013
 
Description Invited presentation at LSHTM, London (January 2014) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact White IR. Invited presentation at LSHTM, London (January 2014) entitled Network meta-analysis
Year(s) Of Engagement Activity 2014
 
Description Presentation at Departement de Biostatistique et Informatique Medicale (DBIM), Hopital Saint-Louis, Paris (April 2012) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact White IR. Presentation at Departement de Biostatistique et Informatique Medicale (DBIM), Hopital Saint-Louis, Paris (April 2012) entitled Multivariate meta-analysis and multiple treatments meta-analysis
Year(s) Of Engagement Activity 2012
 
Description Presentation at ISCB 2014 conference, Vienna 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Snell KIE - Presentation at ISCB 2014 conference, Vienna entitled A meta-analysis framework for summarising the performance of a prognosis model developed and validated across multiple studies
Year(s) Of Engagement Activity 2014
 
Description Presentation at Methods for Meta-analysis conference, London (July 2013) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact RD Riley. Presentation at Methods for Meta-analysis conference, London (July 2013) entitled Meta-analysis of test accuracy studies with multiple and missing thresholds.
Year(s) Of Engagement Activity 2013
 
Description Presentation at Methods in Meta-analysis group, London (June 2014) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact White IR. Presentation at Methods in Meta-analysis group, London (June 2014) entitled a suite of programs for network meta-analysis in Stata
Year(s) Of Engagement Activity 2014
 
Description Presentation at Methods in Meta-analysis meeting, RSS London (December 2014) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact JJ Ensor. Presentation at Methods in Meta-analysis meeting, RSS London (December 2014) entitled an imputation approach for dealing with multiple and missing threshold results in meta-analysis of test accuracy studies
Year(s) Of Engagement Activity 2014
 
Description Presentation at RSS Internal Conference, Newcastle (September 2013) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact RD Riley - Presentation at RSS Internal Conference, Newcastle (September 2013) entitled Multivariate meta-analysis using individual participant data: with application to continuous, survival and surrogate outcomes
Year(s) Of Engagement Activity 2013
 
Description Presentation at Society for Research Synthesis Methods, York (July 2014) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact White IR. Presentation at Society for Research Synthesis Methods, York (July 2014) entitled Network meta-analysis: the frequentist alternative
Year(s) Of Engagement Activity 2014
 
Description Presentation at Symposium on Methods for Evaluating Medical Tests and Biomarkers, University of Birmingham (July 2013) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact RD Riley. Presentation at Symposium on Methods for Evaluating Medical Tests and Biomarkers, University of Birmingham (July 2013) entitled Meta-analysis of test accuracy studies with multiple and missing thresholds
Year(s) Of Engagement Activity 2013