Systematic Reviews and Meta-Analysis of Prognosis Studies (REVAMP): development of core methods, reporting guidelines and a methodology handbook

Lead Research Organisation: Keele University
Department Name: Inst for Primary Care and Health Sci

Abstract

Healthcare research is in an exciting new phase. Researchers now have access to large amounts of information to link an individual's characteristics (such as age, family history or genetic information) with health outcomes (such as death, pain level, depression score). Researchers are using this information to help health professionals and patients understand and predict future outcomes. Such research is called prognosis research, and covers four types: overall prognosis (what is the average outcome risk in a population?), prognostic factors (what characteristics of patients help understand their outcome risk?), prognostic models (can we predict an individual's risk using multiple prognostic factors?), and predictors of treatment effect (can we predict which treatments work best for individual patients?). Prognosis research findings allow health professionals to more accurately predict and therefore intervene to improve the health outcomes of individuals.

Thousands of prognosis studies are published each year, and many have unclear or contradictory findings. This makes it difficult for healthcare professionals to know how best to use these findings to improve the care of their patients. This motivates the need for systematic reviews of prognosis studies. These aim to identify all existing prognosis studies on a particular topic, and to appraise their quality and, if appropriate, combine their results (using meta-analysis) to provide overall answers for healthcare professionals. Systematic review methods are well-established in other areas of medical research, but are only just emerging for prognosis research. In our project, we will develop and make freely available reliable methods to undertake systematic reviews of prognosis studies. We have named this initiative REVAMP, which stands for Systematic Reviews and Meta-Analysis of Prognosis Studies.

We propose a series of five work projects (WPs) to be delivered by researchers with considerable expertise in prognosis research and systematic reviews. In WP1, we will develop and disseminate guidance for reviews of overall prognosis; which aim to provide a summary of the average risk in a particular population (e.g. those diagnosed with breast cancer). A step-by-step guide will be produced, covering aspects such as searching for relevant studies, data extraction, critical appraisal, and meta-analysis.

Clear reporting is essential for those interpreting, critically appraising, and implementing prognosis research findings. However, no reporting guidelines currently exist for systematic reviews of prognosis studies. Therefore, in WP2 we will provide three reporting guidelines (checklists) for prognosis reviews: one for each of overall prognosis, prognostic factors and prognostic models. These will be developed through a consensus process involving researchers, healthcare professionals, organisations developing clinical guidelines, and patients.

In WP3 and 5, we will develop new ways for researchers to calculate how many individuals are needed for a prognosis study. Current studies are often too small and so may have misleading results. We will use mathematical techniques to produce a new way to calculate sample size, so that it will be easier to judge (i) whether an existing prognostic model study was large enough to be fit-for-purpose, and (ii) what size a new prognosis study needs to be to contribute usefully to a future updated review.

In WP4 we will produce step-by-step guidance for conducting reviews of predictors of treatment effect, particularly how to do critical appraisal and meta-analysis when the aim is to summarise whether a treatment's effect depends on patient characteristics such as age.

Our project's findings will be disseminated in a Methodology Handbook for Prognosis Reviews and through a new UK-based research training course, in addition to publications in journals and via blogs and tutorial videos at websites including www.prognosisresearch.com.

Technical Summary

Precision medicine relies heavily on prognosis research findings, which allow health professionals to understand and predict outcomes in patients with a particular disease or condition. The PROGRESS framework defines four types of prognosis research: (i) overall prognosis, (ii) prognostic factors, (iii) prognostic models and (iv) predictors of treatment effect. The implementation of prognosis research findings should be evidence-based, with multiple high-quality studies confirming their accuracy and reliability. This motivates the need for systematic reviews and meta-analyses of prognosis studies to identify, critically appraise, synthesise and summarise the evidence.

Although guidance is well developed for reviews of intervention studies, the methodology for prognosis reviews is only just emerging, and is especially sparse for overall prognosis and predictors of treatment effect. Yet prognosis reviews are rapidly being undertaken, as seen from living reviews of COVID-19 prognosis studies. Hence, there is an urgent need to develop robust methods and guidance to support those undertaking prognosis reviews, including Cochrane, Evidence Based Medicine centres, and decision makers such as NICE. Our grant will address this via five work packages (WPs), covering step-by-step guidance for reviews of overall prognosis (WP1); reporting guidelines for types (i) to (iii) (WP2); equations for appraising the sample size of prognostic model studies for risk of bias classifications (e.g. to ascertain overfitting concerns) (WP3); methods to improve risk of bias and meta-analysis in reviews of predictors of treatment effect (WP4); and sample size calculations for new prognosis studies that subsequently will be included in an updated review (WP5).

Our findings will be disseminated in a Handbook for Prognosis Reviews (co-written with Cochrane Prognosis Methods Group), a UK-based training course, journal articles, statistical software, and tutorials at www.prognosisresearch.com.

Publications

10 25 50

 
Title Reporting guideline (checklist) for reviews of prediction model studies 
Description TRIPOD-SRMA provides a checklist for the items to report when publishing a review of prediction models 
Type Of Material Improvements to research infrastructure 
Year Produced 2023 
Provided To Others? Yes  
Impact Endorsed by the Cochrane Prognosis Methods Group 
URL https://www.equator-network.org/reporting-guidelines/tripod-srma/
 
Description Cochrane Prognosis Methods Group 
Organisation The Cochrane Collaboration
Country Global 
Sector Charity/Non Profit 
PI Contribution We have developed methods and guidance to support those working in the Cochrane collaboration on prognosis reviews, and are developing the Cochrane Handbook for Prognosis Reviews with them
Collaborator Contribution They have provided feedback, guidance and methods to inform and extend the work in REVAMP, and are controbuting to chapters in the Cochrane Prognosis Reviews Handbook
Impact 1. Snell KIE, Levis B, Damen JAA, Dhiman P, Debray TPA, Hooft L, et al. Transparent reporting of multivariable prediction models for individual prognosis or diagnosis: checklist for systematic reviews and meta-analyses (TRIPOD-SRMA). BMJ. 2023;381:e073538. 2. Hudda MT, Archer L, van Smeden M, Moons KGM, Collins GS, Steyerberg EW, et al. Minimal reporting improvement after peer review in reports of COVID-19 prediction models: systematic review. J Clin Epidemiol. 2023;154:75-84. 3. Levis B, Snell KIE, Damen JAA, Hattle M, Ensor J, Dhiman P, et al. Risk of bias assessments in individual participant data meta-analyses of test accuracy and prediction models: a review shows improvements are needed. J Clin Epidemiol. 2024;165:111206.
Start Year 2008
 
Title Software in Stata for calculating the sample size required for validation of prediction models 
Description This calculates the sample size needed for studies evaluating a model, which is important part of risk of bias assessments in reviews of models, or for those planning studies to evaluate models 
Type Of Technology Software 
Year Produced 2023 
Impact Only just released 
URL https://ideas.repec.org/c/boc/bocode/s459226.html#:~:text=pmvalsampsize%20computes%20the%20minimum%2...
 
Title pmvalsampsize: Software in R for calculating the sample size needed to validate a prediction model 
Description This packages calculate the sample size needed to evaluate a prediction model precisely, which is important for those examining a studies risk of bias, or for those designing a new study 
Type Of Technology Software 
Year Produced 2024 
Impact only just released 
URL https://cran.r-project.org/web/packages/pmvalsampsize/index.html
 
Description 3-day training course: Statistical Methods for Risk Prediction & Prognostic Models 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact We run a 3-day training course to educate researchers on how to use statistical methods to develop and evaluate clinical prediction models, covering a huge array of topics and software practicals - this is attended by about 50 people each time, and we run about 3 times per year
Year(s) Of Engagement Activity 2022,2023,2024
 
Description 3-day training course: Statistical Methods for individual Participant Data Meta-Analysis 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact We run a 3-day course to educate researchers on how to undertake an IPD meta-analysis of randomised trials, to evaluate intervention effects, or of multiple datasets aiming to develop/evaluate prediction models or prognosis factors - it is attended by a range of methodologists and clinical academics, and various stages of their careers, across academia and industry
Year(s) Of Engagement Activity 2022,2023,2024
 
Description Cochrane Colloquium Workshops on Prognosis Reviews 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Policymakers/politicians
Results and Impact We run yearly workshops at the Cochrane colloquium to disseminate best practice in reviews of prognosis studies, covering all aspects from initiation to dissemination
Year(s) Of Engagement Activity 2023
 
Description Invited oral presentation (Children's Hospital of Eastern Ontario (CHEO) Research Institute, Canada, June 2023) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Policymakers/politicians
Results and Impact I gave a talk about the Key Steps and Common Pitfalls in Clinical Prediction Model Research, to disseminate current standards and our research outputs / recommendations
Year(s) Of Engagement Activity 2023
 
Description Oral presentation at Society for Research Synthesis Methods Conference 2023, Paris 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Policymakers/politicians
Results and Impact Richard Riley gave an oral presentation on the REVAMP work for power calculations for IPD meta-analysis projects aiming to identify predictors of treatment effect
Year(s) Of Engagement Activity 2023
 
Description Prognosis Research in Healthcare Summer School 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Each year this summer school disseminates best practice in undertaking primary studies and reviews of prognosis research, to a broad clinical and methodological audience from academia and industry, with participants from around the world
Year(s) Of Engagement Activity 2021,2022,2023
 
Description invited oral presentation (Aberdeen, RSS local meeting, October 2023): Clinical prediction models: a playground for healthcare research 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Professional Practitioners
Results and Impact I spoke about 'Clinical prediction models: a playground for healthcare research (Aberdeen, RSS local meeting, invited in-person)', to disseminate our work on instability and the need for better sample sizes, to improve reviews and meta-analyses
Year(s) Of Engagement Activity 2023
 
Description invited oral presentation (March 2022: DAGStat meeting (Education for Statistics in Practice session; Hamburg) 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Policymakers/politicians
Results and Impact I spoke as an invited speaker, giving a 3 session workshop about 'Prognosis Research in Healthcare: initiatives to improve methodology standards', which disseminated much of our REVAMP work on stability of models
Year(s) Of Engagement Activity 2022,2023
 
Description invited oral presentation (Utrecht, October 2023): Stability of clinical prediction models developed using statistical or machine learning methods 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact I gave a talk to disseminate our work on stability of clinical prediction models developed using statistical or machine learning methods
Year(s) Of Engagement Activity 2023