Method - Analysis
Lead Research Organisation:
University College London
Department Name: UNLISTED
Abstract
Appropriate statistical analysis of trial data is key for their correct interpretation. The analysis programme develops and disseminates practical solutions to analysis challenges, principally:
• missing data on patients’ outcomes (ubiquitous, and aggravated by the current pandemic). We continue to address the challenge of how to make accessible, defensible, assumptions about missing patient outcome data, and how to correctly incorporate these in the trial’s interpretation.
• ensuring trials meet their objectives, by thinking carefully about the (i) patient population, (i) outcomes and how they will be measured, and (iii) how to take account of unplanned developments once patients are enrolled in a study (such as inability to tolerate treatment, or switching to another treatment). These issues are particularly important for the Unit’s tuberculosis and cancer trials.
• using computer simulation to effectively understand how useful proposed trials are likely to be, and in particular how resilient to a range of scenarios that may unfold as the trial progresses
• collaborating with the Alan Turing Institute to exploit recent advances in machine learning to identify subgroups of patients who may particularly benefit from certain treatments.
• missing data on patients’ outcomes (ubiquitous, and aggravated by the current pandemic). We continue to address the challenge of how to make accessible, defensible, assumptions about missing patient outcome data, and how to correctly incorporate these in the trial’s interpretation.
• ensuring trials meet their objectives, by thinking carefully about the (i) patient population, (i) outcomes and how they will be measured, and (iii) how to take account of unplanned developments once patients are enrolled in a study (such as inability to tolerate treatment, or switching to another treatment). These issues are particularly important for the Unit’s tuberculosis and cancer trials.
• using computer simulation to effectively understand how useful proposed trials are likely to be, and in particular how resilient to a range of scenarios that may unfold as the trial progresses
• collaborating with the Alan Turing Institute to exploit recent advances in machine learning to identify subgroups of patients who may particularly benefit from certain treatments.
Technical Summary
The Analysis Programme provides and disseminates practical solutions for ongoing and emerging challenges in trial analyses – both for the Unit’s studies and more widely. New analysis methods usually require new software, which we provide for internal and external use.
Our main areas of work are:
1. Estimands and Treatment Switching
Given the time and cost of trials, it is crucial that careful thought is given to precisely what quantities (i.e. estimands) we wish to make inferences for, and in which population. As part of this, we need to ensure that the analysis handles post-randomisation events — such as treatment switching, or withdrawal from therapy (which frequently results in missing data) — in a way that is consistent with the estimand.
We are focusing on estimands for: non-inferiority tuberculosis trials; cancer trials with non-proportional hazards; cluster randomised and stepped wedge trials; and factorial designs.
Going beyond the treatment policy estimand, we need to take account of treatment switching. We are developing and applying suitable methods for our non-inferiority tuberculosis trials.
2. Sensitivity Analysis
The recent ICH-E9 addendum explicitly calls for sensitivity analysis, to explore the robustness of inferences from primary assumptions for post-randomisation events and missing data.
We will continue to develop two broad approaches: (i) eliciting expert opinion about how the distribution of patients’ unobserved outcomes differs from those we observe, and (ii) reference based sensitivity analysis, where – guided by expert opinion – we impute missing patient outcomes by reference to appropriately chosen observed patient outcomes.
3. Missing Outcome Data
We will tackle emerging missing outcome data challenges which often arise from the estimands framework, since after post-randomisation (often termed inter-current) events, patient data are often missing.
We will focus on: composite outcomes, where a naive approach to missing components can cause the composite to be informatively missing; outcome data from patient wearables, where poor patient compliance raises challenging missing data issues; and data truncated by death, where competing methodologies target different estimands, and raise challenges for the specification of the primary analysis.
4. Adjusting for covariates
Adjusting for trial baseline covariates typically improves power, but often changes the estimand. We will develop practical guidance on pre-specifying covariate adjustment methods that maximise power without changing the estimand.
5. Stratified medicine
Often during a study new information emerges leading to an increased focus on a specific stratum. We will show how to gain precision by borrowing information on treatment effects from a larger patient subgroup, informing the strength of borrowing by expert opinion.
In partnership with the Alan Turing Institute, we will develop and evaluate machine learning approaches for detecting — and quantifying the strength of — strata within which treatment effects differ.
6. Simulation studies
Simulation is a key tool for understanding the operating characteristics of complex trial designs, and their resilience to departures from key assumptions. We will provide software tools and practical guidance for trialists.
Our main areas of work are:
1. Estimands and Treatment Switching
Given the time and cost of trials, it is crucial that careful thought is given to precisely what quantities (i.e. estimands) we wish to make inferences for, and in which population. As part of this, we need to ensure that the analysis handles post-randomisation events — such as treatment switching, or withdrawal from therapy (which frequently results in missing data) — in a way that is consistent with the estimand.
We are focusing on estimands for: non-inferiority tuberculosis trials; cancer trials with non-proportional hazards; cluster randomised and stepped wedge trials; and factorial designs.
Going beyond the treatment policy estimand, we need to take account of treatment switching. We are developing and applying suitable methods for our non-inferiority tuberculosis trials.
2. Sensitivity Analysis
The recent ICH-E9 addendum explicitly calls for sensitivity analysis, to explore the robustness of inferences from primary assumptions for post-randomisation events and missing data.
We will continue to develop two broad approaches: (i) eliciting expert opinion about how the distribution of patients’ unobserved outcomes differs from those we observe, and (ii) reference based sensitivity analysis, where – guided by expert opinion – we impute missing patient outcomes by reference to appropriately chosen observed patient outcomes.
3. Missing Outcome Data
We will tackle emerging missing outcome data challenges which often arise from the estimands framework, since after post-randomisation (often termed inter-current) events, patient data are often missing.
We will focus on: composite outcomes, where a naive approach to missing components can cause the composite to be informatively missing; outcome data from patient wearables, where poor patient compliance raises challenging missing data issues; and data truncated by death, where competing methodologies target different estimands, and raise challenges for the specification of the primary analysis.
4. Adjusting for covariates
Adjusting for trial baseline covariates typically improves power, but often changes the estimand. We will develop practical guidance on pre-specifying covariate adjustment methods that maximise power without changing the estimand.
5. Stratified medicine
Often during a study new information emerges leading to an increased focus on a specific stratum. We will show how to gain precision by borrowing information on treatment effects from a larger patient subgroup, informing the strength of borrowing by expert opinion.
In partnership with the Alan Turing Institute, we will develop and evaluate machine learning approaches for detecting — and quantifying the strength of — strata within which treatment effects differ.
6. Simulation studies
Simulation is a key tool for understanding the operating characteristics of complex trial designs, and their resilience to departures from key assumptions. We will provide software tools and practical guidance for trialists.
Organisations
- University College London (Lead Research Organisation)
- Alan Turing Institute (Collaboration)
- University of Pennsylvania (Collaboration)
- HEALTH DATA RESEARCH UK (Collaboration)
- Albert Ludwig University of Freiburg (Collaboration)
- UNIVERSITY OF LEEDS (Collaboration)
- IMPERIAL COLLEGE LONDON (Collaboration)
- UNIVERSITY OF CAMBRIDGE (Collaboration)
- University of Bristol (Collaboration)
Publications
Smith MJ
(2021)
Investigating the inequalities in route to diagnosis amongst patients with diffuse large B-cell or follicular lymphoma in England.
in British journal of cancer
Pham TM
(2021)
A comparison of methods for analyzing a binary composite endpoint with partially observed components in randomized controlled trials.
in Statistics in medicine
Bazo-Alvarez JC
(2021)
Cardiovascular outcomes of type 2 diabetic patients treated with DPP-4 inhibitors versus sulphonylureas as add-on to metformin in clinical practice.
in Scientific reports
Tackney MS
(2021)
A framework for handling missing accelerometer outcome data in trials.
in Trials
Gasparini A
(2021)
INTEREST: INteractive Tool for Exploring REsults from Simulation sTudies.
in Journal of data science, statistics, and visualisation
Khundi M
(2021)
Clinical, health systems and neighbourhood determinants of tuberculosis case fatality in urban Blantyre, Malawi: a multilevel epidemiological analysis of enhanced surveillance data
in Epidemiology and Infection
Lee KJ
(2021)
Framework for the treatment and reporting of missing data in observational studies: The Treatment And Reporting of Missing data in Observational Studies framework.
in Journal of clinical epidemiology
Freeman SC
(2021)
Authors' reply to "Comments on Identifying inconsistency in network meta-analysis: Is the net heat plot a reliable method?".
in Statistics in medicine
Related Projects
Project Reference | Relationship | Related To | Start | End | Award Value |
---|---|---|---|---|---|
MC_UU_00004/01 | 01/04/2021 | 31/03/2026 | £5,186,000 | ||
MC_UU_00004/02 | Transfer | MC_UU_00004/01 | 01/04/2021 | 31/03/2026 | £4,446,000 |
MC_UU_00004/03 | Transfer | MC_UU_00004/02 | 01/04/2021 | 31/03/2026 | £4,999,000 |
MC_UU_00004/04 | Transfer | MC_UU_00004/03 | 01/04/2021 | 31/03/2026 | £5,315,000 |
MC_UU_00004/05 | Transfer | MC_UU_00004/04 | 01/04/2021 | 31/03/2026 | £3,107,000 |
MC_UU_00004/06 | Transfer | MC_UU_00004/05 | 01/04/2021 | 31/03/2026 | £2,889,000 |
MC_UU_00004/07 | Transfer | MC_UU_00004/06 | 01/04/2021 | 31/03/2026 | £2,369,000 |
MC_UU_00004/08 | Transfer | MC_UU_00004/07 | 01/04/2021 | 31/03/2026 | £2,270,000 |
MC_UU_00004/09 | Transfer | MC_UU_00004/08 | 01/04/2021 | 31/03/2026 | £2,160,000 |
Description | NewDAWN |
Geographic Reach | National |
Policy Influence Type | Participation in a guidance/advisory committee |
URL | https://www.diabetes.org.uk/about_us/news/funding-new-type-2-diabetes-remission-research-NewDAWN |
Description | Training course on analysis of longitudinal and hierarchical data commissioned from professional organisation 'Statisticians in the Pharmaceutical Industry' |
Geographic Reach | Multiple continents/international |
Policy Influence Type | Influenced training of practitioners or researchers |
Description | TMRP Doctoral Training Programme |
Amount | £2,518,806 (GBP) |
Organisation | Medical Research Council (MRC) |
Sector | Public |
Country | United Kingdom |
Start | 09/2021 |
End | 09/2026 |
Description | Alan Turing Institute and AI |
Organisation | Alan Turing Institute |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | This collaboration is in two parts: clinical trial monitoring, and treatment effect heterogeneity. 1. We have described the problems in monitoring of clinical trial data. We are supplying data sets, information about monitoring and on-hand expertise to allow an exploration of the use of AI. 2. For the treatment effect heterogeneity project, we are supplying our expertise on treatment effect estimation in clinical trials, new work on performance measures for treatment effect heterogeneity, and data from a trial in treatmnet of severe anaemia in African children |
Collaborator Contribution | Scientists of the Alan Turing Institute are using AI including ML on our clinical trial data and on simulated data to find out what methods can be used in the two settings above. A data study group has been run on the monitoring project (entered separately in Research Fish). |
Impact | A data study group was run in November and December 2021, allowing a group of volunteers to work intensively on the monitoring problem. |
Start Year | 2019 |
Description | Alan Turing Institute and AI |
Organisation | Health Data Research UK |
Country | United Kingdom |
Sector | Private |
PI Contribution | This collaboration is in two parts: clinical trial monitoring, and treatment effect heterogeneity. 1. We have described the problems in monitoring of clinical trial data. We are supplying data sets, information about monitoring and on-hand expertise to allow an exploration of the use of AI. 2. For the treatment effect heterogeneity project, we are supplying our expertise on treatment effect estimation in clinical trials, new work on performance measures for treatment effect heterogeneity, and data from a trial in treatmnet of severe anaemia in African children |
Collaborator Contribution | Scientists of the Alan Turing Institute are using AI including ML on our clinical trial data and on simulated data to find out what methods can be used in the two settings above. A data study group has been run on the monitoring project (entered separately in Research Fish). |
Impact | A data study group was run in November and December 2021, allowing a group of volunteers to work intensively on the monitoring problem. |
Start Year | 2019 |
Description | Collaborative research with MRC BSU |
Organisation | University of Cambridge |
Department | MRC Biostatistics Unit |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | Joint collaborative research in design and analysis of clinical trials |
Collaborator Contribution | Joint collaborative research in design and analysis of clinical trials |
Impact | Several papers including PUBMED id: 19153970; 19452569; 21225900 |
Description | Collaborative research with MRC IEU |
Organisation | University of Bristol |
Department | MRC Integrative Epidemiology Unit |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | Tim Morris has collaborated on the IEU's 'Covidity' project, in particular on work around selection-into-cohort bias |
Collaborator Contribution | Leading the Covidity project |
Impact | Exploring the impact of selection bias in observational studies of COVID-19: a simulation study (https://academic.oup.com/ije/article/52/1/44/6874795) Further manuscripts are currently in preparation. |
Start Year | 2021 |
Description | Collaborative work with Dr Michael Harhay, U Penn |
Organisation | University of Pennsylvania |
Country | United States |
Sector | Academic/University |
PI Contribution | Dr Brennan Kahan and Dr Tim Morris collaborate actively with Dr Micharl Harhay (U Penn) on statistical methods relevant to Dr Harhay's interests |
Collaborator Contribution | Contributed many of the methodological research questions addressed by the collaboration |
Impact | Eliminating ambiguous treatment effects using estimands (https://academic.oup.com/aje/advance-article/doi/10.1093/aje/kwad036/7036583) Increased risk of type I errors in cluster randomised trials with small or medium numbers of clusters: a review, reanalysis, and simulation study (https://link.springer.com/article/10.1186/s13063-016-1571-2) Estimands in cluster-randomized trials: choosing analyses that answer the right question (https://academic.oup.com/ije/article/52/1/107/6644521) Using modified intention-to-treat as a principal stratum estimator for failure to initiate treatment (https://arxiv.org/abs/2206.10453) |
Start Year | 2022 |
Description | IW advisor to Kara-Louise Royle |
Organisation | University of Leeds |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | Advised on developing and submitting a successful NIHR doctoral fellowship proposal and on carrying out the work |
Collaborator Contribution | Developed and submitted a NIHR doctoral fellowship proposal and carried out the work |
Impact | None at present |
Start Year | 2020 |
Description | IW advisor to Suzie Cro |
Organisation | Imperial College London |
Department | Imperial Clinical Trials Unit (ICTU) |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | Advised on development and submission of a successful NIHR Advanced Fellowship application |
Collaborator Contribution | Developed and submitted the successful NIHR Advanced Fellowship application |
Impact | Workshop planned April 2022 to improve how UK clinical trials units approach estimands. |
Start Year | 2019 |
Description | Strengthening the Analytical Thinking for Observational Studies: STRATOS initiative |
Organisation | Albert Ludwig University of Freiburg |
Department | Centre for Medical Biometry and Medical Informatics |
Country | Germany |
Sector | Academic/University |
PI Contribution | Member of the steering committee (James Carpenter) and co-author of the paper in Statistics in Medicine which introduced this initiative Dr Tim Morris is a member of the simulation panel and the visualisation panel. |
Collaborator Contribution | Besides steering group membership, James Carpenter is chair of the Topic Group on missing data, responsible for leading the activities of this group |
Impact | Two papers have been published |
Start Year | 2014 |
Description | Chair, Drug Information Association Scientific Working Group on Estimands and Missing Data |
Form Of Engagement Activity | A formal working group, expert panel or dialogue |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | I chair regular meetings of the Drug Information Association's Scientific Working Group on Estimands and Missing Data, concerned with research, evaluation and dissemination of best practice in handling missing data in the pharmaceutical industry |
Year(s) Of Engagement Activity | 2021,2022,2023 |
URL | https://www.lshtm.ac.uk/research/centres-projects-groups/missing-data#dia-missing-data |
Description | EU-PEARL workshop on non-concurrent controls |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | International workshop on the controversial area of using on patients in a platform protocol not randomised concurrently to the trial. M Parmar and M Sydes contributed presentations and panel discussions. |
Year(s) Of Engagement Activity | 2022 |
URL | https://www.youtube.com/watch?v=nYl-lHtVwxA&ab_channel=EU-PEARL |
Description | Practical Use of Multiple Imputation workshop/shortcourse |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Each workshop is a two-day short course for up to 30 scientists needing to tackle missing data in their research. Feedback has always been outstanding. Many of the attendees have gone on to use multiple imputation based on what they have learned in this course. We haved written two methodological papers with people who have attended the course. We previously published a tutorial paper (>6000 citations at March 2022) based on material from the course and our experiences of teaching the course. |
Year(s) Of Engagement Activity | 2017,2018,2019,2020,2021,2022 |
URL | https://www.ucl.ac.uk/clinical-trials-and-methodology/education/short-courses/missing-data |
Description | Short course for the professional society 'Statisticians in the Pharmaceutical Industry' |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Training professional statisticians in the analysis of longitudinal and hierarchical data |
Year(s) Of Engagement Activity | 2021 |
Description | Simulations course |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | We gave a one-day course on "Using simulation studies to evaluate statistical methods". The audience were researchers either using or planning to use simulation studies in order to develop or evaluate statistical methods. Feedback indicated that the audience felt empowered to start or improve their use of simulation studies. We repeated the course in subsequent years as a 2-day course and also (from 2020) online. |
Year(s) Of Engagement Activity | 2015,2016,2017,2018,2019,2020,2021,2022 |
URL | https://sites.google.com/site/simulationstudies/home |
Description | UCL health methods software showcase |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Professional Practitioners |
Results and Impact | Online workshop for all UCL staff writing software for statistics-related applicatins in health |
Year(s) Of Engagement Activity | 2021 |
Description | UMIT course |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Postgraduate students |
Results and Impact | A series of lectures and practicals on Treatment Switching in Randomised Trials was given in Hall, Austria as part of a 4-day course on Causal Inference Methods. Repeated annually including online in 2021 - 2023 as a 5-day course. |
Year(s) Of Engagement Activity | 2016,2017,2018,2019,2020,2021,2022,2023 |
URL | https://www.umit.at/page.cfm?vpath=departments/public_health/htads-continuing-education-program/caus... |
Description | www.missingdata.lshtm.ac.uk |
Form Of Engagement Activity | A press release, press conference or response to a media enquiry/interview |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Website devoted to analysis of partially observed data Web site |
Year(s) Of Engagement Activity | 2009,2010,2011,2012,2013,2014,2015,2016,2017,2018,2019,2020,2021,2022,2023 |