HMT: Accuracy vs Precision - Developing optimal estimators for trials with multiple hypothesis tests
Lead Research Organisation:
Lancaster University
Department Name: Mathematics and Statistics
Abstract
Accurate estimation of a treatment's effect is vital for expressing its true worth. This is important in early phase trials as this information is routinely used to decide whether further study is warranted and to power future studies. Additionally, health bodies require accurate estimation of the treatment effect to make informed choices about cost-effectiveness relative to alternative options as well as benefit risk assessment. In many current studies multiple hypothesis tests are conducted which has the unwanted side effect that the standard method for estimation is inaccurate.
This project aims to thoroughly evaluate the extent of bias of the standard estimator in studies in which multiple hypothesis tests are conducted. We will also investigate appropriate graphical displays to visualize treatment effects and develop novel approaches for point and interval estimation that balance accuracy and precision of the estimator. The developments will be made in close connection with clinical researchers and will be overseen by an advisory panel.
This project aims to thoroughly evaluate the extent of bias of the standard estimator in studies in which multiple hypothesis tests are conducted. We will also investigate appropriate graphical displays to visualize treatment effects and develop novel approaches for point and interval estimation that balance accuracy and precision of the estimator. The developments will be made in close connection with clinical researchers and will be overseen by an advisory panel.
Technical Summary
Accurate estimation of a treatment's effect is vital for expressing its true worth. This is important in early phase trials as this information is routinely used to decide whether further study is warranted and to power future studies. Additionally, health bodies require accurate estimation of the treatment effect to make informed choices about cost-effectiveness relative to alternative options as well as benefit risk assessment. The treatment effect is usually reported as the maximum likelihood estimator (MLE) which is a precise and readily available estimator. For designs that include interim analyses, it is, however, typically biased since it ignores the sequential nature of the trial. Additional bias is introduced by selecting treatments, subgroups or endpoints based on observed effect sizes.
To address this problem, several classes of estimation procedures have been proposed, most methods are, however, limited to two-stage designs and normally distributed observations. Additionally few estimators offer corresponding confidence intervals, limiting their usefulness in practice. This project aims to thoroughly evaluate the extent of bias of the MLE in studies in which multiple hypothesis tests are conducted and develop novel approaches for point and interval estimation. Specific objectives, which will be discussed in more detail below, are to:
- evaluate the extent of bias for a range of statistical designs;
- develop graphical approaches for displaying treatment effects for correlated hypotheses;
- develop point and interval estimates that yield an optimal trade-off of precision and accuracy;
- develop bias-reducing methods for estimating effects for time-to-event endpoints;
- develop open-source software and associated training tailored to clinical trialists and applied statisticians working on clinical trials.
To ensure practical relevance, the overall research programme will be overseen by an advisory board.
To address this problem, several classes of estimation procedures have been proposed, most methods are, however, limited to two-stage designs and normally distributed observations. Additionally few estimators offer corresponding confidence intervals, limiting their usefulness in practice. This project aims to thoroughly evaluate the extent of bias of the MLE in studies in which multiple hypothesis tests are conducted and develop novel approaches for point and interval estimation. Specific objectives, which will be discussed in more detail below, are to:
- evaluate the extent of bias for a range of statistical designs;
- develop graphical approaches for displaying treatment effects for correlated hypotheses;
- develop point and interval estimates that yield an optimal trade-off of precision and accuracy;
- develop bias-reducing methods for estimating effects for time-to-event endpoints;
- develop open-source software and associated training tailored to clinical trialists and applied statisticians working on clinical trials.
To ensure practical relevance, the overall research programme will be overseen by an advisory board.
Planned Impact
In addition to academic beneficiaries, the work proposed in this grant will have great impact on others. Evaluating the bias of the maximum likelihood estimator and balancing bias and variation in the estimation of treatment effects in studies with multiple hypothesis tests will be of great interest to those using results from a clinical trial. Those planning future trials will be able to use unbiased estimates when synthesizing evidence through techniques like meta-analysis and be able to power subsequent studies appropriately. Regulatory organisations such as the European Medicines Agency or the National Institute for Clinical Excellence will be able to judge the cost-effectiveness of a drug relative to alternative options as well as benefit risk assessment using reliable data.
The development of graphical tools for depicting treatment effects for subgroups and other correlated settings will benefit clinical trialists in communicating the findings to clinical experts by offering an easy to understand tool that does not require in-depth understanding of the underlying methods. Similarly these tools will allow medical doctors to communicate treatment decisions to patients that are member of subgroups that react differently to treatment more easily and effectively.
The development of graphical tools for depicting treatment effects for subgroups and other correlated settings will benefit clinical trialists in communicating the findings to clinical experts by offering an easy to understand tool that does not require in-depth understanding of the underlying methods. Similarly these tools will allow medical doctors to communicate treatment decisions to patients that are member of subgroups that react differently to treatment more easily and effectively.
Publications
Ballarini N
(2021)
subtee : An R Package for Subgroup Treatment Effect Estimation in Clinical Trials
in Journal of Statistical Software
Ballarini NM
(2018)
Subgroup identification in clinical trials via the predicted individual treatment effect.
in PloS one
Ballarini NM
(2021)
Optimizing subgroup selection in two-stage adaptive enrichment and umbrella designs.
in Statistics in medicine
Ballarini NM
(2020)
A critical review of graphics for subgroup analyses in clinical trials.
in Pharmaceutical statistics
Brueckner M
(2019)
Instrumental variable estimation in semi-parametric additive hazards models.
in Biometrics
Brückner M
(2017)
Estimation in multi-arm two-stage trials with treatment selection and time-to-event endpoint.
in Statistics in medicine
Chiu YD
(2018)
Design and estimation in clinical trials with subpopulation selection.
in Statistics in medicine
Dane A
(2019)
Subgroup analysis and interpretation for phase 3 confirmatory trials: White paper of the EFSPI/PSI working group on subgroup analysis.
in Pharmaceutical statistics
Description | A Practical Adaptive and Novel Designs Toolkit |
Amount | £100,000 (GBP) |
Organisation | National Institute for Health Research |
Sector | Public |
Country | United Kingdom |
Start | 06/2019 |
End | 06/2020 |
Description | NIHR Senior Research Fellowship |
Amount | £653,526 (GBP) |
Funding ID | SRF-2015-08-001 |
Organisation | National Institute for Health Research |
Sector | Public |
Country | United Kingdom |
Start | 01/2016 |
End | 12/2020 |
Description | Adaptive designs MUW |
Organisation | Medical University of Vienna |
Department | Centre for Medical Statistics, Informatics and Intelligent Systems |
Country | Austria |
Sector | Academic/University |
PI Contribution | Joint research |
Collaborator Contribution | Joint research |
Impact | Publications. |
Start Year | 2011 |
Description | NWHTMR |
Organisation | University of Liverpool |
Department | Department of Biostatistics |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | Joint research and support in implementation of methods |
Collaborator Contribution | Joint research and implementation of methods |
Impact | Joint papers, funding applications |
Start Year | 2009 |
Description | Adaptive and Bayesian Methods Course |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Annual Short course on adaptive and Bayesian methods |
Year(s) Of Engagement Activity | 2015,2016,2017,2018 |
Description | BSSK-WBS Seminar |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | Presentation on Identification of subgroups and estimation of subgroup effects at BSSK-WBS Seminar. |
Year(s) Of Engagement Activity | 2016 |
Description | Conference (iBright) |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Postgraduate students |
Results and Impact | Conference presentation |
Year(s) Of Engagement Activity | 2019 |
Description | Design of Experiments in Drug Development open for business |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Discussion event around improving drug development through adaptive designs. |
Year(s) Of Engagement Activity | 2015 |
Description | EFSPI Meeting on Oncology and Survival Analysis |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Industry/Business |
Results and Impact | Invited presentation at EFSPI meeting: Treatment selection to improve survival using the predicted individual treatment effect |
Year(s) Of Engagement Activity | 2017 |
Description | ISCB Birmingham |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Presentation of an interactive tool to assess subgroup analyses (and the resulting bias). |
Year(s) Of Engagement Activity | 2016 |
Description | MCP conference Riverside 2017 |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Invited presentation at MCP conference on "Confidence Regions for Treatment Effects in Biomarker Stratified Designs" |
Year(s) Of Engagement Activity | 2017 |
Description | Pre-conference training course on Subgroup Analysis at PSI |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Industry/Business |
Results and Impact | Around 60 participants of the annual PSI (Statisticians in the Pharmaceutical Industry) conference attended the training course on Subgroup analysis. |
Year(s) Of Engagement Activity | 2016 |
URL | http://www.psiweb.org/events/2016-conference |
Description | Short course on Exploratory Subgroup Analysis in Clinical Trials |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Short course on Exploratory Subgroup Analysis in Clinical Trials |
Year(s) Of Engagement Activity | 2017 |
Description | Workshop on DSMBs |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Industry/Business |
Results and Impact | A course on data safety monitoring boards was given to a company. A core topic discussed was bias in estimates due to selection and multiple looks at the data. |
Year(s) Of Engagement Activity | 2017 |