Methodology State-of-the-art workshop: Choosing the target difference (“effect size”) for a randomised controlled trial

Randomised controlled trials (RCT) are considered the optimal study design to assess comparative clinical efficacy and effectiveness. A sample size calculation ensures the study has a reasonable chance to achieve its pre-specified objectives: A compromise is required between the possibility of being misled by chance (there is no true difference between treatments) and the risk of not identifying a difference even when one of the treatments is truly superior.

The “target difference” is the size of the difference the RCT is designed to reliably investigate. It is also commonly referred to as the trial “effect size”.

The critical question is whether a particular difference detected statistically, is of any practical interest. From a scientific and ethical standpoint, selecting an appropriate targeted difference is of crucial importance - specifying too small a target difference could be a wasteful (and perhaps unethical) use of resources, and too large could lead to an important difference being overlooked. Furthermore, the resultant undersized study will not produce reliable evidence.

Despite there being many different approaches available to choose the target difference, few are used regularly in practice.

The project will:
1. Review the existing guidance;
2. Identify key methodological developments or changes in practice;
3. Determine the scope of the guidance required;
4. Produce structured guidance for choosing and reporting the target difference (effect size)

Randomised controlled trials (RCT) are considered the optimal study design to assess the comparative clinical efficacy and effectiveness along with the cost implications of health interventions. Central to the validity of a RCT, is an a-priori sample size calculation which ensures the study has a reasonable chance to achieve its pre-specified objectives. A compromise is required in the design between the possibility of being misled by chance when there is no true difference between treatments and the risk of not identifying a treatment difference when one of the treatments is truly superior.

Under the conventional approach (sometimes called Neyman-Pearson). These two errors are controlled by setting the significance level (Type I error) and statistical power (1 minus Type II error) at appropriate levels. Once these two inputs are set the sample size can be determined given the magnitude of difference to be detected. This difference, the “target difference”, is the magnitude of difference upon which the RCT is designed to reliably investigate. It is also commonly referred to as the trial “effect size”. Appropriate sample size formulae vary depending upon the proposed trial design and statistical analysis though the overall approach is consistent.

Other statistical approaches to defining the sample size include Fisherian/precision based approaches, Bayesian and decision-theoretic Bayesian approaches, and hybrid Bayesian-Neyman Pearson approaches.

Irrespective of the statistical approach used, the critical question is whether a particular difference detected statistically, is of any practical interest. From a scientific and ethical standpoint, selecting an appropriate targeted difference is of crucial importance - specifying too small a target difference could be a wasteful (and perhaps unethical) use of resources, and too large could lead to an important difference being overlooked. Furthermore, the resultant undersized study may not usefully contribute to the evidence base and could detrimentally impact upon decision-making.

Despite there being many different approaches available, few are used regularly in practice. The target difference often appear to be determined upon convenience or other informal basis. Practice is more sophisticated, but no specific guidance exists beyond standard superiority two-arm parallel group trial designs on choosing the target difference and how to report the sample size calculation.

The project will:
1. Review the existing guidance provided by funders;
2. Identify key methodological developments or changes in practice;
3. Determine the scope of the guidance required;
4. Achieve consensus within structured guidance for choosing the target difference (effect size)