The application of interrupted time series designs for the evaluation of public health interventions

What is the context of the research?
Public health interventions (such as smoking legislation, health promotion programmes and health service changes) are generally applied to whole populations rather than individuals. Often, this means that traditional methods for evaluating whether an intervention works, such as a clinical trial for a new drug, cannot be applied. This results in a lack of knowledge about which public health interventions are most effective at improving health, resulting in inappropriate resource allocation with good interventions not being implemented and poor interventions being continued. One powerful method to evaluate interventions, which is frequently used in other fields such as economics, policy research and education research, but less commonly in health research, is interrupted time series (ITS) analysis. The basic principle is to compare the outcome of interest over time before and after an intervention whilst taking into account any underlying trends. Further development of this method in the context of public health will enable more robust evaluation of complex public health interventions.

To which disease(s)/ condition(s), if any, is the research relevant?
This research is relevant to the evaluation of all population level health interventions as well as some individual level health interventions which are not amenable to clinical trials.

What is the research trying to achieve?
This project aims to export and develop the ITS technique as a robust method to evaluate the effectiveness of public health interventions, implement the method in the evaluation of real life public health interventions and develop comprehensive guidelines on the appropriate usage of ITS methods in this field.

Why is this important?
This research will help us to accurately evaluate population level health interventions that would otherwise be difficult to evaluate. This is important for ensuring that the highest quality interventions which lead to the greatest health gains are implemented in practice.

Who is carrying out the research?
Dr James Lopez Bernal (Academic Clinical Fellow in Public Health) will be conducting this research for his PhD at the London School of Hygiene and Tropical Medicine. He will be supervised by Professor Steven Cummins (Professor of Population Health) who is an expert in the evaluation of public health interventions, and Dr Antionio Gasparrini (Lecturer in Biostatistics and Epidemiology) whose main research focus lies in time series analysis. The research will also involve collaboration with the Department of Population Medicine at Harvard University.

How is the research being conducted?
1) Existing knowledge on ITS methods from other disciplines will be collated and reviewed in order to understand where methods may be directly applicable to public health interventions and where there may be differences. Published ITS studies of public health interventions will then be reviewed in order to evaluate specific designs and analytical methods used and any methodological issues arising from them.
2) At least two case studies will be conducted in which an ITS design will be used to evaluate the effectiveness of real life public health interventions, including:
- Evaluation of the effect of a workplace health and wellbeing programme on ill health among employees.
- Evaluation of the effect of the move to GP led commissioning of services in the NHS on referrals to specialist hospital care.
3) Issues identified during the earlier stages of the project will be addressed by developing techniques to improve the robustness and accuracy of the ITS method and testing these on the case studies.
4) The knowledge and experience gained during this research will be used to develop guidelines on the appropriate use of ITS designs for the evaluation of public health interventions.

Background:
Randomized controlled trials are often not possible in the evaluation of population level health interventions due to significant limitations including contamination of the control population, lack of realistic generalisability, no feasible control and other ethical, legal or practical issues. Alternative methods for evaluating such interventions exist, however, these are not well established and there is a lack of clear guidance on their appropriate usage. This is limiting the appropriate evaluation of many public health interventions. The interrupted time-series (ITS) design is one of the most powerful quasi-experimental methods for evaluating the effectiveness of an intervention. A clearer understanding is required of when ITS is most appropriate and how it is best applied in the context of public health intervention research.

Purpose:
To develop the ITS design for the evaluation of complex population level health interventions, apply the design to evaluate the effectiveness of at least two interventions, and generate methodological guidance on the use of ITS designs in public health research.

Methods:
1) A systematic review will be conducted of public health interventions studies using an ITS design in order to identify specific facilitators, barriers and methodological issues.
2) At least two population level health interventions will be evaluated using ITS designs.
3) ITS methods will be adapted and expanded in order to address barriers to validity.
4) The knowledge gained will be used to develop guidelines on the appropriate use of interrupted time series designs for the evaluation of public health interventions

Scientific opportunities:
Appraisal of ITS for public health intervention research; guidelines on the use of ITS for the evaluation of public health interventions; methodological developments; application of ITS to evaluate public health interventions as example case-studies.

The following groups will benefit from the outcomes of this research:

1. Those who evaluate the effectiveness of public health interventions.
This research will appraise, develop and provide guidance on a method to evaluate the effectiveness of interventions which are applied at a population level. This will enable interventions which are not amenable to methods such as randomised controlled experiments to be evaluated more effectively. Beneficiaries in this group will include academics, health think tanks such as the King's Fund and the Nuffield trust and, given that evaluation is a core aspect of developing interventions, public health practitioners and policymakers introducing new interventions.

2. Those who review evidence on the effectiveness of interventions and make recommendations on which interventions to implement.
Greater guidance on methods of evaluation and their appropriateness in different situations will help those who review evidence to appraise whether evaluations have been conducted appropriately and the level of evidence they provide regarding the effectiveness of an intervention. This will benefit systematic reviewers, such as the Cochrane Collaboration, and those who produce guidelines on public health interventions, such as the National Institute of Health and Clinical Excellence (NICE).

3. Those who implement public health interventions.
Public health practitioners are frequently frustrated by a lack of evidence on which interventions are effective to address a given health problem. Improved evaluations of public health interventions will enable public health practitioners, policymakers, health service managers and others who implement population level health interventions to make better informed decisions about which health policies and programmes to employ.

4. Those who stand to benefit from public health interventions.
Better informed decisions will result in the implementation of more effective health interventions which lead to the greatest improvements in health. The beneficiaries will depend on the particular health problem identified and type of intervention required but may include a specific social group, those with a certain disease, or the general population as a whole.

5. Those who pay for interventions.
Improved evaluation of interventions will also help to identify those interventions which are ineffective and thus help to limit inappropriate resource allocation. More efficient use of resources could benefit taxpayers, private organisations or third sector organisations, depending on who is funding the intervention.


Furthermore, there will be beneficiaries from the outcomes of the case studies included in this project:

1. Evaluation of the effect of a workplace health and wellbeing programme on ill health among employees.
The importance of the workplace as a setting for health promotion is being increasingly recognised. The outcomes of this project will help to inform public health practitioners about the effectiveness of workplace health programmes. These findings will also be important to employers in order to maximise staff health and wellbeing and to workers themselves who will benefit most from interventions which have been shown to be effective.

2. Evaluation of the effect of the move to GP led commissioning of services in the NHS on referrals to specialist hospital care.
One of the rationales behind the recent NHS reforms was that GP led commissioning would disincentivise expensive secondary care referrals. This evaluation will help establish whether this has been achieved which could help inform health economists, policymakers and managers nationally and internationally on future health service interventions. Alongside evidence of the most effective pathways of care for different conditions, knowledge of the effect of different commissioning structures will benefit taxpayers by improving efficiency of services.