The application of interrupted time series designs for the evaluation of public health interventions
Lead Research Organisation:
London School of Hygiene & Tropical Medicine
Department Name: Public Health and Policy
Abstract
What is the context of the research?
Public health interventions (such as smoking legislation, health promotion programmes and health service changes) are generally applied to whole populations rather than individuals. Often, this means that traditional methods for evaluating whether an intervention works, such as a clinical trial for a new drug, cannot be applied. This results in a lack of knowledge about which public health interventions are most effective at improving health, resulting in inappropriate resource allocation with good interventions not being implemented and poor interventions being continued. One powerful method to evaluate interventions, which is frequently used in other fields such as economics, policy research and education research, but less commonly in health research, is interrupted time series (ITS) analysis. The basic principle is to compare the outcome of interest over time before and after an intervention whilst taking into account any underlying trends. Further development of this method in the context of public health will enable more robust evaluation of complex public health interventions.
To which disease(s)/ condition(s), if any, is the research relevant?
This research is relevant to the evaluation of all population level health interventions as well as some individual level health interventions which are not amenable to clinical trials.
What is the research trying to achieve?
This project aims to export and develop the ITS technique as a robust method to evaluate the effectiveness of public health interventions, implement the method in the evaluation of real life public health interventions and develop comprehensive guidelines on the appropriate usage of ITS methods in this field.
Why is this important?
This research will help us to accurately evaluate population level health interventions that would otherwise be difficult to evaluate. This is important for ensuring that the highest quality interventions which lead to the greatest health gains are implemented in practice.
Who is carrying out the research?
Dr James Lopez Bernal (Academic Clinical Fellow in Public Health) will be conducting this research for his PhD at the London School of Hygiene and Tropical Medicine. He will be supervised by Professor Steven Cummins (Professor of Population Health) who is an expert in the evaluation of public health interventions, and Dr Antionio Gasparrini (Lecturer in Biostatistics and Epidemiology) whose main research focus lies in time series analysis. The research will also involve collaboration with the Department of Population Medicine at Harvard University.
How is the research being conducted?
1) Existing knowledge on ITS methods from other disciplines will be collated and reviewed in order to understand where methods may be directly applicable to public health interventions and where there may be differences. Published ITS studies of public health interventions will then be reviewed in order to evaluate specific designs and analytical methods used and any methodological issues arising from them.
2) At least two case studies will be conducted in which an ITS design will be used to evaluate the effectiveness of real life public health interventions, including:
- Evaluation of the effect of a workplace health and wellbeing programme on ill health among employees.
- Evaluation of the effect of the move to GP led commissioning of services in the NHS on referrals to specialist hospital care.
3) Issues identified during the earlier stages of the project will be addressed by developing techniques to improve the robustness and accuracy of the ITS method and testing these on the case studies.
4) The knowledge and experience gained during this research will be used to develop guidelines on the appropriate use of ITS designs for the evaluation of public health interventions.
Public health interventions (such as smoking legislation, health promotion programmes and health service changes) are generally applied to whole populations rather than individuals. Often, this means that traditional methods for evaluating whether an intervention works, such as a clinical trial for a new drug, cannot be applied. This results in a lack of knowledge about which public health interventions are most effective at improving health, resulting in inappropriate resource allocation with good interventions not being implemented and poor interventions being continued. One powerful method to evaluate interventions, which is frequently used in other fields such as economics, policy research and education research, but less commonly in health research, is interrupted time series (ITS) analysis. The basic principle is to compare the outcome of interest over time before and after an intervention whilst taking into account any underlying trends. Further development of this method in the context of public health will enable more robust evaluation of complex public health interventions.
To which disease(s)/ condition(s), if any, is the research relevant?
This research is relevant to the evaluation of all population level health interventions as well as some individual level health interventions which are not amenable to clinical trials.
What is the research trying to achieve?
This project aims to export and develop the ITS technique as a robust method to evaluate the effectiveness of public health interventions, implement the method in the evaluation of real life public health interventions and develop comprehensive guidelines on the appropriate usage of ITS methods in this field.
Why is this important?
This research will help us to accurately evaluate population level health interventions that would otherwise be difficult to evaluate. This is important for ensuring that the highest quality interventions which lead to the greatest health gains are implemented in practice.
Who is carrying out the research?
Dr James Lopez Bernal (Academic Clinical Fellow in Public Health) will be conducting this research for his PhD at the London School of Hygiene and Tropical Medicine. He will be supervised by Professor Steven Cummins (Professor of Population Health) who is an expert in the evaluation of public health interventions, and Dr Antionio Gasparrini (Lecturer in Biostatistics and Epidemiology) whose main research focus lies in time series analysis. The research will also involve collaboration with the Department of Population Medicine at Harvard University.
How is the research being conducted?
1) Existing knowledge on ITS methods from other disciplines will be collated and reviewed in order to understand where methods may be directly applicable to public health interventions and where there may be differences. Published ITS studies of public health interventions will then be reviewed in order to evaluate specific designs and analytical methods used and any methodological issues arising from them.
2) At least two case studies will be conducted in which an ITS design will be used to evaluate the effectiveness of real life public health interventions, including:
- Evaluation of the effect of a workplace health and wellbeing programme on ill health among employees.
- Evaluation of the effect of the move to GP led commissioning of services in the NHS on referrals to specialist hospital care.
3) Issues identified during the earlier stages of the project will be addressed by developing techniques to improve the robustness and accuracy of the ITS method and testing these on the case studies.
4) The knowledge and experience gained during this research will be used to develop guidelines on the appropriate use of ITS designs for the evaluation of public health interventions.
Technical Summary
Background:
Randomized controlled trials are often not possible in the evaluation of population level health interventions due to significant limitations including contamination of the control population, lack of realistic generalisability, no feasible control and other ethical, legal or practical issues. Alternative methods for evaluating such interventions exist, however, these are not well established and there is a lack of clear guidance on their appropriate usage. This is limiting the appropriate evaluation of many public health interventions. The interrupted time-series (ITS) design is one of the most powerful quasi-experimental methods for evaluating the effectiveness of an intervention. A clearer understanding is required of when ITS is most appropriate and how it is best applied in the context of public health intervention research.
Purpose:
To develop the ITS design for the evaluation of complex population level health interventions, apply the design to evaluate the effectiveness of at least two interventions, and generate methodological guidance on the use of ITS designs in public health research.
Methods:
1) A systematic review will be conducted of public health interventions studies using an ITS design in order to identify specific facilitators, barriers and methodological issues.
2) At least two population level health interventions will be evaluated using ITS designs.
3) ITS methods will be adapted and expanded in order to address barriers to validity.
4) The knowledge gained will be used to develop guidelines on the appropriate use of interrupted time series designs for the evaluation of public health interventions
Scientific opportunities:
Appraisal of ITS for public health intervention research; guidelines on the use of ITS for the evaluation of public health interventions; methodological developments; application of ITS to evaluate public health interventions as example case-studies.
Randomized controlled trials are often not possible in the evaluation of population level health interventions due to significant limitations including contamination of the control population, lack of realistic generalisability, no feasible control and other ethical, legal or practical issues. Alternative methods for evaluating such interventions exist, however, these are not well established and there is a lack of clear guidance on their appropriate usage. This is limiting the appropriate evaluation of many public health interventions. The interrupted time-series (ITS) design is one of the most powerful quasi-experimental methods for evaluating the effectiveness of an intervention. A clearer understanding is required of when ITS is most appropriate and how it is best applied in the context of public health intervention research.
Purpose:
To develop the ITS design for the evaluation of complex population level health interventions, apply the design to evaluate the effectiveness of at least two interventions, and generate methodological guidance on the use of ITS designs in public health research.
Methods:
1) A systematic review will be conducted of public health interventions studies using an ITS design in order to identify specific facilitators, barriers and methodological issues.
2) At least two population level health interventions will be evaluated using ITS designs.
3) ITS methods will be adapted and expanded in order to address barriers to validity.
4) The knowledge gained will be used to develop guidelines on the appropriate use of interrupted time series designs for the evaluation of public health interventions
Scientific opportunities:
Appraisal of ITS for public health intervention research; guidelines on the use of ITS for the evaluation of public health interventions; methodological developments; application of ITS to evaluate public health interventions as example case-studies.
Planned Impact
The following groups will benefit from the outcomes of this research:
1. Those who evaluate the effectiveness of public health interventions.
This research will appraise, develop and provide guidance on a method to evaluate the effectiveness of interventions which are applied at a population level. This will enable interventions which are not amenable to methods such as randomised controlled experiments to be evaluated more effectively. Beneficiaries in this group will include academics, health think tanks such as the King's Fund and the Nuffield trust and, given that evaluation is a core aspect of developing interventions, public health practitioners and policymakers introducing new interventions.
2. Those who review evidence on the effectiveness of interventions and make recommendations on which interventions to implement.
Greater guidance on methods of evaluation and their appropriateness in different situations will help those who review evidence to appraise whether evaluations have been conducted appropriately and the level of evidence they provide regarding the effectiveness of an intervention. This will benefit systematic reviewers, such as the Cochrane Collaboration, and those who produce guidelines on public health interventions, such as the National Institute of Health and Clinical Excellence (NICE).
3. Those who implement public health interventions.
Public health practitioners are frequently frustrated by a lack of evidence on which interventions are effective to address a given health problem. Improved evaluations of public health interventions will enable public health practitioners, policymakers, health service managers and others who implement population level health interventions to make better informed decisions about which health policies and programmes to employ.
4. Those who stand to benefit from public health interventions.
Better informed decisions will result in the implementation of more effective health interventions which lead to the greatest improvements in health. The beneficiaries will depend on the particular health problem identified and type of intervention required but may include a specific social group, those with a certain disease, or the general population as a whole.
5. Those who pay for interventions.
Improved evaluation of interventions will also help to identify those interventions which are ineffective and thus help to limit inappropriate resource allocation. More efficient use of resources could benefit taxpayers, private organisations or third sector organisations, depending on who is funding the intervention.
Furthermore, there will be beneficiaries from the outcomes of the case studies included in this project:
1. Evaluation of the effect of a workplace health and wellbeing programme on ill health among employees.
The importance of the workplace as a setting for health promotion is being increasingly recognised. The outcomes of this project will help to inform public health practitioners about the effectiveness of workplace health programmes. These findings will also be important to employers in order to maximise staff health and wellbeing and to workers themselves who will benefit most from interventions which have been shown to be effective.
2. Evaluation of the effect of the move to GP led commissioning of services in the NHS on referrals to specialist hospital care.
One of the rationales behind the recent NHS reforms was that GP led commissioning would disincentivise expensive secondary care referrals. This evaluation will help establish whether this has been achieved which could help inform health economists, policymakers and managers nationally and internationally on future health service interventions. Alongside evidence of the most effective pathways of care for different conditions, knowledge of the effect of different commissioning structures will benefit taxpayers by improving efficiency of services.
1. Those who evaluate the effectiveness of public health interventions.
This research will appraise, develop and provide guidance on a method to evaluate the effectiveness of interventions which are applied at a population level. This will enable interventions which are not amenable to methods such as randomised controlled experiments to be evaluated more effectively. Beneficiaries in this group will include academics, health think tanks such as the King's Fund and the Nuffield trust and, given that evaluation is a core aspect of developing interventions, public health practitioners and policymakers introducing new interventions.
2. Those who review evidence on the effectiveness of interventions and make recommendations on which interventions to implement.
Greater guidance on methods of evaluation and their appropriateness in different situations will help those who review evidence to appraise whether evaluations have been conducted appropriately and the level of evidence they provide regarding the effectiveness of an intervention. This will benefit systematic reviewers, such as the Cochrane Collaboration, and those who produce guidelines on public health interventions, such as the National Institute of Health and Clinical Excellence (NICE).
3. Those who implement public health interventions.
Public health practitioners are frequently frustrated by a lack of evidence on which interventions are effective to address a given health problem. Improved evaluations of public health interventions will enable public health practitioners, policymakers, health service managers and others who implement population level health interventions to make better informed decisions about which health policies and programmes to employ.
4. Those who stand to benefit from public health interventions.
Better informed decisions will result in the implementation of more effective health interventions which lead to the greatest improvements in health. The beneficiaries will depend on the particular health problem identified and type of intervention required but may include a specific social group, those with a certain disease, or the general population as a whole.
5. Those who pay for interventions.
Improved evaluation of interventions will also help to identify those interventions which are ineffective and thus help to limit inappropriate resource allocation. More efficient use of resources could benefit taxpayers, private organisations or third sector organisations, depending on who is funding the intervention.
Furthermore, there will be beneficiaries from the outcomes of the case studies included in this project:
1. Evaluation of the effect of a workplace health and wellbeing programme on ill health among employees.
The importance of the workplace as a setting for health promotion is being increasingly recognised. The outcomes of this project will help to inform public health practitioners about the effectiveness of workplace health programmes. These findings will also be important to employers in order to maximise staff health and wellbeing and to workers themselves who will benefit most from interventions which have been shown to be effective.
2. Evaluation of the effect of the move to GP led commissioning of services in the NHS on referrals to specialist hospital care.
One of the rationales behind the recent NHS reforms was that GP led commissioning would disincentivise expensive secondary care referrals. This evaluation will help establish whether this has been achieved which could help inform health economists, policymakers and managers nationally and internationally on future health service interventions. Alongside evidence of the most effective pathways of care for different conditions, knowledge of the effect of different commissioning structures will benefit taxpayers by improving efficiency of services.
People |
ORCID iD |
James Lopez Bernal (Principal Investigator / Fellow) |
Publications
Bernal J
(2020)
The Impact of Social and Physical Distancing Measures on COVID-19 Activity in England
in SSRN Electronic Journal
Bernal JL
(2017)
Interrupted time series regression for the evaluation of public health interventions: a tutorial.
in International journal of epidemiology
Gasparrini A
(2015)
Commentary: On the use of quasi-experimental designs in public health evaluation.
in International journal of epidemiology
Lewis G
(2014)
Mastering Public Health
Lopez Bernal J
(2014)
RE: The effect of the late 2000s financial crisis on suicides in Spain: an interrupted time-series analysis.
in European journal of public health
Lopez Bernal J
(2019)
Difference in difference, controlled interrupted time series and synthetic controls.
in International journal of epidemiology
Lopez Bernal J
(2018)
The use of controls in interrupted time series studies of public health interventions.
in International journal of epidemiology
Lopez Bernal J
(2018)
A methodological framework for model selection in interrupted time series studies.
in Journal of clinical epidemiology
Lopez Bernal JA
(2017)
Association between the 2012 Health and Social Care Act and specialist visits and hospitalisations in England: A controlled interrupted time series analysis.
in PLoS medicine
Title | Model for ITS segmented regression |
Description | Stata and R Model for ITS segmented regression |
Type Of Material | Computer model/algorithm |
Year Produced | 2016 |
Provided To Others? | Yes |
Impact | Most read article of 2017 in IJE |
URL | https://academic.oup.com/ije/article/46/1/348/2622842 |
Description | Collaboration with Harvard Medical School Department of Population Medicine |
Organisation | Harvard University |
Department | Harvard Medical School |
Country | United States |
Sector | Academic/University |
PI Contribution | Provided data and study idea |
Collaborator Contribution | Provided senior supervision and methodological expertise |
Impact | Study still in progress |
Start Year | 2016 |
Description | El Pais newspaper interview |
Form Of Engagement Activity | A press release, press conference or response to a media enquiry/interview |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | Contacted by journalist from El Pais (Spain's most widely published newspaper) on a paper on the effect of the financial crisis on suicides: The effect of the late 2000s financial crisis on suicides in Spain: an interrupted time-series analysis. . |
Year(s) Of Engagement Activity | 2016,2017 |
Description | Full day workshop on interrupted time series at London School of Economics |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Taught on Executive MSc in Health Economics, Outcomes and Management in Cardiovascular Sciences |
Year(s) Of Engagement Activity | 2017 |
URL | http://www.lse.ac.uk/resources/calendar2017-2018/programmeRegulations/taughtMasters/2017_ExecutiveMS... |
Description | Interview with New Scientist |
Form Of Engagement Activity | A press release, press conference or response to a media enquiry/interview |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Media (as a channel to the public) |
Results and Impact | Interview with New Scientist reporter for their magazine |
Year(s) Of Engagement Activity | 2017 |
URL | https://www.newscientist.com/article/2153328-nhs-reforms-have-failed-to-boost-community-based-care-i... |
Description | Invited to teach interrupted time series methods to staff at Marie Stopes International |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Third sector organisations |
Results and Impact | Taught course based on recent tutorial paper to around 20 members of staff from Marie Stopes International |
Year(s) Of Engagement Activity | 2016 |
Description | PLOS Medicine Press Release |
Form Of Engagement Activity | A press release, press conference or response to a media enquiry/interview |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Media (as a channel to the public) |
Results and Impact | Press release by PLOS medicine |
Year(s) Of Engagement Activity | 2017 |
URL | http://journals.plos.org/plosmedicine/article/authors?id=10.1371/journal.pmed.1002427 |
Description | Presentation at International Health Policy Conference 2017 |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Other audiences |
Results and Impact | Presentation at International Health Policy Conference including academics, policymakers and industry professionals |
Year(s) Of Engagement Activity | 2017 |
URL | http://www.lse.ac.uk/LSEHealthAndSocialCare/events/International-Health-Policy-Conference-2017.aspx |
Description | Workshop on interrupted time series at Public Health England London Region |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | Regional |
Primary Audience | Professional Practitioners |
Results and Impact | Workshop for staff at PHE |
Year(s) Of Engagement Activity | 2018 |