HMS: Identifying and Measuring Spillover Effects of Changes to Healthcare Organisation and Delivery

Changes to the way in which healthcare services are organised and delivered have wider consequences beyond the patients or the services targeted. These types of wider changes are called spillovers. They may have positive effects, because more patients benefit than was originally intended, and/or they may have negative effects, because more attention is paid to some patients and activities at the expense of others.

A limited amount of research has shown that these spillover effects can be substantial. They may be substantial enough to affect whether a proposed change in service organisation is good value-for-money. There is therefore a need for a systematic method of identifying and measuring spillover effects.

Our aim is to develop structured guidance for incorporating spillovers in the evaluation of changes to health care organisation and delivery. Our proposal contains three work packages. The first work package will review the published literature to identify and summarise how spillovers have been considered and measured in previous studies.

The second work package will use the example of three case studies to develop the methods. The first case study will focus on a new system introduced for General Practitioners (GPs) under which they received financial bonuses for improving a wide range of quality measures. The second case study will focus on a change to the way hospitals were paid to encourage them to discharge patients on the same day rather than keep them in hospital overnight. We have previously shown that both of these initiatives changed activity as intended and improved the outcomes for the patients that were targeted. However, the wider consequences for other activities and other patients have not been considered.

The third case study will focus on the initiative to provide the population with access to GPs seven-days a week. This is expected to increase the use of general practice services and improve convenience for patients. It is also hoped that it will improve the health of the population and lead to a reduction in visits to Accident and Emergency departments for relatively minor complaints that would be better dealt with by GPs. However, the overall impact of these changes will depend on whether availability at the weekend leads to reduced access to GPs during the week, the size of the reduction in demand on A&E, the benefits to the patients with more severe conditions who continue to visit A&E and the costs of compensating GPs for opening out-of-hours. We will estimate these wider consequences using our spillovers framework and provide an overall assessment of extending access to GP services.

We will re-visit each of these policy changes and examine whether they had spillover effects. We will consider how these spillover effects influence the overall costs and benefits of these policy changes.


In the third and final work package, we will generate structured methodological guidance to evaluate diverse changes to health services. We will work with a project advisory group including decision-makers, health care professionals and academic experts to identify the potential for spillovers, develop the methods and ensure the guidance is comprehensive, authoritative and user-friendly.

The work will provide a consistent way of assessing the wider effects on all patients of changing the way that health and social care services are organised for some patients. We will demonstrate these methods on three important policy changes and examine how considerations of spillovers affect the design of service changes and the way these are evaluated. We will work with decision-makers, professionals and evaluators throughout the project to create guidance that will ensure that the potential for spillover effects is widely recognised and that they are included in the evaluation of changes to the way care services are organised and delivered.

The Highlight Notice on spillover effects notes that the methods for modelling care delivery changes are under-developed. Because inputs, processes and outcomes are inter-related, changes to one part of a system will have multiple consequences for the whole system.

We will develop a methodology to enable structured identification and measurement of spillover effects. We will use econometric evaluation techniques. The research will examine three case studies in depth to understand the longer-term consequences of the equilibrating and offsetting mechanisms affecting health service delivery.

The study comprises three work packages:

WP1 is a systematic review of the spillover effects identified in previous literature and the methodological approaches used to measure them.

WP2 will re-examine three case studies of policy changes to (a) improve quality in primary care through financial incentives, (b) increase the proportion of surgical hospital interventions delivered as daycases through higher reimbursements and (c) extend the availability of general practice services out of hours. Administrative datasets will be re-analysed at patient and organisation level to estimate between and within, and wider system, spillovers of these changes.

WP3 will develop and publish structured methodological guidance on how to identify and measure spillover effects from changes to healthcare organisation and delivery. This guidance will be developed with a project advisory group of evaluation experts, research funders, policy advisors and decision-makers. The full guidance will be published electronically and a summary paper will be submitted to an academic journal.

1. Who will benefit?

Researchers undertaking evaluations of interventions designed to change healthcare organisation and delivery

Commissioners of evaluations of changes to healthcare organisation and delivery, including government agencies, and private and third-sector research funders

Academics, healthcare professionals, service managers and policy makers designing changes to healthcare organisation and delivery. This includes service providers in the private and third sector.

Decision-makers considering the evidence on whether proposed changes to healthcare delivery and organisation should be adopted. This includes decision-makers at various levels in the care system, including decision-makers and their advisers in government and arms-length bodies (including regulators), evidence and improvement agencies (including NICE and NHS Improvement), national and local NHS organisations, and individual healthcare professionals.

The project will also benefit the wider public by promoting better decisions on changes to healthcare organisation and delivery.

2. How will they benefit?
Researchers
There is increasing recognition that service changes require the same level of evaluation scrutiny as new treatments and therapies. However, the methodology is under-developed. Our framework will provide a systematic methodology for identifying and measuring spillover effects. Our structured review will raise awareness in this group and the guidance will affect future work.

Commissioners of evaluations
Changes to healthcare organisation and delivery have wider consequences than explicitly intended. Full evaluation of these changes requires consideration of these wider spillover effects. Those commissioning evaluations require a structured framework to examine whether proposals will capture the full effects on allocative efficiency. Our structured review will raise awareness in this group and the guidance will affect future work.

Designers of changes to healthcare organisation and delivery
Our work will raise awareness of possible spillover effects. Some positive spillovers will want to be promoted. Some negative spillovers will want to be avoided. Designers of changes to healthcare organisation and delivery will consider spillover effects and consider how they can be handled. Our structured review will raise awareness in this group and promotion of the guidance will affect the design of future initiatives.

Decision-makers considering the evidence on programme effectiveness
Some spillovers may affect decisions. When awareness is raised amongst those making decisions and spillover effects become accepted and expected in programme evaluations, this should inform decision-makers at all levels. The retrospective case studies will inform closely-related decisions and the prospective case study on seven day GP services should help to shape this policy.

Wider society and public
Spillovers are a neglected consideration in policy-making. Our framework will promote evaluation of these effects. This should increase the effectiveness of public services and policy, by ensuring that the full costs and benefits of service changes are taken into account. By promoting allocative efficiency this will also ensure that maximum gains in quality and quantity of life are secured from NHS and social care budgets.

All staff working on the project will benefit by developing skills and understanding in systematic reviewing, programme evaluation and econometric analysis. All these skills are transferable to other sectors, such as engineering, defence, manufacturing, and finance. The researchers will also acquire other professional skills such as networking and presentation skills.