Tailoring health policies to improve outcomes using machine learning, causal inference and operations research methods

Lead Research Organisation: University of York
Department Name: Centre for Health Economics


To maximise the impact of health policies on population health and improve the equitable distribution of health, policymakers require answers to questions such as: does the policy work for the intended recipients? Who benefits most? Does the policy reduce health inequalities? Who should be eligible for a programme? To generate evidence to answer these questions, policy evaluations need to go beyond the average population impact and consider how impacts differ across different types of individuals (treatment effect heterogeneity across subgroups). While such subgroup analysis has been done before, the previously used approaches are limited in that they open the door to the researcher cherry-picking the subgroups on the basis of what turns out as statistically significant in the estimates.
By contrast, machine learning techniques - automated algorithms that learn from the data - can reveal patterns in the policy impact that may not be expected beforehand. This is important for policymakers who need to understand who benefits most (and who does less, or not at all) from the implemented policy in question. Once we are able to assess how a given policy affects different sub-groups of the population and by how much, we can take these insights and design the eligibility criteria of a health policies, so that they maximise a decision maker's objective, for example by generating the most total health benefit given a fixed health care budget.

The combination of machine learning with methods that can estimate causal impacts of policy is relatively recent area of research, in particular their application to learn about "treatment effect heterogeneity" and the targeting of policies. Hence, there is no methodological guidance available on how to apply these recent tools in health policy evaluations. The proposed research aims to make a contribution by assessing and extending the available approaches to address the specific challenges that typically arise when evaluating health policies.
These include statistical challenges, such as the need to account for potential biases due to observed and unobserved differences between the treated and control groups; but also challenges to make evaluations relevant to decision making, by considering not just the benefits but also the costs of an intervention (cost effectiveness), and also considering budget constraints or considerations of equity when designing which population subgroups should be targeted with a policy.

This project proposes to address these challenges, by assessing and extending recently proposed machine learning and causal inference methods in the context of health policy evaluations and also by combining tools from different disciplines: causal inference, machine learning and cost-effectiveness modelling, for the first time. By successfully addressing these challenges, this project will deliver methods that will help researchers and policymakers carry-out more comprehensive evaluations of country-wide health policies. This could help support significant improvements to population health and reduce the health gap between the rich and poor within countries. The methodological developments are motivated by two case studies from a low- and middle-income country context, where the gains in terms of improving health and reducing health inequalities are particularly large. The case studies focus on two large scale health policies with ongoing relevance: major public health insurance reform in Indonesia and the country-wide Family Health Programme in Brazil.

To maximise impact on current health policy making, design of the specific research questions in the case studies will benefit from on-going input from Indonesian and Brazilian collaborators as well as policymakers. With extensive communication and impact activities, this project will make its methodological insights available for researchers working on health policy evaluations, in academia and beyond.

Technical Summary

To maximise the impact of health policies, policy evaluations need to go beyond the average effects and consider treatment effect heterogeneity. Recently, machine learning approaches have been proposed to estimate treatment effect heterogeneity in a flexible yet principled way, combined with rigorous causal inference approaches. These methods have been further extended to examine which individuals should be targeted by a policy, based on their observed characteristics (optimal treatment regimes). However, these methods have not yet been translated to the challenging context of the evaluation of large-scale health policies. Here, due to non-experimental evaluation designs, researchers need to adjust for observed and unobserved confounding, and to inform real-world policy making, there is a need to estimate interpretable health policy assignment rules, to account for budgetary and equity constraints, and to consider the long term costs and benefits of a health policy.

To address these challenges, we plan to first compare combined causal inference/machine learning approaches to estimate heterogeneous treatment effects for subgroups and (conditional) individual level treatment effects. The methods will be extended to adjust for confounding in quasi-experimental studies, and we will test their performance by applying them in a case study and in a simulation study. Building on this work, we develop a framework that can answer the overarching question: which subgroups of individuals or communities should be 'treated' to maximise the health policy maker's objective function, subject to constraints? Third, to consider the full health benefits and costs of a policy, we combine with the above framework the tools of decision modelling.

Outputs from this project will equip researchers with advanced tools to generate impact evaluation evidence that can inform the design of efficient health policies to promote population health and health equity objectives.

Planned Impact

The project is motivated by the need to better use existing data to better tailor large scale health policies, with a view to maximise population health given scarce resources. This research is anticipated to benefit a larger audience beyond the academic community.

The potential non-academic beneficiaries include:

1. Direct beneficiaries:

1.a. Researchers and technical analysts involved in impact evaluation, working for policy organisations in Brazil and Indonesia, and in the UK.

The project will benefit this group by resulting in methodological guidance to assess heterogeneity in impact evaluations and economic evaluations of health policies. The methodological guidance will be accompanied by training events.

1.b. Policy makers in Indonesia and Brazil who will be engaged throughout the research, in the following stages

- Refining the policy evaluation questions (e.g. providing inputs on relevant policy levers, equity considerations, and constraints in each country

- Ensuring there is a deep understanding of the contexts in which the case study policies were designed and implemented

- Being informed about emerging results, interpreting final results

In Indonesia, the impact evaluation of the Universal Health Insurance scheme is an ongoing activity by Indonesian and international researchers. With a large portion of the population still not covered by health insurance, learning about heterogeneous policy impacts, and the relative benefit of policy targeting rules can have a real impact on the ongoing re-design of the health insurance programmes. We aim to achieve this impact by engaging Indonesian Policy makers (Ministry of Health [see letter of support], the Health Technology Assessment Committee, and Social Security Administrator for Health) throughout.

In Brazil, potential plans of disinvestment in the Family Health Programme highlights the policy relevance on generating evidence on "for whom" the programme worked best, and can feed into future developments. This will be ensured by the engagement of policy makers such as the Paolo State Secretary of Health, the Ministry of Health, and the Brazilian Primary Care Secretariat, tasked with PSF policy formulation at the federal level. We will be supported in policy maker engagement by the World Bank (Brazil) [see letter of support].

By broadening the scope of evidence generated from policy evaluations and economic evaluations, on the long term, this research will help ensure that scarce societal resources are allocated in the best ways to improve population health.

2. Potential users of the outputs of the project include:

2.a. Researchers and technical analysts involved in impact evaluation in LMICs outside of Brazil and Indonesia (e.g. analysts working for non-governmental organisations (e.g. 3ie, Innovations for Poverty Action), intergovernmental organisations (World Bank [see letter of support], Global Fund, WHO).

2.b Researchers and technical analysts involved in health policy evaluation and health technology assessment in the UK (e.g. analysts working for Public Health England, NICE, Department of Health, Health Foundation [see letter of support]), by providing methodological guidance on the methods developed in the project, which are relevant outside of the LMIC context
Description BMBR Event by NIHR Methodology Incubator 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact Dr Kreif gave a presentation on the lessons learned in the process of obtaining NIRG funding through UKRI, and engaged in discussion with the participants.
Year(s) Of Engagement Activity 2021
Description Indonesian Health Economics Association virtual workshop on new methods 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Policymakers/politicians
Results and Impact We organised a "satellite session" as part of the 2021 Indonesian Health Economics Association conference, with the title: "New methods to estimate heterogenous impacts of the JKN health insurance programme". The aim was to generate a discussion about key analytical choices for our work estimating the heterogenous impacts of the Indonesian health insurance programme, as well as estimating optimal subsidised health insurance assignment rules.

In this 1.5 hour workshop we engaged Indonesian Policy makers, as well as researchers from academia and international organisations, involved in Indonesian health policy making.
We gathered input on several contextual aspects of the evaluations, for example:
- Which variables should we consider as potential source of heterogenous impacts of JKN?
- What are the current targeting rules for enrolling the currently uninsured in JKN? What variables could be used in a potential alternative targeting rule?
- What are the relevant constraints to consider (overall health care budget, fiscal capacity of the districts) when deciding what proportion of the currently uninsured to target for insurance?
- Beyond health insurance, are there other aspects of JKN where heterogenous policy impacts would be interesting to investigate?

Proposed content of the workshop
We also gave a brief snapshot of our research to date, by presenting the paper "Estimating heterogeneous policy impacts using causal machine learning: a case study of health insurance reform in Indonesia", where we use the IFLS data to estimate the impact of the health insurance reforms between 2000 and 2014 on maternal health care utilisation. Professor Budi Hidayat (University of Indonesia) provided discussion to this paper.

We then presented some ongoing work that uses the SUSUNAS data, to estimate heterogeneous impacts of JKN (subsidised health insurance) on adult health care utilisation, and also estimate an illustrative optimal policy allocation rule using this data (proposed time: 20 minutes presentation, 15 minutes discussion). This was discussed by a representative of the National Health Insurance agency BPJS), who provided valuable suggestions for our research design.
Finally, we had an open discussion on the above topics.
Year(s) Of Engagement Activity 2021
Description Short Course Workshop, HESG Conference 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Postgraduate students
Results and Impact Attendees participated in a short course on using machine learning for health policy evaluation, which consisted of lectures and a hands-on coding portion. Participants asked thoughtful questions and discussed the practicalities of using this novel technology in their research. A follow-up survey indicated that attendees felt they had a grasp on the basics of the methods following the workshop, and were interested in implementing them across a variety of research areas.
Year(s) Of Engagement Activity 2022
Description Short Course Workshop, NORCE (Norwegian Research Centre) 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Attendees participated in a short course on using machine learning for health policy evaluation, which consisted of lectures and a hands-on coding portion. Participants asked thoughtful questions and discussed the practicalities of using this novel technology in their research. The small size of the group made in-depth discussions feasible, regarding how to adapt current research techniques to the methods presented.
Year(s) Of Engagement Activity 2022
Description Stanford AI+ Health event 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact Dr Kreif participated in a panel discussion on the use of machine learning methods in health economics and outcomes research.
Year(s) Of Engagement Activity 2021