Policy Evaluation Beyond Averages: Distributional Impact Analysis.

Assessing the impact of social and economic policies is a central concern for policymakers. However the way those policies are evaluated is often not effective, which means that a policy can be put in place that does not benefit to the right people in society, or might even harm those it is meant to help. This work will enable policymakers to make much more informed and accurate decisions in relation to developing and implementing policy, meaning that society as a whole will benefit as policy can be more accurately be designed to meet the right needs of the right people.

The diagnostic tools employed by the vast majority of impact studies focus on average gains or losses from a policy. This is a potentially important shortcoming since, in practice, most individuals will not receive the average benefit or loss. The average effect of a policy is not informative about who loses and who benefits from a policy, and both the direction and magnitude of the effect may vary substantially across individuals. In this research, I develop quantitative techniques which allow for more nuanced policy evaluation. I intend to produce a statistical tool that will enable social scientists and policymakers to go beyond calculating average effects, while ensuring that their analysis captures the
diverse effects of the policy under study.

By allowing for differentiated impact evaluation across individuals, my novel perspective on causal analysis will provide innovative methods that will help improve allocation of resources and the design of policies targeted at specific populations, and has potential to achieve far-reaching benefits. I will focus on three specific tasks:

1. Producing tools capturing the diversity of causal effects. I will introduce statistical methods for causal analysis and policy evaluation that allow for researchers and policymakers to uncover the complexity and diverse impacts of policies and treatments. By generalising a classical method (the so-called instrumental variables method), I will establish new theoretical results necessary for the sound application of my approach, and will illustrate these results with concrete examples from several fields, such as economics and health sciences.

2. Promoting nuanced quantitative policy evaluation. My approach will change the practice of quantitative policy evaluation by allowing policymakers and social scientists to determine who benefits and who loses from a policy or treatment. They will be able to formulate nuanced and accurate policy diagnostics reducing the risks inherent to policy implementation. I will visit leading policy-oriented institutions, and collaborate with practitioners outside the scientific community in order to demonstrate how my methods will allow them to address a range of new policy questions and achieve more nuanced policy diagnostics.

3. Making my methods available. I will provide practitioners with the technology and documentation necessary for the implementation of my methods through a freely available and open source software. I will also advertise my methods directly to practitioners through the organisation of workshops and presentations in leading academic and nonacademic institutions.

Who will benefit from this research?

In addition to Groups 1 (theoretical econometricians and statisticians) and 2 (applied academic researchers) described in the "Academic Beneficiaries" Section, I expect quantitative social scientists and policymakers to benefit from the proposed research, and the general public to benefit from applications of my methods.

GROUP 3: POLICYMAKERS AND RESEARCHERS IN POLICY-ORIENTED INSTITUTIONS.

This group will include policymakers performing quantitative policy evaluation in institutions at the local, national and international levels. Those include institutions focusing on a wide range of issues, such as labour market reform, poverty reduction, the provision of public goods or the impact of welfare programmes. For instance Instrumental Variables methods are currently commonly used at institutions such as the Bank of England, the UK Ministry of Education or the International Monetary Fund.

GROUP 4: GENERAL PUBLIC.

The proposed research will affect the general public by improving policy and decision making. Thus the impact of this research on this group will be indirect but potentially very large. For instance, the applications of my methods to health sciences described in the Case for Support will use Mendelian randomisation, a technique used for assessing the causal impact of various factors, such as Body Mass Index or Selenium intake, on cancer survival or aggressiveness in studies sponsored by institutions such as Cancer Research UK.

How will they benefit from this research?

GROUP 3: In addition to the benefits described for Group 2 (applied academic researchers), this group of users will benefit from the new insights provided by my methods for the design of new policies, when advising governments and decision-makers. By gaining knowledge of distributional causal effects, they will be able to make more accurate and targeted recommendations for allocation of resources and the design of policies.

GROUP 4: By allowing for a more precise understanding of causal relationships my methods will help better predict interventions that are likely to be successful from those that have harmful effects. For instance for the relationship between cancer determinants and disease agressiveness, my methods will improve current Mendelian randomisation studies. This in turn could help prevent cancers by better understanding the determinants of a person's risk of developing the disease.

What will be done to ensure that they have the opportunity to benefit from this activity?

GROUP 3: I will focus strongly on collaborating directly with members of this group. I will present my methods at leading policy-oriented institutions and initiate collaborations on datasets and policy questions of interest to these institutions. For instance, I have been invited to present my methods and visit the World Bank (at the Impact Evaluation Unit) and the European Investment Bank. I will develop specific lectures and examples for policymakers in these institutions for them to be able to apply my methods. Throughout the project I will initiate collaborations with those institutions, and strengthen my links with other institutions contacted in the early stages of this project. I will provide a freely available software and the necessary documentation for Group 3. This software will be advertised during my presentations to this group.

GROUP 4: I will develop applications relevant to the general public. For example, I will collaborate with the MRC Epidemiology Integrative Unit to apply my methods to Mendelian randomisation. The dissemination of my research findings to institutions such as the Department of Health and Cancer Research UK will receive the support of PolicyBristol, a team based at the University of Bristol dedicated to the targeted dissemination of policy research findings, and the Jean Golding Institute. This will help maximise the benefits of the proposed research for the general public.