Programme Evaluation for Policy Analysis

Lead Research Organisation: Institute for Fiscal Studies
Department Name: IFS Research Team

Abstract

Abstracts are not currently available in GtR for all funded research. This is normally because the abstract was not required at the time of proposal submission, but may be because it included sensitive information such as personal details.

Publications

10 25 50

publication icon
Mike Brewer (Author) (2011) Programme evaluation for policy analysis

publication icon
Costa Dias, M. (2012) Microeconometric Policy Evaluation

publication icon
Costa Dias, M. (2012) Causal Inference in Economics

 
Description PEPA was a node of the National Centre for Research Methods. PEPA research focused on various evaluation and inference methods and their empirical implications when addressing substantive policy questions. Here we describe a small part of our research.

We looked at randomised experiments to assess the possibility that individuals might react to the randomisation itself, rendering the causal inference from the experiment irrelevant for policy purposes. We set out a theoretical framework to systematically consider "randomisation bias", and provided the first empirical evidence on this form of bias in the context of the UK ERA study. We found evidence that the 26.6% non-participation rate in the ERA experiment has introduced significant randomisation bias in the experimental estimates in terms of employment but not survey earnings outcomes.

We studied inference in difference-in-difference designs. The literature has been concerned getting correct test-size with DID. Based on simulations, we demonstrated that correct test size can be achieved in most cases with standard methods. When inference is conducted appropriately, the problem is low power. We showed how power can be improved while controlling size by combining Generalised Least Squares estimation with sandwich-type variance-covariance estimates. Our approach indicates that data cannot rule out substantial negative employment effects from the introduction of the National Minimum Wage in the UK. In this context, the difference-in-difference research design simply has very low power.

We investigated the relationship between group size and informal risk sharing using simulations to show that this relationship is theoretically ambiguous. We exploited a social norm among an ethnic group in Malawi (Chewa), which indicates that the well-being of a woman's children is partly insured by her brothers. We find that households where the woman has many brothers are not well-insured against crop loss events. We fail to uncover a similar relationship for the sisters of the woman. Calibrating our theoretical framework to produce a relationship between household risk sharing and group size similar to that uncovered in the data, indicates that the threat of coalitional deviations can explain our empirical findings.

Allowing for flexible forms of unobserved heterogeneity is essential in policy evaluation. We extended instrumental variable methods to problems in which multiple values of unobservable variables can be associated with particular combinations of observed endogenous and exogenous variables. In our Generalised instrumental variable models, the mapping from unobserved heterogeneity to endogenous variables may not be invertible. We developed a method to estimate treatment effects on hazard rates when the instrument is the policy reform, and showed how to implement it. Using this method, we demonstrated that the UK New Deal for Young People had anticipation effects on the exit rates to work.

We developed a lifecycle model to understand the long-term impacts of welfare policies for women, who experience career intermittency. We found that taking time off paid work and short working hours is costly for educated women, for whom working experience is especially valuable, and that the short-term evaluation of working subsidies measures their impact accurately for women with basic education.
Exploitation Route Our outputs are likely to be of interest in academic, practitioner and policy circles.

The new evaluation methods and frameworks developed under PEPA, for instance in instrumental variables and dynamic models, are currently being used in other contexts, to look at the gender wage gap, investments in human capital over the lifecycle or retirement decisions and policy.

PEPA developed software and user manuals to make some of evaluation methods widely assessable; e.g. software on bounds estimators can spread the use of this robust approach; our manual on matching will insure a rigorous use of this popular method.

PEPA supported collaboration with the HMT in discussing their dynamic model for tax analysis and share our experience in working with comparable models. Our research on empirical evaluation methods lead to a collaboration with the DWP on feasible approaches to evaluate the Universal Credit (DWP research report 6, July 2014).

PEPA had a strong TCB programme aimed at improving the design and understanding of evaluations to a wide audience. We ran 19 masterclasses/conferences for advanced audiences, 21 intermediate courses for practitioners in the academy, public and private sectors, and an especially course for civil servants and policy-makers.
Sectors Communities and Social Services/Policy,Education,Government, Democracy and Justice,Other

URL http://www.ifs.org.uk/centres/PEPA/
 
Description Researchers from PEPA have, throughout the award and beyond, been asked by numerouse Government officials, NGOs and charities about how to appropriately commission and/or undertake and/or advice on setting up/undertaking quantitative evaluations. This has been through directly contacting researchers and/or attending one of our courses.
First Year Of Impact 2012
Sector Education,Environment,Healthcare,Government, Democracy and Justice
Impact Types Economic,Policy & public services

 
Description Meeting with officials from the cross-government evaluation group 
Organisation HM Treasury
Country United Kingdom 
Sector Public 
PI Contribution Part of HM Treasury held in London
Start Year 2011
 
Description Female labour supply, human capital and the lifetime effects of in-work benefits 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Policymakers/politicians
Results and Impact Article for Winter edition of NCRM MethodsNews

no information
Year(s) Of Engagement Activity 2012
URL http://eprints.ncrm.ac.uk/2923/1/MethodsNewsWinter2012.pdf
 
Description Financial incentives in health : new evidence from India's Janani Suraksha Yojana 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Policymakers/politicians
Results and Impact PEPA output: 3ie-LIDC Seminar Series

no information
Year(s) Of Engagement Activity 2012
URL http://www.esrc.ac.uk/my-esrc/grants/RES-576-25-0042/outputs/read/ebf7f7ce-34aa-498f-9ab8-6abe82f495...
 
Description Measuring the impact of government interventions 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Policymakers/politicians
Results and Impact Methods News, Spring 2012

Economics has a long tradition of studying causal questions. Over the past few decades causal methods have been widely employed in measuring the impact of government policies.

no information
Year(s) Of Engagement Activity 2012
URL http://eprints.ncrm.ac.uk/2277/1/MethodsNewsSpring2012.pdf
 
Description Programme evaluation for policy analysis 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Participants in your research or patient groups
Results and Impact Talk about PEPA at the NCRM start-up meeting

Section not completed
Year(s) Of Engagement Activity 2011
 
Description Programme evaluation for policy analysis 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Participants in your research or patient groups
Results and Impact Talk about PEPA at a ESRC meeting about a What Works Initiative at NESTA held in London

Section not completed
Year(s) Of Engagement Activity 2011
 
Description Programme evaluation for policy analysis 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Policymakers/politicians
Results and Impact Presented at What Works Action Planning Workshop

no information
Year(s) Of Engagement Activity 2011
URL http://www.esrc.ac.uk/my-esrc/grants/RES-576-25-0042/outputs/read/edda6fa2-d105-45ba-bccc-63321503b5...
 
Description The Gender Wage Gap and the Career Patterns of Men and Women 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Other audiences
Results and Impact Seminar at the University of Greenwich, January 2016.
Year(s) Of Engagement Activity 2016
URL http://www.ifs.org.uk/uploads/publications/conferences/BDEM-Greenwich-Jan2016.pdf