Programme Evaluation for Policy Analysis
Lead Research Organisation:
Institute for Fiscal Studies
Department Name: IFS Research Team
Abstract
Abstracts are not currently available in GtR for all funded research. This is normally because the abstract was not required at the time of proposal submission, but may be because it included sensitive information such as personal details.
Publications
Fitzsimons, E.
(2012)
Household responses to information on child nutrition: experimental evidence from Malawi
Joyce, R.
(2012)
Inference in difference-in-differences
Richard Blundell (Author)
(2012)
The long-term effects of in-work benefits in a lifecycle model for policy evaluation
Malde, B.
(2012)
Collecting Data Electronically in Developing Countries
Fitzsimons, E.
(2012)
Household responses to information on child nutrition: experimental evidence from Malawi
Sianesi, B.
(2012)
Dealing with randomisation bias in a social experiment : the case of ERA
Sianesi, B.
(2012)
Dealing with randomisation bias in a social experiment : the case of ERA
Fitzsimons, E.
(2012)
Household responses to information on child nutrition: experimental evidence from Malawi
Rosen, A.
(2012)
An Overview of Programme Evaluation Methods
Blundell, R.
(2012)
Education, labour supply and welfare
Monica Costa Dias (Author)
(2012)
Progamme evaluation methods
Brewer, M.
(2012)
What is: regression discontinuity design?
Rasul, I.
(2012)
Programme Evaluation and Social Networks
Costa Dias, M.
(2012)
Microeconometric Policy Evaluation
Sianesi, B.
(2012)
What is propensity score matching?
Fitzsimons, E.
(2012)
Household responses to information on child nutrition: experimental evidence from Malawi
Mike Brewer (Author)
(2011)
Programme evaluation for policy analysis
Lorraine Dearden (Author)
(2011)
The role of evaluation in social research : current perspectives and new developments
| Description | PEPA was a node of the National Centre for Research Methods. PEPA research focused on various evaluation and inference methods and their empirical implications when addressing substantive policy questions. Here we describe a small part of our research. We looked at randomised experiments to assess the possibility that individuals might react to the randomisation itself, rendering the causal inference from the experiment irrelevant for policy purposes. We set out a theoretical framework to systematically consider "randomisation bias", and provided the first empirical evidence on this form of bias in the context of the UK ERA study. We found evidence that the 26.6% non-participation rate in the ERA experiment has introduced significant randomisation bias in the experimental estimates in terms of employment but not survey earnings outcomes. We studied inference in difference-in-difference designs. The literature has been concerned getting correct test-size with DID. Based on simulations, we demonstrated that correct test size can be achieved in most cases with standard methods. When inference is conducted appropriately, the problem is low power. We showed how power can be improved while controlling size by combining Generalised Least Squares estimation with sandwich-type variance-covariance estimates. Our approach indicates that data cannot rule out substantial negative employment effects from the introduction of the National Minimum Wage in the UK. In this context, the difference-in-difference research design simply has very low power. We investigated the relationship between group size and informal risk sharing using simulations to show that this relationship is theoretically ambiguous. We exploited a social norm among an ethnic group in Malawi (Chewa), which indicates that the well-being of a woman's children is partly insured by her brothers. We find that households where the woman has many brothers are not well-insured against crop loss events. We fail to uncover a similar relationship for the sisters of the woman. Calibrating our theoretical framework to produce a relationship between household risk sharing and group size similar to that uncovered in the data, indicates that the threat of coalitional deviations can explain our empirical findings. Allowing for flexible forms of unobserved heterogeneity is essential in policy evaluation. We extended instrumental variable methods to problems in which multiple values of unobservable variables can be associated with particular combinations of observed endogenous and exogenous variables. In our Generalised instrumental variable models, the mapping from unobserved heterogeneity to endogenous variables may not be invertible. We developed a method to estimate treatment effects on hazard rates when the instrument is the policy reform, and showed how to implement it. Using this method, we demonstrated that the UK New Deal for Young People had anticipation effects on the exit rates to work. We developed a lifecycle model to understand the long-term impacts of welfare policies for women, who experience career intermittency. We found that taking time off paid work and short working hours is costly for educated women, for whom working experience is especially valuable, and that the short-term evaluation of working subsidies measures their impact accurately for women with basic education. |
| Exploitation Route | Our outputs are likely to be of interest in academic, practitioner and policy circles. The new evaluation methods and frameworks developed under PEPA, for instance in instrumental variables and dynamic models, are currently being used in other contexts, to look at the gender wage gap, investments in human capital over the lifecycle or retirement decisions and policy. PEPA developed software and user manuals to make some of evaluation methods widely assessable; e.g. software on bounds estimators can spread the use of this robust approach; our manual on matching will insure a rigorous use of this popular method. PEPA supported collaboration with the HMT in discussing their dynamic model for tax analysis and share our experience in working with comparable models. Our research on empirical evaluation methods lead to a collaboration with the DWP on feasible approaches to evaluate the Universal Credit (DWP research report 6, July 2014). PEPA had a strong TCB programme aimed at improving the design and understanding of evaluations to a wide audience. We ran 19 masterclasses/conferences for advanced audiences, 21 intermediate courses for practitioners in the academy, public and private sectors, and an especially course for civil servants and policy-makers. |
| Sectors | Communities and Social Services/Policy,Education,Government, Democracy and Justice,Other |
| URL | http://www.ifs.org.uk/centres/PEPA/ |
| Description | Researchers from PEPA have, throughout the award and beyond, been asked by numerouse Government officials, NGOs and charities about how to appropriately commission and/or undertake and/or advice on setting up/undertaking quantitative evaluations. This has been through directly contacting researchers and/or attending one of our courses. |
| First Year Of Impact | 2012 |
| Sector | Education,Environment,Healthcare,Government, Democracy and Justice |
| Impact Types | Economic,Policy & public services |
| Description | Meeting with officials from the cross-government evaluation group |
| Organisation | HM Treasury |
| Country | United Kingdom |
| Sector | Public |
| PI Contribution | Part of HM Treasury held in London |
| Start Year | 2011 |
| Description | Female labour supply, human capital and the lifetime effects of in-work benefits |
| Form Of Engagement Activity | A magazine, newsletter or online publication |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Policymakers/politicians |
| Results and Impact | Article for Winter edition of NCRM MethodsNews no information |
| Year(s) Of Engagement Activity | 2012 |
| URL | http://eprints.ncrm.ac.uk/2923/1/MethodsNewsWinter2012.pdf |
| Description | Financial incentives in health : new evidence from India's Janani Suraksha Yojana |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Policymakers/politicians |
| Results and Impact | PEPA output: 3ie-LIDC Seminar Series no information |
| Year(s) Of Engagement Activity | 2012 |
| URL | http://www.esrc.ac.uk/my-esrc/grants/RES-576-25-0042/outputs/read/ebf7f7ce-34aa-498f-9ab8-6abe82f495... |
| Description | Measuring the impact of government interventions |
| Form Of Engagement Activity | A magazine, newsletter or online publication |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Policymakers/politicians |
| Results and Impact | Methods News, Spring 2012 Economics has a long tradition of studying causal questions. Over the past few decades causal methods have been widely employed in measuring the impact of government policies. no information |
| Year(s) Of Engagement Activity | 2012 |
| URL | http://eprints.ncrm.ac.uk/2277/1/MethodsNewsSpring2012.pdf |
| Description | Programme evaluation for policy analysis |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Policymakers/politicians |
| Results and Impact | Presented at What Works Action Planning Workshop no information |
| Year(s) Of Engagement Activity | 2011 |
| URL | http://www.esrc.ac.uk/my-esrc/grants/RES-576-25-0042/outputs/read/edda6fa2-d105-45ba-bccc-63321503b5... |
| Description | Programme evaluation for policy analysis |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Participants in your research or patient groups |
| Results and Impact | Talk about PEPA at the NCRM start-up meeting Section not completed |
| Year(s) Of Engagement Activity | 2011 |
| Description | Programme evaluation for policy analysis |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Participants in your research or patient groups |
| Results and Impact | Talk about PEPA at a ESRC meeting about a What Works Initiative at NESTA held in London Section not completed |
| Year(s) Of Engagement Activity | 2011 |
| Description | The Gender Wage Gap and the Career Patterns of Men and Women |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Other audiences |
| Results and Impact | Seminar at the University of Greenwich, January 2016. |
| Year(s) Of Engagement Activity | 2016 |
| URL | http://www.ifs.org.uk/uploads/publications/conferences/BDEM-Greenwich-Jan2016.pdf |