Improving school governance and learning outcomes at scale: A randomized evaluation of the Madhya Pradesh School Quality Assessment program

We propose to conduct a randomized controlled trial (RCT) of one of the most ambitious recent attempts by a developing country government to improve education governance at scale. The evaluation will have a particular focus on the state of school governance, processes and learning outcomes in rural tribal areas, which have been one of the most challenging contexts to improve education in India.

The Madhya Pradesh (MP) School Quality Assessment (MPSQA) intervention is a comprehensive program that aims to improve school governance and outcomes through a combination of (a) regular monitoring of schools, (b) creation of school report cards, and a customized school improvement plan, (c) quarterly follow-ups on progress against this plan, and (d) leveraging of ICT tools to collect and report real-time records of all assessment results, school improvement plans, and follow-ups to a dedicated online system. The PI's have worked with the project implementation team and have successfully randomized the rollout of the program across ~20,000 schools in the 2015/16 school year. The government is currently hoping to universalize the program to cover all 120,000 schools in MP by 2019.

The timing of the evaluation is ideal: finding positive results may help accelerate the scaling up across MP and elsewhere in India, while not finding significant positive results (along with data on intermediate factors to understand reasons for non-impact) may prompt a re-evaluation of the program and its implementation before spending resources on a scale up that may not have the desired impacts on processes or outcomes.

The evaluation will assess the effects of the program on key process metrics (e.g. teacher absence, effort and pedagogy) as well as student academic outcomes for two years. The proposed evaluation will focus on the randomized state-wide rollout of the program in 2015/16 and seek to answer three core questions: (a) How do school-level processes, inputs, and learning outcomes differ within the public education system between remote tribal areas and other parts of the state?; (b) To what extent does the MPSQA intervention improve school-level processes and learning outcomes in the program schools and does this differ between schools in remote tribal areas and other parts of the state; and (c) What are the pathways through which program effects are realized and inhibited (examples include improvements in teacher effort, in better school management, in better-focused pedagogical practices, and student attendance and effort)?

In answering these questions, we will seek to use extensive administrative data including state-wide low-stakes assessment results, alongside independently-administered tests and detailed process metrics collected through regular unannounced school visits to a randomly-selected subset of schools at both elementary and secondary levels. These independently-administered tests will guard against the possibility of manipulation of official test results in program schools and also allow for the linking of test results, through careful test design, to other assessments in India and internationally.

The core proposal and the study design are based on the use of quantitative impact evaluation methods using survey data. However, recognizing that large system-wide reforms often have unanticipated facets and multiple layers of interaction between different agents and different levels of governance, we also propose a qualitative component which seeks a complementary nuanced understanding of (a) the context in which the intervention operates, in terms of the interactions, perceptions and motivation of the relevant agents in the education system (students, teachers, headmasters and education civil servants) (b) how the program effects are mediated through these various layers of the education system and learning about what the 'enabling factors' or bottlenecks of such reforms might be.

The proposed evaluation has been carefully conceived to maximize impact. The core contribution of the proposal for policy impact is being able to evaluate, at scale and with experimental variation, a system-level investment in state capacity for better education governance through building both the human resource capacity as well as the system capacity (exemplified by the technology component) needed to improve governance at scale. This is substantially different from existing work because although there have been several RCT's of education interventions that have found positive outcomes, it has been challenging to scale up these findings. Two key facilitators of scalability of this particular intervention include: (a) program rollout is being integrated into existing structures of education governance and not being implemented by short-term external actors on a "project" basis, and (b) the development of a technology platform to collect, aggregate, and disseminate real-time performance data on the school system.

In order to ensure that policy-relevant lessons do in fact get translated into action, the proposal relies on close partnerships focusing on the different levels at which we would like to see impact: with the Government of Madhya Pradesh for state-level impact, with ARK India and the Government of India for impact across different Indian states and with ARK's global operations for policy impact in other countries. These relationships, which are already at an advanced stage, are described in greater detail in the Pathways to Impact document and the Letters of Support from ARK UK and ARK India.

A core channel of impact for research findings is to refine or re-frame the public discourse on specific policy debates around education. Both PI's take this responsibility seriously and attempt to communicate their research findings broadly online. One key channel to do this would be through blog posts targeting specific audiences such as the Ideas for India website (targeting readers interested on evidence-based policy making in India), the World Bank Development Impact website (targeting development researchers and practitioners) and the Guardian Development blog (targeting a more general readership interested in international issues); these are all channels that the PI's have communicated their research through in the past and will be targeted again for dissemination of results as they are obtained.

Finally, we think there is genuine potential for this research to have an impact on education policy-making outside India through demonstration: by showing that it is possible to rigorously and technically evaluate a large systems-level intervention, that such an evaluation generates valuable insights, and that such an intervention (hopefully) generates potential gains, we hope this will be a template which other policy organizations and researchers can use in order to further evidence-driven interventions. The Lead PI's role as the JPAL co-Chair for Education Research is particularly valuable in this regard - both for the credibility it indicates among researchers but also because JPAL itself, along with its sister organizations such as Innovations for Poverty Action, has extensive communications outreach in different developing countries and also regular interactions with policy makers in various nodal departments including Ministries of Education.