Improving Learning: Developing Measures of Accountability and Evaluating their Association with Students' Gains in Achievement in Nepal

Lead Research Organisation: University of Michigan
Department Name: Institute for Social Research

Abstract

We propose to develop and validate measures of accountability to be shared with the Nepal Ministry of Education (MOE) and to use those measures in an analysis of the determinants of accountability and its association with students' gains in achievement. The proposed study will build on the resources of the Chitwan Valley Family Study (CVFS), a 20-year ongoing panel study of 116 schools with 3,000 households with 3,500 school aged children in 151 communities located throughout the Western Chitwan Valley of Nepal. With funding from DFID-ESRC, we are proposing to achieve two aims:
Aim One: To Develop and Pretest a Suite of Nepali Accountability Assessment Tools (NAATs) for Use by the MOE and to Pilot these Tools within the Chitwan Valley of Nepal. Importantly, the tools will be designed so that Nepal's MOE can both assess and potentially improve its current accountability processes at multiple levels of the increasingly decentralized Nepalese education system [4].
To achieve this aim we will: (1) develop a variety of accountability assessment tools for use in Nepal's education system; (2) modify a set of instructional processes and instructional quality measures developed for use in OECD countries for use in the Nepali educational system; and (3) gather data on students' academic achievement using standardized test items developed by Nepal's MOE.
Aim Two: To Investigate How Accountability Processes; Environments for Student Learning in Schools, Families, and Communities; and Student Learning are Related. This involves investigating three main research questions: Are accountability processes systematically related to socioeconomic disparities among communities, schools within communities, and families within schools? In school and community settings where accountability processes are more intensive, is the quality of instructional service delivery higher? And, controlling for socioeconomic disparities related to student achievement is student learning higher in schools and communities where accountability processes are more intensive?
To meet this aim, we will: (1) administer a newly designed PET-QSDS survey to 380 key stakeholders; (2) administer the NASA test at the beginning and end of the school year and a student survey to 1,740 8th graders; and (3) administer a teacher survey to 1,392 teachers and a parent survey to 1,740 parents. The results of this research will be relevant to education policy makers in Nepal and will also contribute directly to comparative education research on school effectiveness.
This study will generate rigorous scientific outcomes: (1) development of a low income context adaptive accountability assessment tool; (2) cross-cultural assessment of the reliability and predictive validity of accountability measures; (3) identification of contextual factors with strong correlation with accountability; (4) potential for identification of new dimensions of accountability in low income settings; and (5) scientific advancement in our understanding of the relationship between accountability, instructional quality and students' gains in achievement. These outcomes will be made widely available to scientists and policy makers. First, we will conduct dissemination workshops at local and national levels to share findings of the study and provide training on the use of the newly designed accountability assessment tool and analysis of the data generated through the various surveys mentioned above. Second, the data will be made available through ICPSR and the UK Data Service. Third, the findings will be disseminated through presentations at national and international conferences and published in scientific articles, and research and policy briefs. Finally, the participation of Nepali faculty, scientists, government representatives and school authorities throughout the project will advance the scientific and analytical capacity of their respective host institutions (DOE,TU, PABSON, PDs).

Planned Impact

The methodological tools and protocols, comprehensive measurement, and empirical findings generated by the proposed study have high potential to impact both education policy makers and academia and also contribute directly to comparative education research on school effectiveness at regional and global levels.
Impact on policy: The proposed study is likely to impact education in Nepal in multiple ways. The NAAT and user protocol to measure accountability will be a great asset to the government of Nepal and an important resource to achieve school sector refom plan aims and enhance education quality. Substantively, the empirical answers to the questions we posed: (1) the extent to which accountability is (i) a significant predictor of instructional quality and (ii) distributed unequally across CVFS strata (geographical and urbanacity), different community contexts and across public and private schools; (2) the extent to which any relationships between accountability and students' gains in achievement is explained by instructional quality; and (3) the extent to which the measures of accountability developed in Nepal is a significant predictor of student performance on NASA will provide crucial information for policy makers struggling to make the best use of scarce resources.
We plan to engage stakeholders in several stages of this program to ensure effective policy influence. The key stakeholders include: local teachers, school management committees, proprietors of institutional schools, parents, district education offices, district development committees, Private and Boarding Schools' Organization Nepal (PABSON-Chitwan), I/NGOs, colleges and universities (district level), Central Department of Education at Tribhuvan University (TU), Kathmandu University (KU), Department of Education, Ministry of Education, National Planning Commission, and development partners engaged in educational programs. We will ensure stakeholder engagement through their participation in consultative meetings, research staff trainings, and design and dissemination workshops.
Impact on academia: The empirical findings will be of great use to a wide range of academic audiences for teaching and research purposes. At the local level, these audiences include affiliates of TU, KU and other local colleges and universities in Nepal. As indicated in the letter of support, the faculty and students at the Central Department of Education at TU will directly benefit from research findings both substantively and methodologically. At the regional level, the findings will have strong relevance to academic audiences studying low-income countries in South Asia. Our UM/ISER-N joint training program on survey methodology in Nepal has strong collaborative ties with various universities and research organizations in South Asia. Faculty, research scholars, and students from Bangladesh, India, Pakistan, and Afghanistan regularly participate in the training we offer. We will use this venue to share our findings. At the global level, the publication of findings will improve and stimulate further research in this area by scholars worldwide.
We plan to make the CVFS panel data and the new data we propose to collect available through the UK Data Service. We will also promote the data at national and international conferences. This will greatly increase the use of these data and the resulting impact on academia. We expect that the larger scientific community will benefit from this research well beyond the current project period by stimulating new research agendas in other areas of science, such as parental investment in child education, teacher incentives and evaluation.
The findings of this study will also have broader implications for the design of education policy worldwide. Our global target audiences are UNESCO, Institute of International Education (IIE), I/NGOs, and donors engaged in educational development worldwide.

Publications

10 25 50
 
Description Analysis reveals high reliability and validity of the student achievement and social accountability measures that we developed for this project. The descriptive analysis from our study, as well as from Nepal's National Assessment Student Achievement (NASA), illustrates low student achievement. Results show an important gap in student cumulative achievement between community and institutional schools. Our preliminary analysis suggests that one third (33%) of the variation in student cumulative achievement by the end of 8th grade is explained by school-level variation. Indeed, further analysis of the annual gain in student achievement revealed great variation between students. As reflected in the mean difference between baseline and endline achievement tests, half of the students (50%) had no gain, almost one third (32%) had some gain, 14% had good gain, and only 4% had very good gain. This suggests that a large fraction of students are not learning in school. This pattern is consistent across all three subjects: Mathematics, Nepali, and Science.

School-level aggregation of achievement scores shows similar patterns-the majority of schools gain little or not at all. The mean difference between baseline and endline Mathematics achievement tests across 114 schools shows very little gain. There is no gain between baseline and endline in Science. However, achievement scores for Nepali show some gain.

Preliminary analyses revealed several important insights on social accountability and student achievement. First, a large portion, approximately one third of student achievement is explained by school-level variation. Second, student's perception of instructional quality is strongly associated with student achievement. Finally, certain social accountability features, including parents' knowledge, judgement about instructional quality, school management, and school principal instructional leadership, each have strong associations with student achievement, net of several individual household and contextual factors. These analyses will be further refined as the analyses progress.

School infrastructure, facilities, and student achievement analyses are motivated by enhancing our understanding of gender differences in student achievement. Preliminary results demonstrate that school infrastructure and facilities have an important influence on student achievement. The existence of separate bathrooms for girls and boys with soap and sanitary pads in the girls' bathroom had a strong positive effect on girls' achievement, suggesting appropriate facilities are important to improving girls' educational achievement.
Exploitation Route The outcomes of this funding-measurement tools, data, and research findings-might be used by parents, schools, stakeholders, researchers, policymakers, and development partners in four important ways.

First, several measurement tools including the unannounced classroom observation, survey questionnaires, and student assessments that were developed and tested during this project, were found to be highly reliable and valid. These tools are now available to the scientific community. Thus, research scholars from Nepal and other low-income settings interested in education research can use and adopt these tools free of cost.

Second, the data generated through this funding can be used to answer several policy questions that were not part of the funded project. For example, the project team is now collaborating with a UNICEF-funded study titled "Global Education Monitoring Report 2021/22: Non-State Actors in Education in Nepal: A Country Study". Using the data collected through the project, we plan to investigate the role of non-state actors in student learning outcomes across institutional schools, school management committees, parent teacher associations, and other non-governmental organizations.

Third, schools in the study area can use the findings of this study to improve student learning outcomes. For example, evidence of higher learning outcomes for girls in institutional schools that have a separate toilet for girls with soap and sanitary products is a relatively simple, cost-efficient strategy to improve learning outcomes for female students. This strategy alone could significantly and immediately impact female student learning outcomes.

Finally, given that students, parents, and families can freely choose schools, these findings can provide much needed information to parents in order for them to make informed choices regarding the types of schools they select for their children.
Sectors Communities and Social Services/Policy,Education

URL http://idea.isernepal.org/
 
Title Assessment 
Description Student Achievement Assessment (SAA). We worked with the Educational Review Office (ERO), the Ministry of Education of Nepal to design the SAA questionnaire for Nepali, Mathematics, and Science to measure student gains in achievement during the school year. We draw on the expertise of ERO to design scientifically valid and reliable measures that are comparable to the National Assessment of Student Achievement (NASA) of Nepal measures. Three subject matter specialists, the chief of ERO, 15 subject teachers (5 teachers per subject), and ISER-N study manager were involved in the design of the SAA. The design was completed in five steps 1) item writing by the respective subject committee; 2) review of the items by a review committee; 3) subjective committee review to prepare final draft for pretest; 4) administered the final draft to 1,241 8th grade students in 6 schools outside of the study area; 5) expert evaluation of the item performance and final item selection and refinement of the items based on analysis of the pretest data. In order to minimize student cheating during the assessment, we reshuffled the order of question items and created 4 modules each for Math and Science and two modules for Nepali. 
Type Of Material Physiological assessment or outcome measure 
Year Produced 2018 
Provided To Others? No  
Impact It is too early to describe notable impact. 
 
Title Observation Forms 
Description The research teams at the University of Michigan and ISER-N, and local stakeholders (primarily school inspectors), jointly designed two separate instruments: the school observation form and the classroom observation instrument. Unannounced School Observation (USO). USO was designed to collect information on school facilities, services, and teaching quality independent of surveys. The main goal of this observation is to verify the information collected through school, teacher, and students' surveys. The school observation measures include school facilities (such as number of buildings, library, laboratory, computer room, playground, toilet, drinking water) and services (such as transportation, hostel etc.). Unannounced Classroom Observation (UCO). Classroom observation measures specifically focus on teacher-student interactions and teaching quality. The purpose of this observation was to gather information on teaching quality including classroom environment, teaching-learning methodology, instructional practice of teachers, students' attitude towards learning, teacher-students interaction, management of physical objects, etc. The measures of UCO were integrated into a windows-based interactive software program specifically designed for this study. The classroom observation instrument is a highly flexible, user-friendly software program (for tablets) that can easily be used by school inspectors in the future. We plan to share this software with the government after rigorous evaluation of its effectiveness. UCO instrument was pretested through 28 school observations and 155 classroom observations. 
Type Of Material Physiological assessment or outcome measure 
Year Produced 2018 
Provided To Others? Yes  
Impact It is too early to describe notable impact. 
URL http://idea.isernepal.org/data/
 
Title Survey Questionnaires 
Description Please see details of the Student Survey, Parent Survey, Teacher Survey, Principal and School Survey, School Management Committee (SMC) Survey, and Parent Teacher Association (PTA) Survey in the Engagement Activities (Instrument Design) section. 
Type Of Material Physiological assessment or outcome measure 
Year Produced 2019 
Provided To Others? Yes  
Impact It is too early to describe notable impact. 
URL http://idea.isernepal.org/data/
 
Title Parent Survey 
Description The parent survey includes two parts: i) the household census and relationship and ii) the parent individual survey. The household census and relationship survey data includes measures of household composition, structure, household size, socio-economic background (ethnicity, social status) and age, gender, marital status, living arrangement, education and occupation of each of household members. The relationship grid records the relationship of each member of the household to every other member of the household. In addition, the survey data also includes household wealth, assets and income including remittance received. The survey also includes parent/caregiver's individual-level measures such as parents' perceptions of teaching quality, parental action related to gathering information about alternative schools, and barriers/facilitators to exercising school choice, and awareness and participation in civil society organizations seeking to influence governance of education. 2020-2021 note: Out of 4886 eligible students' parents, we collected data from 4535 of those students' parents (93% response rate). We increased the sample size--proposed sample size was 1740--to assure sufficient power to detect associations between accountability and student achievement. 
Type Of Material Database/Collection of data 
Year Produced 2019 
Provided To Others? No  
Impact It is too early to describe notable impacts resulting from the development of this data. 
 
Title Parent Teacher Association (PTA) Survey 
Description The PTA data consists of responses to a survey interview from a representative of the PTA. The data consists of measures of short-term accountability (local community level) and include other measures of accountability (delegation, finance, performance, information, and enforcement). These data are currently being cleaned and not yet ready for analysis. 2020-2021 note: These data are now clean and ready for analysis by project researchers. Out of 114 schools, 108 participated in the PTA Survey (95% response rate). 
Type Of Material Database/Collection of data 
Year Produced 2020 
Provided To Others? No  
Impact It is too early to describe notable impacts resulting from the development of this data. 
 
Title Principal and School Survey 
Description The principal and school survey data consists of three parts: i) the principal individual interview with LHC, ii) the school history calendar (SHC), and iii) the school finance survey. The principal's LHC includes annual records of the principal's experience. The individual interview portion includes measures of principal's performance, information, and enforcement. The SCH portion includes updates to the existing SHC that we last updated in 2015. The SCH measures include number of students, number of boys and girls, male and female teachers, non-teaching staff, medium of instruction, and grades offered. The school finance section includes details on income, resource flow and disbursement. 2020-2021 note: Out of 114 eligible schools, the Principal and School Survey was completed with 113 schools (99% response rate). 
Type Of Material Database/Collection of data 
Year Produced 2020 
Provided To Others? No  
Impact It is too early to describe notable impacts resulting from the development of this data. 
 
Title School Location Dataset 
Description Using the existing GPS coordinates of all schools, health services, and urban center within western Chitwan Valley (study area) from the Chitwan Valley Family Study, we have created several school locational measures. These measures included distance from the school to urban center and number of school or health services within certain radius (eg.1 kilo meter). 
Type Of Material Database/Collection of data 
Year Produced 2019 
Provided To Others? No  
Impact It is too early to describe notable impacts resulting from the development of this data. 
 
Title School Management Committee (SMC) Survey 
Description The school management committee (SMC) data includes responses to an interview from a representative of the SMC (community school) and school board (institutional school). This survey includes measures of community-level accountability features. These features include delegation, finance, performance, information, and enforcement and engagement. These data are currently being cleaned and not yet ready for analysis. 2020-2021 note: These data are now clean and ready for analysis by project researchers. Out of 114 schools, 109 participated in the SMC Survey (96% response rate). 
Type Of Material Database/Collection of data 
Year Produced 2020 
Provided To Others? No  
Impact It is too early to describe notable impacts resulting from the development of this data. 
 
Title Student Baseline Assessment 
Description Using the newly developed Student Achievement Assessment tools, we administered a Baseline Achievement Assessment for Nepali, Mathematics, and Science. The goal of this baseline achievement assessment was to create a bench mark before students were exposed to grade 8 curriculum so that we could generate a gain in achievement score once we administered the endline achievement assessment. Out of 4886 eligible students from 114 schools 4734 students participated (96.9% response rate). The raw score of these assessments are now converted in to a scale score with a mean of 500 and standard deviation of 50. In addition to the scale score this data include percent attempted, percent correct, and is now available to the project team for analysis. 
Type Of Material Database/Collection of data 
Year Produced 2018 
Provided To Others? No  
Impact It is too early to describe notable impacts resulting from the development of this data. 
 
Title Student Basic Level Examination (BLE) Test Score (district-level exam) 
Description To cross verify our endline assessment scores with district-level test scores we also collected basic level examination (BLE) test scores from the education section of the Bharatpur Metropolitan City (BMC). BLE is a district level standardized test administered to 8th graders at the end of school year. Out of 4746 eligible students, we acquired test scores for 4746 students (100% response rate) from 114 schools that participated in BLE. The BLE raw scores are converted in 100 full mark. 
Type Of Material Database/Collection of data 
Year Produced 2019 
Provided To Others? No  
Impact It is too early to describe notable impacts resulting from the development of this data. 
 
Title Student Endline Assessment 
Description To assess student gain in achievement, we repeated the baseline student assessment at the end of the school year in the same 114 schools. Out of 4886 total students across all selected schools, a total of 4599 students participated (94.1% response rate). Similar to the baseline score, the raw score of these assessments are now converted into a scale score with a mean of 500 and standard deviation of 50. In addition to the scale score this data set also include percent attempted, percent correct, and is now available to project team for analysis. 
Type Of Material Database/Collection of data 
Year Produced 2019 
Provided To Others? No  
Impact It is too early to describe notable impacts resulting from the development of this data. 
 
Title Student Survey 
Description The student survey data includes measures of students' backgrounds, knowledge, aspirations, satisfaction, and perception of classroom environment and teaching quality. Out of 4886 eligible students, we completed the interview with 4542 students resulting in a response rate of 93%. These data are now available to the project research team for analysis. 
Type Of Material Database/Collection of data 
Year Produced 2019 
Provided To Others? No  
Impact It is too early to describe notable impacts resulting from the development of this data. 
 
Title Teacher Survey 
Description The teacher survey data consists of responses from a survey interview of 8th grade teachers. This dataset also has two parts: i) a life history calendar (LHC) and ii) a survey. The LHC portion of the data includes the measures of teachers' experiences annually. The survey portion of the data includes measures of teachers' background, knowledge, training, instructional practice, classroom management, and parental outreach and reporting. 2020-2021 note: Out of 948 eligible teachers, 905 teachers participated (95% response rate). We increased the sample size slightly--proposed sample size was 928--to assure sufficient power to detect associations between accountability and student achievement. 
Type Of Material Database/Collection of data 
Year Produced 2019 
Provided To Others? No  
Impact It is too early to describe notable impacts resulting from the development of this data. 
 
Title Unannounced Classroom Observation 
Description The unannounced classroom observation was conducted in three waves (each observation lasting 10-minutes, consisting of 12 observations in each wave in each school) in grade 6, 7 and 8. Out of 4104 observations 4085 (99.5% response rate) were conducted using the windows-based interactive software program specifically designed for this study. The unannounced classroom observation data includes measures of teaching quality including classroom environment, teaching-learning methodology, instructional practice of teachers, teachers and students behavior in the classroom, teacher-students interaction, and management of physical objects. 
Type Of Material Database/Collection of data 
Year Produced 2019 
Provided To Others? No  
Impact It is too early to describe notable impacts resulting from the development of this data. 
 
Title Unannounced School Observation 
Description The unannounced school observation data includes measures of schools' environment; physical infrastructure such as buildings and its types; facility of drinking water, toilet and its hygiene; library, computer lab, science lab and their accessibility to students; separate room for head teacher, teachers and administration; etc. 2020-2021 note: We collected data on 114 schools (response rate 100%). 
Type Of Material Database/Collection of data 
Year Produced 2019 
Provided To Others? No  
Impact It is too early to describe notable impacts resulting from the development of this data. 
 
Description The Institute for Social and Environmental Research - Nepal (ISER-N) 
Organisation Institute for Social and Environmental Research - Nepal (ISER-N)
Country Nepal 
Sector Charity/Non Profit 
PI Contribution The University of Michigan (U-M) team has significantly contributed to this program of research, particularly on instrument design during this reporting period. The U-M team continues to contribute scientific and intellectual knowledge on research design and implementation. We have been working closely with the partner, ISER-N, to readjust the project schedule and work through data collection and data cleaning tasks. The U-M team also contributed to the program through monitoring and supervising research activities and through extending managerial and administrative support to ISER-N. Finally, the U-M team is actively conducting data analysis using the newly available NAAT data. U-M and ISER-N staff are well positioned to complete analyses as proposed. However, due to the initial project delay, we are under pressure to complete capacity building and dissemination activities. 2020-2021 Update: The U-M team continues working closely with the partner, ISER-N, to readjust the project schedule and work through data collection and data cleaning tasks. We continue contributing to the program through monitoring and supervising research activities and through extending managerial and administrative support to ISER-N. The U-M team is actively conducting data analysis with the NAAT data. U-M and ISER-N staff are working hard to complete analyses as proposed. However, due to the initial project delay and COVID-19, we were not able to complete all capacity building and dissemination activities.
Collaborator Contribution Our partner, ISER-N, has significantly contributed to this program of research. ISER-N staff has been particularly busy this reporting period, putting forth great effort to complete data collection using the Nepali Accountability Assessment Tools (NAAT), designed with significant stakeholder involvement for this project. The completed NAAT datasets include: 1. School Location Dataset 2. Student Baseline Assessment 3. Unannounced School Observation 4. Unannounced Classroom Observation 5. Student Endline Assessment 6. Student Basic Level Examination (BLE) Test Score (district level exam) 7. Student Survey 8. Parent Survey 9. Teacher Survey 10. Principal and School Survey 11. School Management Committee (SMC) Survey 12. Parent Teacher Association (PTA) Survey Please see details of each dataset under Research Databases, Datasets & Models. 2020-2021 Update: These data (items 1-12 above) are now clean and ready for analyses. ISER-N significantly contributed to coding, re-coding survey responses, inconsistency checking, cleaning, and codebook preparation activities.
Impact The UM/ISER-N collaboration is multi-disciplinary and broadly includes the areas of demographic processes, sociology, education, and gender. UM/ISER-N partnership has generated a series of outputs. The first set of outputs include newly designed instruments referred as the Nepali Accountability Assessment Tools (NAAT). The second set of outputs include a series of multilevel data and documentation (codebooks). The final set of outputs include the results of preliminary analysis. To briefly summarize our output by dataset, NAAT instruments were created (in 2018 and 2019) for each dataset below, data collection is complete, and data is available for preliminary analysis for items1-9 below. (Data for items10-12 are currently being cleaned.) Datasets: 1. School Location (GPS coordinates) 2. Student Baseline Assessment 3. Unannounced School Observation 4. Unannounced Classroom Observation 5. Student Endline Assessment+ 6. Student Basic Level Examination (BLE) Test Score (district level exam)^ 7. Student Survey 8. Parent Survey 9. Teacher Survey 10. Principal and School Survey 11. School Management Committee (SMC) Survey 12. Parent Teacher Association (PTA) Survey +The NAAT instrument used is the same as the Student Baseline Assessment ^The test was conducted by the district education office 2020-2021 Update: Data for items 10-12 (noted above) were cleaned during this reporting period. Our revised sample sizes and collected case counts are significantly larger than the proposed sample sizes; we increased many sample sizes to assure sufficient power to detect associations between accountability and student achievement. All of these data are now available for analysis and are being processed for public release.
Start Year 2017
 
Description Consultative Meetings (Individual and Small Group) at the Local and National Levels 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Professional Practitioners
Results and Impact We shared our project objectives/implementation plan and obtained stakeholder feedback to facilitate stakeholder engagement and support. We held consultative meetings that involved (i) individual meetings with education experts and research scholars, and (ii) small group meetings with local stakeholders.

The individual consultative meetings included meeting with the ERO chief, university faculty, research scholars from New Era, (an NGO that implemented a World Bank School Education study), and an independent consultant to World Bank.

Small group meetings with stakeholders included a series of meetings with the chief of district education and coordination committee, chief of BMC education office, former regional director of education, former district education officer, and school inspectors.
Year(s) Of Engagement Activity 2018
URL http://idea.isernepal.org/consultative-meetings-at-the-local-and-national-levels/
 
Description Instrument Design and Data Collection 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Schools
Results and Impact We continued with a number of (a) instrument design activities and (b) data collection activities. We describe each below:

(a) Instrument Design
We used a rigorous, reiterative mixed methods approach to design data collection instruments. This approach combines substantive, methodological, and operational expertise into a single coordinated team, and guides all aspects of the instrument design process. Our instrument design process builds on relevant literature and our previous research experience. The following steps were completed for each instrument: (1) strategically assembled an interdisciplinary team of research scientists, research staff, and stakeholders; (2) iterative translation/rewording and testing of specific measures; (3) in depth/cognitive interviewing with representatives of the study population to refine each instrument (both qualitative and quantitative); (4) repeated pre-tests and revisions of the rewording/translations; and (5) refinement and finalization.

Instruments that we designed over the course of this project include the following:
1. Student Achievement Assessment (SAA). We worked with the Educational Review Office (ERO), the Ministry of Education of Nepal to design the SAA questionnaire for Nepali, Mathematics, and Science to measure student gains in achievement during the school year. We draw on the expertise of ERO to design scientifically valid and reliable measures that are comparable to the National Assessment of Student Achievement (NASA) of Nepal measures. Three subject matter specialists, the chief of ERO, 15 subject teachers (5 teachers per subject), and ISER-N study manager were involved in the design of the SAA. The design was completed in five steps 1) item writing by the respective subject committee; 2) review of the items by a review committee; 3) subjective committee review to prepare final draft for pretest; 4) administered the final draft to 1,241 8th grade students in 6 schools outside of the study area; 5) expert evaluation of the item performance and final item selection and refinement of the items based on analysis of the pretest data. In order to minimize student cheating during the assessment, we reshuffled the order of question items and created 4 modules each for Math and Science and two modules for Nepali.

2. Unannounced School Observation (USO). USO was designed to collect information on school facilities, services, and teaching quality independent of surveys. The main goal of this observation is to verify the information collected through school, teacher, and students' surveys. The research teams at the University of Michigan and ISER-N, and local stakeholders (primarily school inspectors), jointly designed two separate instruments: the school observation form and the classroom observation instrument. School observation measures include school facilities (such as number of buildings, library, laboratory, computer room, playground, toilet, drinking water) and services (such as transportation, hostel etc.).

3. Unannounced Classroom Observation (UCO). Classroom observation measures specifically focus on teacher-student interactions and teaching quality. The purpose of this observation was to gather information on teaching quality including classroom environment, teaching-learning methodology, instructional practice of teachers, students' attitude towards learning, teacher-students interaction, management of physical objects, etc. The measures of UCO were integrated into a windows-based interactive software program specifically designed for this study. The classroom observation instrument is a highly flexible, user-friendly software program (for tablets) that can easily be used by school inspectors in the future. We plan to share this software with the government after rigorous evaluation of its effectiveness. UCO instrument was pretested through 28 school observations and 155 classroom observations.

4. Parent Survey (PS). This survey was designed following the same rigorous and reiterative process described above. Parent survey measures include parents' perceptions of teaching quality, parental action related to gathering information about alternative schools, barriers/facilitators to exercising school choice, and awareness and participation in civil society organizations seeking to influence governance of education. This instrument was pretested on 87 parents whose children are currently enrolled in school.

5. Student Survey (SS). This survey was designed to measure students' backgrounds, their knowledge and satisfaction, and their perception on classroom environment and teacher quality. Student survey measures are drawn from previous national and international studies including TRIPOD, Evaluating the Design and Impact of School Sector Development Program (SSDP) Training in Nepal (3ie) and School Sector Reform Program (World Bank). The student survey questionnaire was pretested on 120 students.

6. Teacher Survey (TS). This survey was designed following our rigorous reiterative instrument design protocol. The same team of local stakeholders, education professionals and research team was involved in designing the teacher survey questionnaire. Similar to the SS, measures for this survey are drawn from several previous national and international studies including TRIPOD, Evaluating the Design and Impact of School Sector Development Program (SSDP) Training in Nepal (3ie), School Sector Reform Program (World Bank) and the CVFS Life History Calendar. This survey consists of two parts: i) a life history calendar (LHC) and ii) a survey questionnaire. The LHC portion of the interview records teachers' experiences annually since birth until the interview date. The survey includes measures of teachers' knowledge, attitude, perception, training, instructional practice, classroom management, satisfaction and parental outreach and reporting.

7. Principal and School Survey (PSS). This survey was designed to collect school-level accountability measures and consists of three parts: principal interview (LHC and survey questionnaire), school history calendar (SHC), and school finance. The principal's LHC includes annual records of the principal's experiences since birth. The survey portion of the principal interview includes measures of principal's performance, information, delegation and enforcement. The SHC portion of the survey includes updates to the existing SHC that we last updated in 2015. The SHC measures include annual records of number of students, number of boys and girls, number of male and female teachers, non-teaching staff, medium of instruction, and grades. The school finance section includes details on income, resource flow and disbursement. The principal and school survey measures are drawn from several previous national and international studies and was designed by the same team of education experts, local stakeholders and research staff strictly following ISER-N's instrument design protocol.

8. School Management Committee (SMC) Survey. The SMC survey was designed to collect measures of community-level accountability features. The measures of accountability includes delegation, finance, performance, information, and enforcement. Similar to other surveys, measures are drawn from several previous national and international studies and was designed by the same team of education experts, local stakeholders and research staff strictly following ISER-N's instrument design protocol.

9. Parent Teacher Association (PTA) Survey. Similar to the SMC, the PTA is also responsible for overseeing the school management, particularly the instructional aspect. This survey was designed to collect measures of short-term accountability (local community level) that includes measures of delegation, finance, performance, information, and enforcement. The survey questionnaire was designed by the same team of education experts, local stakeholders and research staff strictly following ISER-N's instrument design protocol.

(b) Data Collection
As reported in the previous reporting period, the long delays in the government transition from a centrally controlled governance system to a decentralized governance system (i.e., a republic set up with seven provinces) significantly delayed the start of the project and pushed several data collection activities to later years. More importantly, our original plan was to administer the assessment to only 1,740 students by taking a sample of 15 students from 8th grade in each of the schools. However, since many of the schools have smaller class sizes than expected (and did not have enough 8th graders to select a representative sample), we decided to include all 8th graders from 114 schools in the study, instead of a sample of 8th graders. This decision allowed the study to capture school-level variability and enhanced statistical power to generate reliable estimates to answer our research questions. The investigators have reallocated their budget to increase sample size to ensure enough statistical power to detect associations between students' learning achievement and accountability processes. As a result our data collection activities differ from our original plan.

As described in staff recruitment, an ethnically representative, well trained and highly experienced team of research staff was involved in the data collection. In addition, a high level of supervision and monitoring of field data collection was maintain throughout the data collection period. A summary of data collection activities is below:

1. Student Baseline Assessment. Using the newly developed Student Achievement Assessment tools we administered a Baseline Achievement Assessment for Nepali, Mathematics, and Science. The goal of this baseline achievement assessment was to create a bench mark before students were exposed to grade 8 curriculum so that we can generate a gain in achievement score once we administer the endline achievement assessment. Out of 4886 total students for 114 schools 4543 students (93%) participated in Mathematics, 4552 students (93%) in Nepali, and 4519 students (92%) participated in the Science assessment.

2. Unannounced School Observation. Altogether, we conducted 114 60-minute school observations in paper and pencil format. The unannounced observation collected information on schools' environment; physical infrastructure such as buildings and its types; facility of drinking water, toilet and hygiene; library, computer lab, science lab and accessibility to students; separate room for head teacher, teachers and administration; etc. The data collection process took place between January 21 and Feb 20, 2019.

3. Unannounced Classroom Observation. The classroom observations were conducted using a touch screen interface specifically programmed for classroom observations. We conducted unannounced classroom observations between January 21 and March 22, 2019 in each of the 114 schools. The unannounced classroom observation was conducted in three waves (each observation lasting 10-minutes, consisting of 12 observations in each wave in each school) in grade 6, 7 and 8. Out of 4104 observations, 4085 (99.5%) were conducted using the interactive software program designed for this study. The purpose of the observation was to gather information on teaching quality including classroom environment, teaching-learning methodology, instructional practice of teachers, students' attitude towards learning, teacher-students interaction, management of physical objects, etc.

4. Student Endline Assessment. To assess student gain in achievement, we repeated the baseline student assessment at the end of the school year in the same 114 schools. The end line assessment was conducted in the same 114 schools as the baseline assessment. Out of 4886 total students across all selected schools, a total of 4489 students (92%) participated in the Math subject assessment, 4374 students (90 %) participated in the Nepali subject assessment, and 4452 students (91%) participated in Science subject assessment.

5. Basic Level Examination (BLE) Test Score. To cross verify our endline assessment scores with metro-level test scores we also collected basic level examination (BLE) test scores from the education section of the Bharatpur Metropolitan City (BMC). BLE is a district level standardized test administered to 8th graders at the end of school year. We collected Nepali, Math, and Science test scores for 4746 students from 114 schools who participated in the BLE. Out of 4886 total students across all selected schools, a total of 4647 students (95%) participated in the Math subject assessment, 4667 students (95 %) participated in the Nepali subject assessment, and 4666 students (95%) participated in Science subject BLE test.

6. Student Survey. We conducted 45-minute student interviews in a private setting either at the student's home or a location where the student was comfortable answering the survey questions. This interview was conducted in Nepali using a computer-assisted personal interviewing program specifically designed for this survey. This survey measured students' backgrounds, knowledge, aspirations, satisfaction, and perception of classroom environment and teaching quality. Out of 4886 eligible students, we completed the interview with 4624 students resulting in a response rate of 95%.

7. Parent Survey. We conducted 60-minute interviews of students' parents in a private setting either at the respondent's home or a location where the respondent was comfortable answering the survey questions. This interview was conducted in Nepali using a computer-assisted personal interviewing program specifically designed for this survey. This survey includes household background measures of household size, composition, socio-economic background (ethnicity, social status), education and occupation, wealth, assets and income. The survey also includes individual-level measures such as parents' perceptions of teaching quality, parental action related to gathering information about alternative schools, and barriers/facilitators to exercising school choice, and awareness and participation in civil society organizations seeking to influence governance of education. Out of 4886 eligible parents, we completed this interview with 4611 parents resulting in a response rate of 94%.

8. Teacher Survey. We administered a 60-minute survey to teachers of the 8th graders of all 114 schools in a private setting either at the teacher's home or a location where the teacher was comfortable answering the survey questions. This interview was conducted in Nepali using a computer-assisted personal interviewing program specifically designed for this survey. This survey consists of a life history calendar (LHC) that records teachers' experiences annually as well as a survey portion that includes measures of teachers' background, knowledge, training, instructional practice, classroom management, and parental outreach and reporting. Out of 948 eligible teachers, 906 teachers were interviewed resulting in a response rate of 95%.

9. Principal and School Survey. We administered a 90-minute principal and school survey consisting of three parts: principal interview (LHC and survey portions), school history calendar (SHC), and school finance. The principal's LHC includes annual records of the principal's experience. The survey portion includes measures of principal's performance, information, and enforcement. The SCH portion includes updates to the existing SHC that we last updated in 2015. The SCH measures include number of students, number of boys and girls, male and female teachers, non-teaching staff, medium of instruction, and grades offered. The school finance section includes details on income, resource flow and disbursement. The survey was conducted in Nepali using a paper and pencil instrument by ISER-N senior research staff. Out of 114 schools, 113 principals and schools were interviewed resulting in a response rate of 99%.

10. School Management Committee (SMC) Survey. We administered a 45-minute survey to representatives of the school management committee (community school) and school board (institutional school). This survey includes measures of accountability features (delegation, finance, performance, information, and enforcement). The survey was conducted in Nepali using a paper and pencil instrument by ISER-N senior research staff. Out of 114 schools, 105 SMCs were interviewed resulting in a response rate of 92%.

11. Parent Teacher Association (PTA) Survey. Similar to the SMC, the PTA is also responsible in overseeing the school management, particularly the instructional aspect. To collect measures of short-term accountability (local community level) we administered a 45-minute survey representative of the parent teacher association. This survey includes measures of accountability (delegation, finance, performance, information, and enforcement). The survey was conducted in Nepali language using paper and pencil instrument by ISER-N senior research staff. Out of 114 schools, 105 SMCs were interviewed resulting in a response rate of 95%.
Year(s) Of Engagement Activity 2018,2019,2020
URL http://idea.isernepal.org/category/activities/
 
Description Orientation Workshop with Local Government 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Policymakers/politicians
Results and Impact This workshop had two objectives: (i) to share the project research objectives/implementation plan and obtain participant feedback, and (ii) to ensure local government engagement and support. The workshop was conducted at the Bharatpur Metropolitan City (BMC) office. Eight BMC officials attended: the deputy mayor, chief administrative officer, joint secretary, undersecretary, education division, school inspector, section officer, and information officer. Five research staff from the Institute for Social and Environmental Research, Nepal (ISER-N) also participated in the capacity-building workshop. Dr. Ghimire, PI, presented a detailed introduction of the project, including information on the background and rationale, objectives, research design, implementation, and capacity-building plan. His presentation was followed by a discussion about the research plan and participant feedback on the implementation plan. At the conclusion of this workshop, BMC made the following requests: (i) to expand the study area to include other parts of the BMC, (ii) to extend technical support to update school location maps and records, and (iii) to have ISER-N serve as expert advisors for other BMC programs. Overall, the orientation workshop with the local government was highly successful in fostering the beginning of a collaborative relationship between our research program and the local government in BMC.
Year(s) Of Engagement Activity 2018
URL http://idea.isernepal.org/meeting-conducted-at-bharatpur-metropolitan-city/
 
Description Stakeholder's Workshops 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Schools
Results and Impact We conducted a stakeholder's workshop at ISER-N's central office in Chitwan. We conducted this workshop in three sessions. This workshop had three basic objectives (i) to share project objectives/implementation plan, ii) to obtain stakeholders feedback, and iii) to ensure stakeholder engagement and support.

Altogether, a total of 101 school principals, head teachers and representatives, along with the chief of district education and coordination office, chief of BMC education division, former regional director of education, former district education officer, two BMC education advisors, and three school inspectors participated in these workshops.
Year(s) Of Engagement Activity 2018
URL http://idea.isernepal.org/stakeholders-workshops/