Improving Learning: Developing Measures of Accountability and Evaluating their Association with Students' Gains in Achievement in Nepal
Lead Research Organisation:
University of Michigan–Ann Arbor
Department Name: Institute for Social Research
Abstract
We propose to develop and validate measures of accountability to be shared with the Nepal Ministry of Education (MOE) and to use those measures in an analysis of the determinants of accountability and its association with students' gains in achievement. The proposed study will build on the resources of the Chitwan Valley Family Study (CVFS), a 20-year ongoing panel study of 116 schools with 3,000 households with 3,500 school aged children in 151 communities located throughout the Western Chitwan Valley of Nepal. With funding from DFID-ESRC, we are proposing to achieve two aims:
Aim One: To Develop and Pretest a Suite of Nepali Accountability Assessment Tools (NAATs) for Use by the MOE and to Pilot these Tools within the Chitwan Valley of Nepal. Importantly, the tools will be designed so that Nepal's MOE can both assess and potentially improve its current accountability processes at multiple levels of the increasingly decentralized Nepalese education system [4].
To achieve this aim we will: (1) develop a variety of accountability assessment tools for use in Nepal's education system; (2) modify a set of instructional processes and instructional quality measures developed for use in OECD countries for use in the Nepali educational system; and (3) gather data on students' academic achievement using standardized test items developed by Nepal's MOE.
Aim Two: To Investigate How Accountability Processes; Environments for Student Learning in Schools, Families, and Communities; and Student Learning are Related. This involves investigating three main research questions: Are accountability processes systematically related to socioeconomic disparities among communities, schools within communities, and families within schools? In school and community settings where accountability processes are more intensive, is the quality of instructional service delivery higher? And, controlling for socioeconomic disparities related to student achievement is student learning higher in schools and communities where accountability processes are more intensive?
To meet this aim, we will: (1) administer a newly designed PET-QSDS survey to 380 key stakeholders; (2) administer the NASA test at the beginning and end of the school year and a student survey to 1,740 8th graders; and (3) administer a teacher survey to 1,392 teachers and a parent survey to 1,740 parents. The results of this research will be relevant to education policy makers in Nepal and will also contribute directly to comparative education research on school effectiveness.
This study will generate rigorous scientific outcomes: (1) development of a low income context adaptive accountability assessment tool; (2) cross-cultural assessment of the reliability and predictive validity of accountability measures; (3) identification of contextual factors with strong correlation with accountability; (4) potential for identification of new dimensions of accountability in low income settings; and (5) scientific advancement in our understanding of the relationship between accountability, instructional quality and students' gains in achievement. These outcomes will be made widely available to scientists and policy makers. First, we will conduct dissemination workshops at local and national levels to share findings of the study and provide training on the use of the newly designed accountability assessment tool and analysis of the data generated through the various surveys mentioned above. Second, the data will be made available through ICPSR and the UK Data Service. Third, the findings will be disseminated through presentations at national and international conferences and published in scientific articles, and research and policy briefs. Finally, the participation of Nepali faculty, scientists, government representatives and school authorities throughout the project will advance the scientific and analytical capacity of their respective host institutions (DOE,TU, PABSON, PDs).
Aim One: To Develop and Pretest a Suite of Nepali Accountability Assessment Tools (NAATs) for Use by the MOE and to Pilot these Tools within the Chitwan Valley of Nepal. Importantly, the tools will be designed so that Nepal's MOE can both assess and potentially improve its current accountability processes at multiple levels of the increasingly decentralized Nepalese education system [4].
To achieve this aim we will: (1) develop a variety of accountability assessment tools for use in Nepal's education system; (2) modify a set of instructional processes and instructional quality measures developed for use in OECD countries for use in the Nepali educational system; and (3) gather data on students' academic achievement using standardized test items developed by Nepal's MOE.
Aim Two: To Investigate How Accountability Processes; Environments for Student Learning in Schools, Families, and Communities; and Student Learning are Related. This involves investigating three main research questions: Are accountability processes systematically related to socioeconomic disparities among communities, schools within communities, and families within schools? In school and community settings where accountability processes are more intensive, is the quality of instructional service delivery higher? And, controlling for socioeconomic disparities related to student achievement is student learning higher in schools and communities where accountability processes are more intensive?
To meet this aim, we will: (1) administer a newly designed PET-QSDS survey to 380 key stakeholders; (2) administer the NASA test at the beginning and end of the school year and a student survey to 1,740 8th graders; and (3) administer a teacher survey to 1,392 teachers and a parent survey to 1,740 parents. The results of this research will be relevant to education policy makers in Nepal and will also contribute directly to comparative education research on school effectiveness.
This study will generate rigorous scientific outcomes: (1) development of a low income context adaptive accountability assessment tool; (2) cross-cultural assessment of the reliability and predictive validity of accountability measures; (3) identification of contextual factors with strong correlation with accountability; (4) potential for identification of new dimensions of accountability in low income settings; and (5) scientific advancement in our understanding of the relationship between accountability, instructional quality and students' gains in achievement. These outcomes will be made widely available to scientists and policy makers. First, we will conduct dissemination workshops at local and national levels to share findings of the study and provide training on the use of the newly designed accountability assessment tool and analysis of the data generated through the various surveys mentioned above. Second, the data will be made available through ICPSR and the UK Data Service. Third, the findings will be disseminated through presentations at national and international conferences and published in scientific articles, and research and policy briefs. Finally, the participation of Nepali faculty, scientists, government representatives and school authorities throughout the project will advance the scientific and analytical capacity of their respective host institutions (DOE,TU, PABSON, PDs).
Planned Impact
The methodological tools and protocols, comprehensive measurement, and empirical findings generated by the proposed study have high potential to impact both education policy makers and academia and also contribute directly to comparative education research on school effectiveness at regional and global levels.
Impact on policy: The proposed study is likely to impact education in Nepal in multiple ways. The NAAT and user protocol to measure accountability will be a great asset to the government of Nepal and an important resource to achieve school sector refom plan aims and enhance education quality. Substantively, the empirical answers to the questions we posed: (1) the extent to which accountability is (i) a significant predictor of instructional quality and (ii) distributed unequally across CVFS strata (geographical and urbanacity), different community contexts and across public and private schools; (2) the extent to which any relationships between accountability and students' gains in achievement is explained by instructional quality; and (3) the extent to which the measures of accountability developed in Nepal is a significant predictor of student performance on NASA will provide crucial information for policy makers struggling to make the best use of scarce resources.
We plan to engage stakeholders in several stages of this program to ensure effective policy influence. The key stakeholders include: local teachers, school management committees, proprietors of institutional schools, parents, district education offices, district development committees, Private and Boarding Schools' Organization Nepal (PABSON-Chitwan), I/NGOs, colleges and universities (district level), Central Department of Education at Tribhuvan University (TU), Kathmandu University (KU), Department of Education, Ministry of Education, National Planning Commission, and development partners engaged in educational programs. We will ensure stakeholder engagement through their participation in consultative meetings, research staff trainings, and design and dissemination workshops.
Impact on academia: The empirical findings will be of great use to a wide range of academic audiences for teaching and research purposes. At the local level, these audiences include affiliates of TU, KU and other local colleges and universities in Nepal. As indicated in the letter of support, the faculty and students at the Central Department of Education at TU will directly benefit from research findings both substantively and methodologically. At the regional level, the findings will have strong relevance to academic audiences studying low-income countries in South Asia. Our UM/ISER-N joint training program on survey methodology in Nepal has strong collaborative ties with various universities and research organizations in South Asia. Faculty, research scholars, and students from Bangladesh, India, Pakistan, and Afghanistan regularly participate in the training we offer. We will use this venue to share our findings. At the global level, the publication of findings will improve and stimulate further research in this area by scholars worldwide.
We plan to make the CVFS panel data and the new data we propose to collect available through the UK Data Service. We will also promote the data at national and international conferences. This will greatly increase the use of these data and the resulting impact on academia. We expect that the larger scientific community will benefit from this research well beyond the current project period by stimulating new research agendas in other areas of science, such as parental investment in child education, teacher incentives and evaluation.
The findings of this study will also have broader implications for the design of education policy worldwide. Our global target audiences are UNESCO, Institute of International Education (IIE), I/NGOs, and donors engaged in educational development worldwide.
Impact on policy: The proposed study is likely to impact education in Nepal in multiple ways. The NAAT and user protocol to measure accountability will be a great asset to the government of Nepal and an important resource to achieve school sector refom plan aims and enhance education quality. Substantively, the empirical answers to the questions we posed: (1) the extent to which accountability is (i) a significant predictor of instructional quality and (ii) distributed unequally across CVFS strata (geographical and urbanacity), different community contexts and across public and private schools; (2) the extent to which any relationships between accountability and students' gains in achievement is explained by instructional quality; and (3) the extent to which the measures of accountability developed in Nepal is a significant predictor of student performance on NASA will provide crucial information for policy makers struggling to make the best use of scarce resources.
We plan to engage stakeholders in several stages of this program to ensure effective policy influence. The key stakeholders include: local teachers, school management committees, proprietors of institutional schools, parents, district education offices, district development committees, Private and Boarding Schools' Organization Nepal (PABSON-Chitwan), I/NGOs, colleges and universities (district level), Central Department of Education at Tribhuvan University (TU), Kathmandu University (KU), Department of Education, Ministry of Education, National Planning Commission, and development partners engaged in educational programs. We will ensure stakeholder engagement through their participation in consultative meetings, research staff trainings, and design and dissemination workshops.
Impact on academia: The empirical findings will be of great use to a wide range of academic audiences for teaching and research purposes. At the local level, these audiences include affiliates of TU, KU and other local colleges and universities in Nepal. As indicated in the letter of support, the faculty and students at the Central Department of Education at TU will directly benefit from research findings both substantively and methodologically. At the regional level, the findings will have strong relevance to academic audiences studying low-income countries in South Asia. Our UM/ISER-N joint training program on survey methodology in Nepal has strong collaborative ties with various universities and research organizations in South Asia. Faculty, research scholars, and students from Bangladesh, India, Pakistan, and Afghanistan regularly participate in the training we offer. We will use this venue to share our findings. At the global level, the publication of findings will improve and stimulate further research in this area by scholars worldwide.
We plan to make the CVFS panel data and the new data we propose to collect available through the UK Data Service. We will also promote the data at national and international conferences. This will greatly increase the use of these data and the resulting impact on academia. We expect that the larger scientific community will benefit from this research well beyond the current project period by stimulating new research agendas in other areas of science, such as parental investment in child education, teacher incentives and evaluation.
The findings of this study will also have broader implications for the design of education policy worldwide. Our global target audiences are UNESCO, Institute of International Education (IIE), I/NGOs, and donors engaged in educational development worldwide.
| Description | Results reveal high reliability and validity of the student achievement and social accountability measures that we developed during this project. Results from our study, as well as from Nepal's National Assessment Student Achievement (NASA), also illustrate low student achievement. Findings show an important gap in student cumulative achievement between community and institutional schools. Our preliminary results suggest that one third (33%) of the variation in student cumulative achievement by the end of 8th grade is explained by school-level variation. Indeed, further analysis of the annual gain in student achievement revealed great variation between students. As reflected in the mean difference between baseline and endline achievement tests, half of the students (50%) had no gain, almost one third (32%) had some gain, 14% had good gain, and only 4% had very good gain. This suggests that a large fraction of students are not learning in school. This pattern is consistent across all three subjects: Mathematics, Nepali, and Science. School-level aggregation of achievement scores shows similar patterns-the majority of schools gain little or not at all. The mean difference between baseline and endline Mathematics achievement tests across 114 schools shows very little gain. There was no gain between baseline and endline in Science. However, achievement scores for Nepali show some gain. Preliminary analyses revealed several important insights regarding social accountability and student achievement. First, a large portion (approximately one third) of student achievement is explained by school-level variation. Second, student's perception of instructional quality is strongly associated with student achievement. Finally, certain social accountability features, including parents' knowledge, judgement about instructional quality, school management, and school principal instructional leadership, each have strong associations with student achievement, net of several individual household and contextual factors. These analyses will be further refined as the analyses progress. School infrastructure, facilities, and student achievement analyses are motivated by enhancing our understanding of gender differences in student achievement. Preliminary results demonstrate that school infrastructure and facilities have an important influence on student achievement. The existence of separate bathrooms for girls and boys with soap and sanitary pads in the girls' bathroom had a strong positive effect on girls' achievement, suggesting appropriate facilities are important to improving girls' educational achievement. Closeout report update (March 2022): Additionally, our paper investigating parental migration and student achievement has been accepted for oral presentation at the Annual Meeting of the Population Association of America, which will be held April 6-9, 2022 in Atlanta, Georgia. This paper explores the extent to which father's migration abroad influences school choice for their children. Using students' assessment data, we investigated the difference in student achievement between public and private schools. We also estimated the association between father's international migration status and children's enrollment in private school. Preliminary results suggest a remarkable difference in student achievement in all three subjects (mathematics, science, and Nepali) between students attending public and private schools. We found a strong, positive statically significant association between father's international migration status and children's enrollment in private school. This result remains after adjusting for other factors, including selection into migration that likely affects this relationship. |
| Exploitation Route | The outcomes of this funding-measurement tools, data, and research findings-might be used by parents, schools, stakeholders, researchers, policymakers, and development partners in four important ways. First, several measurement tools, including the unannounced classroom observation, survey questionnaires, and student assessments, that were developed and tested during this project were found to be highly reliable and valid. These tools are now available to the scientific community. Thus, research scholars from Nepal and other low-income settings interested in education research can use and adopt these tools free of cost. Second, the data generated through this project can be used to answer policy questions in unique ways. For example, our team is now collaborating with members of a UNICEF-funded study titled "Global Education Monitoring Report 2021/22: Non-State Actors in Education in Nepal: A Country Study". Using the data collected through the DFID-ESRC-funded project, we plan to investigate the role of non-state actors in student learning outcomes across institutional schools, school management committees, parent teacher associations, and other non-governmental organizations. Third, schools in the study area can use the findings of this study to improve student learning outcomes. For example, evidence of higher learning outcomes for girls in institutional schools that have a separate toilet for girls with soap and sanitary products is a relatively simple, cost-efficient strategy to improve learning outcomes for female students. This strategy alone could significantly and immediately impact female student learning outcomes. Finally, given that students, parents, and families can freely choose schools, these findings can provide crucial information to parents to help them make informed choices regarding the types of schools they select for their children. |
| Sectors | Communities and Social Services/Policy,Education |
| URL | http://idea.isernepal.org/ |
| Title | Assessment |
| Description | Student Achievement Assessment (SAA). We worked with the Educational Review Office (ERO), the Ministry of Education of Nepal to design the SAA questionnaire for Nepali, Mathematics, and Science to measure student gains in achievement during the school year. We draw on the expertise of ERO to design scientifically valid and reliable measures that are comparable to the National Assessment of Student Achievement (NASA) of Nepal measures. Three subject matter specialists, the chief of ERO, 15 subject teachers (5 teachers per subject), and ISER-N study manager were involved in the design of the SAA. The design was completed in five steps 1) item writing by the respective subject committee; 2) review of the items by a review committee; 3) subjective committee review to prepare final draft for pretest; 4) administered the final draft to 1,241 8th grade students in 6 schools outside of the study area; 5) expert evaluation of the item performance and final item selection and refinement of the items based on analysis of the pretest data. In order to minimize student cheating during the assessment, we reshuffled the order of question items and created 4 modules each for Math and Science and two modules for Nepali. |
| Type Of Material | Physiological assessment or outcome measure |
| Year Produced | 2018 |
| Provided To Others? | Yes |
| Impact | Due to COVID-19, it is taking much longer to make a notable impact. |
| URL | https://idea.isernepal.org/products/nepali-accountability-assessment-tool/ |
| Title | Observation Forms |
| Description | The research teams at the University of Michigan and ISER-N, and local stakeholders (primarily school inspectors), jointly designed two separate instruments: the school observation form and the classroom observation instrument. Unannounced School Observation (USO). USO was designed to collect information on school facilities, services, and teaching quality independent of surveys. The main goal of this observation is to verify the information collected through school, teacher, and students' surveys. The school observation measures include school facilities (such as number of buildings, library, laboratory, computer room, playground, toilet, drinking water) and services (such as transportation, hostel etc.). Unannounced Classroom Observation (UCO). Classroom observation measures specifically focus on teacher-student interactions and teaching quality. The purpose of this observation was to gather information on teaching quality including classroom environment, teaching-learning methodology, instructional practice of teachers, students' attitude towards learning, teacher-students interaction, management of physical objects, etc. The measures of UCO were integrated into a windows-based interactive software program specifically designed for this study. The classroom observation instrument is a highly flexible, user-friendly software program (for tablets) that can easily be used by school inspectors in the future. We plan to share this software with the government after rigorous evaluation of its effectiveness. UCO instrument was pretested through 28 school observations and 155 classroom observations. |
| Type Of Material | Physiological assessment or outcome measure |
| Year Produced | 2018 |
| Provided To Others? | Yes |
| Impact | Due to COVID-19, it is taking much longer to make a notable impact. |
| URL | http://idea.isernepal.org/data/ |
| Title | Survey Questionnaires |
| Description | Please see details of the Student Survey, Parent Survey, Teacher Survey, Principal and School Survey, School Management Committee (SMC) Survey, and Parent Teacher Association (PTA) Survey in the Engagement Activities (Instrument Design) section. |
| Type Of Material | Physiological assessment or outcome measure |
| Year Produced | 2019 |
| Provided To Others? | Yes |
| Impact | Due to COVID-19, it is taking much longer to make a notable impact. |
| URL | http://idea.isernepal.org/data/ |
| Title | Parent Survey |
| Description | The parent survey data include parent/caregiver's individual-level measures such as parents' perceptions of teaching quality, parental action related to gathering information about alternative schools, barriers/facilitators to exercising school choice, and awareness and participation in civil society organizations seeking to influence governance of education. Out of 4889 eligible students' parents, data were collected from 4606 parents resulting in a response rate of 94%. The data are archived at the UK Data Service. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2019 |
| Provided To Others? | Yes |
| Impact | Due to COVID-19, it is taking much longer to make a notable impact. |
| Title | Parent Teacher Association (PTA) Survey |
| Description | The Parent Teacher Association (PTA) data consists of responses to a survey interview from a representative of the PTA. The data consist of measures of short-term accountability (local community level) and include other measures of accountability knowledge, information, performance and monitoring. Out of 114 eligible schools, data were collected from PTA representatives from 108 schools resulting in a response rate of 95%. The data are archived at the UK Data Service. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2020 |
| Provided To Others? | Yes |
| Impact | Due to COVID-19, it is taking much longer to make a notable impact. |
| Title | Principal and School Survey |
| Description | The Head Teacher (Principal) survey was administered to the head teachers of the schools that participated in the study. The data include information on the head teacher's general background, school management experience, professional engagement with school leadership, and satisfaction with working conditions. Out of 114 eligible schools, data were collected from Head Teachers (Principals) from 113 schools resulting in a response rate of 99%. The data are archived at the UK Data Service. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2020 |
| Provided To Others? | Yes |
| Impact | Due to COVID-19, it is taking much longer to make a notable impact. |
| Title | School Location Dataset |
| Description | Using the existing GPS coordinates of all schools, health services, and urban center within western Chitwan Valley (study area) from the Chitwan Valley Family Study, we have created several school locational measures. These measures include distance from the school to urban center and number of school or health services within a 1 kilo meter radius. These data have been cleaned and analyzed and will be available only through a restricted data access agreement. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2019 |
| Provided To Others? | Yes |
| Impact | Due to COVID-19, it is taking much longer to make a notable impact. |
| Title | School Management Committee (SMC) Survey |
| Description | The school management committee (SMC) data includes responses to an interview from a representative of the SMC (community school) or school board (institutional school). This survey includes measures of community-level accountability features. These features include governance structure, finance, performance, information, and monitoring and engagement. Out of 114 eligible schools, data were collected from SMCs from 109 schools resulting in a response rate of 96%. The data are archived at the UK Data Service. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2020 |
| Provided To Others? | Yes |
| Impact | Due to COVID-19, it is taking much longer to make a notable impact. |
| Title | Student Baseline Assessment |
| Description | Using the Student Achievement Assessment tools, a Student Baseline Assessment in the subjects of Nepali, Mathematics, and Science was administered. The Student Baseline Assessment data provides bench mark subject scores before students were exposed to the grade 8 curriculum. Comparing the Student Baseline Assessment data to the Student Endline Assessment data provides a score of students' gain in achievement. Out of 4889 eligible students, 4519 students (93%) participated in the Mathematics subject assessment, 4543 students (93%) participated in the Nepali subject assessment, and 4552 students (92%) participated in the Science subject assessment. The raw assessment scores were converted to a scale score with a mean of 500 and standard deviation of 50. In addition to the scale score, this data include percent attempted and percent correct. These data have been cleaned and analyzed and will be available only through a restricted data access agreement. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2018 |
| Provided To Others? | Yes |
| Impact | Due to COVID-19, it is taking much longer to make a notable impact. |
| Title | Student Basic Level Examination (BLE) Test Score (district-level exam) |
| Description | To cross verify the Student Endline Assessment scores with district-level test scores, we also collected Student Basic Level Examination (BLE) Test Scores from the education section of Bharatpur Metropolitan City. The BLE is a district-level standardized test administered to 8th graders at the end of the school year. Out of 4889 eligible students, a total of 4647 students (95%) participated in the Math subject BLE test, 4667 students (95%) participated in the Nepali subject BLE test, and 4666 students (95%) participated in the Science subject BLE test. The BLE raw scores were converted in 100 full mark. These data have been cleaned and analyzed and will be available only through a restricted data access agreement. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2019 |
| Provided To Others? | Yes |
| Impact | Due to COVID-19, it is taking much longer to make a notable impact. |
| Title | Student Endline Assessment |
| Description | To assess student gain in achievement, the Baseline Student Assessment was repeated at the end of the school year in the same schools, creating the Student Endline Assessment data. Out of 4889 eligible students, a total of 4489 students (92%) participated in the Math subject assessment, 4374 students (90%) participated in the Nepali subject assessment, and 4452 students (91%) participated in Science subject assessment. Similar to the baseline score, the raw score of these assessments were converted to a scale score with a mean of 500 and standard deviation of 50. In addition to the scale score, this data also include percent attempted and percent correct. These data have been cleaned and analyzed and will be available only through a restricted data access agreement. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2019 |
| Provided To Others? | Yes |
| Impact | Due to COVID-19, it is taking much longer to make a notable impact. |
| Title | Student Survey |
| Description | The student survey data include measures of students' backgrounds, knowledge, aspirations, satisfaction, and perception of the classroom environment and teaching quality. Out of 4889 eligible students, data were collected from 4588 students resulting in a response rate of 94%. These data are available to the project research team for analysis. The data are archived at the UK Data Service. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2019 |
| Provided To Others? | Yes |
| Impact | Due to COVID-19, it is taking much longer to make a notable impact. |
| Title | Teacher Survey |
| Description | The teacher survey data includes measures of teachers' background, knowledge, training, instructional practice, classroom management, and parental outreach and reporting. Out of 951 eligible teachers, data were collected from 906 teachers resulting in a response rate of 95%. The data are archived at the UK Data Service. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2019 |
| Provided To Others? | Yes |
| Impact | Due to COVID-19, it is taking much longer to make a notable impact. |
| Title | Unannounced Classroom Observation |
| Description | The Unannounced Classroom Observation data contains three waves-each observation lasted 10-minutes and consisted of 12 observations in each school-in grades 6, 7, and 8. Out of 4104 observations, 4085 (99.5% response rate) were conducted using the windows-based interactive software program designed for this study. The Unannounced Classroom Observation data includes measures of teaching quality, including classroom environment, teaching-learning methodology, instructional practice of the teachers, student attitude towards learning, teachers' and students' behavior in the classroom, teacher-student interaction, management of physical objects, etc. These data have been cleaned and analyzed and will be available only through a restricted data access agreement. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2019 |
| Provided To Others? | Yes |
| Impact | Due to COVID-19, it is taking much longer to make a notable impact. |
| Title | Unannounced School Observation |
| Description | The Unannounced School Observation data includes measures of schools' environments, including the physical infrastructure such as buildings and its type; facilities of drinking water and toilets, and hygiene; resources such as library, computer lab, and science lab, and their accessibility to students; separate room for head teacher (principal), teachers, and administration; etc. These data have been cleaned and analyzed and will be available only through a restricted data access agreement. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2019 |
| Provided To Others? | Yes |
| Impact | Due to COVID-19, it is taking much longer to make a notable impact. |
| Description | The Institute for Social and Environmental Research - Nepal (ISER-N) |
| Organisation | Institute for Social and Environmental Research - Nepal (ISER-N) |
| Country | Nepal |
| Sector | Charity/Non Profit |
| PI Contribution | The University of Michigan (U-M) team has significantly contributed to this program of research, particularly on instrument design during this reporting period. The U-M team continues to contribute scientific and intellectual knowledge on research design and implementation. We have been working closely with the partner, ISER-N, to readjust the project schedule and work through data collection and data cleaning tasks. The U-M team also contributed to the program through monitoring and supervising research activities and through extending managerial and administrative support to ISER-N. Finally, the U-M team is actively conducting data analysis using the newly available NAAT data. U-M and ISER-N staff are well positioned to complete analyses as proposed. However, due to the initial project delay, we are under pressure to complete capacity building and dissemination activities. 2020-2021 Update: The U-M team continues working closely with the partner, ISER-N, to readjust the project schedule and work through data collection and data cleaning tasks. We continue contributing to the program through monitoring and supervising research activities and through extending managerial and administrative support to ISER-N. The U-M team is actively conducting data analysis with the NAAT data. U-M and ISER-N staff are working hard to complete analyses as proposed. However, due to the initial project delay and COVID-19, we were not able to complete all capacity building and dissemination activities. 2022 Update: Drs. Rowan and Ghimire collaborated with ISER-N staff to organize the Research Dissemination Workshop, present results, and facilitate discussions. The goals of the workshop were twofold: 1) to share research findings and experiences from the study to foster intellectual debate and interactions among key stakeholders and potential consumers of the empirical evidence generated from this project, and 2) to enhance the scientific and analytical capacity of Nepali students, faculty, and scientists and promote future research collaboration to address important policy issues related to education achievement in Nepal. (Please see details provided in Engagement Activities.) |
| Collaborator Contribution | Our partner, ISER-N, has significantly contributed to this program of research. ISER-N staff has been particularly busy this reporting period, putting forth great effort to complete data collection using the Nepali Accountability Assessment Tools (NAAT), designed with significant stakeholder involvement for this project. The completed NAAT datasets include: 1. School Location Dataset 2. Student Baseline Assessment 3. Unannounced School Observation 4. Unannounced Classroom Observation 5. Student Endline Assessment 6. Student Basic Level Examination (BLE) Test Score (district level exam) 7. Student Survey 8. Parent Survey 9. Teacher Survey 10. Principal and School Survey 11. School Management Committee (SMC) Survey 12. Parent Teacher Association (PTA) Survey Please see details of each dataset under Research Databases, Datasets & Models. 2020-2021 Update: These data (items 1-12 above) are now clean and ready for analyses. ISER-N significantly contributed to coding, re-coding survey responses, inconsistency checking, cleaning, and codebook preparation activities. 2022 Update: ISER-N led all of the logistical efforts for the Research Dissemination Workshop, including contacting potential participants and presenters, setting up and monitoring the virtual meeting, troubling shooting technical issues during the workshop, and writing the workshop report. |
| Impact | The UM/ISER-N collaboration is multi-disciplinary and broadly includes the areas of demographic processes, sociology, education, and gender. UM/ISER-N partnership has generated a series of outputs. The first set of outputs include newly designed instruments referred as the Nepali Accountability Assessment Tools (NAAT). The second set of outputs include a series of multilevel data and documentation (codebooks). The final set of outputs include the results of preliminary analysis. To briefly summarize our output by dataset, NAAT instruments were created (in 2018 and 2019) for each dataset below, data collection is complete, and data is available for preliminary analysis for items1-9 below. (Data for items10-12 are currently being cleaned.) Datasets: 1. School Location (GPS coordinates) 2. Student Baseline Assessment 3. Unannounced School Observation 4. Unannounced Classroom Observation 5. Student Endline Assessment+ 6. Student Basic Level Examination (BLE) Test Score (district level exam)^ 7. Student Survey 8. Parent Survey 9. Teacher Survey 10. Principal and School Survey 11. School Management Committee (SMC) Survey 12. Parent Teacher Association (PTA) Survey +The NAAT instrument used is the same as the Student Baseline Assessment ^The test was conducted by the district education office 2020-2021 Update: Data for items 10-12 (noted above) were cleaned during this reporting period. Our revised sample sizes and collected case counts are significantly larger than the proposed sample sizes; we increased many sample sizes to assure sufficient power to detect associations between accountability and student achievement. All of these data are now available for analysis and are being processed for public release. 2022 Update: All non-restricted-use datasets were deposited at the UK Data Service during this reporting period. (Please see updates in Research Datasets, Databases & Models.) |
| Start Year | 2017 |
| Description | Consultative Meetings (Individual and Small Group) at the Local and National Levels |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Professional Practitioners |
| Results and Impact | We shared our project objectives/implementation plan and obtained stakeholder feedback to facilitate stakeholder engagement and support. We held consultative meetings that involved (i) individual meetings with education experts and research scholars, and (ii) small group meetings with local stakeholders. The individual consultative meetings included meeting with the ERO chief, university faculty, research scholars from New Era, (an NGO that implemented a World Bank School Education study), and an independent consultant to World Bank. Small group meetings with stakeholders included a series of meetings with the chief of district education and coordination committee, chief of BMC education office, former regional director of education, former district education officer, and school inspectors. |
| Year(s) Of Engagement Activity | 2018 |
| URL | http://idea.isernepal.org/consultative-meetings-at-the-local-and-national-levels/ |
| Description | Instrument Design and Data Collection |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Schools |
| Results and Impact | We continued with a number of (a) instrument design activities and (b) data collection activities. We describe each below: (a) Instrument Design We used a rigorous, reiterative mixed methods approach to design data collection instruments. This approach combines substantive, methodological, and operational expertise into a single coordinated team, and guides all aspects of the instrument design process. Our instrument design process builds on relevant literature and our previous research experience. The following steps were completed for each instrument: (1) strategically assembled an interdisciplinary team of research scientists, research staff, and stakeholders; (2) iterative translation/rewording and testing of specific measures; (3) in depth/cognitive interviewing with representatives of the study population to refine each instrument (both qualitative and quantitative); (4) repeated pre-tests and revisions of the rewording/translations; and (5) refinement and finalization. Instruments that we designed over the course of this project include the following: 1. Student Achievement Assessment (SAA). We worked with the Educational Review Office (ERO), the Ministry of Education of Nepal to design the SAA questionnaire for Nepali, Mathematics, and Science to measure student gains in achievement during the school year. We draw on the expertise of ERO to design scientifically valid and reliable measures that are comparable to the National Assessment of Student Achievement (NASA) of Nepal measures. Three subject matter specialists, the chief of ERO, 15 subject teachers (5 teachers per subject), and ISER-N study manager were involved in the design of the SAA. The design was completed in five steps 1) item writing by the respective subject committee; 2) review of the items by a review committee; 3) subjective committee review to prepare final draft for pretest; 4) administered the final draft to 1,241 8th grade students in 6 schools outside of the study area; 5) expert evaluation of the item performance and final item selection and refinement of the items based on analysis of the pretest data. In order to minimize student cheating during the assessment, we reshuffled the order of question items and created 4 modules each for Math and Science and two modules for Nepali. 2. Unannounced School Observation (USO). USO was designed to collect information on school facilities, services, and teaching quality independent of surveys. The main goal of this observation is to verify the information collected through school, teacher, and students' surveys. The research teams at the University of Michigan and ISER-N, and local stakeholders (primarily school inspectors), jointly designed two separate instruments: the school observation form and the classroom observation instrument. School observation measures include school facilities (such as number of buildings, library, laboratory, computer room, playground, toilet, drinking water) and services (such as transportation, hostel etc.). 3. Unannounced Classroom Observation (UCO). Classroom observation measures specifically focus on teacher-student interactions and teaching quality. The purpose of this observation was to gather information on teaching quality including classroom environment, teaching-learning methodology, instructional practice of teachers, students' attitude towards learning, teacher-students interaction, management of physical objects, etc. The measures of UCO were integrated into a windows-based interactive software program specifically designed for this study. The classroom observation instrument is a highly flexible, user-friendly software program (for tablets) that can easily be used by school inspectors in the future. We plan to share this software with the government after rigorous evaluation of its effectiveness. UCO instrument was pretested through 28 school observations and 155 classroom observations. 4. Parent Survey (PS). This survey was designed following the same rigorous and reiterative process described above. Parent survey measures include parents' perceptions of teaching quality, parental action related to gathering information about alternative schools, barriers/facilitators to exercising school choice, and awareness and participation in civil society organizations seeking to influence governance of education. This instrument was pretested on 87 parents whose children are currently enrolled in school. 5. Student Survey (SS). This survey was designed to measure students' backgrounds, their knowledge and satisfaction, and their perception on classroom environment and teacher quality. Student survey measures are drawn from previous national and international studies including TRIPOD, Evaluating the Design and Impact of School Sector Development Program (SSDP) Training in Nepal (3ie) and School Sector Reform Program (World Bank). The student survey questionnaire was pretested on 120 students. 6. Teacher Survey (TS). This survey was designed following our rigorous reiterative instrument design protocol. The same team of local stakeholders, education professionals and research team was involved in designing the teacher survey questionnaire. Similar to the SS, measures for this survey are drawn from several previous national and international studies including TRIPOD, Evaluating the Design and Impact of School Sector Development Program (SSDP) Training in Nepal (3ie), School Sector Reform Program (World Bank) and the CVFS Life History Calendar. This survey consists of two parts: i) a life history calendar (LHC) and ii) a survey questionnaire. The LHC portion of the interview records teachers' experiences annually since birth until the interview date. The survey includes measures of teachers' knowledge, attitude, perception, training, instructional practice, classroom management, satisfaction and parental outreach and reporting. 7. Principal and School Survey (PSS). This survey was designed to collect school-level accountability measures and consists of three parts: principal interview (LHC and survey questionnaire), school history calendar (SHC), and school finance. The principal's LHC includes annual records of the principal's experiences since birth. The survey portion of the principal interview includes measures of principal's performance, information, delegation and enforcement. The SHC portion of the survey includes updates to the existing SHC that we last updated in 2015. The SHC measures include annual records of number of students, number of boys and girls, number of male and female teachers, non-teaching staff, medium of instruction, and grades. The school finance section includes details on income, resource flow and disbursement. The principal and school survey measures are drawn from several previous national and international studies and was designed by the same team of education experts, local stakeholders and research staff strictly following ISER-N's instrument design protocol. 8. School Management Committee (SMC) Survey. The SMC survey was designed to collect measures of community-level accountability features. The measures of accountability includes delegation, finance, performance, information, and enforcement. Similar to other surveys, measures are drawn from several previous national and international studies and was designed by the same team of education experts, local stakeholders and research staff strictly following ISER-N's instrument design protocol. 9. Parent Teacher Association (PTA) Survey. Similar to the SMC, the PTA is also responsible for overseeing the school management, particularly the instructional aspect. This survey was designed to collect measures of short-term accountability (local community level) that includes measures of delegation, finance, performance, information, and enforcement. The survey questionnaire was designed by the same team of education experts, local stakeholders and research staff strictly following ISER-N's instrument design protocol. (b) Data Collection Please see closeout report updates in the Research Datasets, Databases & Models section. As reported in the previous reporting period, the long delays in the government transition from a centrally controlled governance system to a decentralized governance system (i.e., a republic set up with seven provinces) significantly delayed the start of the project and pushed several data collection activities to later years. More importantly, our original plan was to administer the assessment to only 1,740 students by taking a sample of 15 students from 8th grade in each of the schools. However, since many of the schools have smaller class sizes than expected (and did not have enough 8th graders to select a representative sample), we decided to include all 8th graders from 114 schools in the study, instead of a sample of 8th graders. This decision allowed the study to capture school-level variability and enhanced statistical power to generate reliable estimates to answer our research questions. The investigators have reallocated their budget to increase sample size to ensure enough statistical power to detect associations between students' learning achievement and accountability processes. As a result our data collection activities differ from our original plan. As described in staff recruitment, an ethnically representative, well trained and highly experienced team of research staff was involved in the data collection. In addition, a high level of supervision and monitoring of field data collection was maintain throughout the data collection period. A summary of data collection activities is below: 1. Student Baseline Assessment. Using the newly developed Student Achievement Assessment tools we administered a Baseline Achievement Assessment for Nepali, Mathematics, and Science. The goal of this baseline achievement assessment was to create a bench mark before students were exposed to grade 8 curriculum so that we can generate a gain in achievement score once we administer the endline achievement assessment. Out of 4886 total students for 114 schools 4543 students (93%) participated in Mathematics, 4552 students (93%) in Nepali, and 4519 students (92%) participated in the Science assessment. 2. Unannounced School Observation. Altogether, we conducted 114 60-minute school observations in paper and pencil format. The unannounced observation collected information on schools' environment; physical infrastructure such as buildings and its types; facility of drinking water, toilet and hygiene; library, computer lab, science lab and accessibility to students; separate room for head teacher, teachers and administration; etc. The data collection process took place between January 21 and Feb 20, 2019. 3. Unannounced Classroom Observation. The classroom observations were conducted using a touch screen interface specifically programmed for classroom observations. We conducted unannounced classroom observations between January 21 and March 22, 2019 in each of the 114 schools. The unannounced classroom observation was conducted in three waves (each observation lasting 10-minutes, consisting of 12 observations in each wave in each school) in grade 6, 7 and 8. Out of 4104 observations, 4085 (99.5%) were conducted using the interactive software program designed for this study. The purpose of the observation was to gather information on teaching quality including classroom environment, teaching-learning methodology, instructional practice of teachers, students' attitude towards learning, teacher-students interaction, management of physical objects, etc. 4. Student Endline Assessment. To assess student gain in achievement, we repeated the baseline student assessment at the end of the school year in the same 114 schools. The end line assessment was conducted in the same 114 schools as the baseline assessment. Out of 4886 total students across all selected schools, a total of 4489 students (92%) participated in the Math subject assessment, 4374 students (90 %) participated in the Nepali subject assessment, and 4452 students (91%) participated in Science subject assessment. 5. Basic Level Examination (BLE) Test Score. To cross verify our endline assessment scores with metro-level test scores we also collected basic level examination (BLE) test scores from the education section of the Bharatpur Metropolitan City (BMC). BLE is a district level standardized test administered to 8th graders at the end of school year. We collected Nepali, Math, and Science test scores for 4746 students from 114 schools who participated in the BLE. Out of 4886 total students across all selected schools, a total of 4647 students (95%) participated in the Math subject assessment, 4667 students (95 %) participated in the Nepali subject assessment, and 4666 students (95%) participated in Science subject BLE test. 6. Student Survey. We conducted 45-minute student interviews in a private setting either at the student's home or a location where the student was comfortable answering the survey questions. This interview was conducted in Nepali using a computer-assisted personal interviewing program specifically designed for this survey. This survey measured students' backgrounds, knowledge, aspirations, satisfaction, and perception of classroom environment and teaching quality. Out of 4886 eligible students, we completed the interview with 4624 students resulting in a response rate of 95%. 7. Parent Survey. We conducted 60-minute interviews of students' parents in a private setting either at the respondent's home or a location where the respondent was comfortable answering the survey questions. This interview was conducted in Nepali using a computer-assisted personal interviewing program specifically designed for this survey. This survey includes household background measures of household size, composition, socio-economic background (ethnicity, social status), education and occupation, wealth, assets and income. The survey also includes individual-level measures such as parents' perceptions of teaching quality, parental action related to gathering information about alternative schools, and barriers/facilitators to exercising school choice, and awareness and participation in civil society organizations seeking to influence governance of education. Out of 4886 eligible parents, we completed this interview with 4611 parents resulting in a response rate of 94%. 8. Teacher Survey. We administered a 60-minute survey to teachers of the 8th graders of all 114 schools in a private setting either at the teacher's home or a location where the teacher was comfortable answering the survey questions. This interview was conducted in Nepali using a computer-assisted personal interviewing program specifically designed for this survey. This survey consists of a life history calendar (LHC) that records teachers' experiences annually as well as a survey portion that includes measures of teachers' background, knowledge, training, instructional practice, classroom management, and parental outreach and reporting. Out of 948 eligible teachers, 906 teachers were interviewed resulting in a response rate of 95%. 9. Principal and School Survey. We administered a 90-minute principal and school survey consisting of three parts: principal interview (LHC and survey portions), school history calendar (SHC), and school finance. The principal's LHC includes annual records of the principal's experience. The survey portion includes measures of principal's performance, information, and enforcement. The SCH portion includes updates to the existing SHC that we last updated in 2015. The SCH measures include number of students, number of boys and girls, male and female teachers, non-teaching staff, medium of instruction, and grades offered. The school finance section includes details on income, resource flow and disbursement. The survey was conducted in Nepali using a paper and pencil instrument by ISER-N senior research staff. Out of 114 schools, 113 principals and schools were interviewed resulting in a response rate of 99%. 10. School Management Committee (SMC) Survey. We administered a 45-minute survey to representatives of the school management committee (community school) and school board (in |
| Year(s) Of Engagement Activity | 2018,2019,2020 |
| URL | http://idea.isernepal.org/category/activities/ |
| Description | Orientation Workshop with Local Government |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Policymakers/politicians |
| Results and Impact | This workshop had two objectives: (i) to share the project research objectives/implementation plan and obtain participant feedback, and (ii) to ensure local government engagement and support. The workshop was conducted at the Bharatpur Metropolitan City (BMC) office. Eight BMC officials attended: the deputy mayor, chief administrative officer, joint secretary, undersecretary, education division, school inspector, section officer, and information officer. Five research staff from the Institute for Social and Environmental Research, Nepal (ISER-N) also participated in the capacity-building workshop. Dr. Ghimire, PI, presented a detailed introduction of the project, including information on the background and rationale, objectives, research design, implementation, and capacity-building plan. His presentation was followed by a discussion about the research plan and participant feedback on the implementation plan. At the conclusion of this workshop, BMC made the following requests: (i) to expand the study area to include other parts of the BMC, (ii) to extend technical support to update school location maps and records, and (iii) to have ISER-N serve as expert advisors for other BMC programs. Overall, the orientation workshop with the local government was highly successful in fostering the beginning of a collaborative relationship between our research program and the local government in BMC. |
| Year(s) Of Engagement Activity | 2018 |
| URL | http://idea.isernepal.org/meeting-conducted-at-bharatpur-metropolitan-city/ |
| Description | Research Dissemination Workshop |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Professional Practitioners |
| Results and Impact | In total, 184 individuals were invited to participate in the dissemination workshop. Individuals involved in social science, education research and extension, policy formulation, and program implementation representing government and non-governmental development and research organizations participated in the workshop. Altogether, 64 individuals attended the workshop. Presentations sparked questions and discussions, after which participants expressed increased interest in the empirical data and potential uses of the data. One participant asked about the top three policy messages from the research, summarized below: 1. Clear socio-economic and gender differences are observed in school enrollment. This fact should be realized by policy makers and efforts should be taken to equalize school enrollment; 2. Changing the governance structure of private schools away from a purely for profit enterprise to a non-profit organization could lead to greater equality in access to quality education; and 3. Organizational governance of schools and instructional quality seem to have important influence in determining learning quality. |
| Year(s) Of Engagement Activity | 2021 |
| URL | https://idea.isernepal.org/research-dissemination-workshop-for-improving-learning-developing-measure... |
| Description | Stakeholder's Workshops |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Schools |
| Results and Impact | We conducted a stakeholder's workshop at ISER-N's central office in Chitwan. We conducted this workshop in three sessions. This workshop had three basic objectives (i) to share project objectives/implementation plan, ii) to obtain stakeholders feedback, and iii) to ensure stakeholder engagement and support. Altogether, a total of 101 school principals, head teachers and representatives, along with the chief of district education and coordination office, chief of BMC education division, former regional director of education, former district education officer, two BMC education advisors, and three school inspectors participated in these workshops. |
| Year(s) Of Engagement Activity | 2018 |
| URL | http://idea.isernepal.org/stakeholders-workshops/ |