Supplementing measures of social accountability and enhancing understanding of accountability and students' achievement during the COVID-19 pandemic
Lead Research Organisation:
University of Michigan
Department Name: Institute for Social Research
Abstract
This augmentation project will enrich the parent project data funded by DFID-ESRC by adding measures of public expenditure and conducting a follow-up survey of schools, school management committees, and a sub set of students and parents from the parent project. The parent project has already made significant contribution to Nepali school education by developing and testing a set of tools to gather information about school performance and student educational outcomes.
Analysis of these data revealed strong associations between student learning and accountability measures such as parents' knowledge, engagement, and empowerment. However, because of the ongoing transition in Nepal's governance structure from a centralized to a local governance system, including in the education sector, it was not feasible to collect information about public expenditure and service delivery during the parent project. The lack of these important accountability measures has limited our findings. Additionally, since the parent project was completed, COVID-19 related school closures have dramatically changed the learning environment for schools, parents, and students. This change has been disruptive, negatively affecting some schools and students more than others. To enhance our social accountability measures and inform our understanding of how COVID-19 intersects with social accountability and student learning we propose the following four aims:
Aim One. Conduct a public expenditure tracking and service delivery survey and in-depth interview with representatives of local government bodies to add expenditure data to our measures of social accountability.
Aim Two. Conduct follow-up phone surveys with students, parents, school principals, and School Management Committees/School Boards (SMC/SB) that participated in the parent study to understand how COVID-19 related school disruptions have affected them and how disruptions have impacted student learning.
Aim Three. Collect School Education Examination (SEE) scores from Bharatpur Metropolitan City (BMC). This national exam taken at the end of grade 10 will be administered to students in our sample in March 2021.
Aim Four. Link the newly collected data with existing student, parent, and school-level data to estimate the effect of school disruptions and accountability during COVID-19 on student education outcomes. In particular, we will explore how dropout rates, attendance rates, and student achievement (SEE scores) have changed after COVID-19 compared to before the pandemic. We will consider factors like learning environment and support structures at home and school. We will also explore different school coping strategies and whether these strategies are correlated with accountability measures.
We will investigate how school disruptions have a greater negative impact on some students more than others by conducting our analyses among sub-groups of students. For example, we will compare males and females, advantaged
and disadvantaged ethnic groups, students with highly educated and less educated parents, students with and without parents who are international migrants, and students who performed well in earlier assessments compared to those who did not perform well.
This study will generate important scientific resources including: (1) measures of public expenditure tracking in a low income context; (2) follow-up measures of accountability from schools, school management committees, parents, and
students; and (3) scientific advancement in our understanding of the relationship between accountability and students' achievement during COVID-19. We will make these findings widely available to scientists and policy makers through local dissemination workshops to share findings of the study, making the data publicly available through ICPSR and the UK Data Service, and through presentations at national and international conferences and publications in scientific articles and policy brief.
Analysis of these data revealed strong associations between student learning and accountability measures such as parents' knowledge, engagement, and empowerment. However, because of the ongoing transition in Nepal's governance structure from a centralized to a local governance system, including in the education sector, it was not feasible to collect information about public expenditure and service delivery during the parent project. The lack of these important accountability measures has limited our findings. Additionally, since the parent project was completed, COVID-19 related school closures have dramatically changed the learning environment for schools, parents, and students. This change has been disruptive, negatively affecting some schools and students more than others. To enhance our social accountability measures and inform our understanding of how COVID-19 intersects with social accountability and student learning we propose the following four aims:
Aim One. Conduct a public expenditure tracking and service delivery survey and in-depth interview with representatives of local government bodies to add expenditure data to our measures of social accountability.
Aim Two. Conduct follow-up phone surveys with students, parents, school principals, and School Management Committees/School Boards (SMC/SB) that participated in the parent study to understand how COVID-19 related school disruptions have affected them and how disruptions have impacted student learning.
Aim Three. Collect School Education Examination (SEE) scores from Bharatpur Metropolitan City (BMC). This national exam taken at the end of grade 10 will be administered to students in our sample in March 2021.
Aim Four. Link the newly collected data with existing student, parent, and school-level data to estimate the effect of school disruptions and accountability during COVID-19 on student education outcomes. In particular, we will explore how dropout rates, attendance rates, and student achievement (SEE scores) have changed after COVID-19 compared to before the pandemic. We will consider factors like learning environment and support structures at home and school. We will also explore different school coping strategies and whether these strategies are correlated with accountability measures.
We will investigate how school disruptions have a greater negative impact on some students more than others by conducting our analyses among sub-groups of students. For example, we will compare males and females, advantaged
and disadvantaged ethnic groups, students with highly educated and less educated parents, students with and without parents who are international migrants, and students who performed well in earlier assessments compared to those who did not perform well.
This study will generate important scientific resources including: (1) measures of public expenditure tracking in a low income context; (2) follow-up measures of accountability from schools, school management committees, parents, and
students; and (3) scientific advancement in our understanding of the relationship between accountability and students' achievement during COVID-19. We will make these findings widely available to scientists and policy makers through local dissemination workshops to share findings of the study, making the data publicly available through ICPSR and the UK Data Service, and through presentations at national and international conferences and publications in scientific articles and policy brief.
| Description | We have discovered that the dropout rate three years after COVID-19 is associated with online courses provided by schools during lockdowns. Specifically, schools offering no or fewer online courses during the COVID-19 lockdowns experienced higher dropout rates among students. Additionally, we found that during the pandemic, students from all disadvantaged groups (Hill Janajati, Terai Janajati, and particularly Dalits) were more likely to drop out of school. |
| Exploitation Route | The data from this project provide crucial evidence for formulating effective education policy in Nepal. Specifically, the data show that the focus on improving academic performance in Nepali schools should move beyond debates about school control (public vs. private) and accountability, and instead should focus on practical ways to improve the quality of teaching in the classroom. The education community in Nepal likely has valuable insights into this issue and should be actively involved in developing and sharing well-thought-out programs aimed at improving teaching, with the ultimate goal of enhancing student learning. |
| Sectors | Communities and Social Services/Policy Education |
| Description | The project team, consisting of members from the University of Michigan (U-M) and partner organization in Nepal, the Institute for Social and Environmental Research-Nepal (ISER-N), continues to actively engage with policymakers and stakeholders at both local and national levels. Their goal is to maximize the impact of these research findings. Building on previous efforts, the team organized meetings to share their research findings with various entities, including the Bharatpur Metropolitan Education Section, the Chitwan Kachari, a national policy dialogue forum, and the Nepal Planning Commission, which is a national planning body. During these meetings, they explored potential ways to incorporate the findings into policy formulation and practice. This year, two specific findings garnered significant attention from policymakers. First, it was discovered that schools offering no or fewer online courses during the COVID-19 lockdowns experienced higher dropout rates among students. Second, there was a disproportionately higher dropout rate among disadvantaged groups, such as the Hill Janajati, Terai Janajati, and particularly among the Dalits. These findings prompted a strong response from policymakers, who are now more committed to improving online teaching facilities to address these disparities. The continued dialogue between the research team and policymakers aims to ensure that these insights lead to actionable improvements in educational access and equity. |
| First Year Of Impact | 2024 |
| Sector | Communities and Social Services/Policy,Education |
| Impact Types | Policy & public services |
| Title | Survey Questionnaires |
| Description | Please see details of the Student Survey, Parent Survey, Head Teacher Survey, and School Management Committee (SMC) Survey in the Engagement Activities (Instrument Design) section. |
| Type Of Material | Physiological assessment or outcome measure |
| Year Produced | 2024 |
| Provided To Others? | Yes |
| Impact | It is too early to describe notable impact. |
| URL | https://doi.org/10.5255/UKDA-SN-857220 |
| Title | Head Teacher Survey 2023 |
| Description | The Head Teacher dataset consists of two parts: the survey data and the school COVID calendar data. The dataset includes measures the principal's performance, information, and enforcement; actions taken by the school amidst the challenges posed by the COVID-19 pandemic; the management of classes; interactions with students; and the overall operational decisions made to ensure the smooth functioning of the educational institution during COVID-19. Out of 95 eligible head teachers, 79 head teachers completed the survey resulting in a response rate of 83.15%. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2024 |
| Provided To Others? | Yes |
| Impact | It is too early to describe notable impacts resulting from the development of this data. |
| URL | https://doi.org/10.5255/UKDA-SN-857220 |
| Title | Parent Survey 2023 |
| Description | This dataset consists of two parts: the survey questions and the Life History Calendar (LHC). The dataset includes household-level measures of household size, composition, socio-economic background (ethnicity, social status), education, occupation, wealth, assets, and income; challenges faced by parents of school children during the COVID-19 pandemic and its impacts on education; and responses and actions the school undertook for the students and their education. The data set also includes individual-level measures such as parents' perceptions of teaching quality, parental action related to gathering information about alternative schools, barriers/facilitators to exercising school choice, and awareness and participation in civil society organizations seeking to influence governance of education. The LHC data set include the child's parents' life history, including their birth, marriage, details about the child, household chores, employment, and education. Out of 2858 eligible parents, 2082 parents completed the survey resulting in a response rate of 72.84%. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2024 |
| Provided To Others? | Yes |
| Impact | It is too early to describe notable impacts resulting from the development of this data. |
| URL | https://doi.org/10.5255/UKDA-SN-857220 |
| Title | School Management Committee (SMC) Survey 2023 |
| Description | This dataset includes assessments of the school management committee's (for public schools) or the school board's (for private schools) duties and responsibilities, particularly focusing on their actions and responses during the COVID-19 pandemic. It measures various aspects of their accountability, including delegation of tasks, management, performance evaluation, information dissemination, pandemic response strategies, and enforcement of policies to ensure the effective functioning of the school amidst the challenges posed by COVID-19. Out of 95 eligible chairpersons, 78 chairpersons completed this survey resulting in a response rate of 82.10%. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2024 |
| Provided To Others? | Yes |
| Impact | It is too early to describe notable impacts resulting from the development of this data. |
| URL | https://doi.org/10.5255/UKDA-SN-857220 |
| Title | Student Survey 2023 |
| Description | This dataset consists of two parts: survey questions and a Life History Calendar (LHC). The Student Survey measures students' backgrounds, knowledge, aspirations, satisfaction, and perception of their classroom environment. It also measures teaching quality and the impact of lockdowns and remote learning on education during the COVID-19 pandemic, as well as the support provided by schools through online services or other means, and the time allocated for learning. Out of 2858 eligible students, 2052 students completed the survey resulting in a response rate of 71.8%. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2024 |
| Provided To Others? | Yes |
| Impact | It is too early to describe notable impacts resulting from the development of this data. |
| URL | https://doi.org/10.5255/UKDA-SN-857220 |
| Description | The University of Michigan & Institute for Social and Environmental Research-Nepal Partnership |
| Organisation | Institute for Social and Environmental Research - Nepal (ISER-N) |
| Country | Nepal |
| Sector | Charity/Non Profit |
| PI Contribution | The University of Michigan (U-M) team significantly contributed to this program of research, particularly on instrument design. The U-M team continues to contribute scientific and intellectual knowledge on research design and implementation. We have been working closely with the partner, ISER-N, to readjust the project schedule and work through data collection and data cleaning tasks. The U-M team also contributed to the program through monitoring and supervising research activities and through extending managerial and administrative support to ISER-N. Finally, the U-M team is actively conducting data analysis using the newly available data. U-M and ISER-N staff are well positioned to complete the analyses. |
| Collaborator Contribution | Our partner, ISER-N, has significantly contributed to this program of research. ISER-N staff was particularly busy implementing project activities, including instrument design, testing, and data collection using the Nepali instruments designed with significant stakeholder involvement. The completed datasets include: 1. Student Survey, 2. Parent Survey, 3. Head Teacher Survey, and 4. School Management Committee (SMC) Survey. Please see details of each dataset under Research Databases, Datasets & Models. |
| Impact | The U-M/ISER-N collaboration is multi-disciplinary and broadly includes the areas of demographic processes, sociology, education, and gender. The U-M/ISER-N partnership has generated a series of outputs. The first set of outputs includes newly designed instruments. The next set of outputs includes datasets for each survey. Instruments and corresponding datasets: 1. Student Survey, 2. Parent Survey, 3. Head Teacher Survey, and 4. School Management Committee (SMC) Survey. |
| Start Year | 2023 |
| Description | Consultative Meetings (Individual and Small Group) |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Professional Practitioners |
| Results and Impact | We shared our project objectives/implementation plan and obtained stakeholder feedback to facilitate stakeholder engagement and support. We held consultative meetings that involved (i) individual meetings with education experts and research scholars, and (ii) small group meetings with local stakeholders. The individual consultative meetings included meetings with the Chief of Education of Bharatpur Metropitan City (BMC), head teachers, education experts, university faculty, and research scholars. Small group meetings with stakeholders included a series of meetings with the Chief of District Education and the Coordination Committee, Chief of the BMC Education Office, the former Regional Director of Education, former District Education Officer, and School Inspectors. |
| Year(s) Of Engagement Activity | 2023 |
| Description | Instrument Design and Data Collection |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Schools |
| Results and Impact | We implemented a number of (a) instrument design activities and (b) data collection activities. We describe each below: (a) Instrument Design We used a rigorous, reiterative mixed methods approach to design data collection instruments. This approach combined substantive, methodological, and operational expertise into a single coordinated team, and guided all aspects of the instrument design process. Our instrument design process builds on relevant literature and our parent project experience. The following steps were completed for each instrument: (1) strategic assembly of an interdisciplinary team of research scientists, research staff, and stakeholders; (2) iterative translation/rewording and testing of specific measures; (3) in depth/cognitive interviewing with representatives of the study population to refine each instrument (both qualitative and quantitative); (4) repeated pre-tests and revisions of the rewording/translations; and (5) refinement and finalization. Instruments that we designed during this project include: 1. Parent Survey. This survey was designed following the same rigorous and reiterative process described above. This instrument consisted of two parts: the survey questions and the Life History Calendar (LHC). The survey questionnaire was programmed in Nepali using a computer-assisted personal interviewing program. The LHC, including the COVID calendar, was designed in paper and pencil format. The Parent Survey instrument includes household-level measures of household size, composition, socio-economic background (ethnicity, social status), education, occupation, wealth, assets, and income; challenges faced by parents of school children during the COVID-19 pandemic and its impacts on education; and responses and actions the school undertook for the students and their education. This survey instrument includes individual-level measures such as parents' perceptions of teaching quality, parental action related to gathering information about alternative schools, barriers/facilitators to exercising school choice, and awareness and participation in civil society organizations seeking to influence governance of education. The LHC measures include parents' life history, including their birth, marriage, details about the child, household chores, employment, and education. The Parent Survey questionnaire was pretested on 30 parents whose children were enrolled in school. 2. Student Survey. This survey was designed to measure students' backgrounds, their knowledge and satisfaction, and their perceptions of the classroom environment and teacher quality. The Student Survey instrument was largely drawn from the parent study, however, we also designed new COVID-19-related measures. Similar to the Parent Survey instrument, the Student Survey instrument also consists of two parts: survey questions and the LHC. The survey measures include students' backgrounds, knowledge, aspirations, satisfaction, and perception of their classroom environment. It also includes measures of teaching quality and the impact of lockdowns and remote learning on education during the COVID-19 pandemic, as well as the support provided by schools through online services or other means, and the time allocated for learning. The Student Survey questionnaire was pretested on 120 students. 3. Head Teacher Survey. This survey was designed to collect school-level measures and consists of two parts: the head teacher interview and school COVID calendar. The Head Teacher Survey instrument includes the performance of the head teacher, information, and enforcement; actions taken by the school amidst the challenges posed by the COVID-19 pandemic; the management of classes; interactions with students; and the overall operational decisions made to ensure the smooth functioning of the educational institution during the COVID-19 pandemic. 4. School Management Committee (SMC) Survey. This survey was designed to collect community-level measures. This survey encompasses assessments of the SMC's (at community schools) or the School Board's (at institutional schools) duties and responsibilities, particularly focusing on their actions and responses during the COVID-19 pandemic. The measures include various aspects of their accountability, including delegation of tasks, management, performance evaluation, information dissemination, pandemic response strategies, and enforcement of policies to ensure the effective functioning of the school amidst the challenges posed by the COVID-19 pandemic. Similar to the other surveys, measures were drawn from the parent project, and we designed new measures. The measures were designed by the same team of education experts, local stakeholders, and research staff strictly following ISER-N's instrument design protocol. (b) Data Collection The delays in the government approval process significantly delayed the start of the project and pushed several data collection activities to the second half of the year. Our original plan was to administer the survey to only 1618 students by taking a sub-sample of 35% of the students from the 4624 students in the parent project. However, since many of the schools have smaller class sizes than expected (and did not have enough students to select a representative sample), we carefully decided to include 50% of the students from the 113 original schools in the study and over-sample from ethnic minority groups and from schools with smaller class sizes. A total of 2858 students were eligible. This decision allowed the study to capture school-level variability and enhanced statistical power to generate reliable estimates to answer our research questions. The investigators reallocated the budget to increase the sample size to ensure enough statistical power to detect associations between students' learning achievement and accountability processes. As a result, our data collection activities differ from our original plan. An ethnically representative, well-trained, and highly experienced team of research staff was involved in the data collection. In addition, a high level of supervision and monitoring of field data collection was maintained throughout the data collection period. A summary of data collection activities is below: 1. Student Survey. We conducted 60-minute student interviews in a private setting, either at the student's home or a location where the student was comfortable answering the survey questions. This survey consisted of two parts: survey questions and a Life History Calendar (LHC). The interview was conducted in Nepali using a computer-assisted personal interviewing program. The LHC, including the COVID calendar, was designed in paper and pencil format. The Student Survey measures students' backgrounds, knowledge, aspirations, satisfaction, and perception of their classroom environment. It also measures teaching quality and the impact of lockdowns and remote learning on education during the COVID-19 pandemic, as well as the support provided by schools through online services or other means, and the time allocated for learning. Out of 2858 eligible students, 2052 students completed the survey resulting in a response rate of 71.8%. 2. Parent Survey. We conducted 60-minute interviews of students' parents in a private setting, either at the respondent's home or a location where the respondent was comfortable answering the survey questions. This survey consisted of two parts: the survey questions and the Life History Calendar (LHC). The interview was conducted in Nepali using a computer-assisted personal interviewing program. The LHC, including the COVID calendar, was designed in paper and pencil format. The Parent Survey includes household-level measures of household size, composition, socio-economic background (ethnicity, social status), education, occupation, wealth, assets, and income; challenges faced by parents of school children during the COVID-19 pandemic and its impacts on education; and responses and actions the school undertook for the students and their education. The survey also includes individual-level measures such as parents' perceptions of teaching quality, parental action related to gathering information about alternative schools, barriers/facilitators to exercising school choice, and awareness and participation in civil society organizations seeking to influence governance of education. The LHC additionally collects information about the child's parents' life history, including their birth, marriage, details about the child, household chores, employment, and education, and this data is collected using a paper and pencil instrument. Out of 2858 eligible parents, 2082 parents completed the survey resulting in a response rate of 72.84%. 3. Head Teacher Survey. We administered a 90-minute Head Teacher Survey, which consisted of two parts: the survey questions and the school COVID calendar. The interview was conducted in Nepali using a paper and pencil instrument. The Head Teacher Survey measures the principal's performance, information, and enforcement; actions taken by the school amidst the challenges posed by the COVID-19 pandemic; the management of classes; interactions with students; and the overall operational decisions made to ensure the smooth functioning of the educational institution during the COVID-19 pandemic. Out of 95 eligible principals, 79 principals completed the survey resulting in a response rate of 83.15%. 4. School Management Committee (SMC) Survey. We administered a 45-minute survey to representatives of the SMC (at community schools) or the School Board (at institutional schools). This survey encompasses assessments of the SMC's (or School Board's) duties and responsibilities, particularly focusing on their actions and responses during the COVID-19 pandemic. It measures various aspects of their accountability, including delegation of tasks, management, performance evaluation, information dissemination, pandemic response strategies, and enforcement of policies to ensure the effective functioning of the school amidst the challenges posed by the COVID-19 pandemic. The survey was conducted in Nepali using a paper and pencil instrument. Out of 95 eligible chairpersons, 78 chairpersons completed this survey resulting in a response rate of 82.10%. |
| Year(s) Of Engagement Activity | 2023 |
| Description | Orientation Workshop with the Local Government |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Policymakers/politicians |
| Results and Impact | Bharatpur Metropolitan City Program Advisory Committee Meeting: The main program meeting convened on March 21st, drawing attendance from the Deputy Mayor, Chief Administrative Officer, Assistant Chief Development Officer, Chief of Education Division, Social Welfare Division Chief, and eight other officials. This meeting was held to explain the study objectives, activities, potential areas of collaboration, and use of the study findings. This meeting was also intended to get the approval from the local authority for further processing of the federal-level approval from the Social Welfare Council. |
| Year(s) Of Engagement Activity | 2023 |
| Description | Stakeholders' Workshops |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Professional Practitioners |
| Results and Impact | This workshop was primarily designed to solicit feedback from the representatives of the School Management Committee (SMC), school administrator, and teachers on the survey instruments. The workshop discussed the way to collect information on the school closures, alternative teaching approaches used, the problems they faced during the COVID-19 pandemic, and the action they took as a representative of the school. Additionally, these instruments also explore the head teachers' strategies for handling school operations during the COVID-19 pandemic, including their approaches to conducting classes, managing staff, and ensuring the safety and well-being of students. The participants included teachers, head teachers, SMC representatives, and education officers of BMC. The diverse viewpoints enriched the discussions, providing feedback to improve our survey instrument and project activities. |
| Year(s) Of Engagement Activity | 2023 |