Toward the Development of a Rigorous and Practical Classroom Observation Tool: The Uganda secondary school project

Worldwide, there has been growing interest in understanding the nature of quality education. A major key to this quest lies in what goes on inside classrooms, where children derive the bulk of their daily experiences in academic and social learning. While factors like the physical condition of the school building, textbooks, and teacher degrees play a role in children's learning and life outcomes, they are small and indirect. Teacher instructional practices and classroom processes, in terms of supportiveness and organization, play considerable roles in children's learning and well being outcomes. Yet, the focus of many attempts to improve (and evaluate) educational programs has been based on classic, though simple, input-output model. In other words, an intervention takes place, and then the change in child academic or social outcomes are measured. Studies of this type can be viewed as "black box" studies; they tell us little more than whether the program worked or not. They fail to provide us with insights on how to more effectively facilitate deeper learning. To do this, we first need to be able to effectively measure instructional practices and classroom processes.

The most accurate way of measuring instructional practices and classroom processes is with the use of observational methods. To date, available methods have been too labor-intensive and costly for large-scale evaluation studies or for use in daily practice. Reliable, valid, cost-effective, and practically useful tools are needed. Nowhere is this truer than in low-income and fragile states. This is the goal of the proposed investigation.

To achieve these ends, we capitalize on a large-scale experimental school and classroom-based intervention program undertaken in Ugandan public secondary schools by the World Bank (WB), in partnership with the Ministry of Education and Sports (MoES). In a second phase of this project, the WB enlisted New York University (NYU) to supplement the impact evaluation by examining the instructional practices and classroom processes with live observations using an innovative tool, known as TIPPS, before, in the middle, and at the end of the intervention year. Samples of these classrooms are also videotaped for more intensive analysis. This data provides a unique opportunity to further develop and validate an innovative, affordable, scalable, and practically useful tool for assessing teacher practices and classroom processes. It also has the potential to provide feedback to teachers, especially when used in tandem with mentoring and reflected practice for improved teacher performance. We conduct a series of scientific studies to assure the viability, validity, and utility of this instrument.

In addition to the development and validation of an effective classroom observational instrument, we want to assure its use in policy and practice in Uganda and eventually in other low-income and fragile states. Thus, we begin the project year by working closely with the various stakeholder groups - ministry, union officials, school administrators, teachers, and World Bank Africa Region staff - to facilitate buy-in and ownership. We will engage them in interviews and focus groups to both inform them about the instrument and to gain their assistance in structuring the end of the year workshops for maximum effectiveness. The goals of these workshops are to explain our findings with regard to the intervention and the tool, and more importantly, so that the tool can be implemented at policy levels by the ministry, with the aide of the unions. In this manner, this tool could then be put into practical use in secondary schools around the country, and eventually primary schools as well.

In recent years, Uganda has faced challenges in its education system due the advent of Universal Secondary Education (USE). Maximizing teacher impact in the current environment has consistently emerged as an area of focus. Perhaps the most vocal around this issue has been that the ministry itself, expressing the desire for an evidenced-based understanding of the teaching and learning that takes place in Ugandan classrooms. Currently, the MoES continues to cultivate its Digital Science Initiative, a major goal of which is to create a catalogue of classroom content to capture Ugandan teaching practices. Systematic analysis of classrooms through live observation brings one level of interest for the ministry, but the fact that this proposed study is accompanied by a rich library of classroom video content places our analysis at a distinct advantage in terms of depth of insight we can provide. This proposal has the ability to provide a critical window into the instructional practices and classroom processes of interventions as well as everyday practice.

However, it is of paramount importance that a clear message be communicated around teacher observations - both live and video. The recent trend of declining educational outcomes has created pressures for teachers, compounded following the teacher strikes in September 2013. The Ministry of Finance has allocated additional funds for teacher pay raises - a boon for the teacher's union - but this has subsequently put the entire education sector under severe scrutiny to perform well. Classroom observations are not meant to be evidence for punitive action; they should not be used to create a culture of fear. Thus, the inclusion of Uganda National Teacher's Union (UNATU) in these dialogues is necessary to ensure appropriate ownership by another set of key stakeholders - the teachers. We would be remiss not to mention the views of teachers we have encountered through this study, who have often shown as much interest in feedback from observation as the ministry. The feedback mechanism has not been developed as a part of the current intervention, but the need and desire is there. In conjunction with mentoring and reflective practice, practices already somewhat in place through local Center Coordinating Tutors (CCTs), a feedback device for educational practitioners that has been validated could be invaluable for improving practices. Yet in doing so, we must be able to uphold and ensure the rights and privacy of the teachers throughout the process.

We will engage these two key beneficiaries in a consultative process throughout the life of this project. We will begin by cultivating relationships with the groups respectively (i.e. introductory meetings with the MoE and UNATU) at the onset of the project in order to provide individualized attention and sensitization to the study. The culmination of conversations and data analyses would be presented in a workshop/seminar with key stakeholders (i.e., Ministry of Education and Sports, Uganda National Teacher's Union (UNATU), World Bank, civil society organizations), where we expect to bring a common level of understanding around the need, significance, and practicality around classroom observation as well as to bring some consensus on how to collaboratively address any obstacles (see "Pathways to Impact" for further details). In this forum, we can collectively consider and address concomitant issues such as cost and sustainability, understanding that classroom observations are generally more demanding in terms of time and financial resources. Nevertheless, the trade-offs are worthwhile when we consider the countless millions that have been spent in Uganda on other teaching and learning inputs that have not shown any real impact in improving student outcomes. The message must be conveyed that government, and Uganda as a whole, cannot afford to continue to let money go to waste with no results in student learning outcomes.