Toward the Development of a Rigorous and Practical Classroom Observation Tool: The Uganda secondary school project

Lead Research Organisation: New York University
Department Name: Sch for Culture Education and Human Dev

Abstract

Worldwide, there has been growing interest in understanding the nature of quality education. A major key to this quest lies in what goes on inside classrooms, where children derive the bulk of their daily experiences in academic and social learning. While factors like the physical condition of the school building, textbooks, and teacher degrees play a role in children's learning and life outcomes, they are small and indirect. Teacher instructional practices and classroom processes, in terms of supportiveness and organization, play considerable roles in children's learning and well being outcomes. Yet, the focus of many attempts to improve (and evaluate) educational programs has been based on classic, though simple, input-output model. In other words, an intervention takes place, and then the change in child academic or social outcomes are measured. Studies of this type can be viewed as "black box" studies; they tell us little more than whether the program worked or not. They fail to provide us with insights on how to more effectively facilitate deeper learning. To do this, we first need to be able to effectively measure instructional practices and classroom processes.

The most accurate way of measuring instructional practices and classroom processes is with the use of observational methods. To date, available methods have been too labor-intensive and costly for large-scale evaluation studies or for use in daily practice. Reliable, valid, cost-effective, and practically useful tools are needed. Nowhere is this truer than in low-income and fragile states. This is the goal of the proposed investigation.

To achieve these ends, we capitalize on a large-scale experimental school and classroom-based intervention program undertaken in Ugandan public secondary schools by the World Bank (WB), in partnership with the Ministry of Education and Sports (MoES). In a second phase of this project, the WB enlisted New York University (NYU) to supplement the impact evaluation by examining the instructional practices and classroom processes with live observations using an innovative tool, known as TIPPS, before, in the middle, and at the end of the intervention year. Samples of these classrooms are also videotaped for more intensive analysis. This data provides a unique opportunity to further develop and validate an innovative, affordable, scalable, and practically useful tool for assessing teacher practices and classroom processes. It also has the potential to provide feedback to teachers, especially when used in tandem with mentoring and reflected practice for improved teacher performance. We conduct a series of scientific studies to assure the viability, validity, and utility of this instrument.

In addition to the development and validation of an effective classroom observational instrument, we want to assure its use in policy and practice in Uganda and eventually in other low-income and fragile states. Thus, we begin the project year by working closely with the various stakeholder groups - ministry, union officials, school administrators, teachers, and World Bank Africa Region staff - to facilitate buy-in and ownership. We will engage them in interviews and focus groups to both inform them about the instrument and to gain their assistance in structuring the end of the year workshops for maximum effectiveness. The goals of these workshops are to explain our findings with regard to the intervention and the tool, and more importantly, so that the tool can be implemented at policy levels by the ministry, with the aide of the unions. In this manner, this tool could then be put into practical use in secondary schools around the country, and eventually primary schools as well.

Planned Impact

In recent years, Uganda has faced challenges in its education system due the advent of Universal Secondary Education (USE). Maximizing teacher impact in the current environment has consistently emerged as an area of focus. Perhaps the most vocal around this issue has been that the ministry itself, expressing the desire for an evidenced-based understanding of the teaching and learning that takes place in Ugandan classrooms. Currently, the MoES continues to cultivate its Digital Science Initiative, a major goal of which is to create a catalogue of classroom content to capture Ugandan teaching practices. Systematic analysis of classrooms through live observation brings one level of interest for the ministry, but the fact that this proposed study is accompanied by a rich library of classroom video content places our analysis at a distinct advantage in terms of depth of insight we can provide. This proposal has the ability to provide a critical window into the instructional practices and classroom processes of interventions as well as everyday practice.

However, it is of paramount importance that a clear message be communicated around teacher observations - both live and video. The recent trend of declining educational outcomes has created pressures for teachers, compounded following the teacher strikes in September 2013. The Ministry of Finance has allocated additional funds for teacher pay raises - a boon for the teacher's union - but this has subsequently put the entire education sector under severe scrutiny to perform well. Classroom observations are not meant to be evidence for punitive action; they should not be used to create a culture of fear. Thus, the inclusion of Uganda National Teacher's Union (UNATU) in these dialogues is necessary to ensure appropriate ownership by another set of key stakeholders - the teachers. We would be remiss not to mention the views of teachers we have encountered through this study, who have often shown as much interest in feedback from observation as the ministry. The feedback mechanism has not been developed as a part of the current intervention, but the need and desire is there. In conjunction with mentoring and reflective practice, practices already somewhat in place through local Center Coordinating Tutors (CCTs), a feedback device for educational practitioners that has been validated could be invaluable for improving practices. Yet in doing so, we must be able to uphold and ensure the rights and privacy of the teachers throughout the process.

We will engage these two key beneficiaries in a consultative process throughout the life of this project. We will begin by cultivating relationships with the groups respectively (i.e. introductory meetings with the MoE and UNATU) at the onset of the project in order to provide individualized attention and sensitization to the study. The culmination of conversations and data analyses would be presented in a workshop/seminar with key stakeholders (i.e., Ministry of Education and Sports, Uganda National Teacher's Union (UNATU), World Bank, civil society organizations), where we expect to bring a common level of understanding around the need, significance, and practicality around classroom observation as well as to bring some consensus on how to collaboratively address any obstacles (see "Pathways to Impact" for further details). In this forum, we can collectively consider and address concomitant issues such as cost and sustainability, understanding that classroom observations are generally more demanding in terms of time and financial resources. Nevertheless, the trade-offs are worthwhile when we consider the countless millions that have been spent in Uganda on other teaching and learning inputs that have not shown any real impact in improving student outcomes. The message must be conveyed that government, and Uganda as a whole, cannot afford to continue to let money go to waste with no results in student learning outcomes.

Publications

10 25 50
 
Description We begin with a description of the empirical findings, and then turn to how this work has begun to infuse our partner NGO organization and the ministry of education collaborators in Uganda and beyond.

Empirical Findings: As previously noted, TIPPS looks into assessing quality of instruction in Ugandan secondary school classrooms. The first major finding is the we uncovered a conceptually meaningful and interesting 3-factor structure. Initially, this structure was revealed in an exploratory factor analysis (EFA) of the 2013 TIPPS data, with modifications based on the fit parameters, confirmed in a confirmatory factor analysis (CFA) using the 2014 TIPPS data: (i) Instructional Strategies, (ii) Sensitive and Connected Teaching, and (iii) Deeper Learning. The second major finding is that the Deeper Leaning factor was concurrently associated with academic performance. More specifically, performance in Math and English on the National Assessment of Progress in Education (NAPE) was significantly predicted by TIPPS factors, and accounted for primarily by the Deeper Learning factor. Interestingly, the directionality is opposite in these two subjects. Deeper Learning is associated with significantly higher math performance and lower English performance. For Biology, there were no significant findings. Delving deeper into the concepts revealed that the Scaffolding concept was carrying the weight of the Deeper Learning factor. That is, as teachers engage in more scaffolding their classes perform better in Math. When we turn to English classes, the concept that is most strongly associated with performance is the provision of Specific feedback, while Connects activities and subject matter to achieve larger instructional concepts and objectives was inversely associated with average classroom English performance. Collectively, these findings suggest a specificity hypothesis that particular instructional strategies are associated differentially with performance in different subjects.

Impact in Uganda and beyond: The highlight of our impact in Uganda was the series of workshops we co-led with STiR Education, which was attended by hundreds of teachers, administrators, educational policy makers, union officials, including District Education Officers and District Inspectorate Services. We broke down the findings by regions and districts, and discussed what the findings meant for the different leaders and how they might implement them. We also spent considerable time discussing how TIPPS could be used in both in-service training and pre-service education. Our partnership with STiR and the ministry continues and has now expanded to India. We have, and continue to receive many requests for use of TIPPS regarding different developmental levels and cultural contexts. In particular, we developed the primary level TIPPS for a preliminary study with Kaivalya Education Foundation in India. As part of a collaboration with colleagues from University of Pennsylvania and Innovations for Poverty Action, we developed the pre-primary level TIPPS that was used as part of impact evaluation. Preliminary findings indicate improvement of the training program on TIPPS factors -- Supporting student expression and Emotional support and behavior management. We also partnered with Tele-Taleem Pakistan to propose a cluster randomized trial to measure teaching and learning in remote rural areas of Pakistan that was an unfunded finalist.
Exploitation Route We have been working with different academic and practitioner partners to utilize this research on measuring instructional practices and classroom processes in low and middle-income countries. We have and will continue to engage academic colleagues in the further development of this measure. In particular, we have presented at multiple academic forums, have papers under development for publication to disseminate these findings, and initiated discussions with scholars at several universities (e.g., American University-Sharjah, Fundação Getulio Vargas Brazil, University of California-Berkeley, University of Pennsylvania, New York University). In a proposal to ESRC's closed call under review, we plan to take this work forward. With practitioners, we are conducting and supporting colleagues in programmatic evaluation with TIPPS. Such work has been undertaken in Ghana and Uganda in pre-schools, in primary schools in India (KEF), and in secondary schools in Uganda and India (STiR Education). In particular, this work is informing our partners decision making, and the education ministries with whom they work closely, on developing pre-service training and in-service support using TIPPS as a feedback tool to improve performance. This is a shift from stand-alone teacher development initiatives to tailor training to improve the learning conduciveness of the classroom.
Sectors Education

URL http://steinhardt.nyu.edu/global-ties/advanced_methods/tipps
 
Description While on the research end, we continue to refine and understand the critical cultural contextualization that behooves comparative instruments like the TIPPS, the practice objectives have far exceeded our own expectations. There has been immense interest in this instrument from several educational sectors, and NGOs in particular. In several low-and-middle-income countries interest has been expressed in creating feedback mechanisms for the teachers. In the Ugandan context, the Ministry of Education and Sports, the teaching colleges and universities, and the teacher unions will be armed with additional information and techniques to further improve the quality of education in Ugandan classrooms. In part, these Pathways to Influence elements of the proposal will be facilitated by our emerging collaboration with STIR Education Uganda. We are currently working with them to enhance the stakeholders' workshop that is planned for May 2016 where a Learning Improvement Group will be held and include participants from District Education Offices and Departments (DEO, DIS), among others. Both STIR and the TIPPS team think there is much potential in encouraging more evidence-based practices among DEOs, DIS's, etc. in order to stimulate more of a focus on educational quality rather than inspection. In addition to Uganda, the TIPPS is already being used in Ghana and India, and is planned for use Pakistan. With the completion of the remaining planned analyses, the Ministry of Education and Sports, the teaching colleges and universities, and the teacher unions will be armed with additional information and techniques to further improve the quality of education in Ugandan classrooms.
First Year Of Impact 2015
Sector Education
Impact Types Cultural,Societal,Policy & public services

 
Title Adaptation of the TIPPS to different developmental levels and cultural contexts 
Description The TIPPS has been adapted for early childhood and primary grade classrooms in different linguistic/cultural contexts. 
Type Of Material Improvements to research infrastructure 
Year Produced 2015 
Provided To Others? Yes  
Impact We have, and continue to receive many requests for use of TIPPS regarding different developmental levels and cultural contexts. 
URL http://steinhardt.nyu.edu/global-ties/advanced_methods/tipps
 
Description NYU - STIR Education Uganda Partnership 
Organisation STIR Education
Country Uganda 
Sector Charity/Non Profit 
PI Contribution We are currently collaborating with STIR Uganda, in the short-run, to further develop, refine and implement the TIPPS as part of the STIR intervention program (aimed at teacher motivation, empowerment, and social network development) in the secondary schools of Eastern Uganda. We are also collaborating on the development of additional measures of instructional practices and classroom processes as well as a more extensive evaluation of the STIR Network Intervention.
Collaborator Contribution STIR Uganda is interested in understanding the processes at play in classrooms as well as the impact of formal and informal teacher networks in Eastern Uganda. We will take the lead on evaluation, while they will provide in-field support.
Impact Currently there are no outputs or outcomes.
Start Year 2016
 
Description NYU - Tele Taleem (T2) Collaboration 
Organisation TeleTaleem
Country Pakistan 
Sector Private 
PI Contribution We have been collaborating with Tele Taleem with regard to their rural "Learning Boost - Punjab" intervention in two different veins: (a) improving the quality of their observational feedback to teachers, by replacing their current methods with TIPPS-infused feedback; (b) creating an evaluation design to assess this large scale intervention. Together, we are seeking funds for both delivery of services and evaluation research associated with this program.
Collaborator Contribution Tele Taleem is the on-ground intervention delivery entity.
Impact Currently, there are no outputs or outcomes.
Start Year 2015
 
Description NYU-KEF(Kaivalya Education Foundation) Collaboration 
Organisation Kaivalya Education Foundation (KEF)
Country India 
Sector Charity/Non Profit 
PI Contribution The purpose of this project is two-fold: a) to adapt, contextualize and pilot test the Teacher Instructional Practices and Processes Systems (TIPPS) tool for use in government-run primary schools in India; and b) to pilot test the use of the TIPPS as a feedback device to improve pedagogical practices, classroom processes, and ultimately student outcomes. To date, we have adapted the TIPPS tool and trained KEF staff to administer the tool in public primary grade classrooms. KEF staff is currently coding 152 videotaped classrooms, shortly thereafter, we will analyze and write up the report of the data. We will then proceed to pilot test the use of the TIPPS as a feedback device to improve pedagogical practices, classroom processes, and ultimately student outcomes.
Collaborator Contribution KEF is the on-the-ground intervention team and serving as the in-field data collection partner.
Impact Currently, there are no outputs or outcomes, data collection is underway.
Start Year 2015
 
Description NYU-University of Pennsylvania -IPA Collaboration 
Organisation Innovations for Poverty Action (IPA)
Country United States 
Sector Charity/Non Profit 
PI Contribution We adapted the TIPPS instrument developmentally for early childhood classrooms in the Ghanaian context. Second, we trained colleagues at NYU and the University of Pennsylvania as master trainers to be able to train enumerators in Ghana to code classrooms videotapes using the TIPPS as part of IPA's "Improving Kindergarten Quality in Ghana" project in peri-urban Accra, Ghana. We will analyze the psychometric utility of the TIPPS in this context. If the instrument is psychometrically viable in this context, our colleagues at NYU and IPA will evaluate TIPPS as a mediating mechanism of student outcomes.
Collaborator Contribution IPA and our colleagues at New York University and the University of Pennsylvania are conducting the overall evaluation of this project. Our primary work was adapting, training, and analyzing the TIPPS per se, while our partners conducted the RCT and other data collection activities.
Impact As part of a multi-disciplinary team of economists, psychologists, and statisticians, we developed a Pre-Primary version of the TIPPS for an intervention program in Ghana. Again observer reliability was high (based on videotaped observations), and the emergent factor structure indicated a slightly different three-factor structure, along with strong concurrent and predictive validity. Additional data analyses are underway and the intervention is on-going.
Start Year 2015
 
Description The utility of observational tools in LAMIC classrooms for understanding and improving practice. Annual Conference Comparative and International Education Society. 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact This presentation was given at the 60th Annual Conference Comparative and International Education Society on 9 March 2015 as part of a panel presentation on the use and utility of classroom observational instruments in Low-and-Middle-Income Countries (LMICs) .
Dr. Edward Seidman, PI of the grant was the chair and organized this panel presentation, Sharon Kim and Mahjabeen Raza (grand secondments) also served on the panel and discussed TIPPS findings from Uganda.
Year(s) Of Engagement Activity 2016
URL https://ww2.eventrebels.com/er/CFP/AgendaAtAGlance.jsp?CFPID=506&ScreenID=206&DisplayPresenterID=729...
 
Description Describing Profiles of Instructional Practice: A new approach to analyzing classroom observational data using TIPPS data from Uganda 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Dr. Peter Halpin, Co-I presented the paper, "Describing Profiles of Instructional Practice: A new approach to analyzing classroom observational data using TIPPS data from Uganda" at the Workshop on Measurement of Basic Education Outcomes in Washington D.C.
Year(s) Of Engagement Activity 2015
URL http://www.worldbank.org/en/events/2015/05/12/workshop-on-measurement-of-basic-education-outcomes#2
 
Description Teacher-Child interactions in early education classrooms: What are we learning from measuring them in different cultural contexts? Annual Conference Comparative and International Education Society. 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Dr. Seidman, PI of the study served as the discussant for the panel presentation titled, "Teacher-Child interactions in early education classrooms: What are we learning from measuring them in different cultural contexts?" at the Annual Conference Comparative and International Education Society in Vancouver, BC
Year(s) Of Engagement Activity 2016
URL https://ww2.eventrebels.com/er/CFP/AgendaAtAGlance.jsp?CFPID=506&ScreenID=206&DisplayPresenterID=759...
 
Description The Teacher Instructional Practices and Processes System: An Overview and Implementation Guidelines 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Dr. Seidman, Sharon Kim and Mahjabeen Raza presented at the Workshop on Measurement of Basic Education Outcomes. The presentation was titled, "The Teacher Instructional Practices and Processes System: An Overview and Implementation Guidelines"
Year(s) Of Engagement Activity 2015
URL http://www.worldbank.org/en/events/2015/05/12/workshop-on-measurement-of-basic-education-outcomes#2
 
Description WEBINAR: THE UTILITY OF OBSERVATIONAL TOOLS IN LAMIC CLASSROOMS 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact Edward Seidman, Mahjabeen Raza and Sharon Kim presented TIPPS findings in Uganda as well as utility of the instrument in low and middle income countries.
Year(s) Of Engagement Activity 2015
URL http://www.teachersforefa.unesco.org/v2/index.php/en/newss/item/507-webinar-the-utility-of-observati...