Teacher Instructional Practices and Processes System (TIPPS): Cultural extension and testing as a feedback tool to improve pedagogical practices
Lead Research Organisation:
New York University
Department Name: NYU Steinhardt (Culture Edu & Human Dev)
Abstract
Research shows that effective teachers are the most important factor contributing to student achievement. Although curricula, reduced class size, funding, family, and community involvement all contribute to school improvement and student achievement, the most influential factor in the classroom is the teacher. Yet it has become clear that the professional development needed to support teachers and how effectively they function within classrooms is often lacking or ineffective. In many parts of the globe, teachers who need the most professional development (e.g. new or underqualified) often receive the least. Internationally, there is also a recognized need for improved instruments and methodologies to gauge elements of classroom quality and effective teaching. Particularly in low-income and fragile contexts, where many teaching personnel are underprepared and under resourced, more rigorously developed and culturally attuned observation tools have the potential to provide much needed feedback to teachers in a continuous cycle of improvement.
In our previous RLO grant (Toward the Development of a Rigorous and Practical Classroom Observation Tool: The Uganda secondary school project), we developed and validated the Teacher Instructional Practices and Processes System (TIPPS) with learning outcomes in secondary schools in Uganda. Using this observational tool, we examined the quality of teaching practices and classroom processes through live observations. Subsequently, we developed a pre-school version of the TIPPS in Ghana that was found to have meaningful associations with both learning and socio-emotional outcomes. We also piloted a primary school version of the TIPPS in India, where an NGO is using the TIPPS as a guide to provide teacher feedback. To date, we have not had the opportunity to systematically employ TIPPS as a feedback tool in a supportive fashion to improve teaching practices, student learning and teacher outcomes. Creating this cycle of continuous improvement is the goal of the present investigation, albeit in a new cultural context.
To test this in the Honduran context is an idea that grew organically, thanks in large part to the yearly gatherings of RLO colleagues that allowed for cross-pollination of ideas and discussions on topics of interest. Since our first RLO meeting in London, we have been speaking with Erin Murphy-Graham and her team about how we could join forces to augment the impact of the Sistema de Aprendizaje Tutorial program (Tutorial Learning System or SAT) in Honduras. The current proposal represents one-half of two parallel, collaborative but separate investigations. Murphy-Graham's proposal seeks a deeper understanding of which SAT pedagogical practices are effective (as assessed by the TIPPS) in impacting student learning and social and emotional outcomes, as well as how pedagogical practices effect teacher motivation by examining an intensive SAT condition without the addition of feedback. In this way, she hopes to recommend improvements to her partners in the SAT program. Our parallel proposal, which would operate in tandem with her existing intervention work, serves to further our objectives to both extend TIPPS' cross-cultural reach and systematically test its use as a feedback tool in the context of an optimally supportive structure. Using the SAT programming, we seek to develop an empirically-based, robust feedback mechanism that cultivates improvement in a continuous cycle of change
In our previous RLO grant (Toward the Development of a Rigorous and Practical Classroom Observation Tool: The Uganda secondary school project), we developed and validated the Teacher Instructional Practices and Processes System (TIPPS) with learning outcomes in secondary schools in Uganda. Using this observational tool, we examined the quality of teaching practices and classroom processes through live observations. Subsequently, we developed a pre-school version of the TIPPS in Ghana that was found to have meaningful associations with both learning and socio-emotional outcomes. We also piloted a primary school version of the TIPPS in India, where an NGO is using the TIPPS as a guide to provide teacher feedback. To date, we have not had the opportunity to systematically employ TIPPS as a feedback tool in a supportive fashion to improve teaching practices, student learning and teacher outcomes. Creating this cycle of continuous improvement is the goal of the present investigation, albeit in a new cultural context.
To test this in the Honduran context is an idea that grew organically, thanks in large part to the yearly gatherings of RLO colleagues that allowed for cross-pollination of ideas and discussions on topics of interest. Since our first RLO meeting in London, we have been speaking with Erin Murphy-Graham and her team about how we could join forces to augment the impact of the Sistema de Aprendizaje Tutorial program (Tutorial Learning System or SAT) in Honduras. The current proposal represents one-half of two parallel, collaborative but separate investigations. Murphy-Graham's proposal seeks a deeper understanding of which SAT pedagogical practices are effective (as assessed by the TIPPS) in impacting student learning and social and emotional outcomes, as well as how pedagogical practices effect teacher motivation by examining an intensive SAT condition without the addition of feedback. In this way, she hopes to recommend improvements to her partners in the SAT program. Our parallel proposal, which would operate in tandem with her existing intervention work, serves to further our objectives to both extend TIPPS' cross-cultural reach and systematically test its use as a feedback tool in the context of an optimally supportive structure. Using the SAT programming, we seek to develop an empirically-based, robust feedback mechanism that cultivates improvement in a continuous cycle of change
Planned Impact
Feedback to teachers has long been a professional development strategy, but the mechanism that works in different contexts and the roles different stakeholders play on the ground remains understudied. This project will provide valuable, contextual information on the dosage of feedback required, the length and composition of the cycle of continuous improvement needed, as well as the economic costs of implementing such a program and sustaining it. These are key details needed by local stakeholders on promising programming but oftentimes ones that are rarely provided. By tailoring the way research learning is provided to different stakeholders, we aim to bolster their impact - such that stakeholders are provided findings in a palatable, relevant manner. Indeed, the review of SAT programming by Brookings lists this effort to "maintaining discrete roles for each actor" as an effective way to keep partnerships. By working closely with the different stakeholders, maintaining open communication and discourse, as well as providing updates and opportunities for discussion at regular intervals, we will ensure that the direct beneficiaries are continuously in the know and availing the information. These beneficiaries include (i) the teacher-focused wing of the Secretary of Education's office, (ii) colleagues at the National Pedagogical University that actively work with the Honduran school system and SAT, (iii) education specialist staff from multilaterals supporting education delivery in Honduras (DFID, USAID, IADB, IBRD) and (iv) a host of non-governmental organizations involved in implementing the SAT program in Honduras.
The next set of beneficiaries is the network of NGOs that implement SAT across Latin America. A key consideration for them is the sensitive balance they maintain with the Government systems with which they work. SAT programming, while closely linked to government system, maintains independence of action. The examination of feedback structures in Honduras will generate learning about the way partnership with governing structures is maintained. For example, the buy-in of the governance structure is critically important if there are to be any improvements or advancements in education. Our plan for impact is to work with the Murphy-Graham team to create communication that integrates both SAT programmatic findings with feedback to teachers from our TIPPS protocol. Sending out information that is packaged in a way to be useful to other SAT implementers will allow them to better understand the different components, and more importantly the mechanisms, of the program and the way in which they function. We anticipate that sending out electronic newsletters will provide us more opportunities to engage with interested teams and to further impact. If demand exists, we could consolidate all project learnings into an easily accessible website that various partner projects could use to avail this information.
Finally, the TIPPS team is embedded in key research dissemination entities that allow us to be able to reach researchers, donors, multilaterals and practitioners working in the realm of international development and education. In particular, the research focused newsletters of NYU Steinhardt School of Culture, Education, and Human Development and Global TIES for Children have a global membership. Being part of both entities, we routinely use their outreach to share reports, notes and journal articles that both highlight the work as well as share learning with them. In addition, the TIPPS team also routinely presents at the annual CIES conference, and other industry conferences to increase the impact of our findings and learning.
The next set of beneficiaries is the network of NGOs that implement SAT across Latin America. A key consideration for them is the sensitive balance they maintain with the Government systems with which they work. SAT programming, while closely linked to government system, maintains independence of action. The examination of feedback structures in Honduras will generate learning about the way partnership with governing structures is maintained. For example, the buy-in of the governance structure is critically important if there are to be any improvements or advancements in education. Our plan for impact is to work with the Murphy-Graham team to create communication that integrates both SAT programmatic findings with feedback to teachers from our TIPPS protocol. Sending out information that is packaged in a way to be useful to other SAT implementers will allow them to better understand the different components, and more importantly the mechanisms, of the program and the way in which they function. We anticipate that sending out electronic newsletters will provide us more opportunities to engage with interested teams and to further impact. If demand exists, we could consolidate all project learnings into an easily accessible website that various partner projects could use to avail this information.
Finally, the TIPPS team is embedded in key research dissemination entities that allow us to be able to reach researchers, donors, multilaterals and practitioners working in the realm of international development and education. In particular, the research focused newsletters of NYU Steinhardt School of Culture, Education, and Human Development and Global TIES for Children have a global membership. Being part of both entities, we routinely use their outreach to share reports, notes and journal articles that both highlight the work as well as share learning with them. In addition, the TIPPS team also routinely presents at the annual CIES conference, and other industry conferences to increase the impact of our findings and learning.
People |
ORCID iD |
Edward Seidman (Principal Investigator) |
Description | Data analysis is currently underway. However, we have two early findings to report. 1. Cultural contextualization is also integral to classroom infused feedback to teachers. The process of contextualizing the TIPPS to the Honduran context required (i) cultural informants that are proficient in both the culture of the classroom and school, and (ii) a subset of items, targeted for their relevance to the context, SAT programming and teacher interest, must be chosen for feedback purposes. We also note that the interval between feedback sessions is an important component of the feedback process, however, given the COVID disruption, this study is unable to ascertain what is a suitable interval between feedback sessions. 2. The factor structure of the socio-emotional outcomes of students indicates that the theoretically hypothesized structure of socio-emotional sub-constructs (Self-efficacy, self-regulation, social awareness, peer connectedness). We surmise this is due to cultural difference between prevalent SEL literature and the Honduran context. This study makes an useful contribution to the SEL literature by mapping SEL in the Honduran context, especially since this data is from predominantly rural areas of the country. |
Exploitation Route | Interdisciplinary and cultural contextualized learning models can be used to develop educational materials and training programs to diverse populations. We find that cultural differences between the US, Uganda, India, Pakistan, Honduras, and now, as we progress in Nicaragua, are critical variables to be considered in both the classroom observation and feedback components. Linguistic changes are key to the adaptation process, but a considerable emphasis must be placed on the ensuring that the underlying concept must be included through that translation. Building a user friendly methodology and associated data analysis for real-time observation methods is crucial if classroom observations are used for feedback purposes. This project has created methodology and analysis that practitioners can use to ascertain reliability of data for feedback use. |
Sectors | Education Other |
Description | The impact of COVID has been decidedly negative on the routines of schools and classrooms. In the same vein, we anticipated that the enthusiasm and potential uptake of observation and accompaniment would wane. However, that was not the case. Prior to Honduran schools closing due to COVID, we had conducted a week-long intensive training on TIPPS and feedback mechanism that was attended by asesors and enumerators. We didn't anticipate how firmly rooted the methodology and promise of the TIPPS mechanism was in the cohort that was trained. In the ensuing years, some of those asesors are now supervisors and remain committed to the TIPPS styled observation process and feedback. This orientation and commitment to the process has led to sustaining efforts to keep schooling going through COVID. We have kept the Secretaría de Educación (Ministry of Education of Honduras) abreast of the project activities and work closely with the Honduran National Pedagogical University, and the Honduran Bayan Association in Honduras. These partners and affiliates are key stakeholders in Honduras, and afford us the opportunity to share and present findings in a way that can be readily used by them. In particular, the 2018-2030 Strategic Plan for the Educational sector of Honduras focuses on providing quality ongoing teacher professional development opportunities for in-service teachers. As Honduras looks to standardize the process of feedback and regularize the mechanism in teacher training, the commitment of supervisors and asesors to TIPPS provides a great policy recommendation. We will push for more visibility of this finding with ministry and university colleagues. Finally, to our knowledge, there are no behavioral observations to feedback-styled teacher professional development programming in place in remote and rural Honduran areas. The trust that supervisors and asesors demonstrate in this process indicates that the TIPPS infused feedback process could potentially be one of the first programming that scales in remote and rural areas. |
First Year Of Impact | 2023 |
Sector | Education,Other |
Impact Types | Cultural Societal Policy & public services |
Description | TIPPS Infused Feedback in the Nicaraguan context |
Geographic Reach | South America |
Policy Influence Type | Influenced training of practitioners or researchers |
Impact | To our knowledge, this is the first concerted effort to work with practitioners to integrate classroom instructional practices to support teacher professional development in the Nicaraguan context. The study has both piqued curiosity and encouraged practitioner partners to reimagine their programming with a classroom observation component. The initial PIASE study has been extended till 2025 because interest in this programming has been indicated by both practitioners and policy makers. |
Title | Education and Socio-Emotional Outcomes of students |
Description | This dataset contains Science, Math, and Socio-Emotional (SEL) Outcomes of 7th and 8th grade students in rural Honduras |
Type Of Material | Data analysis technique |
Year Produced | 2024 |
Provided To Others? | Yes |
Impact | These datasets are available in the data repository. |
Title | Reliability of Classroom Observations |
Description | Classroom observation data is unique in the way that it is collected, and then used for feedback purposes, we go beyond simply agreement between observers. The observer is trained to quantify the qualitative "Quality" of practices and processes taking place in the classroom (ie, this is not a static or binary measurement). Additionally, a four point Likert Scale employed in the TIPPS measure is a two-step decision making tree (based on the Harter Scale). The broader quality categorization (low or high) and then further refined quality distinction. Thus, the way to analyze classroom observation data is also distinct from usual scale analyses. We leverage Gwet's AC1, a psychometric analysis that accounts for guessing in the quality rating of the classroom. We also compute Omega and ICCs so as to understand the reliability from a multidimensional perspective. |
Type Of Material | Data analysis technique |
Year Produced | 2024 |
Provided To Others? | Yes |
Impact | This dataset is uploaded to the data repository. |
Description | NYU- UC Berkeley Partnership |
Organisation | University of California, Berkeley |
Country | United States |
Sector | Academic/University |
PI Contribution | This is a jointly, integrated and managed project with each institution playing a differential primary role in write-ups of outcomes. |
Collaborator Contribution | This is a jointly, integrated and managed project with each institution playing a differential primary role in write-ups of outcomes. |
Impact | Study still underway. |
Start Year | 2019 |
Description | NYU- UPNFM Partnership |
Organisation | Francisco Morazán National Pedagogical University |
Country | Honduras |
Sector | Academic/University |
PI Contribution | Full collaboration on survey data collection, coding and video-coding. |
Collaborator Contribution | Full collaboration on survey data collection, coding and video-coding. |
Impact | None as of yet. |
Start Year | 2020 |
Description | TIPPS Infused Feedback Workshop |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | On the request of the Platform for Research and Educational Action (PIASE) team, the TIPPS team did a series of informative presentations for a range of stakeholders in Nicaragua. There were two webinars where information on TIPPS and feedback using classroom observation was provided, followed by discussion (35-40 persons in attendance). Next, a three-day workshop was conducted to train a cadre of master trainers and stakeholders (N=30) on the TIPPS instrument. This workshop was also the venue through which the feedback process for use in the field was concretized. The TIPPS team also conducted two listening sessions (N=20) with different in-field partners to support implementation of TIPPS infused feedback on the ground. |
Year(s) Of Engagement Activity | 2023,2024 |
Description | Webinar |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | With the help of Bayan, UPFNM, University of California, Berkeley, a wide array of practitioners and policy makers from Honduras and elsewhere in Latin America (N~150) participated in a TIPPS Webinar. The Webinar generated a great deal of interest including from an educational organization in Nicaragua. |
Year(s) Of Engagement Activity | 2021 |