Levelling the playing field: assessing through equity the quality of Chilean schools

Lead Research Organisation: University of Bristol
Department Name: Education

Abstract

This proposal builds on my PhD "Searching for fairer ways of comparing Chilean secondary schools performance: a mixed methods study investigating contextual value added approaches" (Muñoz-Chereau, 2013a). My research found that pupil intake and contextual features in Chilean secondary schools exert a very powerful influence on students' performance. Therefore, accounting for prior achievement, student background and school context, provides an estimation method that seeks to correct for the extremely unequal Chilean educational playing field. Due to the fact that pupils and teachers are not distributed randomly in Chile, individual student characteristics, are considered to be potentially confounded with those associated with classes or schools groups (Raudenbush & Bryk, 2002). This has proved especially true in the Chilean educational system (Manzi et al, 2011; Meckes & Bascopé, 2012). Given that value added is a technique designed to make fair comparisons between schools (OECD, 2008) by adjusting for various background factors and prior attainment by the individual child, my research challenges the routinely applied rankings implemented in Chile to assess school performance; develops value added models using Chilean secondary schools' data, and uses those results to identify schools practices associated with better performance. Its dissemination will deliver new insights into the quality of Chilean schools; increase local awareness that different 'value added' indicators are needed to assess the quality of schools, and provide methodological tools coupled with evidence-based descriptions of effective schools.
The aim of the fellowship is to improve education -primarily in Chile and potentially in other ODA listed countries in the region such as Argentina- by applying an equity lens when assessing students' performance using value added models. This is needed because although Chile in the last decade has reached the highest per capita average income in the Latin-American region (EEA, 2007), its wealth distribution remains one of the most unequal in the world (Zahler, 2011). Distribution of wealth and its associated inequality, vulnerability and social exclusion remain a serious concern in Chile (OECD, 2011). In this context education is a key element to stop perpetuating inequalities and support sustainable development goals, and the assessment of schools performance plays a crucial role.
During the fellowship, I will carry out activities described in detail in the work plan, including: building on the impacts of the PhD; collaborating with users; disseminating research findings and further extending my publications track record. I will also apply my research related skills, seek opportunities to improve my academic/career development; and conduct additional research directly linked to my PhD.
The outcome of the proposed activities will support social cohesion and will work against educational segregation by describing a more favourable picture of schools that educate pupils coming from more disadvantaged socioeconomic backgrounds. This will help shifting the debate from type of school (public/private), to teaching and quality of school practices. Given that this is not part of the current national assessment framework, Chilean educational policies and practices oriented to tackle educational inequalities will be enhanced and supported. Moreover, recent educational reforms concerning a new evaluation system for schools (Sistema de Evaluación y Orientación del Desempeño de los Establecimientos y sus Sostenedores) as well as a reform for supporting public education (Sistema de Educación Pública) currently under discussion in Parliament, makes this proposal a timely contribution to inform and support these new developments. Finally, by combining equity and quality, Chilean educational system will be provided with information to become a fairer educational system.

Publications

10 25 50

publication icon
Muñoz-Chereau B (2018) Exploring gender gap and school differential effects in mathematics in Chilean primary schools in School Effectiveness and School Improvement

 
Description 1.Significant new knowledge: stakeholders' understandings of value added measures and indicators. Although research on value-added indicators for the Chilean school system has been conducted over the last decade, it has impacted very little the policy arena. Therefore I explored qualitatively the policy context in which this omission takes place, by analysing policy documents and interviewing local policymakers and researchers about their views on value-added measures. Both these and official documents pointed out legal, political, methodological, ethical and pragmatic reasons for and against value-added indicators. Overall, more challenging issues than opportunities of including these indicators into the new accountability system were identified. I have started the dissemination of these results through an impact case story at University of Bristol, and through a paper to be submitted to a high impact factor journal in my field (such as School Effectiveness and School Improvement).


2.Significant new knowledge: Despite the systematic recognition of large gender gaps against girls in mathematics performance in Chile, the role of schools in explaining this gap has been comparatively overlooked. I explored within and between school variation of Chilean primary schools in promoting attainment and progress of girls and boys. In order to reflect the hierarchical nature of the educational data, I used Multilevel modelling with 163,044 students, nested within 3,355 schools, within 310 Municipalities to fit Raw, Contextualised Attainment and Value-Added models. The results indicated a very small but significant gender differential school effect in progress in Chilean primary schools. I disseminated these results through presenting at ICSEI 2018 international conference, a policy Bristol Policy Brief http://bristol.ac.uk/media-library/sites/policybristol/documents/PolicyBristol_Briefing_October_2017_Chile_Schools.pdf, a blog post at the School of Education's research Centre for Assessment and Evaluation Research in Education (CAERe) http://www.bristol.ac.uk/education/news/2017/research-conducted-at-the-soe-focusing-on-value-added-measures-of-school-perform.html, and a paper submitted to School Effectiveness and School Improvement journal.

3.New research networks/collaborations/partnerships: I collaborated with users through a placement at the Unit of Curriculum and Evaluation, Ministry of Education (MoE) and cooperated with Líderes Educativos and Agency of Quality-Chile. During this fieldwork in Chile I worked with local policymakers, particularly at the Ministry of Education and the Quality Agency in Education supporting government's thinking around innovative school accountability and school improvement methods to account for inequality when assessing school performance.

4. Improved research capability generated from training oriented to acquiring advanced research skills in quantitative methods (Introduction to Multilevel Modelling Using MLwiN, 9-11 January 2017, University of Bristol; Introduction to STATA, 23rd January 2017, University of Manchester; Introduction to Experimental Design, 12th June 2017, University of Manchester) and qualitative methods (Qualitative analysis: methodologies for analysing text and talk, 10-20 July 2017, University of Essex; Qualitative data: sharing and reusing secondary data, 10-14 July 2017, University of Essex).
Exploitation Route My research outcomes can be taken forward by local policymakers. I disseminated my research and supported the work on assessment at the Quality Agency of Education (QAE). After delivering 3 seminars (two face-to-face ones and one webinar) with stakeholders, the QAE commissioned a policy brief that drew on my research on assessment incorporating an equity perspective. My work was mainly oriented to change their understandings around the approach and methods used. Overall, the potential value of my research for them is the provision of fairer measures of school performance, which in turn can support school improvement efforts. I also discussed with stakeholders a policy briefing -written with the support from PolicyBristol in English and Spanish.

My work may also impact Chilean schools (Head teachers and Teachers). The seminars I conducted were oriented at head-teachers and teachers working in disadvantaged backgrounds (i.e some state-funded schools). The potential value that can be gained from my research is that value added models applied to education describe a more favourable picture of schools that educate pupils coming from more disadvantaged socioeconomic backgrounds. In this way I promoted a reduction of a sense of failing.
Sectors Education

URL http://www.bristol.ac.uk/research/impact/educational-equity-chilean-schools
 
Description The work supported by this award is intending to increase the effectiveness of education by the incorporation of value-added models into the Chilean school accountability system. However, the current methodology for classifying schools used in Chile does not only exclude value-added approaches, but also has recently limited the possibility of implementing longitudinal approaches to assess school effectiveness, by reducing by half the amount of testing at the national level. Yet this work provided accessible examples of how value-added indicators can be used formatively to complement school self-evaluation by monitoring students' progress and support school improvement efforts. The dissemination of this work across varied user communities has facilitated potential national and regional impact through different engagement activities described in detail, such as one interview for Líderes Educativos' bulletin; participation as a discussant at the International Seminar 'Schools that learn together' organised by Líderes Educativos, University of Valparaíso and Agency of Quality, one webinar (2nd. November, 2017, University of Concepción; 2 seminars with Head Teachers, teachers and policymakers (one in Concepción on 4th November 2017 with 50 participants (18 Head Teachers, 22 Deputy Heads and 15 policymakers) and one in San Felipe, 9th November 2017, with 55 participants (21 Head Teachers, 24 Deputy Heads and 10 policy makers); a policy brief commissioned by Quality Agency of Education (QAE). Also in 2018 I organised the international seminar 'School Differential Effects' at UCL Institute of Education (14th Nov. 2018) and from 15-19 October 2018, I trained Chilean researchers from the Faculty of Education at University Catholic of Concepcion who aims to estimate the value-added by their institution to the initial teaching training of their undergraduate students. By training Chilean researchers on value-added methods to assess educational effectiveness, I supported the potential implementation of value-added models to assess the quality of a Chilean Higher Education Institution.
First Year Of Impact 2017
Sector Education,Government, Democracy and Justice
Impact Types Societal

 
Description Quality Agency of Education policy brief
Geographic Reach South America 
Policy Influence Type Participation in a guidance/advisory committee
 
Description Training of researchers at Faculty of Education Catholic University of Concepción on value-added in Higher Education
Geographic Reach South America 
Policy Influence Type Influenced training of practitioners or researchers
 
Title Interview guideline 
Description As part of the research directly linked to my PhD, I conducted interviews with users (policymakers and researchers) aiming to identify issues -such as difficulties and constraints- faced when encountering and using value added indicators. 
Type Of Material Physiological assessment or outcome measure 
Year Produced 2018 
Provided To Others? Yes  
Impact This novel semi-structured guideline can be used for collecting qualitative data. 
 
Title Qualitative interview data 
Description As part of the research directly linked to my PhD, I conducted interviews with users (policymakers and researchers) aiming to identify issues -such as difficulties and constraints- faced when encountering and using value added indicators. This qualitative dataset is formed by 5 semi-structured interviews conducted in Spanish with local policymakers and researchers. 
Type Of Material Database/Collection of data 
Year Produced 2018 
Provided To Others? Yes  
Impact The new knowledge generated by this award is underpinned by the analysis of this dataset. 
 
Description External examiner of PhD upgrate 
Organisation Pontifical Catholic University of Chile
Country Chile 
Sector Academic/University 
PI Contribution I have been invited to act as an external examiner of a PhD candidate Gonzalo Cifuentes' upgrate to take place on 17/03/20
Collaborator Contribution With chilean and international colleagues we will examine Gonzalo's upgrade proposal to work to apply value added measures in the Chilean educational system. He is a current student of the Faculty of Education, Pontificia Universidad Católica de Chile
Impact The output will be the PhD upgrade outcome of a student
Start Year 2019
 
Description Key local network 
Organisation Educational Leaders
Country Chile 
Sector Learned Society 
PI Contribution My collaboration with Lideres Educativos is directly linked to this award. I supported their work on the strand of Educational Leaders Professional Development working across the country, particularly on the use of data to inform the decisions of headteachers and other educational leaders to enhance school improvement.
Collaborator Contribution Lideres Educativos is a key local network- http://www.lidereseducativos.cl- they brought together primary and secondary schools, national and international universities, and non-profit foundations who share the conviction that public education in Chile must improve. EL also acts as a network for academic development and research in the country. They helped disseminating my work among non-academic users. They organised the seminars, invited participants and disseminated my work through their network.
Impact The main outputs resulting from this collaboration are: -I was interviewed by Líderes Educativos' bulletin, 11 April, 2017 https://www.lidereseducativos.cl/recursos/boletin-no1-2017. This was an opportunity to bring awareness to my research and support the professional opportunities provided by Lideres Educativos. My work provided input to one of their strands aiming to support head teachers and management teams in the use of data to inform their decisions. -I was invited to act as a discussant at the International Seminar 'Schools that learn together' organised by Líderes Educativos, University of Valparaíso and Agency of Quality Education, 24th October 2017, Valparaíso, http://www.agenciaeducacion.cl/noticias/inscribete-seminario-colegios-aprenden-juntos-nuevas-perspectivas-evaluar-orientar-la-mejora-territorio.
Start Year 2017
 
Description Mainstreaming Mental Health 
Organisation University of Leeds
Country United Kingdom 
Sector Academic/University 
PI Contribution I took part in an interview with the lead researcher Anna Madill to as one of the six non-mental health GCRF grants purposefully sampled for their study. We explored the extent to which my GCRF project engaged implicitly with psychosocial well-being impact.
Collaborator Contribution My research was disseminated as a case study in the paper Madill, A., Shloim, N., Brown, B., Hugh-Jones, S., Plastow, J., & Setiyawati, D. (2022). Mainstreaming global mental health: Is there potential to embed psychosocial well-being impact in all global challenges research?. Applied Psychology: Health and Well-Being.
Impact A paper was published reporting the main findings of the study.
Start Year 2021
 
Description BERA Expert pannel 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Policymakers/politicians
Results and Impact I was invited by Prof Gemma Moss, Prof Harvey Goldstein, Prof Pam Sammons, Prof Gordon Stobart, Sean Hayes and Gwen Sinnott to became a member of the BERA Expert panel 'Education Accountability Reform' which t is exploring alternatives to SATs. We will have a series of engagement activities during 2020
Year(s) Of Engagement Activity 2020
 
Description Blog post for CAERe Research Centre, University of Bristol 
Form Of Engagement Activity Engagement focused website, blog or social media channel
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact I wrote a blog post describing my research and its connection with the policy context in Chile. I brought attention to the fact that the new high-stake national school accountability system does not use value-added measures to take proper account of Chilean extremely unequal educational field. Developing approaches that take into account these inequalities when assessing school performance, remains one of the main challenges of the assessment of the quality of Chilean educational system in specific, and other DAC listed countries in the region in general where raw o status 'league tables' are a common practice.
Year(s) Of Engagement Activity 2017
URL https://caeregsoe.wordpress.com/2017/07/07/research-conducted-at-the-soe-focusing-on-value-added-mea...
 
Description Centre for Researching Education Across Boundaries (CREAB) Seminar, School of Education, University of Bristol 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Postgraduate students
Results and Impact I presented the main aims and methodology of my postdoctoral fellowship and how it built on my PhD. I argued that my research challenges the routinely applied rankings implemented in Chile to assess school performance; develops value added models using Chilean secondary schools' data, and uses those results to identify schools practices associated with better performance. This presentation enabled me to critically engage with colleagues and postgraduate students at the School of Education, University of Bristol.
Year(s) Of Engagement Activity 2016
 
Description Debate on International Seminar 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact I lead the debate between speakers in the International Seminar 'Schools that learn together' organised by Líderes Educativos, University of Valparaíso and Quality Agency of Education, 24th October 2017, Valparaíso. I lead the discussion towards the role of value-added indicators to improve school effectiveness and improvement.
Year(s) Of Engagement Activity 2017
URL https://www.lidereseducativos.cl/seminario-colegios-que-aprenden-juntos/
 
Description Impact story website 
Form Of Engagement Activity Engagement focused website, blog or social media channel
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact University of Bristol commissioned a journalist to write an impact case story from my research. It is showcased in the University's research impact website, which is visited by potential and current students, practitioners and the general public. It highlights the aim of the project, the related publications and partners. From this, colleagues hace contacted me interested in the funding application as well as in the outcomes of the research.
Year(s) Of Engagement Activity 2018
URL https://www.bristol.ac.uk/research/impact/educational-equity-chilean-schools/
 
Description Interview by Lideres Educativos bulletin 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Schools
Results and Impact I was interviewed by Lideres Educativos' bulletin published on 11 April, 2017 . This was an opportunity to bring awareness to my research and support the professional opportunities provided by Lideres Educativos. I described and commented on the main findings of my research and the way these linked to school effectiveness and improvement.
Year(s) Of Engagement Activity 2017
URL https://www.lidereseducativos.cl/recursos/boletin-no1-2017
 
Description Policy brief supported by Policy Bristol 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Policymakers/politicians
Results and Impact I discussed with stakeholders (policymakers and practitioners) at the Quality Agency Education (QAE) the policy briefing "Comparing Chilean school performance: improving fairness in the classification of schools" written with the support from PolicyBristol in English and Spanish. This increased the interest in my research and lead the QAE to commission a new policy brief that critically examines the role of assessment (formative and summative) for improving teaching and learning in the Chilean current high-stake test based accountability system.
Year(s) Of Engagement Activity 2017
URL https://www.bristol.ac.uk/policybristol/policy-briefings/chile-schools/
 
Description Promoting educational equity in Chilean schools- an impact case story at University of Bristol 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Media (as a channel to the public)
Results and Impact As part of a broader campaign to raise the profile of the University of Bristol work in the area of global development, I was asked to write a short case study of my individual research project. The case studies are hosted on the institutional website and may be used as part of a promotional campaign. I described the Global Challenge my project responds to, my motivation for working on this, the overall strategy to address the challenge, the potential impact, and my collaborators.
Year(s) Of Engagement Activity 2017
URL http://www.bristol.ac.uk/research/impact/educational-equity-chilean-schools
 
Description Seminar 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Schools
Results and Impact The seminar was delivered on 4th November 2017 at University of Concepcion). 50 participants took part in the activity (18 Head Teachers, 22 Deputy Heads and 15 policymakers). I disseminated my research findings.
Year(s) Of Engagement Activity 2017
 
Description Seminar 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Schools
Results and Impact I conducted a seminar (9th November 2017, San Felipe) with 55 participants (21 Head Teachers, 24 Deputy Heads and 10 policy makers). I disseminated findings of my research.
Year(s) Of Engagement Activity 2017
 
Description Seminar 'Accountability and school differential effects' at UCL Institute of Education and online 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact I organised the Seminar 'Accountability and school differential effects' as a member of the Centre for Educational Evaluation and Accountability at the UCL Institute of Education. Drawing on my research on differential effects I introduced the seminar (https://mediacentral.ucl.ac.uk/Player/40625978) and invited two academics (Professor Steve Strand from University of Oxford and Dr George Leckie from University of Bristol) who presented their work. The seminar was very well attended (i.e 40 participants) and the online video-recording of the presentations expanded the reach of the activity to an international level.
Year(s) Of Engagement Activity 2018
URL https://www.ucl.ac.uk/ioe/events/2018/nov/accountability-and-school-differential-effects
 
Description Webinar Seminars with policymakers, head-teachers, deputy heads and teachers 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Schools
Results and Impact I conducted a Webinar (2nd. November, 2017, University of Concepción) co-organised with Lideres Educativos and oriented to schools to disseminate the results of my research and discuss the use of value-added indicators for school assessment, improvement and self-evaluation. This was delivered within the general topic of how assessment (formative and summative) could enhance teaching and learning within a high-stake test based accountability system as the one under implementation in Chile.
Year(s) Of Engagement Activity 2017
URL https://www.lidereseducativos.cl/conversaciones-en-linea-evaluacion-formativa-mejorando-la-equidad-y...