Levelling the playing field: assessing through equity the quality of Chilean schools
Lead Research Organisation:
University of Bristol
Department Name: Education
Abstract
This proposal builds on my PhD "Searching for fairer ways of comparing Chilean secondary schools performance: a mixed methods study investigating contextual value added approaches" (Muñoz-Chereau, 2013a). My research found that pupil intake and contextual features in Chilean secondary schools exert a very powerful influence on students' performance. Therefore, accounting for prior achievement, student background and school context, provides an estimation method that seeks to correct for the extremely unequal Chilean educational playing field. Due to the fact that pupils and teachers are not distributed randomly in Chile, individual student characteristics, are considered to be potentially confounded with those associated with classes or schools groups (Raudenbush & Bryk, 2002). This has proved especially true in the Chilean educational system (Manzi et al, 2011; Meckes & Bascopé, 2012). Given that value added is a technique designed to make fair comparisons between schools (OECD, 2008) by adjusting for various background factors and prior attainment by the individual child, my research challenges the routinely applied rankings implemented in Chile to assess school performance; develops value added models using Chilean secondary schools' data, and uses those results to identify schools practices associated with better performance. Its dissemination will deliver new insights into the quality of Chilean schools; increase local awareness that different 'value added' indicators are needed to assess the quality of schools, and provide methodological tools coupled with evidence-based descriptions of effective schools.
The aim of the fellowship is to improve education -primarily in Chile and potentially in other ODA listed countries in the region such as Argentina- by applying an equity lens when assessing students' performance using value added models. This is needed because although Chile in the last decade has reached the highest per capita average income in the Latin-American region (EEA, 2007), its wealth distribution remains one of the most unequal in the world (Zahler, 2011). Distribution of wealth and its associated inequality, vulnerability and social exclusion remain a serious concern in Chile (OECD, 2011). In this context education is a key element to stop perpetuating inequalities and support sustainable development goals, and the assessment of schools performance plays a crucial role.
During the fellowship, I will carry out activities described in detail in the work plan, including: building on the impacts of the PhD; collaborating with users; disseminating research findings and further extending my publications track record. I will also apply my research related skills, seek opportunities to improve my academic/career development; and conduct additional research directly linked to my PhD.
The outcome of the proposed activities will support social cohesion and will work against educational segregation by describing a more favourable picture of schools that educate pupils coming from more disadvantaged socioeconomic backgrounds. This will help shifting the debate from type of school (public/private), to teaching and quality of school practices. Given that this is not part of the current national assessment framework, Chilean educational policies and practices oriented to tackle educational inequalities will be enhanced and supported. Moreover, recent educational reforms concerning a new evaluation system for schools (Sistema de Evaluación y Orientación del Desempeño de los Establecimientos y sus Sostenedores) as well as a reform for supporting public education (Sistema de Educación Pública) currently under discussion in Parliament, makes this proposal a timely contribution to inform and support these new developments. Finally, by combining equity and quality, Chilean educational system will be provided with information to become a fairer educational system.
The aim of the fellowship is to improve education -primarily in Chile and potentially in other ODA listed countries in the region such as Argentina- by applying an equity lens when assessing students' performance using value added models. This is needed because although Chile in the last decade has reached the highest per capita average income in the Latin-American region (EEA, 2007), its wealth distribution remains one of the most unequal in the world (Zahler, 2011). Distribution of wealth and its associated inequality, vulnerability and social exclusion remain a serious concern in Chile (OECD, 2011). In this context education is a key element to stop perpetuating inequalities and support sustainable development goals, and the assessment of schools performance plays a crucial role.
During the fellowship, I will carry out activities described in detail in the work plan, including: building on the impacts of the PhD; collaborating with users; disseminating research findings and further extending my publications track record. I will also apply my research related skills, seek opportunities to improve my academic/career development; and conduct additional research directly linked to my PhD.
The outcome of the proposed activities will support social cohesion and will work against educational segregation by describing a more favourable picture of schools that educate pupils coming from more disadvantaged socioeconomic backgrounds. This will help shifting the debate from type of school (public/private), to teaching and quality of school practices. Given that this is not part of the current national assessment framework, Chilean educational policies and practices oriented to tackle educational inequalities will be enhanced and supported. Moreover, recent educational reforms concerning a new evaluation system for schools (Sistema de Evaluación y Orientación del Desempeño de los Establecimientos y sus Sostenedores) as well as a reform for supporting public education (Sistema de Educación Pública) currently under discussion in Parliament, makes this proposal a timely contribution to inform and support these new developments. Finally, by combining equity and quality, Chilean educational system will be provided with information to become a fairer educational system.
Publications
Munoz-Chereau B
(2019)
Value-added indicators for a fairer Chilean school accountability system: a pending subject
in Journal of Education Policy
Muñoz-Chereau B
(2018)
Exploring gender gap and school differential effects in mathematics in Chilean primary schools
in School Effectiveness and School Improvement
Description | 1.Significant new knowledge: stakeholders' understandings of value added measures and indicators. Although research on value-added indicators for the Chilean school system has been conducted over the last decade, it has impacted very little the policy arena. Therefore I explored qualitatively the policy context in which this omission takes place, by analysing policy documents and interviewing local policymakers and researchers about their views on value-added measures. Both these and official documents pointed out legal, political, methodological, ethical and pragmatic reasons for and against value-added indicators. Overall, more challenging issues than opportunities of including these indicators into the new accountability system were identified. I have started the dissemination of these results through an impact case story at University of Bristol, and through a paper to be submitted to a high impact factor journal in my field (such as School Effectiveness and School Improvement). 2.Significant new knowledge: Despite the systematic recognition of large gender gaps against girls in mathematics performance in Chile, the role of schools in explaining this gap has been comparatively overlooked. I explored within and between school variation of Chilean primary schools in promoting attainment and progress of girls and boys. In order to reflect the hierarchical nature of the educational data, I used Multilevel modelling with 163,044 students, nested within 3,355 schools, within 310 Municipalities to fit Raw, Contextualised Attainment and Value-Added models. The results indicated a very small but significant gender differential school effect in progress in Chilean primary schools. I disseminated these results through presenting at ICSEI 2018 international conference, a policy Bristol Policy Brief http://bristol.ac.uk/media-library/sites/policybristol/documents/PolicyBristol_Briefing_October_2017_Chile_Schools.pdf, a blog post at the School of Education's research Centre for Assessment and Evaluation Research in Education (CAERe) http://www.bristol.ac.uk/education/news/2017/research-conducted-at-the-soe-focusing-on-value-added-measures-of-school-perform.html, and a paper submitted to School Effectiveness and School Improvement journal. 3.New research networks/collaborations/partnerships: I collaborated with users through a placement at the Unit of Curriculum and Evaluation, Ministry of Education (MoE) and cooperated with Líderes Educativos and Agency of Quality-Chile. During this fieldwork in Chile I worked with local policymakers, particularly at the Ministry of Education and the Quality Agency in Education supporting government's thinking around innovative school accountability and school improvement methods to account for inequality when assessing school performance. 4. Improved research capability generated from training oriented to acquiring advanced research skills in quantitative methods (Introduction to Multilevel Modelling Using MLwiN, 9-11 January 2017, University of Bristol; Introduction to STATA, 23rd January 2017, University of Manchester; Introduction to Experimental Design, 12th June 2017, University of Manchester) and qualitative methods (Qualitative analysis: methodologies for analysing text and talk, 10-20 July 2017, University of Essex; Qualitative data: sharing and reusing secondary data, 10-14 July 2017, University of Essex). |
Exploitation Route | My research outcomes can be taken forward by local policymakers. I disseminated my research and supported the work on assessment at the Quality Agency of Education (QAE). After delivering 3 seminars (two face-to-face ones and one webinar) with stakeholders, the QAE commissioned a policy brief that drew on my research on assessment incorporating an equity perspective. My work was mainly oriented to change their understandings around the approach and methods used. Overall, the potential value of my research for them is the provision of fairer measures of school performance, which in turn can support school improvement efforts. I also discussed with stakeholders a policy briefing -written with the support from PolicyBristol in English and Spanish. My work may also impact Chilean schools (Head teachers and Teachers). The seminars I conducted were oriented at head-teachers and teachers working in disadvantaged backgrounds (i.e some state-funded schools). The potential value that can be gained from my research is that value added models applied to education describe a more favourable picture of schools that educate pupils coming from more disadvantaged socioeconomic backgrounds. In this way I promoted a reduction of a sense of failing. |
Sectors | Education |
URL | http://www.bristol.ac.uk/research/impact/educational-equity-chilean-schools |
Description | The work supported by this award is intending to increase the effectiveness of education by the incorporation of value-added models into the Chilean school accountability system. However, the current methodology for classifying schools used in Chile does not only exclude value-added approaches, but also has recently limited the possibility of implementing longitudinal approaches to assess school effectiveness, by reducing by half the amount of testing at the national level. Yet this work provided accessible examples of how value-added indicators can be used formatively to complement school self-evaluation by monitoring students' progress and support school improvement efforts. The dissemination of this work across varied user communities has facilitated potential national and regional impact through different engagement activities described in detail, such as one interview for Líderes Educativos' bulletin; participation as a discussant at the International Seminar 'Schools that learn together' organised by Líderes Educativos, University of Valparaíso and Agency of Quality, one webinar (2nd. November, 2017, University of Concepción; 2 seminars with Head Teachers, teachers and policymakers (one in Concepción on 4th November 2017 with 50 participants (18 Head Teachers, 22 Deputy Heads and 15 policymakers) and one in San Felipe, 9th November 2017, with 55 participants (21 Head Teachers, 24 Deputy Heads and 10 policy makers); a policy brief commissioned by Quality Agency of Education (QAE). Also in 2018 I organised the international seminar 'School Differential Effects' at UCL Institute of Education (14th Nov. 2018) and from 15-19 October 2018, I trained Chilean researchers from the Faculty of Education at University Catholic of Concepcion who aims to estimate the value-added by their institution to the initial teaching training of their undergraduate students. By training Chilean researchers on value-added methods to assess educational effectiveness, I supported the potential implementation of value-added models to assess the quality of a Chilean Higher Education Institution. |
First Year Of Impact | 2017 |
Sector | Education,Government, Democracy and Justice |
Impact Types | Societal |
Description | Quality Agency of Education policy brief |
Geographic Reach | South America |
Policy Influence Type | Participation in a guidance/advisory committee |
Description | Training of researchers at Faculty of Education Catholic University of Concepción on value-added in Higher Education |
Geographic Reach | South America |
Policy Influence Type | Influenced training of practitioners or researchers |
Title | Interview guideline |
Description | As part of the research directly linked to my PhD, I conducted interviews with users (policymakers and researchers) aiming to identify issues -such as difficulties and constraints- faced when encountering and using value added indicators. |
Type Of Material | Physiological assessment or outcome measure |
Year Produced | 2018 |
Provided To Others? | Yes |
Impact | This novel semi-structured guideline can be used for collecting qualitative data. |
Title | Qualitative interview data |
Description | As part of the research directly linked to my PhD, I conducted interviews with users (policymakers and researchers) aiming to identify issues -such as difficulties and constraints- faced when encountering and using value added indicators. This qualitative dataset is formed by 5 semi-structured interviews conducted in Spanish with local policymakers and researchers. |
Type Of Material | Database/Collection of data |
Year Produced | 2018 |
Provided To Others? | Yes |
Impact | The new knowledge generated by this award is underpinned by the analysis of this dataset. |
Description | External examiner of PhD upgrate |
Organisation | Pontifical Catholic University of Chile |
Country | Chile |
Sector | Academic/University |
PI Contribution | I have been invited to act as an external examiner of a PhD candidate Gonzalo Cifuentes' upgrate to take place on 17/03/20 |
Collaborator Contribution | With chilean and international colleagues we will examine Gonzalo's upgrade proposal to work to apply value added measures in the Chilean educational system. He is a current student of the Faculty of Education, Pontificia Universidad Católica de Chile |
Impact | The output will be the PhD upgrade outcome of a student |
Start Year | 2019 |
Description | Key local network |
Organisation | Educational Leaders |
Country | Chile |
Sector | Learned Society |
PI Contribution | My collaboration with Lideres Educativos is directly linked to this award. I supported their work on the strand of Educational Leaders Professional Development working across the country, particularly on the use of data to inform the decisions of headteachers and other educational leaders to enhance school improvement. |
Collaborator Contribution | Lideres Educativos is a key local network- http://www.lidereseducativos.cl- they brought together primary and secondary schools, national and international universities, and non-profit foundations who share the conviction that public education in Chile must improve. EL also acts as a network for academic development and research in the country. They helped disseminating my work among non-academic users. They organised the seminars, invited participants and disseminated my work through their network. |
Impact | The main outputs resulting from this collaboration are: -I was interviewed by Líderes Educativos' bulletin, 11 April, 2017 https://www.lidereseducativos.cl/recursos/boletin-no1-2017. This was an opportunity to bring awareness to my research and support the professional opportunities provided by Lideres Educativos. My work provided input to one of their strands aiming to support head teachers and management teams in the use of data to inform their decisions. -I was invited to act as a discussant at the International Seminar 'Schools that learn together' organised by Líderes Educativos, University of Valparaíso and Agency of Quality Education, 24th October 2017, Valparaíso, http://www.agenciaeducacion.cl/noticias/inscribete-seminario-colegios-aprenden-juntos-nuevas-perspectivas-evaluar-orientar-la-mejora-territorio. |
Start Year | 2017 |
Description | Mainstreaming Mental Health |
Organisation | University of Leeds |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | I took part in an interview with the lead researcher Anna Madill to as one of the six non-mental health GCRF grants purposefully sampled for their study. We explored the extent to which my GCRF project engaged implicitly with psychosocial well-being impact. |
Collaborator Contribution | My research was disseminated as a case study in the paper Madill, A., Shloim, N., Brown, B., Hugh-Jones, S., Plastow, J., & Setiyawati, D. (2022). Mainstreaming global mental health: Is there potential to embed psychosocial well-being impact in all global challenges research?. Applied Psychology: Health and Well-Being. |
Impact | A paper was published reporting the main findings of the study. |
Start Year | 2021 |
Description | BERA Expert pannel |
Form Of Engagement Activity | A formal working group, expert panel or dialogue |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Policymakers/politicians |
Results and Impact | I was invited by Prof Gemma Moss, Prof Harvey Goldstein, Prof Pam Sammons, Prof Gordon Stobart, Sean Hayes and Gwen Sinnott to became a member of the BERA Expert panel 'Education Accountability Reform' which t is exploring alternatives to SATs. We will have a series of engagement activities during 2020 |
Year(s) Of Engagement Activity | 2020 |
Description | Blog post for CAERe Research Centre, University of Bristol |
Form Of Engagement Activity | Engagement focused website, blog or social media channel |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Postgraduate students |
Results and Impact | I wrote a blog post describing my research and its connection with the policy context in Chile. I brought attention to the fact that the new high-stake national school accountability system does not use value-added measures to take proper account of Chilean extremely unequal educational field. Developing approaches that take into account these inequalities when assessing school performance, remains one of the main challenges of the assessment of the quality of Chilean educational system in specific, and other DAC listed countries in the region in general where raw o status 'league tables' are a common practice. |
Year(s) Of Engagement Activity | 2017 |
URL | https://caeregsoe.wordpress.com/2017/07/07/research-conducted-at-the-soe-focusing-on-value-added-mea... |
Description | Centre for Researching Education Across Boundaries (CREAB) Seminar, School of Education, University of Bristol |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Postgraduate students |
Results and Impact | I presented the main aims and methodology of my postdoctoral fellowship and how it built on my PhD. I argued that my research challenges the routinely applied rankings implemented in Chile to assess school performance; develops value added models using Chilean secondary schools' data, and uses those results to identify schools practices associated with better performance. This presentation enabled me to critically engage with colleagues and postgraduate students at the School of Education, University of Bristol. |
Year(s) Of Engagement Activity | 2016 |
Description | Debate on International Seminar |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | I lead the debate between speakers in the International Seminar 'Schools that learn together' organised by Líderes Educativos, University of Valparaíso and Quality Agency of Education, 24th October 2017, Valparaíso. I lead the discussion towards the role of value-added indicators to improve school effectiveness and improvement. |
Year(s) Of Engagement Activity | 2017 |
URL | https://www.lidereseducativos.cl/seminario-colegios-que-aprenden-juntos/ |
Description | Impact story website |
Form Of Engagement Activity | Engagement focused website, blog or social media channel |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | University of Bristol commissioned a journalist to write an impact case story from my research. It is showcased in the University's research impact website, which is visited by potential and current students, practitioners and the general public. It highlights the aim of the project, the related publications and partners. From this, colleagues hace contacted me interested in the funding application as well as in the outcomes of the research. |
Year(s) Of Engagement Activity | 2018 |
URL | https://www.bristol.ac.uk/research/impact/educational-equity-chilean-schools/ |
Description | Interview by Lideres Educativos bulletin |
Form Of Engagement Activity | A magazine, newsletter or online publication |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Schools |
Results and Impact | I was interviewed by Lideres Educativos' bulletin published on 11 April, 2017 . This was an opportunity to bring awareness to my research and support the professional opportunities provided by Lideres Educativos. I described and commented on the main findings of my research and the way these linked to school effectiveness and improvement. |
Year(s) Of Engagement Activity | 2017 |
URL | https://www.lidereseducativos.cl/recursos/boletin-no1-2017 |
Description | Policy brief supported by Policy Bristol |
Form Of Engagement Activity | A formal working group, expert panel or dialogue |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Policymakers/politicians |
Results and Impact | I discussed with stakeholders (policymakers and practitioners) at the Quality Agency Education (QAE) the policy briefing "Comparing Chilean school performance: improving fairness in the classification of schools" written with the support from PolicyBristol in English and Spanish. This increased the interest in my research and lead the QAE to commission a new policy brief that critically examines the role of assessment (formative and summative) for improving teaching and learning in the Chilean current high-stake test based accountability system. |
Year(s) Of Engagement Activity | 2017 |
URL | https://www.bristol.ac.uk/policybristol/policy-briefings/chile-schools/ |
Description | Promoting educational equity in Chilean schools- an impact case story at University of Bristol |
Form Of Engagement Activity | A press release, press conference or response to a media enquiry/interview |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Media (as a channel to the public) |
Results and Impact | As part of a broader campaign to raise the profile of the University of Bristol work in the area of global development, I was asked to write a short case study of my individual research project. The case studies are hosted on the institutional website and may be used as part of a promotional campaign. I described the Global Challenge my project responds to, my motivation for working on this, the overall strategy to address the challenge, the potential impact, and my collaborators. |
Year(s) Of Engagement Activity | 2017 |
URL | http://www.bristol.ac.uk/research/impact/educational-equity-chilean-schools |
Description | Seminar |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | Regional |
Primary Audience | Schools |
Results and Impact | The seminar was delivered on 4th November 2017 at University of Concepcion). 50 participants took part in the activity (18 Head Teachers, 22 Deputy Heads and 15 policymakers). I disseminated my research findings. |
Year(s) Of Engagement Activity | 2017 |
Description | Seminar |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | Regional |
Primary Audience | Schools |
Results and Impact | I conducted a seminar (9th November 2017, San Felipe) with 55 participants (21 Head Teachers, 24 Deputy Heads and 10 policy makers). I disseminated findings of my research. |
Year(s) Of Engagement Activity | 2017 |
Description | Seminar 'Accountability and school differential effects' at UCL Institute of Education and online |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | I organised the Seminar 'Accountability and school differential effects' as a member of the Centre for Educational Evaluation and Accountability at the UCL Institute of Education. Drawing on my research on differential effects I introduced the seminar (https://mediacentral.ucl.ac.uk/Player/40625978) and invited two academics (Professor Steve Strand from University of Oxford and Dr George Leckie from University of Bristol) who presented their work. The seminar was very well attended (i.e 40 participants) and the online video-recording of the presentations expanded the reach of the activity to an international level. |
Year(s) Of Engagement Activity | 2018 |
URL | https://www.ucl.ac.uk/ioe/events/2018/nov/accountability-and-school-differential-effects |
Description | Webinar Seminars with policymakers, head-teachers, deputy heads and teachers |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Schools |
Results and Impact | I conducted a Webinar (2nd. November, 2017, University of Concepción) co-organised with Lideres Educativos and oriented to schools to disseminate the results of my research and discuss the use of value-added indicators for school assessment, improvement and self-evaluation. This was delivered within the general topic of how assessment (formative and summative) could enhance teaching and learning within a high-stake test based accountability system as the one under implementation in Chile. |
Year(s) Of Engagement Activity | 2017 |
URL | https://www.lidereseducativos.cl/conversaciones-en-linea-evaluacion-formativa-mejorando-la-equidad-y... |