How should we measure school performance and hold schools accountable? A study of competing statistical methods and how they compare to Progress 8

Lead Research Organisation: University of Bristol
Department Name: Education

Abstract

In 2016, the Department for Education radically overhauled their secondary school accountability system and introduced 'Progress 8', arguing it to be the simplest and fairest school performance measure to date. Progress 8 aims to quantify and communicate the average academic value each school adds to their pupils' learning. Specifically, Progress 8 measures how much higher each school's pupils score in their age 16 GCSE examinations than expected given their age 11 KS2 test scores when they started secondary schooling. Progress 8 scores are used to hold schools to account, with the lowest scoring schools judged 'underperforming' and 'coasting'; classifications that trigger intense scrutiny and intervention from the school inspectorate, Ofsted. Given the high-stakes involved, research is urgently needed to first evaluate the strengths and weaknesses of the statistical method underlying Progress 8, and second, to explore the potential benefits of alternative methods for measuring school performance. Our proposed research will address these needs and in doing so will advance scientific understanding about school performance measurement.

The first aim of the proposed research is to study how sensitive Progress 8 and other value-added measures' scores, rankings and classifications are to the statistical modelling decisions and assumptions implicit in their design. These include fundamental decisions over whether value-added measures should additionally consider pupil non-academic outcomes and whether they should account for the substantial differences in pupils' demographic and socioeconomic characteristics between schools. We will also explore an assumption implicit in Progress 8 and many other value-added measures which is that there is no relationship between pupils' family backgrounds and the quality of the schools they attend (i.e., confounding or selection into school). We will study the extent to which alternative value-added approaches can address this (using fixed- and random-effects models). We will also explore another assumption of Progress 8 and many other value-added measures which is that the GCSE outcomes of low scoring pupils at KS2 are no more variable than those of high scoring pupils at KS2 (i.e., homoskedasticity). We will study different approaches proposed in the literature to address this concern.

The second aim of the proposed research is to adapt and evaluate a range of cutting edge statistical methods not yet applied to measuring school performance. We will study their advantages and disadvantages over standard value-added approaches including Progress 8. These include methods that allow users to compare not just the average progress made by pupils in each school, but to also study the variability in pupils' progress (i.e., mixed-effects location scale models and multilevel quantile regression). These also include methods that are less reliant on extrapolating the relationships in the data and may therefore lead to fairer and more meaningful school comparisons (i.e., propensity score and other matching methods for comparing multiple treatments).

Whereas the above aims will advance the academic research in the school effectiveness and methods literatures, our third aim is to make fundamental contributions to the wider non-academic understanding of alternative methods for measuring school performance in England. We will therefore disseminate our findings to the users and producers of school and public-sector institution performance measures. These groups include policy makers and their advisers, statisticians and researchers in relevant Government departments and charities, as well as schools, the media and parents. Planned activities include: knowledge exchange meetings, research briefings, a data visualization website, an end-of-grant symposium, online training materials, and a face-to-face short training course.

Planned Impact

WHO WILL BENEFIT FROM THIS RESEARCH?

GROUP A are the USERS OF PROGRESS 8 who need to be able to understand the strengths and weaknesses of the published statistics. This group includes: Department for Education (DfE) policymakers and their advisors who use Progress 8 to classify schools as 'underperforming', 'coasting' or 'high progress'; Ofsted who use these data to target and inform their school inspections; third sector organisations, such as the Education Endowment Foundation, who use Progress 8 to select schools for their randomised control trials and to assess the effectiveness of their interventions; local authorities, academy chains and schools, who use Progress 8 for performance monitoring and resource allocation; the media, who present and explain Progress 8 scores to the public; and parents who are encouraged to use Progress 8 to choose which schools to send their children to.

GROUP B are the PRODUCERS OF SCHOOL AND OTHER PUBLIC-SECTOR INSTITUTION PERFORMANCE MEASURES who need to understand the different statistical methods, choices and issues involved when designing indicators. This group includes: DfE statisticians responsible for Progress 8 and the Fischer Family Trust and other commercial and charitable organisations who produce alternative school performance measures to support target setting and self-evaluation. Internationally, this group includes an increasing number of school systems using pupil test score data to hold schools to account. In terms of other public-sector areas, this group includes: the Higher Education Funding Council for England who are developing university performance measures of 'learning gain', potentially to be included as part of the Teaching Excellence Framework; the Department of Health who publish case-mix adjusted hospital mortality rates; and the Home Office who predict context adjusted police force crime rates.


HOW WILL THEY BENEFIT FROM THIS RESEARCH?

GROUP A will benefit from our substantive research which will carefully explain what Progress 8 attempts to do, how and why it ignores pupil demographic and socioeconomic characteristics and the implications this has for making fair comparisons across schools; and the influence this and other decisions surrounding the design and presentation of Progress 8 have on schools' scores, ranking and classifications. This group will also benefit from our investigations into the risk of over interpreting small differences in schools' performances, and how this will likely prove particularly important when comparing within school socioeconomic and other Progress 8 performance gaps. The novel statistical methods we shall explore will generate new substantive insights into the effectiveness of schools in England and this will further benefit this group and feed directly into national debates and policy surrounding school improvement and equity as well as school accountability.

GROUP B will benefit from our methodological research which explores the potential statistical strengths and weakness of not only different value-added approaches, but also of a range of cutting edge methods not yet applied to measuring school performance. These include mixed-effects location scale models, multilevel quantile regression and matching methods. This will enable this group to make more informed choices when developing their own measures. This group will also benefit from the training materials we shall develop as well as any versions of these adapted by others to their disciplines. The materials will provide step-by-step instructions on how to implement the different statistical methods in standard software. They will also act as exemplars as to how to richly interpret the estimated institution effects generated by the different methods as well as good practices in communicating institution performances and their uncertainty to lay audiences.


WHAT WILL BE DONE TO ENSURE THAT THEY BENEFIT FROM THE RESEARCH?

See PATHWAYS TO IMPACT.

Publications

10 25 50
 
Description This grant is funding three years of intensive research on England's Government school performance tables. The grant is focusing on statistically critiquing and comparing various school performance measures published in these tables, especially value-added and progress measures. We are particularly focussing on the influence of choice of statistical modelling and estimation approach on the resulting performance measures.


First paper published:
Leckie, G. and Goldstein, H. (2019) The importance of adjusting for pupil background in school value-added models: A study of Progress 8 and school accountability in England. British Educational Research Journal. DOI: 10.1002/berj.3511.

Abstract:
In the UK, USA and elsewhere, school accountability systems increasingly compare schools using valueadded measures of school performance derived from pupil scores in highstakes standardised tests. Rather than naïvely comparing school average scores, which largely reflect school intake differences in prior attainment, these measures attempt to compare the average progress or improvement pupils make during a year or phase of schooling. Schools, however, also differ in terms of their pupil demographic and socioeconomic characteristics and these factors also predict why some schools subsequently score higher than others. Many therefore argue that valueadded measures unadjusted for pupil background are biased in favour of schools with more 'educationally advantaged' intakes. But others worry that adjusting for pupil background entrenches socioeconomic inequities and excuses lowperforming schools. In this article we explore these theoretical arguments and their practical importance in the context of the 'Progress 8' secondary school accountability system in England, which has chosen to ignore pupil background. We reveal how the reported low or high performance of many schools changes dramatically once adjustments are made for pupil background, and these changes also affect the reported differential performances of regions and of different school types. We conclude that accountability systems which choose to ignore pupil background are likely to reward and punish the wrong schools and this will likely have detrimental effects on pupil learning. These findings, especially when coupled with more general concerns surrounding highstakes testing and school valueadded models, raise serious doubts about their use in school accountability systems.
Exploitation Route Our findings might be used by Government (Department for Education) in how they better design future school performance measures as well as the increasingly large number of charity and commercial companies who use the published school performance measures in performance monitoring systems which they sell to schools (e.g., Fischer Family Trust). Ofsted could change the way they view and use DfE data when inspecting schools.
Sectors Education

URL http://www.bristol.ac.uk/cmm/research/school-performance/
 
Description KEY IMPACT RELEVANT PAPER PUBLISHED: Leckie, G. and Goldstein, H. (2019) The importance of adjusting for pupil background in school value-added models: A study of Progress 8 and school accountability in England. British Educational Research Journal. DOI: 10.1002/berj.3511. Invited to talk at National Education Union annual conference, Liverpool April 2019 Invited to give evidence at Education Select Committee: Wednesday March 13th 2019 Face-to-face meeting with Ofsted to discuss this work. February 2019 Discussions with Co-Op Multi-academy Trust and individual schools about our work. Radio coverage: January 2019 Heart radio (pre-recorded interview); Love sport radio (live interview); BBC World at One (invited to live interview, declined as I was abroad). Newspaper coverage: January 2019 Bristol Post, Daily Express, The Guardian, The Independent, Leigh Journal, The Star, The Times, Times Educational Supplement, Warrington Guardian, Wigan Today, Yorkshire Post Other media coverage: January 2019 Ekklesia, Greensheets, Humanists UK, Local Gov, My Science, Reclaiming Schools, Schools Improvement, Schools Week, World News Live 4 PolicyBristol briefing and press release done for this work. Presented this work at local high-school: Bristol Cathedral School (September 2018). Presented this work at conferences attended by non-academics in London (November 2018; disseminated via YouTube) and Manchester (September 2019) MORE GENREALLY: More generally, the work we have done on the grant and our programme of dissemination has raised our profile within the field of value-added modelling and school league tables. This has led to impact activities informed by our key findings where we have advised a number of bodies on these issues including: The Bulgarian Ministry of Education department; Swedish Education department; Trinidad and Tobago education department; World Bank.
First Year Of Impact 2018
Sector Education
Impact Types Policy & public services

 
Description Discussions with Bulgarian minister of education and heads of directorates as expert member of World Bank team
Geographic Reach National 
Policy Influence Type Participation in a advisory committee
 
Description Office for Standards in Education, Children's Services and Skills (Ofsted) knowledge exchange meeting about potential Ofsted projects and Progress 8 research
Geographic Reach National 
Policy Influence Type Participation in a advisory committee
 
Description Office for Standards in Education, Children's Services and Skills (Ofsted) knowledge exchange meeting about potential Ofsted projects and Progress 8 research
Geographic Reach National 
Policy Influence Type Participation in a advisory committee
 
Description Office for Student (OfS) Benchmarking review advisory group
Geographic Reach National 
Policy Influence Type Participation in a advisory committee
 
Description Presentation of our Progress 8 research at UK Parliament Education Select Committee
Geographic Reach National 
Policy Influence Type Gave evidence to a government review
 
Description Sweden Department for Education: Advising on school value-added models
Geographic Reach National 
Policy Influence Type Participation in a advisory committee
 
Description Trinidad and Tobago Department for Education: Advising on school value-added models
Geographic Reach National 
Policy Influence Type Participation in a advisory committee
 
Description World Bank Expert advisor on school value-added modelling project
Geographic Reach National 
Policy Influence Type Participation in a advisory committee
 
Description 'Progress 8' National Education Union annual conference, Liverpool 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Schools
Results and Impact April 2019
Year(s) Of Engagement Activity 2019
 
Description A talk of presentaton: 'Accountability and school differential effects' Department of Learning and Leadership, IOE UCL, London. 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Schools
Results and Impact November 2018
Year(s) Of Engagement Activity 2018
URL https://mediacentral.ucl.ac.uk/Play/15633
 
Description A talk or presentation: 2018 AERA Annual Meeting, New York 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact Apri 2018
Year(s) Of Engagement Activity 2018
 
Description A talk or presentation: Bristol Cathedral School, Bristol 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Schools
Results and Impact September 2018
Year(s) Of Engagement Activity 2018
 
Description A talk or presentation: Educational Effectiveness EARLI SIG 18 & 23 meeting. Groningen: 25 years of school league tables, accountability and choice: Lessons from England 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact August 2018
Year(s) Of Engagement Activity 2018
 
Description A talk or presentation: Q-step Seminar Series, University of Exeter 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Postgraduate students
Results and Impact May 2018
Year(s) Of Engagement Activity 2018
 
Description A talk or presentation: Royal Statistical Society Annual Conference, Cardiff 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Postgraduate students
Results and Impact September 2018
Year(s) Of Engagement Activity 2018
 
Description DfE funded NPD User Group meeting (1 day), Bristol. Co-organiser with Rebecca Allen 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Third sector organisations
Results and Impact September 2018
Year(s) Of Engagement Activity 2018
 
Description Discussion with Head of Co-Op Academies about Progress 8 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Schools
Results and Impact April 2019
Year(s) Of Engagement Activity 2019
 
Description Discussion with Headmaster of school in North West about Progress 8 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Schools
Results and Impact October 2018
Year(s) Of Engagement Activity 2018
 
Description Newspaper coverage: Bristol Post, Daily Express, The Guardian, The Independent, Leigh Journal, The Star, The Times, Times Educational Supplement, Warrington Guardian, Wigan Today, Yorkshire Post 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Media (as a channel to the public)
Results and Impact January 2019
Year(s) Of Engagement Activity 2019
 
Description Other media coverage: Ekklesia, Greensheets, Humanists UK, Local Gov, My Science, Reclaiming Schools, Schools Improvement, Schools Week, World News Live 4 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Media (as a channel to the public)
Results and Impact January 2019
Year(s) Of Engagement Activity 2019
 
Description Policy Briefing: Leckie, G., Goldstein, H. (2019). The Government's Progress 8 school performance measure needs to account for pupil background. PolicyBristol. Policy Briefing 66: 01/2019. 
Form Of Engagement Activity Engagement focused website, blog or social media channel
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Media (as a channel to the public)
Results and Impact January 2019
Year(s) Of Engagement Activity 2019
 
Description Postgraduate short course: Introduction to Multilevel Modelling (1 day). Bamberg. Course Instructor 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact October 2018
Year(s) Of Engagement Activity 2018
 
Description Postgraduate short course: Multilevel Modelling in School Effectiveness Research (0.5 days). EARLI Special Interest Group 18 Educational Effectiveness, Groningen. Course Instructor 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact August 2018
Year(s) Of Engagement Activity 2018
 
Description Radio coverage: Heart radio (pre-recorded interview); Love sport radio (live interview); BBC World at One (invited to live interview, declined as I was abroad). 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Media (as a channel to the public)
Results and Impact January 2019
Year(s) Of Engagement Activity 2019